LLM Evaluation & AI Quality Engineering: Measure, Regression-Test, Release, and Monitor LLM, RAG, and Agent Systems with Python: 7 (QA Testing) - Softcover

Book 7 of 7: QA Testing

Corvick, Braylen ; Mondal, Masud

9798161271766: LLM Evaluation & AI Quality Engineering: Measure, Regression-Test, Release, and Monitor LLM, RAG, and Agent Systems with Python: 7 (QA Testing)

Softcover

ISBN 13: 9798161271766

Publisher: Independently published, 2026

View all copies of this ISBN edition

0 Used

4 New

From � 26.27

Stop shipping LLM, RAG and agent features on gut feel. This is the hands-on AI testing and LLM evaluation playbook for QA engineers, SDETs and AI engineers who must prove a release is ready.

Most teams "evaluate" an LLM application by eyeballing a few outputs or trusting one dashboard score. Then a prompt tweak, a model upgrade or a retrieval change quietly breaks production. LLM Evaluation & AI Quality Engineering shows you a better way: AI quality is an engineering lifecycle, not a single evaluation score. You will learn to Define, Measure, Evaluate, Diagnose, Improve, Regression Test, Release, Monitor and Learn, and to treat every measurement as evidence, not truth.

What you will build and learn

A vendor-neutral Python evaluation package you own, backed by a free public companion repository with 119 automated tests
Quality objectives, golden datasets, error analysis and failure taxonomies
Deterministic code evaluators, metrics and rubrics, human annotation with Cohen's kappa and Krippendorff's alpha
LLM-as-a-judge evaluators you have actually validated, with TPR/TNR meta-evaluation
Statistics for non-deterministic systems: Wilson confidence intervals, bootstrap, McNemar paired comparison, sample size, pass@k and pass^k
RAG evaluation: recall@k, precision@k, MRR, nDCG, groundedness, citation validity and retrieval-vs-generation attribution
AI agent evaluation: sandboxed outcome checks, trajectory and tool-use evaluation, reliability
Safety and red teaming: prompt injection, canaries, refusal and over-refusal, mapped to the OWASP Top 10 for LLM Applications 2025
Evaluation-driven development, regression suites for model migration, and cost, latency and reliability trade-offs
AI quality gates in CI/CD with GitHub Actions, and production observability with OpenTelemetry GenAI semantic conventions, drift detection and triage

Who this book is for

QA engineers, SDETs and test leads moving into AI quality; AI/ML engineers who own LLM, RAG or agent features; and technical leads who sign off AI releases. You need intermediate Python, basic pytest and Git.

Why this book is different

Written by a software QA professional, it turns evaluation results into release decisions you can defend. DeepEval, Ragas, Inspect AI, Arize Phoenix, MLflow, Langfuse, LangSmith and Promptfoo are compared fairly, but you are never locked into a vendor. The final part shows how to run an AI quality engineering programme: RACI, release sign-off, a maturity model, and a practical mapping to NIST AI RMF and ISO/IEC 42001.

23 chapters, 5 parts, 21 figures, 24 tables and 10 appendices. Part of the author's AI Quality Engineering series. Pair it with AI Testing with Python for foundations and RAG in Production for building the systems you will evaluate.

"synopsis" may belong to another edition of this title.