LLM Evaluation

1. What is it?

LLM evaluation is the discipline of measuring whether your AI system produces correct, useful, safe outputs — continuously, not anecdotally. It spans retrieval metrics, generation faithfulness, end-to-end answer correctness, and cost/latency budgets.

Without it, “the model got better” is folklore. With it, prompt and model changes become ordinary, gated engineering changes.

2. Why does it exist?

Three uncomfortable properties of LLM systems break traditional QA:

1. Non-determinism — the same input can yield different valid outputs; exact-match testing is meaningless.
2. Open-endedness — many valid answers per question; many failure shapes per answer.
3. Invisible regressions — a prompt tweak that fixes one case can silently break five others. Without a fixed benchmark, you will find out from users.

Evaluation converts an unreliable oracle (a vibe) into a measurable system with regression gates — the same transformation unit tests gave ordinary software.

3. How does it work?

The standard anatomy:

• Golden set — a curated set of inputs with known-good answers (and known-bad traps). Small and honest beats large and synthetic.

5. Production considerations

• Version everything the evaluation depends on — golden set, prompts, model versions. An unversioned benchmark is an anecdote generator.
• Cost of evals — model-graded metrics cost money per run; size the set to the change being made (smoke set per commit, full set per release).
• Data drift — production queries wander from your benchmark. Mine real (privacy-safe) queries periodically and refresh the golden set deliberately.
• Gaming — optimizing against the benchmark stops measuring the product; keep a held-out slice no one optimizes against.
• Tracing — evaluation without traces (LangSmith or equivalent) tells you that quality dropped, not where. Trace everything; evaluate node-by-node in agentic systems.

6. How I approach it

On the recipe-authoring and test-script-generation platforms, evaluation came in this order:

1. Golden sets from real work — actual questions, actual expert-approved outputs, plus deliberately adversarial cases (conflicting sources, unanswerable questions).
2. Measure retrieval before generation — if the right context is not retrieved, nothing downstream matters.
3. Deterministic checks first — schema compliance, citation presence, terminology. Cheap, objective, run on every commit.
4. Model-graded rubrics for judgment calls, calibrated against expert human labels.
5. Regression gates in CI — a prompt/model change that drops the benchmark is a failed build. This moved model changes from “drama” to “pull request”.

The habit that paid for itself most: capturing every production disagreement (expert rejects an output) as a future eval case. The golden set becomes an institutional memory of failure.

7. Architecture

pull request / prompt change
   → run smoke eval set (deterministic checks)        [seconds]
   → run full benchmark on changed layers only        [minutes]
   → compare against last-good baseline
   → gate: pass → ship · regress → blocked with traces

Plus continuous, sampled human review feeding judge calibration and the golden set.

8. Common mistakes

• Evaluating only end-to-end averages — node-level regressions hide inside them.
• Synthetic golden sets (generated by the same model being tested — circularity).
• Model-graded judges, uncalibrated, drifting silently.
• Measuring faithfulness but never refusal quality.
• No CI gate, so improvements are stories rather than deltas.

9. Interview perspective

• “How do you evaluate a RAG system?” — layered: retrieval metrics on labeled chunks, faithfulness and correctness on generation, refusal behavior on traps, plus cost/latency.
• “How do you know your eval isn’t broken?” — human-labeled calibration samples, held-out slices, and judge-agreement rates reported like any other metric.
• “A model update shipped and users say it got worse — what do you do?” — with a benchmark: bisect layers, replay traces, produce a delta. Without one: build the golden set now, from the complaints.

10. Related topics

• RAG — evaluation applied to retrieval.

• Agentic AI — per-node evaluation of agent graphs.

• AI Observability — traces as evaluation input (see AI index).

• Metrics per layer — retrieval (did we get the right context?), generation (is the answer faithful to context? correct?), system (latency, cost, refusal behavior).

• Judges — automated graders: deterministic checks (schema, citations present) and model-graded rubrics for fuzzy qualities. Calibrate judges against human labels.

• Regression gates — a change ships only if the benchmark holds or improves. This is the entire game.

4. Important concepts

• Faithfulness — is the claim supported by the provided context? (Grounding, not truth.)
• Answer correctness — is the claim actually right? (Truth, not just grounding.)
• Retrieval metrics — hit rate, MRR, recall@k against labeled relevant chunks.
• Refusal quality — does the system say “I don’t know” when it should? The most underrated metric in the entire field.
• Human evaluation — the calibration source for every automated judge; sampled continuously, not once.