Evaluating a large language model application is harder than evaluating a classifier: outputs are free text, many answers can be correct, and small prompt changes shift behaviour in unexpected places. This page lays out the evaluation stack that works in practice, from public benchmarks through task-specific eval sets, LLM-as-judge and human review, to regression suites that gate every change.
"Evaluation" means two different things. Model evaluation asks how capable a model is in general, and is what public benchmarks measure. System evaluation asks whether your application, with its prompts, retrieval, tools and guardrails, does your task well. Practitioners spend almost all their effort on the second, because a model that tops a leaderboard can still fail on your documents, your output format, or your users' phrasing.
Without evaluation, every prompt tweak, model upgrade or retriever change is a guess. Teams fall into "vibe checking": try five examples, feel good, ship, then discover that the change fixed one case and broke twenty others. A good eval suite turns that into an engineering loop: change, measure, compare, decide. It is also how you justify switching to a cheaper model, because you can show the quality difference on your own task rather than relying on vendor claims.
Public benchmarks are standardised test sets: MMLU (multi-subject multiple choice), GSM8K (grade-school math), HumanEval and SWE-bench (coding), and many more. They are useful for shortlisting base models. They have known weaknesses: contamination (test questions leak into training data), saturation (top models cluster near the ceiling), and mismatch (multiple-choice accuracy says little about writing a support reply). Treat leaderboard numbers as a starting filter, then build your own set.
Deterministic graders are code: exact match, regex, JSON schema validation, running generated SQL and comparing results, executing generated code against unit tests. Use them whenever the task allows it; they are free, fast and unambiguous.
LLM-as-judge uses a strong model with a rubric to score open-ended outputs, either on an absolute scale (1 to 5), pairwise (which of A or B is better), or against a reference answer. Research such as Zheng et al. (2023, "Judging LLM-as-a-Judge") found strong judges agree with human preferences at rates comparable to human-human agreement on their benchmark, but also documented biases: position bias (preferring the first answer), verbosity bias (preferring longer answers), and self-preference. Mitigate with swapped-order pairwise runs, specific rubrics, short reasoning before the score, and periodic calibration against human labels.
Human evaluation remains the ground truth for nuanced quality, safety and domain correctness. It is slow and costly, so use it to build the golden set and to audit judges, not to grade every run.
Start with 50 to 200 cases that represent real usage: sample production queries, add known hard cases, and add a case for every bug you fix. Tag cases by slice (topic, language, difficulty, user tier) so you can see where a change helps or hurts. For each case store the input, any context, and what "good" means: a reference answer, required facts, a JSON target, or a rubric. Keep the set versioned like code. For retrieval systems, evaluate retrieval and generation separately; see RAG Evaluation with RAGAS.
Run the suite on every change to prompts, model versions, retrieval settings or tool definitions. Compare against the stored baseline per slice, not only in aggregate, since an average can hide a collapse in one category. LLM outputs are stochastic, so fix temperature where possible, run several samples for important cases, and treat small score differences with suspicion: on 100 binary cases, a 2-point change is well within noise. A bootstrap confidence interval over cases is a simple way to judge whether a difference is real.
# Minimal LLM eval harness: deterministic checks + LLM-as-judge + baseline gate.
import json, statistics
CASES = [
{"id": "refund-1", "input": "Can I return shoes after 40 days?",
"must_contain": ["30 days"], "rubric": "States the 30-day policy; polite; no invented exceptions."},
{"id": "json-1", "input": "Extract the city: 'Ship to 9 Elm St, Austin TX'",
"expect_json": {"city": "Austin"}},
]
def call_model(prompt: str) -> str:
"""Replace with your LLM client call (OpenAI, Anthropic, vLLM, ...)."""
canned = {"Can I return shoes after 40 days?":
"Returns are accepted within 30 days of delivery, so 40 days is outside the window.",
"Extract the city: 'Ship to 9 Elm St, Austin TX'": '{"city": "Austin"}'}
return canned[prompt]
def judge(question, answer, rubric) -> int:
"""LLM-as-judge: ask a strong model to score 1-5 against the rubric.
Stubbed here; in practice send JUDGE_PROMPT and parse the integer."""
JUDGE_PROMPT = (f"Rubric: {rubric}\nQuestion: {question}\nAnswer: {answer}\n"
"Score 1-5. Reply with the number only.")
return 5 if "30 days" in answer else 2
def grade(case, out):
scores = {}
if "must_contain" in case:
scores["contains"] = float(all(s in out for s in case["must_contain"]))
if "expect_json" in case:
try: scores["json"] = float(json.loads(out) == case["expect_json"])
except json.JSONDecodeError: scores["json"] = 0.0
if "rubric" in case:
scores["judge"] = (judge(case["input"], out, case["rubric"]) - 1) / 4
return statistics.mean(scores.values())
results = {c["id"]: grade(c, call_model(c["input"])) for c in CASES}
score = statistics.mean(results.values())
BASELINE, TOLERANCE = 0.90, 0.02
print(results, f"mean={score:.3f}")
assert score >= BASELINE - TOLERANCE, "regression: block the merge"The harness mixes graders per case: a substring check and a judge score for the open-ended answer, and exact JSON comparison for the extraction task. Scores are normalised to 0 to 1, averaged, and compared with the baseline using a tolerance; the assert makes it usable as a CI step. Replace the stubs with real API calls. Frameworks such as OpenAI Evals, promptfoo, DeepEval, Inspect, and RAGAS offer the same structure with caching, parallelism and reports.
| Method | Best for | Cost | Main risk |
|---|---|---|---|
| Public benchmarks | Shortlisting base models | Free to read | Contamination, task mismatch |
| Deterministic checks | Formats, extraction, code, SQL | Very low | Only covers checkable properties |
| LLM-as-judge | Open-ended quality at scale | Moderate (API calls) | Judge bias, drift when judge model changes |
| Human evaluation | Golden labels, safety, expert domains | High | Slow; inter-rater disagreement |
| Online A/B and feedback | Real user impact | Needs traffic | Noisy, slow, lagging signal |
Agents and tool-using systems need extra dimensions beyond answer quality. Score the trajectory: did the agent call the right tools with valid arguments, in a sensible order, without unnecessary steps? Score the outcome: did the final state match the goal, for example the ticket was created with the right fields or the test suite passes? Outcome checks are usually deterministic and should be preferred; trajectory checks catch wasteful or risky behaviour that still happens to succeed. Because agents are multi-step and stochastic, run each task several times and report the success rate, not a single pass or fail.
Safety evaluation is a separate suite with its own cases: prompt injection attempts hidden in retrieved documents, requests for disallowed content, leakage of system prompts or personal data, and over-refusal of legitimate requests. Track both directions, since tightening guardrails often increases false refusals. Red-team findings should flow into this suite the same way production bugs flow into the quality suite.