The eval maturity model

Ask an engineering leader whether their team “does evals” and the answer is almost always yes. Ask what would happen if a model update degraded a core behaviour tomorrow — who would know, when, and by what mechanism — and the answers diverge sharply. Some teams would know within the hour from an automated signal. Others would find out from a customer.

Both teams “do evals”. The difference between them is maturity, and maturity is not a function of how many eval frameworks are installed or how large the golden set is. It is a function of what the evaluation practice can actually detect, how fast, and with what degree of human heroics.

This whitepaper lays out a five-stage maturity model for evaluation practice in LLM systems. For each stage we describe what it looks like from the inside, the failure modes characteristic of that stage, and the exit criteria that mark real — rather than nominal — progress to the next. The model is descriptive before it is prescriptive: most teams will recognise their current stage immediately, and that recognition is the useful part. Two caveats before we start. First, stages are per-behaviour, not per-organisation: a team can be at Stage 4 on safety refusals and Stage 1 on summarisation quality, and usually is. Second, later stages contain earlier ones. Continuous production evaluation does not replace gated CI; it sits on top of it.

Stage 1: Ad hoc spot checks

What it looks like. Evaluation is a human looking at outputs. An engineer changes a prompt, runs a handful of inputs they remember being tricky, reads the results, and forms a judgement. The inputs live in a scratch file, a notebook, or nowhere. The judgement lives in the pull request description: “tested a few cases, looks better.” When a stakeholder asks whether the system is good, the answer is a demo.

Nearly every team starts here, and it is worth saying that Stage 1 is not worthless. Human inspection of real outputs is the raw material of every later stage; a team that reads its outputs attentively at Stage 1 will build better golden sets at Stage 2 than a team that skipped straight to tooling. The problem is not the activity but its properties: it is unrepeatable, unrecorded, and biased towards whatever the engineer happened to remember.

Failure modes.

• Regression by anecdote. Changes are accepted because the three cases someone checked improved. The forty cases nobody checked are unmeasured, and some of them got worse. There is no mechanism by which anyone would notice.
• The demo distribution. The cases people spot-check are the cases they can recall, which skews heavily towards recent incidents and impressive examples. Boring, high-volume traffic — where most production value lives — is barely represented.
• Unfalsifiable quality claims. “The new prompt is better” cannot be contested, because there is no shared basis for the claim. Technical arguments about prompts become arguments about taste, and seniority wins them.
• Knowledge evaporation. The tricky cases exist in individual engineers’ heads and scratch files. When those engineers move on, the accumulated knowledge of how the system fails leaves with them.

Exit criteria. The exit from Stage 1 is not a tool purchase; it is an act of writing things down. The team has left Stage 1 when a named, versioned set of inputs exists in the repository; when those inputs have expected outputs or at least written judging criteria; and when the set is drawn substantially from real traffic and real failures rather than invented examples. A hundred well-chosen cases with clear criteria beat a thousand synthetic ones.

Stage 2: Golden sets

What it looks like. The team has a curated evaluation set and a script that runs the system against it and produces scores. Evaluation is now repeatable: the same inputs, the same scoring, comparable numbers across runs. Before significant changes, someone runs the suite and quotes the results in the pull request. Incidents start feeding the set — when something breaks in production, the failing case gets added.

This is a genuine phase change. For the first time, quality claims are contestable and regressions are detectable in principle. The characteristic weakness of Stage 2 is the phrase “in principle”: everything depends on a human remembering to run the suite, and on humans behaving well when the results are inconvenient.

Failure modes.

• The optional gate. The suite runs when someone remembers, which reliably excludes the moments of highest risk: the urgent fix, the Friday deploy, the change that “couldn’t possibly affect model behaviour”. The changes most likely to regress are the least likely to be evaluated.
• Score erosion by negotiation. A change drops the score from 0.86 to 0.83 and ships anyway, because the deadline is real and the drop is “probably noise”. The next baseline is 0.83. Three quarters later the score is 0.71 and nobody can identify a single decision that was wrong.
• The stale set. The product surface grows — new document types, new languages, new tools — but the golden set still describes the product as it was when the set was built. Scores stay green while coverage quietly rots.
• Judging drift. Scoring is often semi-manual at this stage, and different engineers apply the written criteria differently. Run-to-run comparability, the whole point of a golden set, degrades without anyone deciding it should.

Exit criteria. The team has left Stage 2 when running the suite is not a human decision. Concretely: the suite runs automatically in CI on every change that can affect behaviour; thresholds are agreed in advance and encoded in configuration rather than negotiated per-merge; a breach blocks the merge by default, with overrides recorded and reviewed; and the measurement noise of the suite has actually been measured, so thresholds sit outside it. If shipping a regression requires an explicit, logged override rather than mere inattention, you are at Stage 3.

Stage 3: Gated CI

What it looks like. Evals are merge gates. Every behaviour-affecting change runs the suite; blocking thresholds are versioned in configuration; the baseline updates only when a change is accepted. Provider version bumps go through the same gate as prompt edits, and a model migration produces a per-case diff rather than a debate. The golden set has an owner, incident cases flow into it as a matter of process, and quarantine handles flaky cases visibly.

Stage 3 is where evaluation stops being a practice individual engineers maintain and becomes a property of the delivery pipeline. It is also, for many teams, a comfortable place to stall — because the machinery works, and the ways it can be wrong are quiet.

Failure modes.

• Gate theatre. Thresholds set loose enough never to block, or overrides so routine they are muscle memory. The pipeline diagram shows a gate; the merge history shows a formality.
• Trusting an uncalibrated judge. To gate at scale, most teams adopt automated scoring — often an LLM judging outputs against criteria. At Stage 3 that judge is typically unvalidated: nobody has measured whether its verdicts agree with careful human judgement. The gate is now enforcing the judge’s biases with perfect consistency. Verbose outputs score higher because the judge favours length; confident hedging passes because the judge cannot detect it; and the team optimises, change by change, towards what the judge rewards rather than what users need.
• Overfitting to the suite. When a fixed set both measures and gates, the system slowly specialises to it. Scores improve; production complaints do not. This is Goodhart’s law applied to golden sets, and it is invisible from inside the suite.
• Offline-only confidence. The gate certifies behaviour on the golden distribution. Production traffic drifts away from that distribution — new user cohorts, new content, new attack patterns — and the gate, run only at merge time against fixed data, cannot see any of it.

Exit criteria. The exit from Stage 3 runs through the judge. The team has left Stage 3 when automated judges are treated as measurement instruments that require calibration: judge verdicts have been compared against a human-labelled sample and the agreement rate is known, tracked, and acceptable for each metric that gates; known judge biases (length, position, self-preference, sycophancy towards confident phrasing) have been tested for and mitigated where found; judge prompts and versions are pinned, and a judge change is treated as a change to the measurement system — requiring recalibration — rather than a routine edit. When someone asks “how much do we trust this score?” and the answer is a number rather than a shrug, you are at Stage 4.

Stage 4: Calibrated judges

What it looks like. The team maintains a human-labelled calibration set alongside the golden set. Every automated judge has a measured agreement rate against it, refreshed on a schedule and whenever the judge changes. Judges are engineered artefacts: prompt-versioned, pinned to specific models, tested for known biases, and decomposed — separate judges for factual grounding, instruction adherence, and tone, rather than one judge emitting a single holistic score. Disagreement between judge and human labels is itself a tracked metric, and rising disagreement triggers investigation.

The practical consequence is that evaluation scales without silently degrading. The team can gate on nuanced behaviours — faithfulness to retrieved context, appropriateness of refusals, tool selection quality — with known error bars, because the instrument has been checked against ground truth.

Failure modes.

• Calibration decay. The judge was validated once, eighteen months ago, against outputs from a model two generations old. Output style has shifted; the judge’s error profile has shifted with it; the agreement number everyone quotes is a historical artefact.
• Calibrating on the easy cases. Human labels are expensive, so the calibration set skews towards cases where labelling is quick — which are exactly the cases where the judge was already reliable. Agreement looks excellent; the ambiguous cases where judge error concentrates are underrepresented in the very set meant to expose them.
• The last mile of subjectivity. For some behaviours, careful humans disagree with each other at rates comparable to judge–human disagreement. Teams burn effort pushing judge agreement past the human–human ceiling, which is not a measurement problem but a criteria problem: the rubric is underspecified, and no judge can be calibrated against a target that humans cannot agree on.
• Pristine offline, blind online. Everything above still happens at merge time, on curated data. The system’s behaviour on this hour’s production traffic remains uninstrumented beyond latency and error rates.

Exit criteria. The team has left Stage 4 when evaluation runs against production, not only before it. Concretely: a sampled fraction of live traffic is scored continuously by the calibrated judges; scores are tracked as time series with alerting on drift, segmented by the strata that matter; production failures surfaced by this monitoring flow back into the golden set on a defined path; and at least one class of behavioural regression has actually been detected from the production signal before users reported it. The last criterion matters — a pipeline that has never caught anything is unproven, not mature.

Stage 5: Continuous production evaluation

What it looks like. Evaluation is an always-on property of the running system. Sampled production traffic is judged continuously; behavioural metrics sit on dashboards next to latency and error rate, with the same alerting discipline. Offline and online evaluation share judges and criteria, so a merge-time score and a production score are commensurable. Model updates roll out progressively, with behavioural metrics compared across cohorts before full traffic exposure. Drift in inputs — not just outputs — is monitored, so the team learns the golden set is stale from data rather than from an incident. The loop closes structurally: production anomalies become golden set cases, golden set changes recalibrate thresholds, and the offline suite remains an honest rehearsal of the online reality.

Failure modes. Stage 5 has its own pathologies, mostly of scale and attention:

• Alert fatigue, behavioural edition. Behavioural metrics are noisier than infrastructure metrics, and a team that alerts on every wobble trains itself to ignore the channel. The noise-floor discipline from gating applies doubly here.
• Cost creep. Judging even a few percent of production traffic with capable models is a real line item, and the quiet response — sampling less, judging with cheaper models — degrades the signal without anyone deciding to degrade it. Sampling rates and judge quality are budget decisions and should be made explicitly.
• Privacy and data handling. Production traffic contains user data; routing it through judges and into eval sets is a data-processing activity with retention, consent, and residency implications. Mature teams involve their privacy function before the pipeline exists, not after.
• Instrument sprawl. Dozens of judges, hundreds of metrics, and no ranking of which ones page a human. Maturity here looks like fewer, better metrics with clear ownership — the same consolidation observability teams went through a decade ago.

There is no exit from Stage 5, only maintenance. The steady state is a loop that keeps itself honest: production feeds the golden set, the golden set gates changes, calibration keeps the judges truthful, and monitoring confirms that what was certified offline is what is happening online.

Using the model

Three closing observations for teams locating themselves on this ladder.

Do not skip stages. The recurring failure we see is a team at Stage 1 buying Stage 5 tooling: production traffic flowing into an evaluation platform, scored by judges nobody has calibrated, against criteria nobody has written down. The dashboards render; the numbers mean nothing. Each stage exists because it builds an asset the next stage consumes — the golden set feeds the gate, the gate’s judges demand calibration, calibrated judges make production scoring meaningful. Tooling can accelerate a transition; it cannot substitute for the asset.

Advance where it pays. Not every behaviour deserves Stage 5. Push your highest-risk behaviours — safety, compliance, the failure modes that constitute incidents — furthest up the ladder, and let low-stakes behaviours rest at Stage 2 or 3 deliberately rather than by neglect. Maturity spent where it does not reduce risk is cost, not progress.

Audit against exit criteria, not activity. “We have evals in CI” is an activity claim. “A behavioural regression cannot reach main without a logged override, and our judge agrees with human labels 92% of the time on a calibration set refreshed this quarter” is a maturity claim. The exit criteria in this model are written to be auditable — as a leader, ask for the evidence behind each one, and treat the stage where evidence runs out as your actual position. Most teams are one honest stage below where they believe themselves to be, and the gap between the two is precisely where the next incident is waiting.