Essay
·Evals·5 min read
Evals are the unit test suite of AI systems. How regression gates, golden sets, and honest LLM-as-judge practice keep model behaviour shippable — and why eval suites rot if you let them.
By Dr Gareth Roberts
There is a moment in every AI project where someone changes a prompt, eyeballs three outputs, declares them “better”, and merges. Six weeks later a customer finds the case that got worse. Nobody can say when it broke, because nothing was measuring it.
Conventional software solved this problem decades ago. We do not eyeball whether a sorting function still sorts; we run the tests. The equivalent discipline for AI systems exists — it is just unevenly adopted. We call it eval-driven development, and we think it is the single highest-leverage practice a team shipping LLM features can adopt.
An eval is a specification of behaviour, expressed as data: inputs your system will face, paired with a way of judging the outputs. Run the suite, get a score, compare it to the last known-good score. That is the whole idea. Everything else is engineering around it.
The comparison to unit tests is not a loose analogy. It is close to exact:
The one place the analogy bends: unit tests are deterministic and evals usually are not. That changes the statistics, not the discipline. You run more samples, you track distributions rather than single pass/fail bits, and you set thresholds with some tolerance. The workflow — write the check before you trust the change — is identical.
An eval suite that runs when someone remembers is a dashboard. An eval suite that blocks merges is infrastructure.
The practice we recommend is straightforward: wire your evals into CI so that any change to a prompt, model pin, tool definition, or harness configuration triggers the relevant suites, and the change cannot ship if scores drop below threshold. Treat a threshold breach exactly as you would a failing test — either the change is wrong, or the expectation was, and a human decides which.
Two details matter more than teams expect. First, gate on the diff, not just the aggregate. A change that improves the mean score while destroying performance on a critical slice — refund requests, say, or non-English inputs — should fail loudly. Aggregates hide exactly the failures that hurt. Second, version everything: the eval set, the judge, the model, the prompt. A score is meaningless unless you can say precisely what produced it.
The heart of a good suite is a golden set: a curated collection of inputs with known-good expected behaviour, maintained with the same care as the code.
Golden sets should be drawn from reality wherever possible. Production traces — appropriately scrubbed — are worth more than invented examples, because they capture the actual distribution of ambiguity, typos, hostility, and edge cases your system faces. Synthetic cases have a place, particularly for rare-but-critical scenarios you cannot wait to observe in the wild, but a suite built entirely from imagined inputs tests an imagined system.
Every incident should leave a trace in the golden set. The case that broke in production last month is the regression test for next quarter. Over time the suite becomes a fossil record of everything that has ever gone wrong — which is exactly what you want a test suite to be.
Keep golden sets small enough to be curated and fast enough to run on every change. A few hundred well-chosen cases with clear grading criteria beat fifty thousand unlabelled transcripts that nobody has read.
Many behaviours worth evaluating — helpfulness, tone, faithfulness to sources — resist exact matching. Using a model to grade outputs is the practical answer, and it works well enough to be standard. It also fails in specific, well-documented ways, and using it honestly means engineering around them:
LLM-as-judge is a measurement instrument. Instruments need calibration certificates.
Test suites decay, and eval suites decay faster. The system drifts away from the distribution the suite was built on: new features, new user populations, new failure modes the golden set has never heard of. Scores stay green while production quietly degrades, and the suite’s chief output becomes false confidence.
There is a second, subtler failure: overfitting. Iterate against a fixed eval set long enough and you will optimise for the set rather than the behaviour it samples. This is Goodhart’s law wearing a lab coat.
The defences are unglamorous and effective. Refresh golden sets from recent production traces on a schedule, not when someone gets around to it. Hold out a portion of cases that are never used during development, only for final checks. Retire cases that no longer reflect reality, and record why. And watch coverage: if a class of production failure has no corresponding eval, that is a gap in the suite, not bad luck.
None of this is conceptually difficult. It is the same discipline the industry already applies to code, applied to behaviour. What makes it hard in practice is plumbing: versioning eval sets alongside prompts, wiring suites into CI, pinning judges, tracking slices, refreshing golden sets from traces. That plumbing is a large part of what Hyperpriors builds — evals as a first-class part of the control plane, not a script someone runs before the demo.
If you take one thing from this post: the next time you change a prompt, write the eval first. Future you, reading an incident report at 2am, will be glad the suite existed.
Share
More
A five-stage maturity model for LLM evaluation practice — from ad hoc spot checks to continuous production evaluation — with the failure modes of each stage and the exit criteria that mark genuine progress to the next.
Evals · Whitepaper
How to make evals block merges the way tests do: golden set construction, threshold design, flake management, and the gate discipline that makes provider migrations survivable.
Evals
