Regression gates for LLM systems

Most teams that run evals do not gate on them. The suite runs nightly, someone glances at a dashboard, and a regression discovered on Thursday gets bisected across a week of merged changes. The eval exists; the enforcement does not. This post is about closing that gap: making evaluation results a condition of merging, the way test results already are.

A regression gate is a simple contract. Every change that can affect model behaviour — prompt edits, model version bumps, retrieval changes, tool schema changes, context assembly changes — runs the eval suite in CI, and the merge is blocked if agreed thresholds are breached. The contract is simple; the engineering that makes it trustworthy is not. Gates fail in two directions. A gate that blocks spuriously gets overridden, then ignored, then deleted. A gate that passes everything is a dashboard with extra steps. The work is in the calibration.

The golden set is the gate’s foundation

A gate is only as good as the data it runs against. The golden set — the fixed collection of inputs with reference outputs or judging criteria — is where most gates are actually won or lost, long before anyone argues about thresholds.

Three sourcing rules that hold up in practice:

• Draw from production, not imagination. Cases invented in a workshop encode what the team expects users to do. Production traces encode what users actually do, including the malformed, ambiguous, and adversarial inputs that cause real incidents. Sample from logs, then curate.
• Overweight failures. A golden set assembled from random production traffic is dominated by easy cases, and easy cases cannot regress far. Every resolved incident, every support escalation, every case a human corrected should be a candidate. The set should be harder than average traffic, deliberately.
• Version it like code. The golden set lives in the repository, changes go through review, and every eval result records the set version it ran against. A score is meaningless if you cannot say which set produced it, and an engineer quietly removing a failing case is a change that reviewers must see.

Size matters less than composition, but it still matters: statistics on twenty cases are noise. A useful floor for a gated suite is a few hundred cases per behaviour you care about, stratified so that a regression in one segment — one language, one document type, one tool path — cannot be masked by stability elsewhere. Report scores per stratum, and gate on the worst stratum, not the aggregate.

One discipline that separates durable gates from decorative ones: when a production incident occurs, the fix is not complete until the failing case is in the golden set. This is the eval equivalent of a regression test, and it is how the set grows in proportion to what actually goes wrong rather than what the team finds interesting.

Threshold design

The naive gate — fail if the score drops at all — is unusable, because LLM outputs are stochastic and judged scores carry their own noise. The workable pattern has three components.

A baseline, not an absolute. Gate on the delta against the last accepted run on the main branch, not on a fixed number. Absolute thresholds rot: the suite gets harder as incident cases accumulate, and a fixed “must exceed 0.85” either blocks legitimate suite improvements or was set with slack that hides real regressions.

A noise floor, measured rather than guessed. Run the suite several times against an unchanged system and observe the spread. That spread is your measurement noise, and the blocking threshold must sit outside it. If aggregate accuracy varies by two points between identical runs, a one-point drop tells you nothing, and a gate that blocks on it will be overridden within a fortnight.

Two tiers. Not every metric deserves to block. A practical split:

• Blocking: safety behaviours (refusal correctness, data handling), tool call validity, output schema conformance, and the primary quality metric dropping beyond the noise floor. These are the behaviours where a regression is an incident.
• Warning: secondary quality metrics, latency and cost drift, per-stratum movements inside the noise band. These annotate the pull request and accumulate into trend review, but do not block.

The tier assignment is a product decision expressed in CI configuration, and it should be reviewed like one. A sketch of the shape:

gates:
  - metric: tool_call_validity
    type: absolute
    min: 0.99            # structural failures block outright
  - metric: task_success
    type: delta_vs_baseline
    max_drop: 0.03       # measured noise floor: 0.02
    samples_per_case: 3
  - metric: p95_latency_ms
    type: delta_vs_baseline
    max_increase_pct: 20
    action: warn

Flake management

Flaky gates die socially before they die technically. The first spurious block gets an override; the fifth gets a Slack norm of “just re-run it”; after that the gate is ceremony. Treating flake as a first-class engineering problem is not optional.

The mechanics that keep flake down:

• Pin everything pinnable. Temperature at or near zero for gate runs, fixed seeds where the provider honours them, pinned model versions, pinned judge versions. A gate run should differ from the previous one only by the change under test.
• Sample per case, not per suite. Running each case three to five times and scoring the aggregate costs more than one pass but converts coin-flip cases from flake into signal. Concentrate repeats on cases with historically high variance rather than uniformly.
• Quarantine, visibly. A case that flips on identical inputs across runs is measuring noise, not behaviour. Move it to a quarantine list that is reported on every run and reviewed on a schedule — quarantine that only grows is deletion with better manners.
• Track override rate as a health metric. Every gate override is recorded with a reason. If overrides exceed a few percent of gated merges, the gate is miscalibrated and the calibration work happens now, not after the team has learned to ignore it.

Provider and model migrations

The gate earns its keep most visibly when you change models — a version bump from your provider, or a migration between providers. This is the highest-risk routine change an LLM system undergoes: everything can shift at once, including behaviours you never wrote down.

The gate turns migration from a judgement call into a diff. Run the full suite against both configurations, same golden set, same judges, and produce a per-case comparison rather than two aggregate scores. Aggregates routinely conceal the real story: a model that is two points better overall and catastrophically worse on one input class is a common outcome, and the per-stratum, per-case diff is the only artefact that shows it.

Two migration-specific cautions. First, judge consistency: if your judge is served by the provider you are migrating away from, judge and candidate change together and the comparison is contaminated — pin the judge independently of the system under test. Second, format drift: new models fail old output-parsing assumptions in ways that register as quality regressions but are actually harness bugs. Separate structural conformance metrics from quality metrics so the diff tells you which problem you have.

When to block and when to warn

The dividing line, stated once, plainly: block on behaviours where a regression reaching production constitutes an incident; warn on everything you would want to know about but would ship anyway. Safety behaviours, structural validity, and noise-floor-exceeding drops in the primary metric block. Trend-level drift warns.

Err on the side of fewer blocking gates, honestly enforced. A small set of blocking conditions the team genuinely will not override survives; an ambitious set that blocks weekly for defensible reasons gets routed around, and then nothing is gated at all. Start with two or three blocking metrics, measure your noise floor before setting a single threshold, and promote warnings to blocks only when they have demonstrated, over weeks of runs, that they fire on real regressions and nothing else.

A gate is a promise the team makes to itself about what it will not ship. Keep the promise small enough to keep.