Writing your first eval suite

What an eval suite is for

An eval suite is CI for model behaviour. Code changes are caught by tests; prompt changes, model upgrades, and provider drift are caught by evals. If a harness is in production without an eval suite, you are relying on users to file the regression report.

Step 1: Build a golden set

A golden set is a collection of representative inputs paired with expected outcomes. Start small — twenty to fifty cases is genuinely useful — and grow it from production traces rather than inventing examples.

1. Open the Traces view and filter to the harness you want to cover.
2. Select traces that represent typical inputs, known edge cases, and past failures.
3. Export them with hyperpriors eval add-case, which appends each one to your suite’s dataset.

Past failures are the most valuable cases. Every incident should leave a golden case behind so the same failure cannot return silently.

Step 2: Choose assertions or judged comparisons

Each case needs a way to decide pass or fail. Hyperpriors supports two kinds of check, and most suites use both.

Assertions are deterministic checks: the output contains a required field, parses as valid JSON, stays under a length limit, or never mentions a competitor. They are fast, cheap, and unambiguous. Use them wherever the requirement can be stated mechanically.

Judged comparisons use a model as the judge, scoring the output against a rubric or a reference answer. Use them for qualities that resist mechanical definition — tone, faithfulness to a source document, helpfulness. Judged scores are probabilistic, so set thresholds over the suite rather than expecting every case to pass every run.

A case in the suite file looks like this:

- id: refund-policy-question
  input: dataset/refund-policy-question.json
  checks:
    - assert: json_valid
    - assert: contains
      value: "14 days"
    - judge: faithfulness
      reference: dataset/refund-policy-source.md
      min_score: 0.8

Step 3: Run locally, then in CI

Run the suite against your current configuration:

hyperpriors eval run

The CLI prints a per-case breakdown and an overall pass rate, and stores the run so you can compare against previous results. Once the suite passes locally, add the same command to your CI pipeline. A non-zero exit code on failure means a regression blocks the merge, exactly as a failing unit test would.

Keeping the suite honest

Review the suite whenever behaviour changes intentionally — a stale golden set fails good changes and erodes trust. Treat eval cases like tests: version them, review them in pull requests, and delete the ones that no longer describe behaviour you want.