Evals are only useful if they can say no. This release wires Hyperpriors evaluation suites directly into your merge workflow, so a prompt change, harness update, or model swap cannot land if it degrades behaviour your team has decided to protect.
Highlights
- Merge gates. Attach an eval suite to a repository via our GitHub and GitLab integrations. Pull requests report pass/fail status the same way unit tests do, and can be required before merge.
- Golden sets. Curate reference inputs with expected behaviours — exact matches, rubric scores, or judge-model assessments — and version them alongside your code.
- Regression thresholds. Set tolerances per metric: fail the gate on any drop in task success, or allow a bounded trade-off in verbosity against a latency win. Thresholds are declared in config, reviewed like code.
- Provider-migration diffs. Run the same suite against two models or providers and get a case-by-case diff of where behaviour changes — before the migration ships, not after.
- Flake handling. Non-deterministic cases can be sampled multiple times with majority scoring, so a gate failure means a regression rather than a bad draw.
Why it matters
Most teams discover behavioural regressions from users. Treating evals as CI moves that discovery to the pull request, where the change is small, the author is present, and reverting costs nothing. The gate does not make your models better; it stops them getting quietly worse.