Case study

A Series-B fintech platform

A Series-B fintech platform providing credit underwriting software to mid-market lenders. Their underwriting copilot drafts risk assessments and surfaces relevant policy criteria for human underwriters, operating in a regulated environment where every recommendation must be explainable after the fact.

01 — In their words

“The provider shipped a model update on a Tuesday. Our eval gate failed the deploy on Wednesday morning, before a single underwriting decision was affected. That was the moment the whole team stopped arguing about whether we needed this.”

Recorded verbatim · Attribution withheld

02 — The work

Challenge

The copilot depended on hosted models that changed underneath it. Provider updates arrived unannounced and shifted behaviour in ways that spot checks missed: risk narratives drifted in tone, edge-case policies were applied inconsistently, and a decimal-formatting change once altered how debt-to-income ratios were summarised. Nothing crashed, so nothing alerted. Regressions surfaced weeks later through underwriter complaints, by which point tracing a behavioural change back to a specific model version was guesswork.

Solution

The team adopted Hyperpriors evaluation suites as a gate in their deployment pipeline. A regression suite built from anonymised historical underwriting cases now scores every model version, provider update, and prompt change before promotion, with hard thresholds on policy-application accuracy and output structure. Hyperpriors tracing records the full context of every production call — model version, prompt, retrieved policy documents, and output — so any behavioural question can be answered from the trace rather than from memory.

03 — The system

Fig. 1 — System architectureFinancial services · Illustrative topology

04 — Results

The team runs Hyperpriors evaluation gates in CI and full-trace observability across their underwriting copilot, with every model change scored against a regression suite before it reaches production.

01
Three provider-side regressions caught at the eval gate before reaching production, each traced to a specific model version within the hour
02
Complete per-decision audit trails, cutting compliance evidence-gathering for model-related queries from days to minutes
03
Provider migration time reduced from a six-week manual review to under a week, using the same eval suite to qualify candidate models

05 — The record

Industry: Financial services
Location: United Kingdom
Model providers: Two hosted frontier-model providers
Workloads: Underwriting copilot, document analysis

06 — The full account

The situation

The platform’s underwriting copilot had been in production for over a year and was, by most measures, a success. Underwriters trusted its draft risk assessments, and it had become part of the daily workflow at dozens of lending teams.

The problem was that the copilot’s behaviour was not entirely the team’s to control. It ran on hosted models, and those models changed. Provider updates arrived without notice, and each one nudged behaviour: a shift in how hedged the risk narratives sounded, a change in which policy criteria were cited for borderline applications, once a difference in how ratios were rounded in summaries. None of these threw errors. Latency and cost dashboards stayed green while the thing that actually mattered — what the model said — drifted quietly.

Regressions were discovered the slow way, through underwriter feedback aggregated over weeks. In a regulated environment, “we noticed eventually” is not a control.

What changed

The engineering team integrated Hyperpriors in two stages.

First, evaluation as CI. They built a regression suite from several hundred anonymised historical cases with known-good assessments, scoring policy-application accuracy, output structure, and narrative consistency. The suite runs on every prompt change, every configuration change, and — critically — on a schedule against the live provider endpoint, so unannounced model updates are caught even when nothing in the team’s own repository has changed. A failing score blocks promotion.

Second, tracing. Every production call now records the model version, full prompt, retrieved policy documents, and output. When behaviour is questioned, the answer comes from the trace, not from someone’s recollection of what was deployed that week.

Where it landed

In the first six months the eval gate caught three provider-side regressions, each identified and pinned to a model version before customers saw it. Compliance queries about specific underwriting decisions, previously a multi-day archaeology exercise, are now answered from audit trails in minutes.

The less expected benefit was mobility. When the team evaluated a second model provider, the same regression suite qualified the candidate in days rather than the six weeks their previous manual review had taken. Having an objective definition of “behaves correctly” turned provider choice from a leap of faith into a comparison.

07 — Begin

Put an eval gate between your models and your customers.

Request access All case studies