Skip to content

Case study

A legal-tech research company

A legal-technology company whose research agents answer complex legal questions for law firms by decomposing them into multi-step plans — searching case law, reading judgments, cross-referencing statutes, and assembling cited memoranda. A single query can involve dozens of model calls and tool invocations.

01 — In their words

“Before, a failed research run was a support ticket and a shrug. Now it's a trace we can replay step by step. The first week we had it, we fixed a retrieval bug that had been costing us for months and that nobody had been able to reproduce.”

Recorded verbatim · Attribution withheld

02 — The work

Challenge

Agent runs failed invisibly in the middle of the chain. A retrieval step would return weak results, a summarisation step would quietly drop a dissenting judgment, or a tool call would time out and the agent would improvise around the gap — and the run would still complete, producing a memorandum that looked finished but was built on a broken step. With no visibility below the level of the final answer, failures could not be reproduced, debugging meant re-running the whole query and hoping, and retries silently multiplied token spend.

Solution

The engineering team moved the agents into a Hyperpriors harness. Every step — each model call, tool invocation, retrieval, and intermediate decision — is traced with its full inputs and outputs, and any run can be replayed deterministically from any step. Guardrail checks validate that cited cases exist in the retrieved sources and that quotations match the underlying judgments. Retry and fallback behaviour moved out of ad-hoc application code into declared harness policy, making the cost of every recovery path visible and bounded.

03 — The system

HYPERPRIORSCONTROL PLANEHARNESSGUARDRAILSEVAL GATESTRACESMATTER INTAKEQUERIES · BRIEFSAUTHORITY DBCASES · CITATIONSCLAUDE — AGENT STEPSMODEL CALLSVERIFIER — CITATIONSSOLICITORSCITATION SIGN-OFFUNCERTAINTY ROUTES TO PEOPLEAUDIT RECORDREPLAYABLE RESEARCHEVERY STEP, REPLAYABLE
Fig. 1 — System architectureLegal technology · Illustrative topology

04 — Results

The company runs its multi-step research agents inside a Hyperpriors harness, with step-level tracing, deterministic replay, and automated citation checks on every production run.

  1. 01

    Previously unreproducible mid-chain failures now replayed and diagnosed from traces, cutting typical time-to-diagnosis from days to under an hour

  2. 02

    Cost per query down 38 percent after traces exposed redundant retrieval calls and unbounded retry loops

  3. 03

    Automated citation checks on every run, with unverifiable citations blocked from final memoranda rather than discovered by clients

05 — The record

Industry
Legal technology
Location
North America
Model providers
Two hosted frontier-model providers
Workloads
Multi-step research agents, retrieval, cited drafting

06 — The full account

The situation

The company’s research agents are genuinely agentic: a single legal question becomes a plan, the plan becomes dozens of steps, and the steps span case-law search, judgment analysis, statute cross-referencing, and drafting. When everything works, the output is a cited memorandum a junior associate would be pleased with.

When something failed, it failed in the middle — and invisibly. A retrieval step returning thin results, a summarisation quietly losing a dissent, a timed-out tool call the agent papered over. The run still completed. The memorandum still looked like a memorandum. The defect was buried four steps deep in a chain nobody could inspect.

Debugging consisted of re-running the query and hoping the failure recurred, which — this being a non-deterministic system — it usually declined to do. Meanwhile retry logic scattered through the codebase re-ran expensive steps with no budget or record, and token spend per query crept steadily upward without anyone being able to say where it went.

What changed

The team rebuilt the agent runtime on a Hyperpriors harness. Three things followed.

Every step became observable: model calls, tool invocations, retrievals, and intermediate decisions are traced with complete inputs and outputs, so a finished run is no longer a black box with an answer attached.

Every run became replayable. An engineer can take a failed production trace and replay it deterministically from any step, changing one variable at a time — the debugging loop that had been impossible now takes minutes.

And recovery became policy rather than folklore. Retries, fallbacks, and step budgets are declared in the harness, applied uniformly, and accounted for in the trace, instead of living in whichever try/except block someone added during an incident.

Guardrail checks were added at the drafting stage: every citation in a memorandum must resolve to a document the agent actually retrieved, and quotations must match the source text.

Where it landed

Mid-chain failures that had been open mysteries for months were replayed, diagnosed, and fixed within the first weeks — typical time-to-diagnosis fell from days to under an hour. Traces exposed redundant retrieval calls and one unbounded retry loop; removing them cut cost per query by 38 percent. And citation checking now runs on every production memorandum, so an unverifiable citation is a blocked run and an internal alert, not an awkward email from a client.

07 — Begin

See every step your agents take, and replay the ones that go wrong.

Request accessAll case studies