Evaluation: Build a Test Suite Before You Build Trust
An agent without evals is a demo. It may look competent until a lease addendum, angry tenant message, or incomplete data room exposes the exact behavior that matters. Evaluation turns “it seems good” into a measurable contract: these inputs, these outputs, these safety rules, this acceptable regression rate.
Use three layers. Offline evals test curated examples before deployment. Online evals watch production traces, outcomes, and sampled reviews. Human evals cover nuanced correctness, policy, and usefulness that automated checks cannot decide. All three need versioned datasets, prompt and model identifiers, and a baseline to compare against.
For Adam’s projects, create task-specific evals. Lease extraction needs field accuracy and citation support. Tenant triage needs emergency recall and unsafe-action recall. Deal diligence needs blocker recall and false-complete rate. Generic “helpfulness” scores hide the failure modes that cost money or trust.
The supporting guides show how to run a harness, where an LLM judge is valid, and how to prevent a prompt or model change from quietly damaging a working workflow. Evals are not a final QA phase; they are the specification that drives architecture.
Before adding complexity, run this design through a small representative eval and inspect the trace with the operator who will own failures. Make the boundary, escalation, and rollback visible in the product. That discipline will expose more useful work than another round of prompt cleverness.
An eval harness is a repeatable runner that feeds versioned cases to a workflow and scores the outputs. Each case includes input, permitted context, gold answer or rubric, tags, risk tier, and expected abstention behavior. The runner records model, prompt version, tool versions, latency, tokens, trace, output, and scores. Without this metadata you cannot explain a regression.
Start with deterministic checks. Validate JSON Schema, required citations, valid clause types, no cross-tenant IDs, and policy gates. Then add task metrics: field-level exact match for lease clauses; routing accuracy and emergency recall for OTB messages; checklist-status accuracy and blocker recall for Deal Leverage. Normalize formats before comparing dates and money, but never normalize away substantive differences.
Partition cases into smoke (fast, must-pass), core (representative), edge (messy), and adversarial (hostile). Keep a locked regression set that builders cannot tune against casually, plus a development set for iteration. Tag slices such as scans, addenda, Spanish messages, emergency keywords, missing documents, and ambiguous requests. Aggregate scores are useful; slice failures decide whether you ship.
Run the harness on every model, prompt, retrieval, tool, or schema change. Define gates in advance: zero cross-tenant leaks, 100% citation support for legal facts, no decline in emergency recall, and no more than a defined cost increase. Store failing cases as first-class work items. An eval that merely produces a dashboard is not finished; it must block unsafe changes.
An LLM judge is useful when the criterion is linguistic or comparative and a human rubric can be written clearly. It is not an oracle. Use it to rate whether a tenant draft follows tone and policy, whether an answer is supported by supplied evidence, or which of two summaries is more complete. Do not use it as the only judge of financial, legal, safety, or exact extraction facts.
Give the judge a narrow packet: task, rubric, candidate answer, approved evidence, and required JSON verdict. Ask it to identify specific violated criteria and evidence, not to produce vague praise. Example fields: grounded, citation_supported, policy_safe, missing_requirements, severity, and rationale. Blind the judge to model name and experiment hypothesis to reduce bias.
Calibrate it against a human-labeled set. Measure agreement by category, inspect disagreements, and tune the rubric—not the score threshold alone. Use multiple judges or pairwise comparison for consequential subjective work. A small high-quality human panel remains the arbiter for borderline results.
Never let a judge grade its own output in the same context. Keep generation and judgment separate. Watch for verbosity bias, position bias, style bias, and a tendency to reward confident unsupported prose. For RAG, force the judge to mark each claim against the given passages; otherwise it may use its own background knowledge and falsely approve a hallucination.
Treat judge prompts and models as governed dependencies. Version them, evaluate them, and periodically re-calibrate.
Put evaluation in the change path. A pull request that changes a prompt, model, retrieval setting, tool definition, schema, or policy must run a smoke eval before merge. The result should compare candidate versus baseline by metric and critical slice, with links to representative traces. “Green” means the predefined gates passed; it does not mean the output looked polished.
Use tiers. Fast CI runs 20–50 deterministic and high-risk cases. Nightly runs the full regression set, multiple seeds where relevant, retrieval metrics, and a cost/latency report. Weekly, sample production failures into a candidate set for human review. Keep external-model drift in mind: a model alias can change without a code deploy, so pin versions where possible and schedule baseline re-runs.
Set explicit failure policy. Block immediately for schema violations, data-boundary failures, unsafe action proposals, citation failures on high-stakes answers, or emergency-recall drops. Warn for small noncritical movement, but require an owner and follow-up. Permit intentional behavior changes only with updated requirements, dataset changes, and documented approval.
Save artifacts, not just scores: input snapshot, retrieved chunks, tool outputs, output, trace, prompt, model settings, and normalized score. This makes failed examples debuggable instead of mystical. Do not automatically rewrite the golden answer after every production complaint; investigate whether the complaint identifies a true requirement or a one-off preference.
CI is how you keep the agent you shipped yesterday from becoming a different agent tomorrow.