level-2-intermediate/shared/agent-brief-template.md

Agent Brief Template

Template

Name and owner: [agent name], [business owner], [technical owner].
User and job: [who uses it] needs to [specific outcome].
Non-goals: [what it must not decide or do].
Risk tier and data: [low/medium/high], [permitted data], [forbidden data].
Inputs / outputs: [typed input] → [typed output schema].
Architecture: [workflow, router, ReAct, plan-execute]; stop rules.
Tools: [name, read/write/propose, authorization, row/result cap].
Human gates: [exact triggers, approver role, fallback].
Quality contract: [critical metrics and thresholds].
Cost/latency: [unit budget, p95 target, model route].
Rollout/rollback: [shadow plan, flag, kill switch].

Worked example: Belle Lease Clause Extractor

Owner: Adam; technical owner: Belle Realty platform. User/job: leasing operations turns a signed lease into cited clause records. Non-goals: legal interpretation, tenant communication, database writes by the model. Risk/data: medium; current authorized leases only; never payment instruments or unrelated tenant data.

Input/output: {lease_id, document_version} → typed clause proposals with normalized value, exact quote, page, source chunk, confidence, and status. Architecture: deterministic ingestion and retrieval, structured extractor, one verifier pass; stop after two retrieval attempts. Tools: find_clause and get_page, read-only and lease-scoped; create_review_task, propose-only. Human gates: low confidence, OCR below 0.88, conflicts, and all final publication. Quality: 98% critical-field precision, 100% citation support, zero wrong-lease sources. Cost: target under $0.20 per lease; p95 under 90 seconds. Rollout: 30-lease shadow set, then one property; per-agent feature flag and writes disabled by default.

level-2-intermediate/shared/eval-set-template.md

Intermediate Eval Set Template

Case record

case_id, task, version, owner, data classification, source provenance, input, authorized context, expected output or rubric, expected evidence IDs, expected abstention/escalation, tags, risk tier, and notes. Keep immutable versions and split into development, locked regression, adversarial, and production-sampled sets.

Intermediate rubric

Score deterministic requirements first: valid schema; correct organization and entity scope; required fields; source citation presence; tool policy compliance; no prohibited side effect. Then score task behavior: factual correctness, evidence support/faithfulness, groundedness, completeness, calibrated confidence, correct abstention, escalation correctness, and usable tone. Record latency, tokens, model route, cache status, tool calls, and cost.

Required slices

Include normal cases, messy OCR, missing evidence, conflicting addenda, stale documents, ambiguous requests, irrelevant retrieved content, prompt injection, cross-tenant requests, malformed tool results, and low-confidence cases. For tenant triage include emergencies, fair-housing concerns, payment disputes, threats, attachments, and multilingual messages.

Release gates

Define critical failures as zero-tolerance: cross-tenant data, unsupported legal claim, unsafe write proposal, failed emergency escalation, or leaked canary. Compare candidate against baseline by slice, not only average. Use LLM judges only with documented calibration against human labels; require human adjudication for high-risk disagreements. Add every verified production failure to a candidate queue, then curate it before it becomes a regression case.

level-2-intermediate/shared/mcp-server-brief-template.md

MCP Server Brief Template

Server and owner: [name], [technical owner], [on-call owner].
Purpose: [specific capabilities], not [explicit exclusions].
Callers and auth: [clients], token audience, scopes, organization binding.
Data boundary: allowed entities/fields; forbidden fields; retention/log redaction.
Tools/resources: for each: name, purpose, strict input schema, output schema, read/propose/write classification, authorization rule, result cap, timeout, audit event.
Write policy: proposal shape, approver, commit endpoint, idempotency, rollback.
Rate limits: actor/org/tool limits, response-byte and query-cost caps.
Safety: prompt-injection assumptions, input validation, error taxonomy, kill switches.
Observability: trace fields, metrics, alerts, sampled review.
Eval/launch: isolation tests, malformed input tests, canaries, shadow plan, rollback.

Example outline: Belle Read Server

Purpose: provide authorized lease and maintenance facts to internal assistants. Callers use short-lived organization-bound tokens. Tools: get_lease_clause, list_open_maintenance, and get_property; each is read-only, RLS-backed, returns a fixed schema, and logs safe IDs plus row count. propose_create_ticket returns a draft only; the Belle app repeats authorization before persistence. Rate limits apply by actor and property. Tests include guessed lease IDs, cross-tenant semantic requests, oversized searches, stale addenda, and repeated proposal approval. Success means zero isolation failures, predictable p95 latency, and complete audit traces.

Completion check: the brief is not complete until an unauthorized-ID test, a rate-limit test, a proposed-write approval test, and a server-disable test have recorded expected results.

level-2-intermediate/shared/rag-pipeline-brief-template.md

RAG Pipeline Brief Template

Question and user: [who asks what decision-support question].
Corpus: source types, owner, authority, access labels, refresh target, retention.
Non-goals: [what RAG must not answer or decide].
Ingestion: source ID, checksum, extraction/OCR, normalized metadata, raw storage.
Chunking: structural rule, size/overlap, required metadata, version.
Indexing: embedding model, lexical index, namespace/filter strategy, active version.
Retrieval: authorization filters, query rewrite policy, top-k, hybrid method, reranker.
Answer contract: source citations, abstention threshold, freshness display, forbidden claims.
Evaluation: gold queries, recall@k, citation precision, groundedness, latency, cost.
Operations: reindex trigger, deletion/revocation path, SLOs, owner, rollback.

Example: Belle lease Q&A

Users: authorized staff answering lease-policy questions. Corpus: signed leases and governing addenda, with tenant/property scope and effective dates. Chunk by numbered clause and heading, preserve pages and precedence. Filter by organization and active lease before hybrid retrieval; rerank top 20 and answer from 3–6 passages. Every claim cites document/page; if no active evidence exists, return an abstention and review path. Targets: 100% authorized-result correctness and citation support; measure gold-clause recall@5, p95 latency, and cost per grounded answer. An addendum upload triggers selective reindex; permission revocation immediately removes serving eligibility.

Completion check: include one unanswerable query and one cross-tenant request in the launch set; both must fail safely with an observable trace.