Decision title: Status: proposed / accepted / superseded / retired Date / owner / approvers: Decision expiry or review date:
Context
Describe the customer problem, production constraints, and decision boundary. Include the affected workflows, properties or tenants, data classes, dependencies, security and safety considerations, expected volume, latency target, and unit-cost target. Link the relevant system card, evaluations, incident history, and vendor terms.
Options considered
Option
Benefits
Risks / constraints
Evidence
Estimated cost
Reversibility
Do not compare only model quality. Include operational ownership, data residency, permission model, observability, failure modes, migration effort, and customer impact. For example, an open-weight deployment may lower marginal cost while increasing on-call and security burden.
Decision
State exactly what will be used, by whom, for which cohort, and under which controls. Record the acceptance criteria: offline evaluation threshold, staged rollout plan, SLOs, guardrails, cost cap, and rollback condition. Name the person authorized to stop or reverse the decision.
Consequences and follow-up
List accepted tradeoffs, unresolved risks, documentation updates, migrations, training, and time-bound follow-up actions. Link implementation pull requests, dashboards, feature flags, and launch evidence. A decision record should let a future operator understand why a configuration exists and when it must be reconsidered.
Title: Incident ID: Severity / start / end: Incident lead / communications lead / technical lead: Affected feature, properties, tenants, and channels:
Executive summary
Describe what users experienced, how many were affected, the worst plausible harm, and the final state. Use concrete language: “The maintenance assistant showed one tenant another tenant’s ticket summary for 17 minutes,” not “there was an issue with retrieval.” State whether any unsafe tool action, legal obligation, or data notification process was triggered.
Timeline
Time
Event
Evidence / decision
Detection
Containment
Customer communication
Recovery
Impact and scope
List confirmed impact, uncertain impact, and population checked. Segment by property, tenant boundary, model/prompt/retrieval version, provider, region, and release cohort. Include customer contacts, financial effect, support load, and safety or privacy implications.
Technical and control analysis
What changed? What condition allowed the failure? Which controls detected it, failed to detect it, or limited blast radius? Link traces, logs, deployment records, policy decisions, eval evidence, and the exact configuration. Do not attribute root cause to “the model hallucinated”; name the missing validation, ambiguous contract, bad data, unsafe permission, or untested dependency.
Corrective actions
Action
Type: containment / corrective / preventive
Owner
Due date
Verification evidence
Lessons and follow-up
Record what should change in the runbook, eval suite, dashboard, release gate, vendor strategy, or training. The incident is closed only after actions are verified, affected users receive required follow-up, and the learning is shared with the relevant team.
Use this checklist as a release gate. Mark each item yes, not applicable, or blocked, with evidence and an owner. A launch is not complete because every box is checked; it is complete when the remaining risk is understood and explicitly accepted.
Product and scope
[ ] User problem is written in one sentence.
[ ] Target user and excluded users are named.
[ ] High-impact decisions are identified.
[ ] Non-AI alternative is documented.
[ ] Success metric is measurable.
[ ] Harm metric is measurable.
[ ] Customer-facing limitations are defined.
[ ] Human escalation path exists.
[ ] Feature owner is named.
[ ] Domain approver is named.
Data and privacy
[ ] Data-flow diagram is current.
[ ] Every source has an owner.
[ ] Tenant isolation is tested.
[ ] Authorization is enforced before retrieval.
[ ] PII inventory is complete.
[ ] Minimum-necessary fields are used.
[ ] Sensitive fields are redacted from logs.
[ ] Retention periods are defined.
[ ] Deletion path is tested.
[ ] Consent requirements are documented.
[ ] Vendor data terms are reviewed.
[ ] Regional residency is enforced.
[ ] Backups follow retention policy.
[ ] Dataset provenance is recorded.
[ ] Data freshness is visible.
Model and prompting
[ ] Model version is pinned.
[ ] Provider fallback is defined.
[ ] Prompt version is pinned.
[ ] System instructions state scope.
[ ] Untrusted content is labeled.
[ ] Context budget is bounded.
[ ] Structured output schema is validated.
[ ] Temperature and limits are intentional.
[ ] Model card is reviewed.
[ ] Known limitations are recorded.
[ ] Fine-tuning data is approved.
[ ] Retrieval index version is recorded.
[ ] Citation behavior is specified.
[ ] Refusal behavior is specified.
[ ] Model change procedure exists.
Retrieval and tools
[ ] Chunking strategy is documented.
[ ] Retrieval access is tenant-scoped.
[ ] Metadata filters are tested.
[ ] Stale documents have expiry rules.
[ ] Reindex plan is reversible.
[ ] Tool list is allowlisted.
[ ] Tools use least privilege.
[ ] Tool parameters are typed.
[ ] Irreversible actions require confirmation.
[ ] Idempotency keys protect writes.
[ ] Rate limits are set.
[ ] Tool retries are bounded.
[ ] Tool results are sanitized.
[ ] Audit events exist for writes.
[ ] Manual fallback works.
Safety and security
[ ] Threat model is approved.
[ ] Prompt injection tests pass.
[ ] Jailbreak tests pass.
[ ] Exfiltration tests pass.
[ ] Cross-tenant tests pass.
[ ] Identity spoofing tests pass.
[ ] Retrieval poisoning tests pass.
[ ] Abuse rate limits are tested.
[ ] Policy decisions have reason codes.
[ ] Guardrail bypass is monitored.
[ ] Secrets are not in prompts or logs.
[ ] Credentials are short-lived.
[ ] Vulnerability owner is named.
[ ] Red-team findings are tracked.
[ ] Security incident path is rehearsed.
Evaluation and release
[ ] Offline eval set is versioned.
[ ] Eval set is representative.
[ ] Hard negatives are included.
[ ] Safety evals are included.
[ ] Quality threshold is predeclared.
[ ] Cost threshold is predeclared.
[ ] Latency threshold is predeclared.
[ ] Results are segmented.
[ ] Human review rubric exists.
[ ] Statistical uncertainty is considered.
[ ] Shadow run is completed.
[ ] Canary cohort is defined.
[ ] Feature flag exists.
[ ] Dark-launch behavior is safe.
[ ] Rollback has been rehearsed.
Reliability and operations
[ ] SLOs are documented.
[ ] Error budget policy exists.
[ ] Availability is measured at the user edge.
[ ] Time to first useful response is measured.
[ ] Quality outcome is sampled.
[ ] Traces propagate across services.
[ ] Logs are structured.
[ ] Dashboard links to traces.
[ ] Alerts have runbooks.
[ ] Alert owner is named.
[ ] On-call rotation is staffed.
[ ] Incident severity model exists.
[ ] Status communication is prepared.
[ ] Queues have backpressure limits.
[ ] Provider outage behavior is tested.
[ ] Regional failover is tested.
[ ] Capacity is measured.
[ ] Cost per outcome is tracked.
[ ] Drift baseline exists.
[ ] Drift alerts have response actions.
UX, governance, and follow-through
[ ] Loading state is honest.
[ ] Cancellation works.
[ ] Error message offers a next step.
[ ] User corrections are captured.
[ ] Confidence language matches evidence.
[ ] Citations appear where decisions happen.
[ ] Accessible interaction is tested.
[ ] Admin controls are documented.
[ ] Change approval is recorded.
[ ] Decision record is stored.
[ ] Training is provided to operators.
[ ] Support team has a playbook.
[ ] Post-launch review is scheduled.
[ ] Success and harm metrics are reviewed.
[ ] Unused data is removed.
A failed item is useful information. Do not paper over it with an exception; convert it into a conscious launch decision, a compensating control, a named owner, and an expiry date.
Assessment name / date / owners: Systems and versions assessed: Authorization and rules of engagement: Test data classification and retention:
Objective and scope
State the harm hypotheses: cross-tenant disclosure, prompt injection through listing documents, unsafe maintenance escalation, tool misuse, identity spoofing, policy bypass, or denial of service. Name the in-scope model, retrieval sources, tools, identities, environments, and explicit exclusions. A red-team report without scope cannot be safely reproduced or prioritized.
Method
Describe attack families, test cases, automation, human review, success criteria, and stop conditions. Preserve exact prompts, documents, identities, configuration, timestamps, and traces in a restricted evidence store. Separate reproducible facts from the tester’s interpretation.
Findings
ID
Severity
Attack path
Preconditions
Impact
Reproduction
Evidence
Owner
Critical / high / medium / low
Score severity using exploitability, blast radius, sensitivity of data or action, likelihood, detection time, and effectiveness of existing controls. Demonstrate impact without accessing real customer data or causing irreversible action.
Remediation and retest
For each finding, specify immediate containment, durable fix, compensating control, due date, and retest criteria. A model-prompt adjustment alone is rarely sufficient for a permission-boundary problem. Close a finding only after the exact attack fails, regression tests are committed, telemetry can detect recurrence, and risk acceptance is recorded if any residual exposure remains.
level-3-advanced/shared/slo-worksheet.md
AI SLO Worksheet
Use one worksheet per customer-facing journey. An SLO is an internal target for reliable, valuable behavior. It should be strict enough to protect trust and practical enough that the team can measure and improve it.
Avoid a vague “accuracy SLO.” For a leasing assistant, define whether the answer was based on an authorized source, whether it disclosed uncertainty, whether it avoided protected or sensitive inference, and whether the next action was correct. Quality often needs sampled human labels, so state sampling rate, reviewer rubric, lag, and the operational response to a bad result.
Operational plan
Containment action when fast burn occurs:
Owner and escalation path:
Feature flag or rollback control:
Customer communication path:
Manual fallback:
Dependencies whose failures count toward the SLO:
Dependencies whose failures are excluded, with reason:
Dashboard and trace links:
Review monthly and after every material incident. If a target cannot drive a decision, it is a report, not an SLO.