level-1-essentials/company/08-observability-evals/README.md

08 — Observability & Evals

You can't manage what you can't measure. Every LLM call is logged, every workflow has evals.

Files

• logging-and-redaction.md — what to log, what to redact, where it goes, who can see it
• eval-framework.md — how the company scores its AI workflows
• model-bake-off-protocol.md — quarterly protocol to test whether to switch models

Non-negotiables

1. Every LLM call flows through the gateway
2. PII/secrets redacted before storage
3. Every production workflow has an eval set
4. No model or prompt change without a regression run
5. Dashboards visible to Steering Committee

level-1-essentials/company/08-observability-evals/eval-framework.md

Eval Framework

The framework the company uses to score AI workflows. Extends the individual framework in shared/eval-templates/.

What we eval

For every deployed AI workflow, we track:

1. Correctness — does it produce the right output?
2. Format compliance — does it match the expected output shape?
3. Voice / brand — does it sound like us?
4. Safety — does it violate policy or reveal sensitive data?
5. Cost — dollars per resolved item
6. Latency — user-perceived speed
7. Adoption — % of eligible users using it
8. ROI — measured value against measured cost

Eval cadence

Ownership

• Workflow owner — the department lead that uses it. Owns the eval set content.
• AI Champion — technical partner that maintains the eval runner.
• Steering Committee — reviews quarterly aggregate; approves model changes.

Eval set structure

Per workflow:

evals/
└── <workflow-name>/
    ├── cases.jsonl
    ├── rubric.md
    ├── owner.md
    ├── history/
    │   ├── 2026-08-01_baseline.json
    │   └── 2026-08-15_v2.json
    └── results/
        └── [per-run scoreboards]

Minimum eval set

Per workflow, minimum: - 10 core cases — cover the workflow's main scenarios - 5 edge cases — inputs that reveal drift - 5 adversarial cases — prompt injection, boundary inputs, permission tests - 5 regression cases — historical failures now fixed

Total: 25 cases minimum. Some workflows have 200+.

Rubric mapping

Every case scored on 0-3 across dimensions:

• Correctness
• Format
• Voice
• Length
• Faithfulness (for grounded workflows)

Plus safety pass/fail: - No PII leak - No policy violation - No unauthorized action - Correct escalation

Judging

• Human judges for calibration (initial 3 rounds)
• LLM-as-judge with human spot-check (10% sample) for ongoing
• Automated checks for format compliance and safety (regex/structural)
• A/B test for anything user-facing, using real user feedback

Judge quality control

• Judges score their inter-rater agreement quarterly
• Judge prompt is versioned like any other prompt
• Judge model is one tier below production model (avoid conflict of interest)
• Judge output is auditable

Regression protocol

When model or prompt changes: 1. Run current eval set against baseline 2. Run against candidate 3. Compare per-dimension deltas 4. Ship only if: zero safety regressions, no Priority-1 correctness regressions, aggregate delta ≥ 0

Metric definitions

Correctness — output matches ground truth (exact for structured, judged for open-ended) Aggregate score — mean 0-3 across dimensions, safety-fail excluded Pass rate — % of cases where aggregate score ≥ 80% of max Cost per resolved item — total tokens × price + amortized platform cost / resolved items Adoption rate — % of eligible users making ≥ 1 call per week Deflection rate — % of tasks completed without human hand-off (for agentic workflows)

Publishing eval results

Every workflow has a public (within-company) dashboard:

• Current pass rate
• Trend over 8 weeks
• Latest model + prompt version
• Baseline pass rate
• Cost / resolved
• Adoption

Steering Committee reviews these monthly. Any workflow below its threshold is put on remediation.

level-1-essentials/company/08-observability-evals/logging-and-redaction.md

Logging and Redaction

Every LLM call is logged. Sensitive fields are redacted before storage. This is non-negotiable.

What to log

For every LLM request:

{
  "id": "req_abc123",
  "timestamp": "2026-08-19T15:30:00Z",
  "team": "sales",
  "project": "email-triage",
  "user_id_hash": "sha256-of-user-email",
  "tool": "chatgpt-enterprise",
  "model": "gpt-5-2026-06-01",
  "provider": "openai",
  "system_prompt_hash": "sha256-of-system-prompt",
  "input_tokens": 1234,
  "output_tokens": 567,
  "latency_ms": 3420,
  "cost_usd": 0.023,
  "success": true,
  "error_code": null,
  "cache_hit": false,
  "input_redacted": "[redacted prompt with PII scrubbed]",
  "output_redacted": "[redacted response]",
  "tool_calls": [...]
}

Redaction rules

Before any prompt or response is stored, run redaction:

Regex-based (fast, first pass)

• Email addresses → <EMAIL>
• Phone numbers → <PHONE>
• SSNs → <SSN>
• Credit cards → <CARD>
• IBANs → <IBAN>
• API keys (common patterns: sk-, ghp_, AKIA) → <KEY>
• IP addresses → <IP> (optional)

Model-based (slower, second pass for high-sensitivity)

• Use a small classifier or LLM to redact:
• Person names in restricted contexts
• Company names when restricted
• Sensitive medical/legal terms

Configurable per team

• Some workflows must retain data to be useful (e.g., a lease-extraction workflow retains names). Configure exceptions per project, not per request.

Where logs go

• Hot storage: LLM gateway's DB or observability platform for 30 days (Langfuse, Helicone, LangSmith, or in-house)
• Warm storage: Company data warehouse for 90 days (analytics)
• Cold storage: S3 / Glacier for 1 year (audit)
• Deletion: After retention window; secure deletion documented

Who can access

• Individual user: their own logs (rolling 30 days)
• Team lead: their team's logs (redacted content, full metadata)
• Steering Committee: aggregate + drill-down with justification
• Security: incident investigation with full access
• Everyone else: no access

Access logged and reviewed quarterly.

Retention conflict with debugging

Engineers sometimes need to see full prompts to debug bugs. Process:

1. Engineer files a debug request with the ticket ID
2. Security team approves
3. Full-fidelity log accessible for the specific request ID, for a bounded time (e.g., 24 hours)
4. Access logged

What NOT to log

• Passwords, keys, tokens even if user pastes them (redact first)
• Full customer PII in the clear
• HIPAA-protected content in the clear
• Anything the user has explicitly requested to be deleted (Right to be Forgotten under GDPR)

Compliance touchpoints

• GDPR: logs are personal data; retention limited; deletion honored
• HIPAA: if PHI is logged, BAA covers it and access controls enforce minimum-necessary
• SOC 2: logs demonstrate access control and monitoring
• CCPA: users can request their data

Dashboards

Standard dashboards published to the Steering Committee:

• Total spend per team, per week
• Cost per request per tool
• Model usage mix
• Error rate by provider
• Latency percentiles
• Cache hit rate
• Top workflows by volume
• Anomaly alerts (spend spike, error spike, unusual model use)

level-1-essentials/company/08-observability-evals/model-bake-off-protocol.md

Model Bake-Off Protocol

Run this every quarter. Determines whether to switch a workflow's default model.

Why

Providers release new models constantly. A model that was best last quarter may not be this quarter. A quarterly bake-off keeps you honest.

The protocol

Step 1 — Pick workflows to bake off

Pick the top 3 workflows by cost or by strategic importance. Not every workflow needs a quarterly bake-off.

Step 2 — Identify candidate models

For each workflow: - Baseline: current production model - Candidates: 2–4 alternatives (usually 1 same-provider newer + 1 competitor + 1 open-weight cheaper option)

Example for a summarization workflow: - Baseline: Claude Sonnet 4 - Candidate 1: Claude Sonnet 4.5 (newer version) - Candidate 2: GPT-5 (competitor) - Candidate 3: Llama 4 70B on Groq (open-weight cheap)

Step 3 — Freeze the eval set

Use the workflow's current eval set. Do not add cases mid-bakeoff.

Step 4 — Run each candidate

• Same system prompt
• Same eval inputs
• Fresh, not cached
• Multiple runs to average out variance (3 runs minimum)

Step 5 — Score everything

For each candidate:

Step 6 — Decide

Ship the candidate if: - Zero safety regressions - No Priority-1 correctness regressions - Aggregate quality ≥ baseline - Cost within budget - Latency within SLO

Reasonable trade-offs: - Small quality bump for material cost savings — often yes - Material quality bump for equal cost — yes - Large cost cut for small quality drop — case by case (only for low-stakes workflows)

Do NOT switch if: - Any safety fail - Voice degrades on brand-critical workflows - Latency spike breaks user experience

Step 7 — Roll out safely

• Canary: 5% of traffic for 3 days
• If clean, expand to 50% for a week
• If clean, full rollout
• Keep baseline available for rollback for 30 days

Step 8 — Document

Publish a bake-off report to Steering Committee: - Workflows evaluated - Candidates - Scores - Decision + reasoning - Rollout plan - Rollback plan

Automation

The bake-off should be scripted. A typical script:

for candidate in candidates:
    for case in eval_set:
        for run in range(3):
            output = model.generate(system_prompt, case.input, model=candidate.id)
            score = judge(output, case.expected, rubric)
            record(candidate.id, case.id, run, score, latency, cost)

report = aggregate(records)
publish(report, to="steering-committee-slack")

Anti-patterns

• Baking off with a tiny eval set (< 25 cases) — noisy signal
• Baking off only on benchmark data — not your real workflow
• Judging with the same model family that's being tested — biased
• Ignoring cost or latency — they're real user impact
• Making the decision by vibes — write it down