Company AI should run on a shared platform, not dozens of hidden API keys and copy-pasted prompts. The platform owns model access, identity, routing, observability, configuration, budgets, and incident controls. Product teams own workflow behavior and evals.
Start small: one LLM gateway, one tracing convention, one secrets boundary, and one approved configuration path. Every request gets an organization, workflow, prompt version, model route, trace ID, cost record, and policy outcome. That is enough to compare workflows across teams without conflating data or spend.
The goal is not a grand internal platform. It is a paved road that makes the safe option the easy option: typed schemas, tenant-scoped tools, versioned prompts, auditable retrieval, and reversible rollouts.
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team's regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more customer or internal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
Put all provider calls behind a gateway. The gateway accepts a normalized request: authenticated actor, organization, workflow, model policy, messages or structured inputs, schema, budget, and trace context. It chooses an allowed provider/model, applies retries and timeouts, records tokens and cost, and returns a normalized response. Applications never hold broad provider credentials.
Use policies, not arbitrary model names. tenant_triage_classifier can use a fast approved model with a cost cap; lease_verifier can use a higher-accuracy model; sandbox experiments require an explicit flag. Enforce maximum context, output, requests, and concurrent calls. Support fallback only where schemas and evals prove compatibility.
The gateway must preserve provider response IDs, model version, cache status, latency, stop reason, and error class. Redact sensitive content in logs by default. Add per-workflow kill switches and organization budgets. Test provider outage, malformed schema, throttling, runaway retry, and model alias drift.
A gateway is where model choice becomes governable infrastructure instead of an application secret.
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
Trace every agent run from user request through retrieval, tool calls, model calls, validation, human review, and final outcome. Use one trace ID and standard span names: ingest, retrieve, rerank, model, tool, validate, approve, send. Record versions, timing, token/cost data, policy decisions, and safe metadata.
Do not default to storing raw tenant messages, leases, or hidden reasoning. Store references, hashes, redacted excerpts, and access-controlled samples. Give operators a trace viewer that answers: what evidence was used, which tool was called, why was this escalated, what did it cost, and which version produced it?
Create dashboards for success rate, safety-gate rate, retrieval quality, schema failures, latency, cost per successful task, human overrides, and error classes. Alert on cross-tenant denials, emergency false negatives, cost spikes, retrieval freshness lag, and unusual tool enumeration. Observability without an owner and response playbook is a museum.
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
Read this if: You are on call and an agent, RAG pipeline, or LLM gateway is misbehaving. You need the shortest path to "safe" — not a full incident write-up.
Stand-alone. You don't need the rest of this program to use it.
Severity, in one paragraph
SEV-1: The agent is actively doing harm right now — sending wrong emails, leaking data, spending money. Kill it. Then read the rest of this doc.
SEV-2: The agent is producing bad output but not acting on it (drafts a human is reviewing). Contain, do not necessarily kill.
SEV-3: Cost, latency, or eval regression. Fix during business hours.
SEV-4: A single user complaint. File a ticket. Do not page anyone.
If you're reading this at 2 AM, the question is: is anything reaching external humans or systems right now? If yes, SEV-1.
SEV-1: kill first, investigate second
Flip the kill switch. Every production agent must have one. If you don't know where it is, that's your first bug and it's serious.
Prefer a config flag (enabled: false) over deleting the deployment. You want to preserve state for post-mortem.
If the flag doesn't exist, revoke the agent's credentials at the source (IAM, API keys). Slower to recover, but effective.
Announce. Post in your incident channel: what agent, what symptom, what you just did.
Freeze inputs. If the agent is fed by a queue or webhook, pause the source too. You don't want a backlog to replay through a fixed agent that hasn't earned trust yet.
Snapshot state. Grab the last 100 traces, the current config, the eval history. Store in the incident folder before anything gets rotated.
Assign an owner if you're not it. The named owner of the agent is on point. If they're unreachable, their manager is.
SEV-2: contain, don't kill
The agent is drafting, not acting. You have time.
Flip auto-send to false on every tool the agent has, if it isn't already.
Increase the sampling rate on the human review — every draft gets reviewed, not one in ten.
Enable verbose logging so you can see what changed.
Do not roll back yet. You don't know what to roll back to. Diagnose first.
The diagnostic checklist
Work through these in order. Stop at the first one that explains the symptom.
Did anything deploy in the last 24 hours? Model, prompt, retrieval index, tool permissions, config. Check the change log. Rollback beats debug.
Is the underlying model behaving? Hit the model directly (no agent, no RAG) with a known-good prompt. If the raw model is bad, you have an upstream problem — check provider status page.
Is retrieval returning garbage? Log the top-K chunks for a failing query. If the chunks are irrelevant, the index is stale, poisoned, or the embedding model changed. See ../03-rag-at-scale/refresh-orchestration.md.
Did the tool interface change? External APIs the agent calls sometimes silently change response shape. Check the tool call logs for HTTP 200 with unexpected payload shape.
Is a specific user or tenant triggering it? Filter traces by user_id. If it's one tenant, you have a data problem in their scope — see ../03-rag-at-scale/multi-tenant-rag.md.
Is it prompt injection? If the failing inputs contain instructions targeting the agent ("ignore previous instructions", "forget your rules"), you're being attacked. Escalate to security.
Keep secrets in a managed secret store, injected at runtime with least privilege and rotation support. Never place provider keys, Supabase service keys, webhook secrets, or tenant credentials in prompts, client bundles, logs, eval fixtures, or source control. Give each environment separate credentials and make production writes impossible from local development by default.
Configuration is different from secrets. Store model routes, prompt versions, thresholds, feature flags, budgets, and tool allowlists in versioned, reviewed configuration. Validate config schemas at deploy time. Every request should record the effective config version so behavior can be reproduced.
Scope keys by service and purpose. An ingestion worker should not be able to send tenant messages; an MCP read server should not have service-role write authority. Rotate after exposure, revoke on offboarding, and audit secret access. Use canary credentials to detect accidental disclosure.
The operational test is simple: if a trace, screenshot, or support export leaks, does it expose something an attacker can use? Design for no.
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
SSO and access: who can use, deploy, and stop your agents
Read this if: You are about to give the third engineer on your team access to a production agent, or the AI vendor you signed up for is asking about SAML, or you can't answer "which employees can talk to our RAG?" in under 30 seconds.
Stand-alone. No prerequisites in this program.
The three access questions every AI system has to answer
Who can use it? (Runtime access — sending prompts, calling tools)
Who can change it? (Build access — modifying prompts, tools, retrieval indexes, model choice)
Who can stop it? (Kill switch access — pausing, disabling, or retiring the agent)
If those three groups are the same group, you have a governance problem. If any of them is "everyone," you have a bigger one.
The minimum bar for any production agent
[ ] Authentication is SSO-backed (SAML or OIDC through your identity provider)
[ ] Authorization uses groups, not individual users
[ ] Group membership is auditable (who added, when, why)
[ ] Deprovisioning is automatic when someone leaves (via SCIM or an equivalent)
[ ] The kill-switch is reachable by at least two named humans who are always on shift or reachable
An agent with hardcoded API keys or shared logins is not production-ready. It's a demo.
Recommended group layout
Copy this into your identity provider. Adjust naming to your org's conventions.
Group
Purpose
Typical members
ai-users-<agent-name>
Runtime access — send prompts, receive drafts
End users of that specific agent
ai-reviewers-<agent-name>
Review drafts before send (Category 2 approval)
People with the authority to send on the company's behalf
Every agent gets its own ai-users- and ai-reviewers- group. Owners groups can be shared across small related agents. Admins and auditors are org-wide.
Vendor-side SSO checklist
Every AI vendor you use should support the below. If they don't, know what you're accepting.
[ ] SAML or OIDC, not password + 2FA
[ ] SCIM provisioning so deprovisioning happens without a support ticket
[ ] Audit log export so you can see who accessed what, when
[ ] IP allowlist for API tokens (defense in depth)
[ ] Least-privilege service accounts — no personal accounts running production
[ ] Session timeout appropriate to the sensitivity (short for anything that acts, longer for read-only)
[ ] MFA required for the underlying identity provider (SSO is only as strong as the IDP)
If your vendor doesn't support SCIM, you need a manual quarterly access review, and you should assume someone will keep access longer than they should.
Service accounts and API keys: the actual dangerous surface
SSO covers humans. Service accounts and API keys are where breaches happen. Rules:
One key per agent per environment. Not "the shared prod key."
Named owner on every key. Rotate when they leave.
Short-lived where possible. OAuth client credentials with 1-hour token expiry beats a static bearer token.
Scoped, not root. The agent's key can access exactly the tools it needs, nothing else.
Stored in a secrets manager. Never in git, never in .env files committed to shared drives, never in Slack.
Rotated on a schedule — quarterly minimum, immediately on any incident, immediately on team departure.
If you have keys older than a year and haven't rotated because "it might break something," you already know it will break at the worst possible time. Rotate now during business hours.