Cost Engineering: Spend Like It Is Production Infrastructure
AI spend becomes accidental when requests, retries, prompts, context, and models are invisible. Cost engineering makes each workflow intentional: measure cost per successful outcome, remove unnecessary tokens, route easy work to cheaper paths, and refuse to trade away safety for pennies.
Track costs by application, organization, workflow, model, input tokens, output tokens, cache reads, cache writes, tool calls, retries, and human-review outcome. “Monthly model bill” is not a metric. For Belle Realty, compare cost per correctly extracted lease, per safely triaged message, and per diligence checklist completed. Add avoided staff time only after measuring it.
The cheapest model is not always the low-cost model. A weak classifier that sends half of tenant messages to review can be more expensive than a stronger one. Likewise, a giant prompt may cost little per request but create slow, unsupported answers that staff must correct. Optimize in order: eliminate calls, eliminate context, cache safe results, cascade models, then tune output length.
The guides cover semantic caching, model cascading, prompt compression, and workflow accounting. Cost constraints belong in the product contract and CI gates, not in a cleanup sprint after adoption.
Before adding complexity, run this design through a small representative eval and inspect the trace with the operator who will own failures. Make the boundary, escalation, and rollback visible in the product. That discipline will expose more useful work than another round of prompt cleverness.
Calculate cost per completed, correct workflow—not cost per chat. For each run, record model input/output tokens, cache reads/writes, embeddings, reranking, OCR, tool infrastructure, retries, and human-review minutes. Attribute shared platform costs separately so an individual agent is not falsely cheap because observability and storage are invisible.
Define a successful unit. The lease extractor succeeds when required fields are correct, cited, and accepted without material rework. Tenant triage succeeds when the disposition and escalation are correct, not when a response is merely sent. Deal diligence succeeds when checklist rows are evidence-backed and blockers are surfaced. Then calculate: total variable cost divided by successful units; human minutes per successful unit; and cost of prevented errors where that can be measured.
Build a dashboard by workflow, organization, version, model route, and outcome. Set a budget envelope and trigger investigation when cost per successful unit rises, cache hit rate drops, retry rate increases, or escalations change. Compare against the baseline manual process, including delay and error cost—not a fantasy of zero labor.
Use this data to choose architecture. If a second verifier adds 8% cost but removes 70% of unsupported lease clauses, keep it. If a planner adds tokens without improving checklist completeness, remove it. Cost engineering is product judgment backed by telemetry, not a mandate to choose the smallest model.
Model cascading sends easy work to a cheap model and escalates only when uncertainty, risk, or complexity demands it. It works when the handoff criteria are measurable. “Use the smart model when the cheap one feels unsure” is not an engineering plan.
For OTB triage, use deterministic emergency rules first, then a low-cost classifier for intent and confidence. Escalate to a stronger model when confidence is below threshold, multiple intents compete, attachments need interpretation, or policy-sensitive labels appear. For lease extraction, use a cheap locator to select candidate clauses and a stronger structured extractor only when text is noisy or the verifier fails. The final safety gate remains policy and human review, not model tier.
Make escalation visible in the trace: stage, model, reason, confidence, result, and cost. Calibrate confidence against labeled examples; raw model probability is not automatically reliable. Use an abstain route for cases where a better model will not create safe certainty.
Evaluate the cascade as one system. Report quality, escalation rate, cost per successful task, tail latency, and performance across risk slices. A cascade that saves 60% but misses emergencies is a failure. Limit tiers to two or three; complex chains are hard to debug and can hide repeated calls.
Default policy: cheap model for classification and formatting; capable model for ambiguity and synthesis; human for irreversible or policy-sensitive outcomes.
Prompt compression means removing tokens that do not change a correct outcome. It does not mean stripping the evidence, policy, or schema that makes the outcome safe. Most waste comes from repeating product lore, full conversation histories, entire documents, and verbose examples on every call.
Separate stable instructions from dynamic context. Put stable system rules in a versioned template and use provider context caching when it is safe. Replace prose policy with compact decision tables. Retrieve the 3–6 relevant lease chunks instead of inserting a whole lease. Summarize old conversation turns into a structured state: open issue, prior commitments, unresolved question, sentiment, and last staff action. Preserve source links for anything that will be relied on.
Use structured outputs to eliminate “explain your reasoning” text when the application needs only a label or object. Cap output tokens. Ask the model to return a concise draft plus required fields, not an essay followed by JSON. Remove duplicate instructions and contradictory examples; they cost money and make behavior less predictable.
Test compression on the eval harness. Compare quality, citation support, token count, latency, and cost by slice. If a shorter prompt fails on addenda or ambiguous notices, restore the missing context only for that route. Do not compress away adversarial instructions, escalation rules, or human-approval requirements.
Good compression changes the shape of context. It makes relevant facts denser, not merely shorter.
Semantic caching answers a new request from a previous, meaningfully similar result instead of calling a model again. It is valuable for repetitive low-risk questions: “What is the office number?”, “How do I submit a maintenance request?”, or standardized lease policy explanations. It is dangerous when the answer depends on tenant, date, lease version, account state, or live work-order status.
A cache key must include the organization, authorization scope, workflow version, model/prompt version, policy version, source version, and normalized intent—not just the text embedding. Store the final answer, cited sources, confidence, creation time, and expiration. Never share entries across tenants or properties unless the content is deliberately public and identical.
Use a two-stage lookup: exact cache first, then semantic similarity above a conservative threshold. Require a matching intent and metadata scope. For Belle Realty, cached answers about a property’s office hours can be safe; a cached answer about late fees must include the exact lease/template version and may still require retrieval. Invalidate on policy, document, or prompt change. Time-bound live-data answers aggressively.
Measure hit rate, false-hit rate, cache cost, latency saved, and downstream correction rate. Review sampled hits before expanding scope. A wrong cached answer is worse than a slow fresh one because it appears confident and repeats at scale. Start with pre-approved FAQ answers and retrieval snippets, not open-ended agent outputs.
The rule: cache stable meaning, never cached assumptions about a person’s current situation.