Cost should be attributable to a team, project, or user — never a black hole. Set this up before you scale.
Attribution model
Every AI call carries three tags:
- Team (Sales, Support, Engineering, etc.)
- Project / workflow (email-triage, lease-generation, etc.)
- User (email of the requester)
The LLM gateway enforces this by rejecting untagged calls.
Budget tiers
Start with modest per-team monthly caps, review quarterly.
Team size
Suggested monthly AI budget
Individual power user
$50–$200
5-person team
$500–$1,500
20-person department
$3,000–$8,000
100-person org
$10,000–$40,000
These are starting points. Actual = seats × usage × tool mix.
Two budget lines per team
Subscriptions (seats × per-seat cost)
API usage (tokens × prices)
Report both. Ratio typically 60–70% subs / 30–40% API for knowledge workers; inverse for engineering teams building AI features.
Cost controls
Hard cap on API keys (provider dashboards)
Soft alerts at 50% and 80% of hard cap
Rate limits per team via LLM gateway
Weekly cost report to team leads
Monthly review by Steering Committee finance seat
Chargeback vs shared cost
Two philosophies:
Chargeback: each team's spend hits their budget line
- Pros: Motivates efficient use, exposes ROI
- Cons: Discourages experimentation
Shared: IT/Finance owns the total budget
- Pros: Encourages adoption
- Cons: Free-rider problem
Recommendation: Start with shared during 90-day rollout. Move to chargeback once teams have baselined their usage.
ROI tracking
For each department pilot, define:
- Hours saved / week (self-reported + spot-checked)
- Quality delta (eval scores before vs after)
- Employee NPS on the AI experience
- Cost per resolved item (ticket, deal, deliverable)
Publish a quarterly ROI report. Kill any workflow with poor ROI.
Cost anti-patterns
Every engineer with a Claude API key. Use one org account.
Frontier models used for tasks a workhorse could handle (e.g., Opus for classification)
Chatting in a UI when batch API would do
No prompt caching enabled on providers that support it
Duplicated automations across teams
Cost optimization playbook
When a budget line is over:
1. Audit top 10 workflows by cost — is model choice appropriate?
2. Turn on prompt caching where supported
3. Cache retrieval results
4. Move batch workloads to batch API tier (50% off on major providers)
5. Swap frontier → workhorse where quality allows
6. Kill low-ROI automations
An LLM gateway is middleware between your apps and AI providers. It handles:
- Routing (which provider / model for which request)
- Logging (every prompt + response, redacted)
- Cost attribution (per team / project / user)
- Rate limiting (protect from runaway spend)
- PII redaction (before hitting provider)
- Fallback (auto-retry with alternate provider on failure)
- Caching (semantic + prompt caching)
Why you need one
Without a gateway:
- No cost attribution per team
- No unified logs for audits or evals
- No PII scrubbing
- No fallback when a provider is down
- Each team manages their own keys
With a gateway:
- Central point of control, observability, and policy
Options
Portkey (SaaS)
Best for: Mid-market to enterprise wanting a managed solution
Pros: Fast setup, deep observability, good fallback logic, virtual keys per team
Cons: SaaS (data crosses their systems — get DPA), cost scales with volume
Pricing: Free tier; paid starts ~$99/mo
LiteLLM (open source, self-hosted)
Best for: Engineering-forward teams that want full control
Pros: Self-hosted (data stays with you), free, active community, wide provider support
Cons: You operate it (uptime, upgrades, scaling)
Pricing: Free; hosting cost
Vercel AI Gateway
Best for: Teams already on Vercel
Pros: Zero setup, integrated with Vercel apps, good observability
Cons: Vercel platform lock-in
Pricing: Usage-based
Cloudflare AI Gateway
Best for: Cost-sensitive routing at edge
Pros: Cheap, edge caching, easy setup
Cons: Lighter feature set than Portkey / LiteLLM
Pricing: Free tier generous
OpenRouter (as a gateway)
Best for: Small teams experimenting
Pros: One API key, 300+ models, dead simple
Cons: Not a full gateway (limited redaction, weaker auth model for enterprise)
Pricing: Small margin over provider prices
Roll your own
Best for: Large orgs with unique requirements or existing infra
Pros: Perfect fit
Cons: Non-trivial build; ongoing maintenance
Decision matrix
Requirement
Portkey
LiteLLM
Vercel
Cloudflare
OpenRouter
Roll-your-own
Managed (no ops)
✅
❌
✅
✅
✅
❌
Self-hosted option
⚠️
✅
❌
❌
❌
✅
Cost attribution per team
✅
✅
⚠️
⚠️
⚠️
✅
PII redaction
✅
✅
⚠️
⚠️
❌
✅
Semantic caching
✅
✅
✅
✅
❌
✅
Multi-provider routing
✅
✅
✅
✅
✅
✅
Enterprise SSO
✅
Config
✅
✅
⚠️
✅
DPA + zero-retention
Ask
You control
Yes (Vercel)
Yes (CF)
Ask
You control
Recommended defaults
Startup / < 50 employees: Cloudflare AI Gateway or Portkey Free
Mid-market / 50–500: Portkey Paid or self-hosted LiteLLM
Enterprise: Self-hosted LiteLLM or roll-your-own on Azure/AWS with existing security tooling
Setup checklist
Regardless of choice:
- [ ] Gateway sits between all apps and all providers
- [ ] Every request tagged with team, user, purpose
- [ ] Costs report daily to owners
- [ ] Rate limits per team
- [ ] PII redaction before send (regex + optional model-based)
- [ ] Full request/response logging (encrypted, with retention policy)
- [ ] Fallback rules per model
- [ ] Alerting on cost spikes, error rate, 5xx