token-cost-optimization
verifiedc94f3e9e-cc32-437f-9da7-70de303ca43f
Cut LLM API costs — prompt compression, caching, model routing, batching, and streaming without hurting quality.
Metadata
Skill file
# Cutting LLM Token Costs
Use when LLM API spend is climbing and you want real savings without visibly
degrading output quality.
## Measure first
- Log per-call input/output tokens and cost (most SDKs expose `usage`).
- Aggregate by endpoint, model, and caller. Fix what you can *see*.
## Highest-leverage levers
1. **Prompt caching** — cache stable system+few-shot prefixes; repeated
prefixes cost ~90% less on cache hits.
2. **Model routing** — cheap/small model for easy calls (classification,
routing, extraction), big model only for hard reasoning (generation,
multi-step). Save 10–50x on the routed share.
3. **Compress context** — retrieve *less* (top-k tuning, hybrid w/ rerank),
truncate history, summarize old turns.
4. **Batch** — combine many independent calls into one request where the
API supports it; amortize fixed overhead.
5. **Stream** — don't wait for full completion if you only need a token;
enable streaming and early-exit where possible.
## Watch the quality/cost tradeoff
Routing and compression change outputs. Re-run your eval harness (see
`llm-eval-harness-rag`) on any cost optimization before shipping.
## Pitfalls
- Cutting context so hard that retrieval relevance falls and answers get wrong.
- Routing "just everything to the small model" — it drifts on hard tasks.
- Ignoring output tokens (often priced higher than input) when trimming.
- No per-caller attribution, so you can't find the expensive path.
## Verify
- Before/after cost per solved task on your prod mix.
- Eval scores stable (or better) after each cost change.
- Cache hit rate and routed share visible in dashboards.