## Token Cost Optimization Reference

### Lever prioritization (impact x effort)
| Lever | Impact | Effort | Notes |
|-------|--------|--------|-------|
| Prompt caching | 5-10x on hits | Low | Cache stable prefixes at front |
| Model routing | 10-50x on routed share | Medium | Cheap model for easy calls |
| Context compression | 2-5x | Medium | Retrieve less, summarize history |
| Batch calls | 1.2-2x | Low | Amortize fixed overhead |
| Stream + early exit | 1.5-3x | Medium | Stop when answer is sufficient |

### Prompt caching rules
- Cache stable system prompt + few-shot examples
- Put cached content FIRST (providers cache prefixes)
- Keep cached prefix byte-identical (no timestamps, no dynamic values)
- Monitor cache hit rate in dashboards

### Model routing decision tree
```
Query -> classifier
  ├─ easy (classification, extraction, routing) -> gpt-4o-mini / small model
  ├─ medium (summarization, simple Q&A) -> gpt-4o / mid model
  └─ hard (multi-step reasoning, code) -> strong model
```

### Cost attribution
```python
# Log per-call usage
{"model": "gpt-4o", "endpoint": "rag_generation",
 "caller": "user_123", "input_tokens": 5000, "output_tokens": 1000,
 "cached_input_tokens": 4500, "cost": 0.01375}
```
- Aggregate by endpoint, model, AND caller
- Fix what you can see; set budget alerts per endpoint

### Quality guard
- Run eval harness before/after ANY cost change
- Routing and compression change outputs — verify quality didn't regress
- Watch for "route everything to cheap model" drift on hard tasks