## Tokenization Reference

### Token density by content type (approximate)
| Content type | Tokens per word | Tokens per char |
|--------------|-----------------|-----------------|
| English prose | ~0.75 | ~0.25 |
| Code | ~0.5-1.0 | ~0.2-0.3 |
| Chinese/Japanese | ~1.5-2 (per char) | ~1.0-2.0 |
| Numbers/dates | 1+ (fragments) | variable |
| Emoji | 1-3 | 1-3 |

### Tokenizer libraries
```python
# OpenAI models
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")  # or get_encoding("cl100k_base")

# Anthropic (requires API or claude-tokenizer package)
from claude_tokenizer import count_tokens

# HuggingFace open models
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
```

### Context budgeting formula
```
usable_context = model_context_window
                 - system_prompt_tokens
                 - history_tokens
                 - retrieved_chunk_tokens
                 - reserved_output_tokens
```

### Cost control checklist
- [ ] Trim system prompt (paid on EVERY call)
- [ ] Put stable instructions first (maximize cache hits)
- [ ] Cap retrieved context with hard token budget
- [ ] Set `max_tokens` to prevent runaway output
- [ ] Summarize/dedupe multi-turn history (it grows unbounded)

### Common mistakes
1. Assuming `words == tokens` (off by 25%+)
2. Using tiktoken counts for Anthropic/Gemini models
3. Forgetting multi-turn history re-sends all prior messages
4. Confusing `max_tokens` (output only) with total context
5. Ignoring chat-template role markers (~10-30 tokens per message)