Metadata

Skill ID

e729af2a-d752-46c3-a8a6-fd8e1bd4ffda

Version

1

Owner

387274b7-2891-478b-81b8-e11d5adb9319

Tags

tokenizationtokensbpetoken-countcostllmprompt-design

Signature

verified

Integrity

OK

Content hash

88da8c609dba497d6c7c6c741b95963515cb6fc57dd845eab930491fe1bd4655

Created

2026-08-10T09:23:43Z

Skill file

# Tokenization & Token Economics

Use when you're trying to understand **why a prompt costs what it costs**, estimate
billing accurately, or design prompts/pagination that respect context limits. Tokens
— not characters or words — are the unit of LLM pricing, context, and latency.

## How tokenizers split text

Most modern LLMs use a **BPE (Byte-Pair Encoding)**-style subword tokenizer. It's not
word-per-token; it's a learned vocabulary of common subword pieces.

Rough intuitions (verify per model — every family has its own tokenizer):

- **English ≈ 0.75 tokens per word** — a 1000-word prompt is roughly 750 tokens.
- **Code and math tokenize more densely** (many source tokens map to ~1 token; a line
  of code can be very few tokens) — often closer to 1 token per word or fewer.
- **Non-English and rare/varied text tokenize thinner** — more tokens per character —
  e.g. some languages, emoji, and unusual spellings are several tokens each.
- **Numbers/dates and run-ons** can fragment, inflating counts.
- **A token is roughly 4 characters** of English text on average (rule of thumb).

Because tokenizers are model-specific, **the same text can have a very different
token count across models.** Don't assume one model's count for another.

## Token counts dictate price and context

- **Price** is per token: input + output. A "16x" difference between a large and a
  small model (e.g. 4o vs 4o-mini) is often the *token price*, not the model size.
- **Context window** is in tokens — a 128k model holds ~a lot of tokens, but your
  *usable* context is reduced by system prompt + history + retrieved chunks + the
  response space you reserve.
- **Output tokens** are usually priced per token too — long generations cost more than
  the prompt in some plans.

## Estimate and measure accurately

- **Never estimate by characters/words** for billing or context budgeting.
- Use the **provider's tokenizer** (OpenAI `tiktoken`, Anthropic tokenizer, HF
  `tokenizers` for open models) — exact and cheap.
- Quick heuristics are fine for planning but **confirm on the exact model** before
  committing to a budget.

```python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
len(enc.encode("Your prompt text"))       # exact token count for that model
```

## Controlling token cost

- **Trim the system prompt** to essentials; every instruction token is paid on *every*
  call, so a verbose system prompt is a recurring tax.
- **Use prompt caching** so repeated prefixes aren't re-priced/billed (see the prompt
  caching skill) — put stable instructions at the *front* to maximize cache hits.
- **Retrieved context:** only send the top-k relevant chunks, not everything; set a
  hard token budget for context.
- **Capping output:** set `max_tokens` so a model can't generate a runaway long answer.
- **Don't repeat context in history** — dedupe what's already been sent or summarized.

## Pitfalls

- **Assuming words == tokens** — off by 25%+ grossly mis-budgets context and bills.
- **Cross-model token counting** — using tiktoken counts to budget an Anthropic/other
  model. Use that model's tokenizer.
- **Ignoring multi-turn growth** — history re-sends full messages each turn; watch
  context (and cost) grow across a long session. Summarize or truncate history.
- **Tokens in `max_tokens` meaning response tokens** — a low `max_tokens` cuts off
  long answers mid-sentence; a high one risks cost.
- **Forgetting the special/chat-template overhead** — role markers and chat template
  tokens add a small but real constant on top of raw text. It's why `288` often shows
  up in tiny request counts (the fixed overhead of chat messages).

## Verify

- Run several real prompts through the model's tokenizer and validate your cost model
  against the provider's billed token counts.
- Confirm a long multi-turn conversation stays within the usable context (compute
  prompt + reserved output < max context).
- After trimming the system prompt / using caching, verify billed input tokens drop
  accordingly.

Attached files

• script token_count_demo.py c75c1a675b7f… 2026-08-15T02:33:11Z
• reference tokenizer-guide.md 84e86f9c8341… 2026-08-15T02:33:11Z