llm-context-window-management
verifieda84e1ae0-1126-47fc-9d51-f5317b089a4d
Keep long agent and chat conversations inside the LLM context window — truncation, summarization, eviction, and two-layer memory.
Metadata
Skill file
# LLM Context Window Management
Use when an agentic loop or a long chat conversation grows until it overflows the
model's context window — request fails, cost explodes, or the model starts
ignoring old but important instructions. You need a strategy to keep working
context in, working memory out, without losing what matters.
## The core tension
Context windows are finite but conversations (and tool-call histories) grow
without bound. Truncating is free but destructive; summarizing loses detail;
both change the retrieved content the model reasons over. Pick the least lossy
strategy that keeps you inside the window with margin.
## Options, roughly in increasing sophistication
1. **Truncation** — keep the most recent N tokens, drop the oldest. Simplest,
zero LLM cost, but the model loses all older context and any instruction that
lived there. Fine for short sessions; dangerous for multi-step agents.
2. **RAG / retrieval** — instead of stuffing everything, retrieve only relevant
chunks per turn. This is the right answer for *knowledge* that's large and
queryable. It does not replace conversational memory.
3. **Sliding-window eviction** — split history into equal-mass token chunks, drop
the oldest chunk(s) once usage crosses a threshold (e.g. 50% of the window),
keeping the most recent messages. Repair any tool-call/result pairs whose
context got split. Cheap, predictable.
4. **Summarization / compaction** — when estimated tokens exceed `window –
reserve` (e.g. trigger at 25–90% depending on agent), have the LLM generate a
structured summary of the older conversation — session intent, artifacts
created, key decisions, next steps — then prepend that synthetic summary as a
message and keep a recent tail of raw messages. The full original history is
written to disk/DB as the canonical record; only the working set lives in
context. This is how "deep agent" harnesses keep long tasks going.
5. **Two-layer agent memory** — short-term (in-context raw tail + summary) plus
long-term persistent memory (vector-embedded facts/preferences/decisions
retrievable across sessions). This is what extends an agent beyond a single
session's token limit. Long-term memory is typically semantic (facts),
episodic (past interactions), or procedural (how-to steps).
## Rules of thumb for choosing
- **Very long single documents** (books, contracts): hierarchical summarization
so you keep a layered outline, not the whole text.
- **Long multi-session conversations** (coaching, assistants): memory buffering +
summarization with durable long-term storage.
- **Cost-sensitive big-window usage**: context compression / summarization.
- **Regulated content where every word matters**: prefer RAG with exact retrieval
and avoid lossy summarization.
## Concrete implementation notes
- Reserve headroom, not the full window: trigger compaction well before the
limit (some harnesses default to ~85–90%, but triggering earlier, e.g. 25–50%,
can reduce the frequency and size of big blocking compactions).
- Summarization output becomes a synthetic user/summary message prepended to the
kept tail — the model still sees the outline plus recent verbatim turns.
- Preserve artifacts and decisions in the summary so a later turn can pick up the
thread; a bare "we discussed X" summary is nearly useless.
- Always persist the original transcript to disk/db as the canonical record —
the summary is lossy and you must be able to reconstruct on demand.
- For pure-play short chats that rarely overflow, truncation with a sane budget
beats adding summarization machinery.
## Pitfalls
- Compacting at 90%+ of the window and paying for it in quality when the next few
turns blow up immediately.
- A lossy summary that silently drops a critical prior instruction or tool
result — then the agent "forgets" and misbehaves.
- Summarizing knowledge that should have been retrieved by RAG instead.
- Not repairing tool-call/result pairing after eviction, so a result is orphaned.
- Relying only on the volatile in-context copy and losing the canonical record.
## Verify
- Reproduce a session that previously overflowed; confirm the managed version
stays under the window for N turns.
- Preserve an important mid-conversation fact and confirm it's still answerable
after a compaction event.
- Measure that compaction doesn't break downstream tool-call chains.