agent-memory-architecture
verified7b1201a7-37d9-492d-a4c3-69953a8a44a0
Give an LLM agent long-term memory done right — working vs episodic vs semantic vs procedural tiers, RAG-backed recall, consolidation from episodes to facts, TTL/staleness, and the write/consolidate/recall loop.
Metadata
Skill file
# Agent Memory Architecture
Use when a single conversation context window is no longer enough — you're
building an assistant that must remember the user across sessions, recall past
tool outcomes, or accumulate facts about a domain. This is how you design memory
so it helps rather than turns into noise.
## The four memory tiers
AI-agent memory is usually modeled as four tiers, each with a different write
path, retention policy, and purpose:
- **Working memory** — the active context available for the *current* decision
cycle: the task, the running plan, recent observations, intermediate results.
In an LLM agent this is largely the context window plus a structured
scratchpad you keep in front of the model. Written implicitly; **aggressively
pruned** within a session (truncate/summarize as it grows).
- **Episodic memory** — records of specific past experiences: what happened, when,
in which session, with what outcome. Implementation: session logs, transcripts,
structured "episode" entries. Written automatically by logging the agent's
actions. **Expires on a TTL** — it's a timeline, not a fact store.
- **Semantic memory** — distilled facts and knowledge independent of time: user
preferences, domain rules, summarized takeaways. Implementation: often RAG over
a vector store holding distilled facts/notes. Written by a **background
extraction step** that consolidates episodes into facts. Needs **staleness
management** because facts change.
- **Procedural memory** — skills and routines for *how* to do things: policies,
prompts, and agent code (e.g. "always verify before deleting"). The one tier
that should be written by a **deliberate promotion step with validation**, never
fully automatically. **Version and deprecate**, don't delete.
## The core loop: write → consolidate → recall
A production system runs three phases:
1. **Write (during a session).** Episodes are captured automatically — every
user turn, tool call, and outcome appended to a session log. Working memory
stays in the context; anything needing survival past the session is promoted
to episodic storage.
2. **Consolidate (after a session / asynchronously).** A background LLM step reads
the session log and *distills* it into semantic facts: "user prefers kebab-case
function names", "the deploy is down this week", "Churn model is v3 now".
Episodic detail (exact timestamps, raw logs) fades or moves to cold storage.
This episodic→semantic consolidation is what turns a chatbot into an agent
that *learns*. Without it you accumulate episodes but never knowledge.
3. **Recall (at the start of a session).** Before the first turn, retrieve the
relevant semantic facts and recent episodes and inject them as context, e.g.:
- semantic facts via vector similarity to the current task,
- procedural memory by loading the matching skill/policy,
- optionally a short "recent history" summary of the last session for
continuity (pronoun resolution, "you were working on X").
This is RAG applied to agent history — with retrieval quality as the bottleneck.
If your embeddings don't capture the right intent, you surface stale facts and
miss the ones that matter.
## Retrieval and ranking choices
- Prefer **hybrid retrieval** (dense + keyword) for memory, because memory queries
are often exact-phrase ("that bug we hit last Tuesday" needs literal terms plus
semantics).
- Store **metadata** on each memory entry: timestamp, source session id, topic,
confidence/importance score. Use it to boost recent or high-importance facts.
- Weight **recency + relevance** together — people care what changed recently.
A stale fact ("DB is at v5") overriding the current one ("we migrated to v6")
is a classic silent failure.
- For episodic recall, retrieving by similarity works poorly for time-bound
questions ("what happened last Monday") — keep a separate structured index by
date when you need that.
## Forgetting is a feature
- Working memory: prune inside the session (turn the oldest turns into a summary).
- Episodic memory: TTL expiry (e.g. 30 days) or consolidation into semantic.
- Semantic memory: **contradiction handling** — when a new fact conflicts with an
old one, the new fact should win and the old one be marked superseded, not
silently duplicated.
- Procedural memory: version and deprecate; keep a changelog.
Define this up front. If you don't decide what goes to which tier and how it
expires, you accumulate noise, get contradictions, and the system degrades —
memory becomes a liability instead of an asset.
## Pitfalls
- **No consolidation step** — you store raw episodes forever, recall gets noisy,
nothing becomes "knowledge".
- **Recall everything** — stuffing the whole memory store into context blows the
window and buries relevant facts ("lost in the middle"). Retrieve, don't dump.
- **Ignoring staleness** — stale semantic facts silently override current truth.
- **Automatic procedural promotion** — letting the model write permanent "how to"
rules on its own bakes in bugs. Human/validated promotion only.
- **No per-user isolation** in a multi-user system — memory leaking across users
is both a correctness and a privacy bug.
- **Retrieval-only memory for time-bound queries** with no date index.
## Verify
- Run a multi-session test: tell the agent a fact in session 1, confirm it uses
it correctly in session 2 (after consolidation+recall).
- Confirm contradiction handling: change a fact, confirm the new one wins and
recall doesn't return both.
- Measure retrieval precision on a memory eval set (does recall surface the
*relevant* fact, not just *a* fact?).
- Confirm the context injected at session start fits your window and the agent
uses it (traceback: the fact appears in the retrieved context and in the answer).