AI Agent Memory: Short-Term Context vs Long-Term State
The first version of a long-running assistant we built simply appended every message to the context window, and it worked fine for the first hour of a session and degraded steadily after that — not from hitting the token limit, but from the model losing track of what actually mattered as older, less relevant turns diluted the signal from recent ones.
We split memory into three tiers: the active conversation window, a session summary that compresses older turns into key facts, and a persistent long-term store for information that should survive across sessions entirely — user preferences, prior decisions, account details. Each tier gets refreshed on a different schedule, and only the long-term store gets written to deliberately rather than by default.
Deciding what belongs in long-term memory turned out to be the hard design problem, not the storage mechanism. We use an explicit extraction step after each session — a smaller, cheaper model reviews the conversation and proposes facts worth persisting, which a rules layer then filters before they're written, rather than letting the primary agent decide unilaterally what's worth remembering.
Retrieval into long-term memory needs the same relevance discipline as document RAG — pulling every stored fact into context regardless of relevance recreates the original dilution problem at a different layer. For a client's coaching app, limiting retrieved memory to the five most relevant facts per turn, ranked by recency and relevance, performed better in user testing than surfacing everything the system knew about the user.
The failure mode we watch for now is stale memory contradicting fresh input — an agent confidently referencing a preference the user stated three months ago and has since changed. We added explicit conflict resolution: newer statements override older ones automatically, and the agent surfaces the update ("noted, I'll use this instead") rather than silently overwriting history without acknowledgment.
