Context Window Management for Long-Running AI Conversations
For a customer support assistant handling multi-turn conversations that sometimes ran to 40 or 50 messages, naively appending the full history to every LLM call worked at first and then became a real problem — both cost and latency climbed roughly linearly with conversation length, and past a certain point the model started losing track of the actual current question among older resolved sub-topics.
Our first fix was a sliding window, keeping only the most recent 10 messages. That was simple and fast but caused a specific, visible failure: users who established an important detail early in the conversation, like an order number or a stated preference, would get asked for it again once it aged out of the window, which read as the assistant not listening.
We replaced that with a two-tier approach: a rolling window of recent raw messages for immediate conversational context, plus a running structured summary — key facts established, unresolved questions, decisions made — that gets updated after every few turns and stays in context regardless of conversation length. The summary is generated by a cheap, fast model call separate from the main response generation, so it doesn't add meaningfully to response latency.
Deciding what belongs in the persistent summary versus the discardable window took iteration. We initially tried to have the model decide what was important in real time, which was inconsistent — it kept some irrelevant details and dropped genuinely important ones. Switching to an explicit extraction schema (order ID, stated issue, prior resolution attempts, customer sentiment) gave us consistent, structured memory instead of freeform summarization guesswork.
The measurable result was a drop in 'I already told you that' complaints in user feedback, from roughly 8% of long conversations to under 2%, and average per-turn cost stayed flat instead of growing with conversation length. The general principle: context isn't just a length problem to truncate, it's an information-prioritization problem, and the two need different solutions.
