Shipping a RAG Agent to Production: Lessons From Our First Deployment

Every RAG tutorial starts the same way: embed some documents, stuff them into a vector store, retrieve the top-k chunks, and hand them to an LLM. It works beautifully in a notebook. It gets a lot messier once real users start asking real questions.

The first thing that broke for us was retrieval quality. Cosine similarity on raw chunks surfaced technically relevant but practically useless passages — the right keyword, the wrong context. We ended up layering in a re-ranking step and chunking more aggressively around semantic boundaries instead of fixed token windows.

The second thing was latency. Chaining retrieval, re-ranking, and generation sequentially pushed response times past what users would tolerate. We moved to streaming the generation step as soon as the first relevant chunks came back, rather than waiting for the full context to assemble.

The third — and most important — was failure handling. An LLM will confidently answer a question even when nothing relevant was retrieved. We added an explicit low-confidence path: if retrieval scores fall below a threshold, the agent says so instead of guessing.

None of this is exotic engineering. It's the same discipline we'd apply to any production system — measure, find the failure mode, fix the failure mode, repeat. The models get the headlines, but the systems around them are where the real work happens.