Building a Document Q&A System for Enterprise Clients
A legal services client wanted a Q&A system over roughly 80,000 internal documents — policies, contracts, prior case notes — spanning a decade of revisions. The retrieval and generation pieces were the easy part. The hard part was that different users had different access rights to the same document set, and a naive vector index doesn't know or care who's asking the question.
We solved access control by attaching document-level permission metadata directly to each vector at ingestion time and filtering at the database layer before retrieval ever ran, rather than filtering results after the fact. Filtering after retrieval was our first approach and it was both slower and riskier — if the top five retrieved chunks all happened to be restricted, the user got no usable answer even though relevant unrestricted material existed further down the ranking.
Versioning was the second real problem. Contracts got amended, and old and new versions of the same clause existed side by side in the index, sometimes actively contradicting each other. We added an effective-date field to every chunk and, by default, filtered retrieval to the current version of each document while keeping a separate 'as of' query mode for the handful of users who needed historical answers for litigation research.
We also had to handle documents that intentionally present conflicting positions, like internal memos debating two interpretations of a regulation. Early versions of the system would blend both positions into a single confident-sounding answer, which was actively dangerous in a legal context. We changed the prompt to explicitly require citing which document each claim came from and to flag disagreement between sources rather than resolving it silently, and we display citations inline rather than as a footnote list at the end.
The system now handles about 3,000 queries a week with a client-measured accuracy rate above 92% on their internal audit sample. The infrastructure lessons — permission-aware filtering at the database layer, explicit versioning, and forcing the model to attribute rather than synthesize — turned out to matter more to the final result than any change we made to chunking or embedding model choice.
