Retrieval Quality Metrics That Predict User Satisfaction
After shipping a RAG-based knowledge assistant, we had the usual retrieval metrics dashboard — precision@k, recall@k, mean reciprocal rank — all looking healthy, while support tickets about wrong answers kept coming in. That gap told us the metrics we were tracking weren't the ones that mattered to actual users, so we ran a study correlating retrieval metrics against a sample of 400 real query sessions with human-labeled satisfaction ratings.
Precision@k turned out to be a weak predictor on its own. A query could retrieve five technically relevant chunks and still produce an unsatisfying answer if none of them contained the specific fact the user needed. What correlated much more strongly with satisfaction was what we started calling 'answer coverage' — whether the union of retrieved chunks actually contained enough information to fully answer the question, not just touch on the topic.
The second strong predictor was source diversity relative to query ambiguity. For ambiguous queries, retrieving five chunks all saying the same thing scored worse than retrieving chunks representing the two or three plausible interpretations of the question, because the latter let the generation step either ask a clarifying question or address both interpretations explicitly. We started weighting our re-ranker to favor diversity when query embeddings showed high similarity to multiple distinct clusters in the index.
We also found mean reciprocal rank was actively misleading for multi-fact questions. A query asking about three related requirements would get a great MRR if the single most relevant chunk ranked first, even if the other two facts never got retrieved at all. We replaced it with a coverage-weighted score that checks whether all sub-claims in a question have supporting retrieved evidence, which tracks much closer to what a human reviewer flags as a complete versus partial answer.
None of this replaced human evaluation — we still sample and review real sessions weekly. But shifting the automated metrics toward answer coverage and diversity-aware relevance meant we could catch regressions in CI before they reached users, instead of finding out from a support ticket three weeks after a re-ranker change shipped.
