We started one enterprise RAG project with an in-memory vector store because it was fast to prototype with, and it worked great — right up until the client's document set grew past a few hundred thousand chunks and query latency started climbing past two seconds. That's the trap: most vector databases perform well at demo scale, and the differences only show up once you're indexing millions of vectors under real query load.
The decision that mattered most wasn't raw benchmark speed, it was filtering support. Real RAG systems almost always need metadata filters — by tenant, by document type, by date range — alongside similarity search, and databases that bolt this on as an afterthought see query latency spike badly under filtered queries. We ended up choosing based on filtered-query benchmarks against our own data, not the vendor's published numbers on generic datasets.
For a multi-tenant SaaS client with strict data isolation requirements, we moved to a database supporting native namespace partitioning rather than post-query filtering by tenant ID, which cut cross-tenant query latency by more than half and, more importantly, removed an entire category of "did we leak another tenant's data" risk from the architecture.
Reindexing cost is the thing teams underestimate. Embedding models get upgraded, chunking strategies change, and every time either happens you're re-embedding and reindexing the full corpus. We now budget for that explicitly in project timelines after a client's 4-million-document reindex took 30 hours longer than estimated, mostly due to embedding API rate limits we hadn't load-tested against.
Managed versus self-hosted came down to operational maturity more than cost for most clients. Teams without dedicated infrastructure staff are almost always better served by a managed vector database, even at a premium, because the 2 a.m. page for a self-hosted index running out of memory is a cost too, just one that doesn't show up on the invoice.
