Chunking Strategies for RAG: Why Naive Splitting Fails
Fixed-size chunking is the natural starting point because it's simple to implement, and it's also where most of the retrieval quality problems we see actually originate. Splitting a document every 500 tokens regardless of content structure routinely cuts a table in half, separates a heading from the paragraph it introduces, or splits a single coherent idea across two chunks that then compete against each other in retrieval instead of forming one strong match.
Semantic chunking — splitting on natural boundaries like paragraphs, sections, or detected topic shifts rather than a fixed token count — improved retrieval precision noticeably on a technical documentation project, but it wasn't free: it required real preprocessing investment to detect those boundaries reliably across a corpus with inconsistent formatting, PDFs converted from Word docs, and legacy content with barely any structure at all.
The chunk size versus context trade-off runs in both directions and neither extreme works. Chunks too small lose surrounding context and retrieve fragments that don't make sense on their own; chunks too large dilute the embedding with irrelevant surrounding content and hurt precision. We landed on roughly 300-600 tokens per chunk as a starting range across most projects, then tuned per corpus based on actual retrieval eval results rather than picking a number up front and assuming it would generalize.
Overlap between chunks solved a specific failure mode we kept seeing — answers that depended on content spanning a chunk boundary getting lost entirely because neither chunk on its own contained the full answer. A 10-15% overlap between adjacent chunks recovered most of those boundary-spanning failures on a legal document project, at the cost of modest additional storage and embedding expense.
The highest-leverage change on most projects wasn't chunk size at all, it was attaching metadata to each chunk — document title, section heading, source date — and including that context in the embedded text itself rather than storing it separately. A chunk that says "Section 4.2, Refund Policy: ..." retrieves and reads far better in isolation than the same text with the surrounding context stripped away, because the embedding itself now captures what the fragment is actually about.
