What is chunk overlap, and why does it actually matter...

Overlap means consecutive chunks share a slice of tokens right at their boundary — if one chunk ends at token 500, the next might start back at token 450, so 50 tokens appear in both. The reason it matters: meaning often crosses chunk boundaries that a splitter can’t see coming. A sentence stating a headline number, followed by the very next sentence qualifying it with an important caveat, can end up split across two chunks with no overlap — and whichever chunk gets retrieved, the answer looks either overconfident or incomplete depending on which half surfaced.

With overlap, that boundary content appears in both chunks, so the retrieval step finds the complete thought regardless of which side of the original cut it lands on. The typical setting is 10-20% of chunk size — for a 512-token chunk, 50-100 tokens of overlap. Push it much higher and you’re paying storage and embedding cost for redundant text; push it to zero and you’re back to the original information-loss problem at every boundary.