Semantic chunking improved recall@5 on our evaluation set. Should we ship it?

Before shipping, verify that the evaluation set isn’t biased; for example, concatenating unrelated documents can create artificial boundaries that flatter semantic chunking. Compare the results against a structure-aware splitter, which is a cheaper control. Finally, account for the cost: semantic chunking often leads to a significant increase in index tokens and creates a tighter coupling between the embedding model and document boundaries.