A policy says a service is available on weekends, followed by an exception for a particular location. If an AI search retrieves only the first sentence, the answer can be confidently incomplete even though the source document was correct.
Chunking divides a document into smaller units for retrieval. Microsoft’s Azure AI Search guidance discusses chunk size, structure and overlap. The choice depends on the material and task; there is no single size that makes every document safe to summarise.
Inspect the boundary cases
Build a small evaluation set containing questions whose answers rely on a qualification, a table heading, a definition or a paragraph just over a page break. Keep the correct source passage beside each question.
Then inspect the retrieved text before judging the generated answer. Did the system retrieve the answer and its exception together? Did a table row retain the column names? Did the chunk preserve the document title and version?
- Small chunks can be focused but lose context.
- Large chunks can retain context but include distracting material.
- Overlap can preserve continuity but also repeat passages in the model’s context.
The Databricks data-pipeline guide also recommends considering semantic coherence and useful metadata. Headings and product identifiers can matter as much as raw length.
A boundary test you can read by hand
Create a fictional policy paragraph: “Weekend collections are available at staffed depots. The Riverside depot has weekday collections only.” Ask which days Riverside offers collection. If the retrieved chunk contains only the first sentence, the answer lacks the condition that matters.
Try keeping the paragraph intact, then try a split with overlap. Inspect the actual text returned in each case. A correct answer produced from missing evidence may be a lucky guess, so record evidence coverage separately from answer correctness. Repeat with a table whose unit appears only in its heading to test another kind of lost context.
Change one part of the experiment
Compare two chunking approaches while holding the question set and other settings steady. Record whether the required evidence was retrieved, not merely whether the answer sounded fluent. Include cases where the source does not contain an answer.
Keep the original document accessible for verification and link the answer to the relevant passage. A retrieved chunk is an excerpt, not a new authoritative document. Better boundaries make useful evidence easier to find; they do not remove the need to handle uncertainty and check what the model says.