Retrieval Chunking for Production Agents: How Infrastructure Teams Split Documents So the Right Passage Comes Back

Nearly every retrieval stack in production started life as four lines of tutorial code: a recursive character splitter, 1,000 characters per chunk, 200 characters of overlap, and whatever embedding model was the default that quarter. Those numbers were chosen to make a demo work against a sample PDF, and they survive into production because nothing about them ever fails loudly.

When an agent answers from the paragraph beside the one you needed, no exception is raised. The vector search returned 200 OK, top-k came back populated, and the model wrote fluent prose over the wrong span.

Chunk size and boundary strategy decide whether the clause your agent needs is retrievable at all — everything downstream is working with whatever the splitter handed it. What follows covers the three chunking families running in production today and the evaluation loop that tells you which one your corpus actually requires.

Why Chunk Boundaries Decide Retrieval Quality Before The Embedding Model Does

A chunk becomes exactly one vector. Everything inside that span — the answer, the boilerplate around it, the unrelated procedure two paragraphs down — is averaged into a single point in embedding space.

This is why swapping embedding models rarely fixes a retrieval problem, even though it is a one-line change and the leaderboard says the new model is better. A ranker can only rank the spans your splitter created, which is why retrieval behaves like an infrastructure layer with its own failure modes rather than a model-quality question.

Teams that have already worked through their retrieval blind spots tend to arrive at the same conclusion from the other direction. The corpus, not the benchmark, sets the ceiling.

What Breaks When A Boundary Lands In The Wrong Place

Chunking failures are unusually hard to see because they produce complete, confident answers instead of errors. The failure modes worth naming include but are not limited to:

• Split obligations. A contract chunk ends with “Provider shall maintain commercially reasonable safeguards” and the exception that guts the clause lands in the next chunk. Retrieval returns the obligation without its condition.
• Orphaned references. A chunk opens with “It must be reported within 72 hours,” where “it” was defined two chunks earlier. The embedding has no idea the passage is about breach notification.
• Diluted signal. One sentence of answer inside a 1,500-token chunk of unrelated procedure produces a vector dominated by the procedure. The chunk then ranks below shorter, less relevant text.
• Header amnesia. A table row survives the split as “Tier 2 — 4 hours” with no column headers attached. Nothing in the chunk says whether four hours is a response target, a resolution target, or a billing increment.
• Near-duplicate crowding. Repeated footers, confidentiality notices, and page furniture embed almost identically across hundreds of chunks. Top-k fills with the same boilerplate and the answer chunk sits just below the cutoff.

All of these share a shape: the retrieval call succeeds, the latency budget is met, and the answer is wrong in a way only a domain reader catches. That is what makes chunking a reliability concern rather than a tuning preference.

Fixed-Size Chunking: Still Correct More Often Than Teams Admit

Fixed-size chunking splits on token or character count with a fixed overlap, usually after a recursive pass that prefers paragraph and sentence boundaries. It costs no model calls, runs deterministically, and re-indexes a large corpus in minutes rather than hours.

For prose with even information density — support macros, product descriptions, call transcripts, knowledge-base articles — that determinism is worth more than a smarter boundary. A defensible starting point is 300 to 500 tokens with 10 to 15 percent overlap, which keeps roughly one idea per vector and fits five results comfortably inside a context budget.

Overlap exists so that a sentence straddling a boundary stays recoverable from at least one side. Push it past roughly 20 percent and you are paying for index size and near-duplicate results without buying much recall.

Semantic Chunking: Cutting Where The Meaning Turns

Semantic chunking embeds each sentence or sliding window, measures similarity between consecutive units, and cuts where that similarity drops past a threshold — commonly the 90th or 95th percentile of observed distances. Boundaries then follow topic shifts instead of character counts.

That cost is real and it recurs. A corpus of 50,000 sentences means 50,000 extra embedding calls on every re-index, and the percentile threshold becomes one more tuned parameter that drifts as the corpus changes.

Semantic splitting earns its keep on documents with no reliable structure to exploit: meeting transcripts, incident write-ups, clinical narratives, and long-form research where the topic turns without announcing itself. Keep in mind that variable-length output breaks context budgeting, so cap the range — a 128-token floor and an 800-token ceiling, with outliers merged or re-split — before this reaches production.

Structure-Aware Chunking: Let The Document Tell You Where To Cut

Structure-aware chunking parses the hierarchy a document already carries — Markdown headings, the HTML DOM, a PDF outline, numbered contract clauses, XML sections, table rows — and cuts at the smallest structural unit that still stands on its own. For enterprise corpora this is usually the highest-yield change available, because policy manuals, SOPs, benefit plans, and API references were authored with the boundaries built in.

The technique that makes it work is prepending the heading path to the chunk text before embedding, so a fragment carries “Security Policy > Incident Response > Notification Timelines” ahead of its own sentences. That single move fixes header amnesia and orphaned references in one pass.

Metadata is the second half of the strategy. Document id, section path, tenant, effective date, and jurisdiction let you pre-filter the candidate set before the vector search runs at all, which is how you stop a 2023 policy version from answering a 2026 question.

Tables deserve their own rule. Serialize each row with the header row repeated inline, so “Tier 2 | response time: 4 hours | resolution: 24 hours” survives as an independent, self-describing chunk.

Which Strategy Does Your Corpus Need?

The choice is a property of your documents rather than a matter of taste. Here is how the three families compare on the dimensions that decide a production rollout:

Strategy	Best corpus	Index cost	Characteristic failure	Tuning surface
Fixed-size	Uniform prose, transcripts, support macros	Lowest — no model calls	Splits clauses and tables mid-thought	Size, overlap
Semantic	Unstructured narrative, notes, incident write-ups	One embedding per sentence, every re-index	Wildly variable chunk lengths	Threshold percentile, min and max caps
Structure-aware	Policies, contracts, API docs, SOPs, tabular data	Parser build and maintenance per format	Degrades to fixed-size on malformed documents	Unit depth, heading-path template, metadata schema
Hybrid: structure first, fixed fallback	Mixed enterprise corpora	Moderate	Two code paths to debug	All of the above, plus routing rules

Most production stacks converge on the hybrid row. Structure-aware where the parser is confident, fixed-size with a heading path prepended where it is not.

The Evaluation Loop That Settles The Argument

Nothing above tells you what your corpus needs — only measurement does. The loop that produces a defensible answer runs in five steps:

• Build a golden set. Pull 50 to 200 real queries from your logs and label each with the exact passage that answers it, not merely the document containing it. Below 50, a two-point difference is indistinguishable from sampling noise.
• Score retrieval before answers. Measure recall@k — does any retrieved chunk contain the labeled span — plus mean reciprocal rank and precision at the k you actually ship. Answer quality is a downstream confound at this stage.
• Sweep the grid rather than guessing. Run the same golden set across sizes of 256, 384, 512, 768, and 1,024 tokens, overlaps of 0, 10, and 20 percent, and each strategy, changing one variable at a time.
• Then grade answers. Feed the retrieved context to the model and score responses against a rubric, remembering that a model grading its own retrieval carries position bias and order effects that will flatter whichever candidate happens to be listed first.
• Shadow the winner. Run the candidate index beside production on live traffic, log both result sets for the same queries, and diff them before cutting over.

This is the same discipline you already apply to agent evaluation, pointed one layer down the stack. The output is a number you can defend in an architecture review rather than a preference you can only assert.

What To Instrument Once It Ships

Chunking regressions arrive quietly, usually behind a document-format change or a silent embedding-model upgrade. The signals worth putting on a dashboard include:

• Recall proxy on a live sample. Re-score a rotating slice of production queries against the golden set weekly, and alert on the drop rather than the absolute value.
• Rank of the chunk actually used. If the passage the model cited keeps landing at rank 4 or 5, your ordering is degrading before your recall does.
• Retrieved tokens at P50 and P95. This is the line item where oversized chunks show up as spend rather than as a quality complaint.
• Empty and near-duplicate top-k rate. A rising share of results that are all the same boilerplate is the crowding failure coming back.
• Per-tenant recall. A parser that works on one customer's document template can quietly fail on another's without moving the global average.

The cost dimension is worth sizing explicitly, because it is usually the argument that gets the work funded. Retrieving k=8 at 1,000 tokens puts 8,000 tokens of context in front of every call, and moving to k=5 at 400 tokens removes 6,000 of them — at 20,000 agent calls a day, that is 120 million input tokens a day off the bill.

That number belongs in your agent cost governance review, and it usually arrives alongside better recall rather than in tension with it. Smaller, well-bounded chunks tend to rank more precisely and cost less at the same time.

Route these signals into the same place as the rest of your agent observability stack. A retrieval metric nobody looks at is not instrumentation.

Treat The Chunking Config As A Versioned Artifact

Changing a chunker changes every vector in the index, which makes it a migration rather than a config tweak. Pin the chunker version, the parser version, and the embedding model version in index metadata, then write the new configuration into a separate namespace instead of mutating in place.

Silent version drift is worse than the deliberate change. An embedding model upgraded underneath you leaves two generations of vectors in one index, and cosine distance between them means nothing in particular.

Handle the cutover the way you handle any other production change, with the staging, shadow, and rollback path you already use for agent release management. Note that the rollback artifact here is the old index, so budget the storage to keep it live through the window.

Start With The Corpus You Actually Have

Open ten documents your agent retrieves from and find where a default splitter would cut them. If the answer to a real user question spans a boundary in more than one of the ten, you have your first measurable problem and your first golden-set entry.

If you're scoping a retrieval pipeline and can't yet tell whether plausible-but-wrong answers come from the chunker, the index, or the prompt, the team at iSimplifyMe builds and operates production retrieval systems across Bedrock, Postgres, and managed vector stores every week. Reach out for a working session — we'll build your golden query set, sweep the chunking grid against it, and leave you with a re-index plan and the recall numbers that justify it.