Sit in on any enterprise AI architecture review this year and count how long it takes for someone to ask which vector database the team should standardize on. Then count how long it takes for anyone to ask where the documents are actually coming from.
In most rooms, the second question never lands. The index gets designed, the embedding model gets picked, the agent gets scoped — and then somebody discovers that two thirds of the knowledge that agent needs is sitting in a scanned PDF on a Windows file share nobody has touched since a 2019 migration.
That discovery lands squarely on the ingestion architecture, and it lands there before a single downstream agent behavior can be evaluated. Accordingly, the ingest path deserves the same design rigor you would give a payments path: explicit quality gates, explicit idempotency, explicit failure handling.
Document ingestion is the pipeline that turns scanned PDFs, file shares, and email attachments into chunked, embedded, permission-tagged records. Without it, retrieval only ever sees what was already digital.
Where Enterprise Knowledge Actually Lives
Before you architect anything, inventory what you are actually pointing at. The document estate at a typical mid-market enterprise splits into populations that each fail differently, including but not limited to:
- Scanned PDFs with no text layer. Executed contracts, signed change orders, and inspection reports — anything that passed through a multifunction printer between roughly 2005 and last Tuesday. A naive extractor returns an empty string on these and does not raise an error.
- SMB and DFS file shares. Decades of nested folders with NTFS ACLs that encode who is allowed to see what. Often the folder structure is the only metadata that exists anywhere.
- Email attachments. The same quote or spec sheet duplicated across forty threads, each copy with a different filename and no canonical version.
- Line-of-business exports. CSV and XLSX dumps out of ServiceNow, Workday, or a homegrown ERP, where the meaning lives in a header row that a text chunker will happily strip.
- Wikis, tickets, and Confluence spaces. Already machine-readable, already the population everyone starts with — and usually the smallest slice of the estate by knowledge value.
Note that these populations have wildly different unit economics. OCR on a 400-page scanned deposition costs real money, while pulling a Confluence space through an API costs essentially nothing.
What's more, they carry different security postures. A Confluence space has one access model; a file share has thousands of them, expressed as inherited ACLs you will have to carry all the way into the index.
File Shares And Mailboxes Need Different Connectors
Getting bytes off a legacy share is its own engineering problem. AWS DataSync over an SMB endpoint, FSx for Windows File Server, and Storage Gateway will all move the files, and each one differs in whether it preserves the NTFS metadata you are going to need downstream.
Verify that preservation explicitly before you commit to a connector. A sync that silently drops ACLs will not fail loudly — it will simply produce an index you cannot filter, which you will discover during the first security review rather than during the build.
Mailboxes are a different shape entirely. Pull messages and attachments through the Microsoft Graph API with delta queries instead of exporting PST archives, because delta queries give you the incremental story for free and a PST gives you a one-time snapshot that starts rotting immediately.
Treat each attachment as its own document with the parent message ID, sender, and thread ID attached as metadata. Then deduplicate by content hash, because the same specification sheet will appear dozens of times across one mailbox and forty near-identical chunks competing in a single retrieval query is its own quality problem.
Why Scanned PDFs Break Retrieval Before The Model Sees Them
The failure mode here is quiet, which is exactly what makes it expensive. Point pdfplumber or PyPDF at a scanned document and you get back a zero-length string, the chunker emits nothing, and the job exits 0.
No exception is raised. No dead-letter queue fires, and the document simply is not in the index — nobody finds out until an agent confidently answers a contract question using the one digital-native memo that happened to mention the same vendor.
This is the class of problem we have written about as retrieval blind spots, where the corpus looks complete because the pipeline reported success on every file it touched. Keep in mind that "processed successfully" and "produced retrievable content" are two different assertions, and only one of them is usually being measured.
The rule that catches this: every document must emit a character count, and any file that extracts to fewer characters than a floor you set — 50 characters per page is a reasonable starting point — routes to OCR instead of to the chunker.
Anything that fails OCR too lands in a review queue with its source path attached. Silence is never a passing result.
OCR Quality Gates: The Number You Have To Pick
Once you accept that OCR is a required stage rather than an optional one, you inherit a decision most teams skip: what confidence score is good enough to embed. Amazon Textract returns per-word confidence values from 0 to 100, and those values are the only signal you have about whether the text you are about to index means anything at all.
Textract returns per-word confidence from 0 to 100. A workable gate routes pages averaging below 90 to human review and hard-fails anything below 80 rather than embedding unreadable text as fact.
Use both the mean and the minimum. A page can average 94 while a single 61-confidence token turns a dollar amount, a policy limit, or a dosage into something the agent will later quote back with full confidence.
For instance, on clean 300 DPI scans of typed text, character-level accuracy typically lands in the 98 to 99 percent range. Drop to 200 DPI, add a skew, or run a third-generation fax copy through the same pipeline and estimates put that figure well down into the 80s.
Cost is the other half of the decision. Plain text detection runs roughly $1.50 per 1,000 pages on Textract, while turning on table, form, or query extraction moves that to approximately $15 to $50 per 1,000 pages depending on which features you enable.
That gap is why blanket "analyze everything" configurations blow up quietly. Classify first and spend the expensive extraction only on document types that actually contain tables — the same routing discipline behind AI agent cost governance on the inference side applies just as literally on the ingestion side.
Layout Preservation Is A Retrieval Problem
A scanned document is a picture of a layout, and that layout is carrying meaning. Flatten a rate table into a run of prose and you get a chunk containing the numbers 1,250 and 4,800 with no surviving indication of which row or column either one belonged to.
Flattening a table to prose destroys the row-and-column relationship a retrieval query depends on. Preserve tables as structured markup so every chunk keeps its header row and its units.
The fix is to run structured extraction on documents you have classified as table-bearing, then serialize the result as markdown or HTML inside the chunk itself. Embedding models handle pipe-delimited tables well enough, and more importantly, the model reading the retrieved chunk can still see the header.
Multi-column layouts need the same care. Reading order in a two-column inspection report is column-then-row, and a naive left-to-right pass will interleave two unrelated paragraphs into a single sentence that reads like nonsense and embeds like noise.
Chunk boundaries should follow document structure wherever that structure is legible. Section headings, table boundaries, and page breaks are all better split points than "every 1,000 characters," and this is precisely the mechanism our work on the data and retrieval layer found sets the practical ceiling on answer quality — what got indexed constrains everything downstream of it.
Metadata Capture: The Fields You Cannot Backfill
Metadata is the stage teams defer and then cannot recover. Once a file has been copied out of a legal contracts folder on fileserver01 into a flat S3 prefix, the path that encoded the practice area, the year, and the access group is gone unless you captured it on the way through.
Capture source path, ACL or AD group, document date, version, and page number at ingest time. These fields cannot be reliably reconstructed once the file has left its original share.
The access control field is the one with teeth. A file share expresses permissions as inherited NTFS ACLs, and an index that drops them turns a well-governed share into a flat, fully-readable corpus that any authenticated agent user can query.
Carry the ACL or the mapped AD group onto every chunk as a filterable field, then enforce it as a pre-filter on the query rather than a post-filter on results. Post-filtering still lets the retriever score and rank documents the caller was never permitted to see, and even the result counts leak structure.
Two date fields matter, not one. Document date drives recency ranking and "what is the current version" questions, while ingest date drives your re-processing cadence and your agent data retention logic — conflating them makes both unreliable.
Page number belongs on every chunk as well. Without it an agent can cite a document but cannot cite a location, and a citation nobody can verify in ten seconds is a citation the compliance team will not accept.
Incremental Re-Ingest And The Stale-State Read
The first full crawl is the easy part. The hard part is the second Tuesday, when 900 files changed, 40 were deleted, 12 were renamed, and re-processing the entire 2.4 million page corpus is no longer a defensible use of the budget.
Hash every source file with SHA-256 and store the digest as the idempotency key. Re-ingest only when the digest changes, and delete the superseded chunks in the same operation that writes the new ones.
Modification timestamps are insufficient on their own. A DataSync run, a backup restore, or a permissions change will touch mtime on files whose bytes never changed, and content hashing is the only cheap way to tell the difference.
Deletes need explicit handling or the index accumulates ghosts. When a source file disappears, write a tombstone in the document registry and remove its chunks, because an agent surfacing a terminated contract that was pulled from the share for a reason is a governance incident rather than a bug.
Structurally the path is unremarkable, and it should stay that way: S3 event notification to EventBridge, EventBridge to SQS, SQS to a Lambda worker with a visibility timeout longer than your worst-case OCR job, and a dead-letter queue behind it. Then instrument it the way you would instrument any production consumer, because queue depth, age of oldest message, and DLQ count are the three numbers that tell you whether the corpus is current.
Those numbers belong on the same dashboard as your agent observability signals. Retrieval quality degrades from stale ingest exactly the way it degrades from a bad prompt, and only one of those two is usually being watched.
Model-Version Pinning And The Re-Embed Migration
Embedding models change, and changing one is a migration with all the planning that word implies. Amazon Titan Text Embeddings V1 emits 1,536-dimension vectors while V2 emits a configurable 1,024, 512, or 256, and the two geometries cannot coexist in a single index.
Swapping an embedding model invalidates the entire index, because vector dimensions and geometry both change. Pin the model version in your document registry and treat a swap as a full re-embed migration.
This is the argument for keeping extracted text as a durable artifact in S3, held separately from the vectors. OCR is the expensive stage and plain text does not go stale, so re-embedding 2.4 million pages from stored text costs a small fraction of re-scanning them — Titan V2 runs on the order of $0.02 per million input tokens.
Record the extraction engine version, the OCR confidence, the chunking strategy, and the embedding model on every record. When retrieval quality shifts, those four fields are how you separate a model regression from an ingest regression, and without them you will be guessing.
A Reference Path, Stage By Stage
Here is how the stages fit together, along with the specific control each one hands you and the failure that control exists to prevent:
| Stage | Typical service | Control you own | Failure it prevents |
|---|---|---|---|
| Landing | S3 with DataSync from SMB or DFS | Content hash on arrival | Reprocessing unchanged files |
| Classification | Lambda with MIME and type sniffing | Text-layer detection | Silent empty extraction |
| OCR | Textract detect or analyze | Confidence threshold plus review queue | Garbage text embedded as fact |
| Structure | Table and form extraction | Structured markup inside the chunk | Lost row and column relationships |
| Metadata | DynamoDB document registry | ACL, version, date, page number | Unfilterable, unattributable chunks |
| Embedding | Bedrock Titan or Cohere | Pinned model version | Mixed-geometry index |
| Serving | OpenSearch, Pinecone, or pgvector | Tenant and ACL pre-filter on query | Cross-tenant retrieval |
Overall, the pattern is that every stage owns exactly one gate and every gate emits a metric. A pipeline in which any stage can pass silently is a pipeline you will not be able to debug six months from now.
How To Sequence This Without Stalling The Agent Program
You do not have to ingest the whole estate before shipping anything, and attempting it is how these programs stall. Pick the narrowest document population that answers a question somebody is currently paying a human to answer, and instrument that one end to end first.
Then build the golden set before you build the pipeline. Fifty to a hundred documents spanning your worst scans, your densest tables, and your ugliest multi-column layouts — each with three to five questions and known-correct answers — becomes the agent evaluation harness that tells you whether an ingestion change helped or hurt.
Run every change in shadow mode against that set. A new OCR threshold, a new chunking strategy, or a new embedding model gets scored offline against the golden answers before it touches the live index, the same way you would gate any other production change.
Finally, expect the first pass to surface documents nobody knew existed. In practice the inventory stage is where the real scope gets discovered, and budgeting for a re-scope after week two is more honest than pretending the first estimate will hold.
Get A Second Set Of Eyes On Your Ingestion Architecture
If you are standing up retrieval for a production agent and most of your source material is scanned, shared, or emailed, the sequencing above is what decides whether the program pays off or quietly stalls at 60 percent coverage. The team at iSimplifyMe builds and operates ingestion and retrieval pipelines across file shares, ticketing systems, and data warehouse environments every week.
Reach out for a working session on your document estate. We will inventory what you are actually pointing at, name the extraction and permission failures you are about to hit, and leave you with a staged ingest plan covering quality gates, metadata schema, and re-ingest cadence.