Will a better model fix wrong answers on our PDFs?

Only if the retrieved chunk actually contained the answer. Open the chunk behind each wrong answer. If the answer is not in it, or a table is split in half, the failure happened at ingestion and a bigger model reads the same broken fragment.

Do we really have to convert everything to PDF first?

The paper argues for it, treating PDF as a universal visual interchange format so one conversion path handles every input. It is a defensible choice, not the only one. The point you should not skip is having a single, deterministic normalization step instead of five extractors with five failure modes.

Does the 95.7% token reduction mean our bill drops by that much?

No. That figure is chunking-stage output tokens, and the paper reports chunking cost falling 77.8% to 85.6% under its pricing assumptions. If you re-ingest a small library rarely, the absolute saving is minor. The accuracy gain is the reason to care.

Bad PDF ingestion is why your document chatbot gives wrong answers

Most "we tried a chatbot on our documents and it gave wrong answers" complaints are not model failures. They are ingestion failures. The PDF was flattened into text before anyone checked what happened to the tables, the two-column spec blocks and the heading that scoped the clause.

A new paper puts a method and some numbers behind that step. Document Retrieval-Aware Chunking (D-RAC) treats ingestion as the thing to engineer, and extends an earlier web-focused framework (W-RAC) to arbitrary document formats.

What the paper says

The pipeline has four stages: normalize and render, convert, parse and section, plan and reconstruct. PDF is treated as a universal visual interchange format, so other formats are converted to PDF first using deterministic tooling. The multimodal model touches document content exactly once, and everything downstream is deterministic or identifier based.

The reported results: the full 236-document PDF subset of the RAG-Multi-Corpus benchmark was ingested in 72 minutes with zero conversion or chunking errors, chunking-stage output tokens dropped 95.7%, and chunking cost dropped 77.8% to 85.6% depending on pricing assumptions (paper).

Read that as a benchmark corpus, not as your scanned RFQs. The paper does not test a 2003 fax scan of a drawing with a handwritten revision in the margin.

Why this lands on quoting and support first

This is my reading, not the paper's claim: the documents that hurt a small business the most are the ones with the most structure per page. An RFQ with a tolerance table. A spec sheet in two columns. A terms page where the clause only means what it means because of the heading above it.

Naive text extraction destroys exactly those three things. The retriever then returns a fragment that reads fluently and answers a different question, and the model states it with confidence. That is the failure your estimator saw and called "the AI is dumb".

A decision table before you buy anything

What your documents look like	What plain text extraction does to them	First move
Single column, digital, mostly prose	Survives mostly intact	Skip this work, fix your prompts and your retrieval count
Tables of dimensions, tolerances, prices	Rows and headers separate, numbers lose their column	Preserve tables as markdown at conversion time, keep the header row in every chunk
Two-column specs, side-by-side layouts	Lines interleave across columns	Layout-aware or multimodal conversion, not a text dump
Scans, faxes, photographed pages	Empty or garbage text, silently	Detect and quarantine, do not index blind
Mixed formats (docx, xlsx, email, PDF)	Five different extractors, five different failure modes	Normalize to one interchange format first, which is the paper's argument

The failure log, which costs you nothing

The cheapest diagnostic I know, and the one nobody sells you, takes about 40 minutes.

Collect the next 20 real questions your team asks the tool. Not test questions. Real ones. (about 15 minutes of collecting over a week)
Mark each answer right, wrong or partly right. (about 10 minutes)
For every wrong answer, open the chunk the system retrieved. (about 15 minutes)

Then split the wrong answers into two piles. If the retrieved chunk did contain the answer and the model still got it wrong, that is a generation problem and a better model or a tighter prompt may fix it. If the retrieved chunk did not contain the answer, or contained half a table, that is an ingestion problem and no model upgrade will touch it.

In my experience the second pile is the bigger one on document-heavy work. I have not measured the split across a set of clients, so treat that as opinion.

The arithmetic, on your numbers

Do this with your own inputs, not mine. If your team answers R document questions a week, the tool is wrong on a fraction W of them, and each wrong answer costs M minutes to catch and redo, your weekly loss is R x W x M minutes. Twelve questions a day, wrong a quarter of the time, eight minutes to recover, gives 60 x 0.25 x 8, which is 120 minutes a week. If a chunking fix halves W, you recover about an hour a week. That is a number worth one afternoon of work and not worth a six-figure platform.

The cost side of the paper matters here too. A 95.7% cut in chunking-stage output tokens only shows up on your invoice if you are re-ingesting a large corpus often. For a 400-document library you refresh monthly, the absolute saving is small. The accuracy argument is the one that pays.

I have built a datasheet pipeline for a lighting manufacturer across 6 product families and 32 dimension sheets with a daily catalogue refresh, and the whole difficulty of that job lived in the conversion step, not in the model. More on how I think about document work is on the blog.

Limitations

The benchmark result is on a curated PDF subset. Nothing in the paper says it holds on scans, handwriting, stamps or bad photocopies, and I have not tested it.

Converting everything to PDF first is a real cost: you add a rendering step, and a bad render is now a silent failure you have to detect. Deterministic tooling helps, it does not eliminate this.

Touching content once with a multimodal model means the conversion is a single point of failure. If it misreads a table, every chunk downstream inherits the error and nothing later catches it.

And the harder half of quoting is not retrieval at all. Knowing which clause applies to this customer, this year, this revision, is a data problem your documents may not encode anywhere.

My take

The useful idea here is not the pipeline, it is the ordering. Ingestion quality sets the ceiling on everything after it, and most teams buy the layer above the ceiling. If your documents are messy, spend the first week on conversion fidelity and section structure, and only then argue about models.

Next step: run the 20-question log this week. If most of the failures are missing or broken chunks, you know what to fix, and you know you were about to spend money on the wrong thing.