Extracting Text from PDFs: Horror Stories for RAG Developers

In the article on Retrieval Augmented Generation we looked at how to split documents into chunks, embed them, search them and rerank the results. One fundamental question I deliberately left out there: how do you get at the text in the first place? With Markdown or HTML, that's a non-issue. But probably the most common case is the SharePoint folder full of PDFs, and PDFs come with challenges of their own that, as I wrote back then, would deserve an article of their own. This is that article.

Everything that comes afterwards -- chunking, embeddings, hybrid search, reranking -- assumes that you have clean text. Parsing errors propagate through the whole pipeline, and no reranker will fix a chunk that reads "Ejf [bimvoh" instead of "Die Zahlung" ("The payment").

The example is constructed, but not far-fetched. It comes from a test document that Claude and I built for this article: a 37-page PDF that looks perfectly harmless in the viewer and hides a different pitfall for text extraction on every page. I unleashed a range of tools on this document, from pypdf to a multimodal LLM. The document is in German, so most examples below are German too; the effects don't depend on the language. You can get the PDF here.

Since Halloween is coming up, you can read the rest of this article as a collection of horror stories about documents that behave nicely in the viewer and wreak havoc in the index. For anyone who is just pushing their first PDFs into a RAG pipeline, it's a list of places worth a closer look. And those who have been doing this for a while may find the explanation for some strange observation from the past, such as why a search for "Müller" doesn't find a document that contains "Müller".

What a PDF actually is (and what it isn't)

PDF is a brilliant format for presenting documents exactly the way they should look on paper. But as soon as you try to extract the text, you get the feeling that someone went to great lengths to design a format that makes text extraction as nerve-racking as possible. A PDF doesn't describe running text, but which glyph gets painted where on the page. In the best case, a paragraph stays together. In the worst case, every letter is positioned individually. In the very worst case, not even in the right order.

The easiest way to see this is to unpack the content of a page, for example with qpdf --qdf --object-streams=disable in.pdf out.pdf or mutool show. A sentence like "Die Lieferung erfolgt frei Haus." ("Delivery is free of charge.") can then look like this (abridged from the test document):

BT 1 0 0 1 72      697.89 Tm (D) Tj ET
BT 1 0 0 1 80.664  697.89 Tm (i) Tj ET
BT 1 0 0 1 83.328  697.89 Tm (e) Tj ET
BT 1 0 0 1 93.336  697.89 Tm (L) Tj ET
BT 1 0 0 1 100.008 697.89 Tm (i) Tj ET
...

Every letter gets its own text block (BT … ET), its own position (Tm) and is drawn individually (Tj). There is no space character. Between "e" and "L", the gap between the x coordinates is just a bit larger (10 pt instead of 2.7 pt between "i" and "e"), and whether that is a word gap is up to the extractor to guess. To do so, it has to know the width of each glyph, because the coordinate only says where a glyph starts. Likewise, there are no lines, paragraphs, headings or tables.

On top of that, the order in the content stream is arbitrary. A generator may draw all the headings first and then the body text, the footer first, or the lines of a paragraph from bottom to top. The viewer doesn't care, but for an extractor, it makes things more complicated.

A glyph is not a character

The second big misconception concerns the characters themselves. The D in (D) Tj is a code that the font maps to a glyph. For simple fonts, the code is usually the ASCII value. For embedded subset fonts (practically everything Office or a browser produces), it is often simply the number of the glyph in the font. That's perfectly sufficient for rendering. Anyone who wants to extract text additionally needs to know which Unicode character is behind the glyph, and that information lives in a separate, optional table, the ToUnicode CMap.

If the CMap is missing, the result is gibberish -- often a kind of Caesar cipher, because the glyph numbers are shifted relative to ASCII. If it is wrong, you get plausible-looking nonsense, like the "Ejf [bimvoh" from the introduction. In both cases, though, the page looks perfect, because the viewer doesn't need the table.

HTML, DOCX or Markdown, by comparison, carry semantics: an <h2> is a heading and a <td> is a table cell, while PDF only describes appearance. With Tagged PDF (and PDF/UA for accessibility), there is a structure tree with reading order, headings and tables. But in the wild, most PDFs aren't (meaningfully) tagged, and most parsers don't read the tags anyway.

The candidates

Before we look at the pitfalls, a quick look at the tools I compared. They fall into three groups that behave fundamentally differently.

• Text layer extractors: pypdf, pdfminer.six, PyMuPDF, pdftotext from Poppler, and MarkItDown (which calls pdfminer internally, and pdfplumber for pages that look like tables; since it behaves like pdfminer almost everywhere, it is left out of the overview further down). They only read what's in the content stream and reconstruct words and lines with heuristics. That's fast, cheap and deterministic.
• Layout pipelines: Docling and Unstructured (strategy hi_res). They also read the text layer, but put ML models for layout, reading order and tables on top, and run OCR where there is no text. The output is Markdown or a list of typed elements, respectively.
• Image only: Tesseract as classic OCR on the rendered pages (300 dpi), and Claude Haiku 5.5 as a multimodal LLM on the page images (prompt roughly "transcribe the page as Markdown"). Both only see what gets rendered and ignore the text layer completely.

This selection is arbitrary, and the comparison based on my constructed pitfall PDF is certainly not exhaustive. I still find it useful, because it gives a feel for which type of tool is suited to which problems. Further down there is a big overview in which I rated, for every pitfall and every candidate, whether the text arrives intact.

If you have control over what kind of PDFs go into the RAG system, you can build a tailored, fast and cheap solution. For example, I could imagine that large collections of legal texts work very well with a fast text layer extractor and possibly some special post-processing for headers and footers. If, on the other hand, you have relatively few documents, but with a complex structure and many images, the expensive processing with a multimodal LLM may pay off. And if your data consists of PDFs exported from a company wiki, throw them away and index an HTML export or the wiki's markup source directly.

The pitfalls

The test document contains 33 pitfalls plus two variants of the whole file (corrupted and encrypted). The complete catalogue with observations per tool is in the repository. Here I'll go through the ones that seem most important for RAG, roughly from small to large: characters, words, page, structure, images and finally malicious files.

Characters

Ligatures. Many fonts set "fi", "ffi" or "fl" as a single glyph. In the best case, the ToUnicode CMap maps them to the individual letters. Careful typesetting systems like LaTeX do this, and if you only test with such PDFs, you won't even notice the problem. Often, however, the CMap points to the ligature code point (U+FB03 for "ffi"; Chrome does it this way, for instance), and then the extracted text reads "Effizienz" (efficiency). For embeddings, that's a small loss of quality; for BM25 and full-text search, it's a different word. On the ligature page of the test document, none of the text layer extractors normalized it. The countermeasure is a one-liner, NFKC normalization (unicodedata.normalize("NFKC", text)), and it generally does no harm. If the ligature is missing from the CMap entirely, even NFKC won't help: then you get "EᎴzienz" or "E zienz", depending on the extractor. The two layout pipelines each fail on the other variant. Docling normalizes the ligature code point, but outputs "EᎴzienz" for the missing ligature. Unstructured leaves the code point in place, but recognizes the gibberish for the missing ligature and switches to OCR.

Missing or wrong ToUnicode CMap. The case from the introduction. Without a CMap, pypdf, PyMuPDF and pdftotext return glyph numbers as characters (".¾QGLJXQJVIULVW" for "Kündigungsfrist", notice period). pdfminer reconstructs the text as long as the embedded font still has its own character map, and otherwise returns the cryptic (cid:37)(cid:72)….

With a wrong CMap, all text extractors and both layout pipelines return the shifted text. Such CMaps result from buggy generators or are deliberately built in as "copy protection". Tesseract and Haiku read both pages correctly, because they only see the image.

Unstructured is interesting. With a missing CMap and with the Type 3 font without a character mapping, it returns the correct text, because it detects the gibberish and falls back to OCR for those parts. With the wrong CMap, this doesn't kick in; the shifted text looks like normal text to the heuristic. Docling outputs the gibberish in all cases.

Null bytes. A variant of this is a CMap that maps umlauts to U+0000. pypdf and pdfminer then return Gr\x00\x00e aus D\x00sseldorf ("Grüße aus Düsseldorf", greetings from Düsseldorf). That's not just ugly: PostgreSQL (and thus pgvector), for example, can't store \x00 in text columns at all. So you should always strip control characters after extraction.

Unicode look-alikes. Here the extraction is correct, and it still goes wrong: "Müller" with a combining diaeresis (u + ¨) instead of a precomposed ü, "Pass​wort" with a zero-width space, "Kаffee" with a Cyrillic а. In the viewer and in the extracted text, everything looks the same, but a search for "Müller" doesn't find the word. NFKC fixes the diaeresis, the invisible space you have to remove yourself, and the Cyrillic а can only be detected by checking the scripts within a word.

Words and lines

Spaces. As we saw in the content stream above, every extractor has to guess from gaps where a word ends. Each one does this with its own thresholds, which is why no tool gets all variants right. Letter-spaced text becomes "W i c h t i g e r" with pdfminer and PyMuPDF, and overly tight word gaps (1 pt) turn into "RechnungsadresseundLieferadressesindidentisch." with almost all of them. In the latter case, even Tesseract slips up ("Rechnungsadresseund Lieferadresse sindidentisch."), because the gap is just as narrow in the image. A simple quality signal is the mean word length: very short points to letter-spaced text, very long to missing spaces.

Fonts that aren't embedded. To measure the gap between two glyphs, the extractor needs the width of the first glyph, and that is stored in the font or in the /Widths array of the font dictionary. But it does happen that both are missing: the font is only referenced by name and not embedded, and /Widths is missing, even though the specification requires it except for the 14 standard fonts. The viewer picks a substitute font from the system, and the page looks reasonably normal. The extractors, on the other hand, have to guess the widths. pypdf simply assumes half the font size as the width of every glyph. For narrow letters like i, l or t that's too wide, the computed gaps shrink, and with individually positioned words you get "Die LieferungerfolgtfreiHaus". Docling and Unstructured glue the words together as well. pdfminer assumes a width of 0, and then the page falls apart into one character per line, even when real spaces are present in the stream. Such "letter columns" are of course useless for RAG. PyMuPDF and pdftotext read both variants correctly, Tesseract and Haiku anyway. Incidentally, nothing happens with Arial, because pypdf and pdfminer ship substitute metrics for it. In the test document, we trigger this problem with Calibri instead.

Hyphenation and line breaks. Hyphenated words ("Donaudampfschiff-\nfahrt") break into two parts, and full-text search no longer finds the whole word. All text extractors keep hyphens and line breaks. Docling joins hyphenated words and falls into the obvious trap: "E-Mail" at the end of a line becomes "EMail".

Superscripts and subscripts. "Der Umsatz stieg 2023¹ um 12 Prozent" (revenue rose by 12 percent in 2023¹) becomes "Der Umsatz stieg 20231 um 12 Prozent" with pdfminer, PyMuPDF and pdftotext, and "250 m²" becomes "250 m2". No LLM can untangle that afterwards. Tesseract doesn't do any better. The footnote turns into an apostrophe, "m²" into "m*" and "10⁻³" into "10°".

Page and layout

Multi-column layout. This mainly affects papers, brochures and magazines. Anyone who reads lines across the full width or follows the stream mixes both columns.

The chunk then contains sentences like "Jeder Mitarbeitende hat Reisekosten: Dienstreisen werden mit der Anspruch auf 30 Urlaubstage pro Jahr." -- roughly "Every employee is travel expenses: business trips are entitled to 30 days of vacation per year." Two things surprised me here. pdfminer, the classic tool, separates the columns correctly with its layout analysis. And Docling, despite its ML layout model, interleaves them like pypdf, because the model only detects a single text block on this page. Unstructured separates the columns correctly. It's a similar story when the lines of a paragraph are drawn bottom to top in the stream: Docling outputs the paragraph backwards, pdfminer and Unstructured get it right.

Headers and footers. "VERTRAULICH -- nur für den internen Gebrauch" (confidential -- internal use only) is printed on every page and ends up in every chunk with all text extractors. That makes chunks more similar to each other than they are, and the question "What is confidential?" matches every page. Docling classifies headers and footers and drops them, Unstructured keeps them. Without a layout model, you can remove lines that recur at the same position on many pages.

Page boundaries. If you parse page by page (e.g. so that every chunk has a page number for the citation), you cut sentences that run across the page boundary. And even when parsing continuously, footnote, footer and header end up in the middle of the sentence.

Tables across page boundaries. When a table spans two pages, the header row is usually repeated on the second. Lists of amounts also often have a "carried forward" row with the subtotal at the end of the first and the beginning of the second page. None of the tools merged the two halves back together. The second half is a separate chunk, and without a label like "(continued)" there's no way to tell which table it belongs to. The carry-forward rows look like data, and anyone summing over all rows counts the amounts from the first page several times. But even a cleanly merged table causes problems later: as described in the RAG article, text-based RAG is bad at answering questions that aggregate over a large table. A data set that largely consists of such multi-page tables may therefore need a different solution than text-based RAG, such as extracting the tables as structured data (ideally from a source other than the PDF) and querying them via SQL.

Page numbers. The page number the parser returns is the physical index (and, depending on the library, 0-based). The printed page number can be something entirely different thanks to page labels (ISO 32000-1, section 12.4.2): "iv" in the preface, or "2" on the third physical page. If your citations refer to "page 2" and you want to jump to the location in the viewer, you need to know which of the two is meant. Incidentally, a language model stumbled over this on its own with the test document and initially thought the table of contents was wrong, because it compared image numbers with printed page numbers.

Watermarks and stamps. A diagonal "ENTWURF" (draft) lands right between two paragraphs, and a rotated "GEPRÜFT" (approved) stamp may be taken by the LLM as a statement about the content. All text extractors output both.

Structure

Headings and lists. In a PDF, a heading is just a piece of text in a larger, bolder font. Without detected headings, there is no structure-aware chunking like the Markdown splitter I described in the RAG article, and no heading as context in the chunk. All text extractors output headings as ordinary lines. Docling detects them but puts them all on the same level (##), Unstructured marks headings as Title, without a level. Bullets from symbol fonts turn into arbitrary letters.

Tables. There are no tables in a PDF, just text at positions and a few lines (or not even those). How much of it survives depends heavily on the tool.

pdfminer groups by column, and does so across both tables on the page: first all regions and line items, then all numbers. The question "What was the revenue in the south in 2024?" can no longer be answered reliably, especially since the empty cell for "West 2024" shifts the assignment. pypdf and PyMuPDF output each cell on its own line, which at least preserves the row order, and pdftotext -layout preserves the rows best visually. Docling detects both tables (including the borderless one) perfectly and resolves the merged header cell into "Umsatz (Mio. €) - 2023". Unstructured disappointed me here: of table A, only one row is left in the HTML output, and in the borderless table, the cell detection cuts off the first letters ("Zeratung", "zeisekosten", "_izenz" instead of "Beratung", "Reisekosten", "Lizenz"). The revenue figures from table A are missing from the output entirely.

Forms. Filled-in values in AcroForm fields are not in the content stream, but in the fields and their annotations. pypdf and pdfminer only return the labels ("Name:", "Eintrittsdatum:"), PyMuPDF and Docling append the values at the end of the page, without any association. Haiku assigns them correctly. If you index forms, read the fields separately (reader.get_fields() in pypdf).

Images and scans

Scanned pages. A scan without a text layer yields an empty string with all text extractors, without any error. Without a check, the document silently ends up in the index as an empty chunk and can never be found. Docling, Unstructured, Tesseract and Haiku read the scan well. Incidentally, the same can happen when a page was "printed" via a PostScript printer driver. PostScript has no notion of transparency, and pages with transparency (e.g. text shadows) are rasterized completely in the process. Then, too, the text extractors return an empty page.

Existing OCR layer. Nastier is the scan on which the scanner has already placed an invisible OCR text, with typical errors ("l.Z34,S6" instead of "1.234,56"). Since there is text, no "empty page" check fires, and all text extractors take over the wrong amounts. Docling ignores the existing layer and runs its own OCR, Unstructured takes over the text layer like the text extractors. The fact that OCR noise carries through to answer quality is well studied, by the way; see OHRBench.

Text as outlines, and charts. Text from graphics programs is often "converted to outlines", i.e. stored as vector paths. For text extractors, it's unreadable. With charts, the information lies in the bar heights; what's left are axis labels ("0 5 10 15 20 25 Q1 Q2 Q3 Q4") that match any question about quarters but answer none. Haiku reads off the bars and marks the values as "approx.", which in my opinion is the best solution, but comes at the price of an LLM.

Malicious files

Hidden text. This is where a quality problem turns into a security problem. Parsers extract everything that's in the content stream, including what no human sees: white text on a white background, 0.5 pt font size, text outside the page, text behind a white rectangle or in invisible render mode 3.

Research on prompt injection in automated applicant screening shows that this happens in reality. For RAG, it means documents are an input channel that often bypasses all the guardrails for user input. All text extractors and layout pipelines output five or six of the six hidden messages. Tesseract and Haiku are immune; they only see the visible text.

Redaction that isn't. The classic data leak: a black bar over name, IBAN and salary, with the text underneath still fully present.

The viewer only shows the black bars. But the index contains the full text, and the LLM will happily quote it when asked. This applies to every tool that reads the text layer, including both layout pipelines. At those spots, Tesseract only reads a few characters of noise from the bars ("B", "]]"), Haiku writes "[geschwärzt]" ("[redacted]").

Corrupted and encrypted files. Any good viewer displays broken PDFs tolerantly; parsers vary in how forgiving they are. Anecdotally, LangChain's PDFMiner loader once crashed on a missing creationdate metadata field for me. In the test document, all tools except Haiku, which I didn't run on these variants, repaired the damaged xref table and still read the encrypted variant (empty user password, extraction prohibited). Whether you have to respect that prohibition is an organizational question that should be decided deliberately. In general, uploaded PDFs are untrusted input. Timeouts, memory limits and separate processes for large files are therefore a must.

All tools at a glance

Here is a comparison of the small selection of tools I tested. The categorization is perhaps somewhat subjective, but overall the difference between the three approaches comes out quite clearly.

The text layer extractors fail together and in the same places, namely wherever the text layer doesn't match the image (CMaps, Type 3 fonts, scans, hidden text, redaction). The layout pipelines repair structure, Docling mostly tables and headers, but hardly any broken characters. Unstructured gets more check marks thanks to its OCR fallback, which, however, only kicks in for obvious gibberish. Tesseract is immune to everything concerning the text layer, and in exchange has the typical OCR weaknesses: no structure, wrong superscripts, unusable formulas, rotated text, and it doesn't see the form values at all. Haiku doesn't get a single red cross.

Don't let the many check marks for Haiku lull you, though. A multimodal LLM interprets what it sees. It writes rendering notes in square brackets ("[geschwärzt]"), and they aren't always right: Haiku describes the printed scan as "handwritten/printed". Moreover, the result is not deterministic; two runs don't produce the same text. Classic parsers fail loudly, with gibberish and (cid:…), whereas a VLM fails quietly and plausibly. It can invent text that isn't in the document. Then there's the high cost of sending an image of every PDF page to a language model, and the question of whether the documents are even allowed to leave the company.

Benchmarks like olmOCR-Bench, OmniDocBench or OHRBench help you get your bearings, but they don't reflect your documents, and the rankings change every few months. It's also worth looking at the licenses: PyMuPDF, for instance, is under the AGPL.

There is a fourth option that I didn't try but want to mention: skipping parsing altogether. Visual retrieval like ColPali embeds the page images directly and then searches in image space. CLIP, mentioned in the RAG article, performs poorly on text pages here, because it isn't trained to read text in images. ColPali instead produces one vector per image patch and compares every word/token of the question with these patches. The models are openly available on GitHub and Hugging Face. Retrieval then no longer returns a text chunk but a page image, and the language model gets that image directly for the answer (VisRAG has studied this systematically). So if you work entirely without text extraction, you'll need a multimodal LLM at that point at the latest.

PDF as an intermediate format? Better not

If you index Markdown, HTML or source code alongside PDFs, it's tempting to convert everything to PDF first and then parse it uniformly. After all, a PDF has pages, Markdown doesn't. That way every chunk gets a page number as metadata, and a PDF-based viewer can jump straight to the right page for a citation.

The price for this is high. The conversion throws away the structure the original comes with. Headings, code blocks and tables become positioned glyphs that the parser then has to guess back. So you take on all the pitfalls from this article to produce a single piece of metadata, and on top of that you wait for the conversion.

It's better to parse the original and generate the PDF only for the preview. For formats without pages, there are other anchors a viewer can jump to: the section heading, a line range, an HTML anchor. If the source exists in a structured format (DOCX, HTML, Markdown, a Confluence export), use the source. What gets lost in conversion is shown by the "realistic" pages in the test document, which I didn't build myself but generated with real programs. A LibreOffice export turns a struck-through "zwei" (two) and an inserted "drei" (three) in tracked changes into "bis zu zweidrei Tage" (up to twothree days), and Chrome draws text shadows as a second copy of the text, so every heading appears twice in the index ("Ablauf der AbrechnungAblauf der Abrechnung").

What I recommend

From the catalogue above, you can derive a pipeline that catches the known pitfalls: text layer extractor, quality check per page, layout pipeline, OCR as a cross-check, and a VLM for the rest. Something like this is laborious to build and even more laborious to operate, because every stage brings its own dependencies, models and failure modes. For most applications, that's too much. It makes more sense to estimate which errors can occur in your own documents at all and which of them actually do harm.

Look at the extracted text. Almost all of the pitfalls above stand out on a first read. Take samples across different document types, and from the text that actually ends up in the index, not from the preview in the viewer.

Find out where the PDFs come from. If you produce the PDFs yourself, say from your own reporting or with a fixed Office export, you can rule out entire classes of errors. Whether the fonts are embedded and have a correct CMap, whether scans or hidden text occur -- you check that once and can rely on it afterwards. Then a simple text layer extractor is enough. Even better if the original is available: then you parse the original. With PDFs from outside (invoices, job applications, scanned mail), on the other hand, any pitfall from this article can show up.

Weigh the errors. A duplicated header or an "EMail" degrades retrieval a little, and you can usually live with that. It gets expensive with a redaction that isn't, because it becomes a data leak, and with hidden text in a job application that ends up as an instruction in the LLM's context. These errors are worth catching, even if it takes effort. On the other hand, some checks on the text layer are very cheap: characters per page (close to zero means a scan), the share of (cid:, U+FFFD and control characters, the mean word length. They catch a large share of the problems, and it's often best to flag the page instead of silently indexing it as an empty or broken chunk. No signal from the text layer helps against a wrong CMap or hidden text; for those, you have to compare it with the image, and Tesseract is a comparatively cheap cross-check for that.

Do the math on costs. The text layer extractors need milliseconds per page. Docling and Unstructured run on the CPU, but take several minutes for the test document and come with large models. Tesseract costs about two and a half seconds per page (300 dpi, 8 CPU cores). The VLM is the most expensive. Every page is sent to a model as an image and costs money, and it does so again with every reindexing, for example when the model or the prompt changes. For a few hundred pages, that doesn't matter; for a SharePoint with hundreds of thousands of pages, it does. A small model like Haiku lowers the costs, but showed errors of its own in the test. Then there's the question of whether the documents are even allowed to leave the company. A VLM is therefore mainly worthwhile for pages where everything else fails, i.e. scans, charts and formulas, and only if their content justifies the price. And the option of hosting multimodal LLMs yourself (e.g. Gemma or Qwen) can shift this trade-off once more.

Normalize and record the page. A few things cost next to nothing and belong in every pipeline: NFKC, removing control characters and null bytes, stripping soft hyphens, and recording the page with every chunk, both the index in the document and the printed label.

Evaluate the parsing. The retrieval evaluation from the RAG article shows that something isn't being found, but not whether the error lies in parsing, chunking or retrieval. A small gold data set "PDF → expected text" from your own documents is much more telling for that, and can be extended step by step with every problem document.

Parsing is the most inconspicuous part of a RAG pipeline, but every later step can only work with what comes out of it. It therefore deserves considerably more attention than it usually gets. Building RAG systems, and most recently building the test document, has in any case taught me more about PDF than I ever wanted to know. Ever since, I shudder every time someone dismisses reading PDFs as a solved problem.