PDF-to-structured data conversion & table extraction (document parsing for RAG)

The quality of every RAG assistant, enterprise search and analytics pipeline is capped by the quality of its document parsing: a table flattened into a text soup or a two-column page read in the wrong order produces confidently wrong answers. Modern layout models and compact vision-language models turn PDFs, scans and Office files into structured Markdown/JSON with tables, formulas, reading order and source coordinates — locally on a single GPU or via an API for a few dollars per 1,000 pages.

Problem & value

• Today: PDF is a print format, not a data format. Copy-paste or classic text extraction loses structure (columns, tables, headers/footers); scans need OCR; spreadsheets and slides are flattened. Companies keep knowledge in manuals, reports, datasheets, tenders and contracts that search and RAG tools cannot use reliably.
• AI value: layout-aware parsing keeps headings, tables and reading order, enables citations back to page and bounding box, and makes archives machine-readable at scale. Benchmark research on RAG found that OCR/parsing noise measurably lowers RAG answer quality and that no tested OCR solution was good enough for building high-quality knowledge bases at the time [6] — a motivation for the 2025–26 wave of document VLMs.
• Cost: parsing is now cheap: Mistral OCR 3 lists $2 per 1,000 pages ($1 via batch) [3]; the olmOCR project states "less than $200 USD per million pages" self-hosted (vendor claim, no method given) [5].
• Scope boundary: this use case is the generic parsing layer. Domain-specific field extraction with business validation sits in DOC-001 (invoices), DOC-003 (forms), DOC-012 (financial statements), DOC-013 (drawings). Retrieval and answering sit in KNOW-001/002.

Variants

• RAG ingestion: parse → chunk along headings/tables → embed → index; chunks carry page and box metadata for citations (KNOW-002).
• Table-to-data: extract financial or technical tables into CSV/SQL/dataframes, including tables spanning pages; datasheet and parts-list extraction in manufacturing.
• Scientific & technical documents: formulas to LaTeX, chart-to-table, figure captions.
• Archive digitisation: mass conversion of scanned archives for public administration and libraries (handwriting: DOC-005).
• Multi-format normalisation: one pipeline for PDF, DOCX, PPTX, XLSX, HTML, email.
• Private use: converting personal scans (manuals, contracts, tax papers) into searchable notes.

How it works

1. ingest (file, email, DMS, SharePoint)
2. triage (born-digital? scan? password? language? page count)
3. layout analysis (regions: text, title, table, figure, formula, header/footer; reading order)
4. recognition (text layer or OCR; table structure; formula
5. LaTeX; chart
6. table; figure caption)
7. assembly (Markdown/HTML/JSON, merge cross-page tables, drop headers/footers)
8. quality gate (confidence, table-sum checks, sample review, fallback to stronger parser)
9. chunking with heading/table-header context + metadata (page, bbox, doc id, ACL)
10. embedding / index / structured store (DOC
11. KNOW-001/002, analytics)

• Three technical families: (1) pipeline parsers with separate layout, OCR and table models (classic Docling, MinerU pipeline, Azure/AWS/Google layout services); (2) end-to-end document VLMs that emit a whole page as Markdown/DocTags (Granite-Docling, PaddleOCR-VL, olmOCR, Mistral OCR, MinerU2.5); (3) general frontier VLMs prompted with page images, flexible but costlier and able to paraphrase or hallucinate content.
• Human-in-the-loop: sample-based review per document class, review queue for low-confidence pages, and numeric checks on extracted tables (row/column sums, units) before data enters reporting.

Data preparation

This use case is itself a data-preparation step, but it needs prepared inputs. Source documents sit on file shares, in DMS/ERP attachments, mail archives and paper. Typical gaps: scans without text layer, rotated or low-contrast pages, password-protected PDFs, duplicates and old versions, mixed languages, and complex tables or forms. Access rights get lost when files are copied to a parsing queue.

Steps: (1) inventory formats and volumes, see PREP-001; (2) scan remaining paper at 300 dpi, see PREP-006; (3) convert legacy formats and mail archives, see PREP-007; (4) deduplicate and keep only valid versions; (5) carry over source path, owner, date and permissions as metadata, see PREP-010; (6) draw a 100-page sample, parse it with two engines, and compare tables by hand against a gold set, see PREP-018. Tools: Docling, PaddleOCR or Mistral OCR, a queue such as Celery or n8n, and Paperless-ngx as store. Plan for 20-30 % of effort in preparation and 30 % in ongoing quality checks of the output.

Implementation paths

Which path for whom

• SMB: Path C — Docling or similar runs on one HW2 box (or CPU for born-digital PDFs), keeps confidential documents in-house, and the same parser feeds many other use cases (Baukasten effect).
• Enterprise: Path C — high page volumes make per-page API costs significant, and a shared internal parsing service avoids duplicated pipelines.
• Developer: Path C — open parsers are scriptable, testable and versionable; use Path B APIs only for long-tail layouts.
• Public: Path C — documents often contain citizen data and procurement rules favour sovereign operation; Path B on a German/EU sovereign cloud is the fallback.

Path A — Buy (off-the-shelf)

• Parsing-as-a-service / platforms: LlamaParse (LlamaIndex; agentic-OCR tiers, 130+ formats, regional deployment and EU data-protection pages documented) [4], Unstructured platform (partition, extract, enrich, chunk, embed; 65+ file types, 35+ connectors) [7], Reducto, Mistral Document AI (FR, EU provider).
• IDP vendors (EU marked): ABBYY, Hypatos (DE), Klippa (NL), Mindee (FR), Rossum (CZ), DocuWare and d.velop DMS suites (DE) — strong where parsing is coupled with classification and workflow.
• Built into RAG products: Microsoft 365 Copilot/Azure AI Search, Google Vertex AI Search, AWS Bedrock Knowledge Bases, and EU-hosted enterprise-search products embed their own parsers.
• Choose A when: parsing is not a differentiator, documents may leave the company, volume is moderate.

Path B — Build on cloud APIs

• Layout/OCR services: Azure AI Document Intelligence layout model (Markdown output, tables with spans, selection marks, handwritten-style detection, up to 2,000 pages per PDF/TIFF) in EU regions [2]; Google Document AI Layout Parser (hierarchy-aware, RAG-ready chunks with ancestral headings and table headers; Gemini-generated figure/table descriptions; online limit 15 pages, batch up to 500) [8]; AWS Textract tables/layout (cells, merged cells, column headers, table titles/footers) in Frankfurt [9].
• Document VLM APIs: Mistral OCR 3 (mistral-ocr-2512, Markdown with HTML tables incl. colspan/rowspan, $2/1,000 pages; vendor-reported 74 % win rate over OCR 2 on its internal benchmark) [3].
• Frontier multimodal LLMs (Claude, Gemini, GPT) with page images and JSON schema for long-tail layouts and for chart/figure interpretation; use EU regions and zero-retention terms.
• Orchestration: n8n / Power Automate / Airflow; store raw file + parsed JSON + page images for audit.
• Choose B when: you need quality fast, mixed document types, and have no on-prem mandate.

Path C — Self-host (open source)

• Docling (IBM Research project, MIT): PDF, DOCX, PPTX, XLSX, HTML, images, audio; lossless DoclingDocument JSON, Markdown/HTML export; local/air-gapped; docling-serve API, MCP server, LangChain/LlamaIndex/Haystack integrations; Granite-Docling-258M (Apache 2.0, Sept 2025) as a compact VLM — English strong, Japanese/Arabic/Chinese experimental [1][10].
• PaddleOCR-VL-1.6 (0.9B, Apache 2.0, May 2026): 100+ languages, seals, cross-page table merging, 96.3 on OmniDocBench v1.6 (self-reported) [11].
• MinerU (OpenDataLab; own licence based on Apache 2.0 with extra conditions — check before commercial use), olmOCR 0.4 (AI2, Apache 2.0; 82.4 on olmOCR-Bench) [5], Marker (code Apache 2.0, model weights under a modified OpenRAIL-M licence that restricts commercial use above a revenue threshold) [12], Unstructured open-source library (Apache 2.0) [13].
• Serving: vLLM / SGLang / llama.cpp; PDF rendering with pdfium; queue (Celery/Temporal) for batches.
• Hardware: HW2 (one 24–32 GB GPU) handles sub-1B document VLMs at roughly several pages per second (Marker's published throughput ranges 2.9–7.4 pages/s per GPU depending on mode, other systems lower [12]); CPU-only (HW1) works for born-digital PDFs with pipeline parsers. HW3 for 7B+ VLMs.
• Choose C when: confidential documents (health, legal, HR, defence-adjacent), very high volume, or sovereignty requirements. Ops effort: moderate (GPU server, model updates, regression tests).

Path D — Custom training

Rarely needed. Worth it for unusual scripts, stamps/seals, domain-specific table formats (e.g. legacy engineering lists, historical ledgers) or to compress cost at very high volume. Start by fine-tuning a small document VLM (PaddleOCR-VL, Granite-Docling, Qwen-VL class) on a few thousand page/Markdown pairs, seeded with outputs of a strong parser and corrected by humans. Evaluate on a held-out set of your own documents; do not rely on public leaderboards alone.

State of the art (2026)

• End-to-end document VLMs dominate public leaderboards: on OmniDocBench (CVPR 2025; 1,651 pages, 10 document types, 5 layout types, 5 languages; leaderboard v1.6) the top end-to-end scores are in the mid-90s (e.g. PaddleOCR-VL-1.6 96.34, MinerU2.5-Pro 95.75 as listed on the project page) [14]. Sub-1B models now rival far larger ones; the dataset licence is research-only.
• olmOCR-Bench (1,400 PDFs, 7,000+ test cases) shows unit-test-style scoring: olmOCR 0.4 82.4, PaddleOCR-VL 80.0, Chandra 83.1; pipeline tools such as Docling (50.3 on the Marker-reported run) and MinerU pipeline (72.7) trail the VLMs there — numbers depend on benchmark and version, so test yourself [5][12].
• Trends 2025–26: compact VLMs (0.3–1B) with structured tags (DocTags, OTSL tables), cross-page table merging, chart-to-table, formula-to-LaTeX, parser tiers ("fast" text-layer vs. "agentic" VLM), hybrid designs that keep OCR grounding to limit hallucination (Google's stated rationale) [8], and standard outputs (Markdown + JSON with bboxes) feeding MCP-based agents.
• Limits & failure modes: multi-page and nested tables, merged headers, rotated or skewed scans, small print and stamps, handwriting, mixed languages, scanned forms with checkboxes; VLM-based parsers can silently drop or invent rows; reading order on complex multi-column layouts; benchmark leaderboards are mostly English/Chinese; vendor numbers are self-reported. Mitigate with numeric checks, bbox citations, dual-parser comparison on critical documents and regression sets.

Shared resources & combinations

• Same stack: ocr-docparse + vlm serve DOC-001 (invoices), DOC-003 (forms), DOC-005 (handwriting), DOC-008 (mailroom), DOC-012 (financial statement spreading), DOC-013 (drawings); one GPU parsing service (HW2/HW3) can back all of them.
• Downstream: KNOW-001/002/004 (search, RAG, technical documentation assistants) consume the chunks; embedding-text, reranker and vector-db are shared with them.
• Upstream/adjacent: DOC-018 (anonymise before indexing or before sending to external APIs), DOC-008 (classify and route first).
• Shared building blocks: document store with page images and bboxes, parse cache keyed by file hash, ACL propagation to chunks, evaluation set of 100–300 own pages, workflow engine (n8n).

Effort, cost & KPIs

• Path A: days–weeks; per-page or per-credit SaaS pricing.
• Path B: 2–6 weeks for ingestion pipeline with caching, retries, quality gates; API cost roughly $1–15 per 1,000 pages depending on service and features (check current price lists).
• Path C: 1–3 months incl. serving, evaluation and monitoring; one GPU server, low marginal cost.
• Path D: 2–4 months with annotation effort.
• KPIs: table-cell accuracy (TEDS-style or own cell F1), reading-order errors per page, text edit distance on gold pages, share of pages routed to review, retrieval hit rate / answer correctness of downstream RAG before and after switching parser, cost and latency per page, throughput pages/hour.

People & change

Involve: management for the decision and budget, IT for the GPU box and queue, IT security for the data flow, and the data owners of each source (finance, HR, engineering) because they know which documents are sensitive. The department lead of the first consuming use case (e.g. a knowledge assistant) acts as process owner.

Works council: usually no co-determination under §87(1) no. 6 BetrVG, as parsing documents does not monitor behaviour; inform it under §90(1) no. 3 when it is part of a wider AI rollout. Data protection: HR files, contracts and customer mail contain personal data; involve the DPO, check purpose and legal basis, and sign an AVV with any API provider, see GUIDE-005. Run a DPIA when HR or health documents are in scope.

AI literacy: train the few operators on error types (merged cells, reading order). Pilot: one document class, two reviewers, spot-check 5 % of output. Concerns: "who sees my documents?" - preserve source permissions and log access. Humans review tables that feed numbers. See GUIDE-003 and GUIDE-013.

Risks, regulation & compliance

• EU AI Act: minimal risk for the parsing layer itself (no Annex III use); AI literacy (Art. 4) applies. Risk class is inherited from the downstream use: parsing CVs, loan files or medical records for decisions pulls the system into Annex III obligations (high-risk applicability moved to 2 Dec 2027 under the 2026 omnibus) [15]. Generated summaries/figure descriptions published to the public may require Art. 50 labelling (transparency rules apply from Aug 2026) [15].
• GDPR: documents often contain personal data (contracts, HR, health); sending them to cloud APIs needs a DPA, EU region, no-training/zero-retention terms and, for non-EU providers, a transfer mechanism. Self-hosting (Path C) removes the transfer question. Consider anonymisation (DOC-018).
• Security: PDFs can carry hidden text or prompt injections that reach downstream LLMs; treat parsed content as untrusted input, scan files, sandbox rendering.
• Records law: if the original is a tax-relevant record (GoBD), keep the original unchanged; the parsed copy is a derivative.
• Licensing: check model-weight licences (Marker weights, MinerU licence, OmniDocBench research-only dataset terms) [12][14].
• Quality risk: silent errors in tables propagate into reports; keep citations to page/bbox and validation checks.

Getting started

1. Build a gold set: 100–200 pages of your own documents (worst tables, scans, multi-column) with expected Markdown/CSV for key tables.
2. Run 3–4 candidates (e.g. Docling, PaddleOCR-VL, Mistral OCR, Azure/Google layout) and compare cell accuracy, reading order, cost and latency; look at failures, not only averages.
3. Choose a primary parser plus a fallback for hard pages; add numeric/structural quality gates.
4. Chunk along document structure (headings, whole tables with header context) and store page/bbox metadata; connect to the RAG/search stack (KNOW-002).
5. Cache by file hash, re-run the gold set on every parser/model upgrade, track KPIs monthly.

Sources

1. Docling project (IBM Research, MIT) — https://github.com/docling-project/docling
2. Microsoft — Azure AI Document Intelligence layout model — https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/layout
3. Mistral AI — Mistral OCR 3 — https://mistral.ai/news/mistral-ocr-3
4. LlamaIndex — LlamaParse documentation — https://developers.llamaindex.ai/python/framework/llama_cloud/llama_parse/
5. AI2 — olmOCR repository (v0.4.0, olmOCR-Bench) — https://github.com/allenai/olmocr
6. Zhang et al. — OCR Hinders RAG: Evaluating the Cascading Impact of OCR on RAG (OHRBench) — https://arxiv.org/abs/2412.02592
7. Unstructured — platform documentation — https://docs.unstructured.io/welcome
8. Google Cloud — Document AI Layout Parser and chunking — https://docs.cloud.google.com/document-ai/docs/layout-parse-chunk
9. AWS — Amazon Textract tables — https://docs.aws.amazon.com/textract/latest/dg/how-it-works-tables.html
10. IBM — Granite-Docling-258M model card — https://huggingface.co/ibm-granite/granite-docling-258M
11. PaddleOCR repository (PaddleOCR-VL-1.6) — https://github.com/PaddlePaddle/PaddleOCR
12. datalab-to — Marker (licence, olmocr-bench comparison) — https://github.com/datalab-to/marker
13. Unstructured open-source library — https://github.com/Unstructured-IO/unstructured
14. OmniDocBench (CVPR 2025) — https://github.com/opendatalab/OmniDocBench
15. European Commission — AI Act regulatory framework and application dates — https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai