Every company has them: ten years of .doc and .xls files, scanned PDFs without a text layer and Outlook PST archives on old file servers. Search tools and AI assistants cannot read most of it. A scripted conversion pipeline turns this content into searchable, text-based formats and keeps the metadata (date, author, sender, original path). It is the unglamorous first step before any assistant can answer questions about your own history.
Problem & value
- Who: every SMB with a long-lived file server, departed employees' mailboxes or a retired system. Typically IT manages this reluctantly, "when there is time".
- Today: old files sit on shares, PST files live on laptops and backup drives, image-only PDFs are invisible to search. Nobody knows what is inside.
- Value: content becomes findable and usable for RAG assistants (PREP-013) and classification. The conversion is also a good moment to delete what must not be kept (retention, GDPR).
Variants
- Mail archives: PST/OST, MSG and EML folders of former staff and shared mailboxes.
- Office legacy: binary .doc/.xls/.ppt, password-protected files, files from retired suites.
- Image-only documents: scanned PDFs and multi-page TIFFs (the scanning itself is PREP-006).
How it works
- inventory & copy (read-only)
- type detection
- convert
- OCR for image-only files
- extract text + metadata
- write target format + metadata table + log
- sample check
- index / archive
- Conversion is mostly classic, deterministic tooling (LibreOffice headless, mail extraction tools), not AI. AI is optional: OCR with layout models, or a small LLM to repair broken text and propose titles/tags.
- Human-in-the-loop: random sample review per file type, a review list for files that failed or have low OCR confidence, and a legal or DPO decision on what is deleted rather than converted.
Data preparation
Which data: the legacy files themselves plus a list of where they are (see PREP-001, PREP-003).
Where it lives: file servers, NAS devices, USB disks, laptops, M365/Exchange mailboxes, old backup sets.
Typical gaps: duplicates, password-protected or corrupt files, missing dates/owners, private content mixed in, and PST files of departed staff (retention unclear).
Steps: 1. Copy to a read-only staging area; keep originals untouched until the result is accepted. 2. Remove obvious duplicates by hash; classify by file type and age. 3. Decide with legal/DPO what is deleted, kept in original form or converted (PREP-025). 4. Convert in batches; record original path, hash, dates and sender/recipient in a metadata table. 5. Run OCR only on files without a text layer; check a sample per language. 6. Pseudonymise or restrict access where needed (PREP-022, PREP-023).
Tools: LibreOffice (headless), OCRmyPDF, Docling, readpst/libpst, Paperless-ngx, n8n or PowerShell/Python scripts.
Effort: for an SMB with a few hundred GB, expect roughly 3-10 days of IT work including sample checks, plus calendar time for the batch runs. Clarifying retention and ownership with department leads often takes longer than the conversion. See GUIDE-003.
Implementation paths
Which path for whom
- SMB — Path C: a one-off, data-sensitive job (mail, HR, customer files) that open-source tools handle on a laptop or small server (HW1); no per-page fees and no data leaves the company. The same pipeline later feeds Paperless-ngx or a RAG index (Baukasten effect).
- Enterprise — Path C: the pipeline is scripted once and reused across archives; sensitive mail stays on premises.
- Developer — Path C: open-source libraries give full control over metadata handling.
- Public — Path C: archives contain personal and administrative data; self-operated or sovereign-hosted processing fits procurement and records rules.
Path A — Buy (off-the-shelf)
- Microsoft 365: the Purview Import service bulk-imports PST files into Exchange Online mailboxes, including ANSI and Unicode formats; network upload goes to a temporary Azure storage location (check region availability and costs for your tenant), PST files should stay under 20 GB each, and imported mailboxes go on retention hold until you change it [1]. This rescues mail into a searchable, governed place but does not by itself convert it for your own AI pipeline.
- Archiving/migration vendors: e.g. MailStore (German vendor); check EU hosting and AVV (GUIDE-010).
- Choose A when: the goal is only to bring mail into M365 and the standard tools suffice.
Path B — Build on cloud APIs
- Cloud OCR (Azure AI Document Intelligence EU regions, Mistral OCR as EU provider) for hard scans; orchestration with Power Automate or n8n.
- Send only files that passed the DPO check; mail with personal data should not go to a cloud API by default.
Path C — Self-host (open source) (recommended)
- Office conversion: LibreOffice headless (
soffice --convert-to docx|xlsx|pdf) for .doc/.xls/.ppt. - OCR: OCRmyPDF adds a text layer and produces PDF/A by default using Tesseract, with 100+ languages; plugins allow other engines such as PaddleOCR [2].
- Parsing to Markdown/JSON: Docling (MIT) reads PDF, DOCX, XLSX, PPTX, HTML, images (PNG, TIFF, JPEG), EML and MSG, with OCR for scans [3].
- Mail: readpst (libpst) converts PST to mbox/EML per folder [4]; then script attachment extraction.
- Landing zone: Paperless-ngx (GPL-3.0) for scan, index and archive, or Nextcloud for sharing [5].
- Hardware: HW1 (a normal server or workstation) is enough; a GPU is only useful for VLM-based OCR.
Path D — Custom training
Not needed. Only for rare scripts or damaged historical documents (see DOC-005).
State of the art (2026)
- Deterministic converters are mature; the main progress is in OCR and parsing models that deliver layout-aware Markdown, which suits RAG.
- Limits: OCR errors in poor scans, lost formatting in complex tables, broken macros and embedded objects, encrypted files that cannot be opened, mail threads that lose context when split into single files, and inline images in mail.
- LLM post-processing can invent text; never let it overwrite the original or the OCR layer without a check.
Shared resources & combinations
- Same pipeline serves PREP-006 (scanning), DOC-014 (table extraction), PREP-010 (metadata), PREP-013 (RAG knowledge base) and DOC-018 (redaction).
- One small conversion server with n8n can also run other everyday automations.
Effort, cost & KPIs
- Path A: hours to days for the M365 import itself; licence costs depend on your plan.
- Path C: about 3-10 days of IT time plus batch run time; no licence fees, costs are staff time and storage.
- KPIs: share of files converted without errors, share with a usable text layer, duplicates removed, files deleted under retention rules, search hit rate in a sample test.
People & change
Involve early: management (scope, deletion decisions), department leads and data owners (what is still relevant), IT and IT security (staging, access), DPO and legal (retention, personal data, mail of former staff) (GUIDE-005).
Works council: depends. Mail archives and old files contain employee communication; if converted content becomes searchable by AI, it may be suited to monitor behaviour or performance (§87(1) no. 6 BetrVG), and §90(1) no. 3 requires information when AI-supported work processes are planned. Agree early that private and personal mail is excluded and that searches are not used for performance evaluation (GUIDE-004). Public bodies involve the Personalrat.
AI literacy: Art. 4 AI Act requires measures that support staff's AI literacy; a short briefing for IT and archivists on OCR errors is enough here (GUIDE-006).
Communication and pilot: explain what is converted, deleted and searchable; pilot with one share (GUIDE-008, GUIDE-009).
Humans in control: originals stay until a sampled check passes; deletion decisions are made by named owners, not by scripts.
Risks, regulation & compliance
- EU AI Act: minimal risk; conversion is not an Annex III use. Art. 4 literacy measures apply.
- GDPR: old mail and files contain personal data, sometimes special categories (health, HR). Check legal basis and retention (Art. 5, Art. 6); delete what is not needed. DPIA is likely only if AI search later evaluates people systematically. AVV (Art. 28) for any cloud step.
- Mail secrecy and retention: for business mail, German commercial and tax retention rules (HGB, AO) may require unaltered archiving; keep originals or a GoBD-compliant archive. Private mail needs special care.
- Security: PST files may contain malware or credentials; scan before processing, limit access to the staging area (GUIDE-013).
- Mitigations: read-only staging, hash log, sample review, deletion approval.
Getting started
- Find the legacy sources: file shares, PST files, old disks (PREP-001, PREP-003) and write down owners.
- Decide retention and deletion with legal/DPO before converting anything.
- Take 200-500 representative files and run LibreOffice, OCRmyPDF and readpst on a test machine; measure failures.
- Define the target format and metadata table; load the result into Paperless-ngx or your search tool.
- Run the full batch, review a sample, then archive the originals read-only.
Sources
- Microsoft Learn — Learn about importing organization PST files (Purview) — https://learn.microsoft.com/en-us/purview/pst-import-overview
- OCRmyPDF project — https://github.com/ocrmypdf/OCRmyPDF
- Docling project — https://github.com/docling-project/docling
- libpst / readpst — https://www.five-ten-sg.com/libpst/
- Paperless-ngx project — https://github.com/paperless-ngx/paperless-ngx