Handwriting recognition & historical document transcription

Archives, libraries, companies and families hold enormous amounts of handwritten and historical material that is not searchable and can be read only by a few specialists. Handwritten text recognition (HTR) models and vision-language models (VLMs) turn page images into text with coordinates and confidence values. A human corrects and approves the result. The benefit is access: whole collections become full-text searchable, and routine handwritten forms no longer need to be typed in.

Problem & value

• Today: transcription is manual (volunteers, paleographers, data-entry staff) or limited to printed text by classic OCR, which fails on cursive script, Kurrent/Sütterlin, abbreviations and damaged paper. Collections stay unindexed; handwritten forms and notes are retyped.
• AI value: automatic line segmentation and recognition, searchable even with an imperfect error rate (keyword spotting and fuzzy search), faster first drafts for human correction, and structured extraction from registers.
• Scale evidence (vendor-reported): Transkribus states more than 200 million pages transcribed, 300+ community models and 100+ languages/scripts including Kurrent and Sütterlin; one cited university project transcribed 20,000 pages in two months [1]. These are vendor claims, not independent measurements.
• Independent evidence: an academic benchmark (modern and historical handwriting in English, French, German, Italian) found multimodal LLMs very good on modern handwriting, better on English than other languages, with neither LLMs nor Transkribus models consistently ahead; LLMs could not reliably correct their own zero-shot errors [6].

Variants

• Archives, libraries, museums: mass digitisation of manuscripts, church books, civil registers, with ALTO/PAGE XML output and digital editions.
• Digital humanities / genealogy: research projects and family historians (including Kurrent/Sütterlin letters) — private use is realistic via subscription tools.
• Public administration: legacy paper files, handwritten applications, land and building registers.
• Business: handwritten forms, delivery-note annotations, lab and maintenance logs, field service notes (overlap with DOC-003, DOC-006, DOC-007).
• Healthcare: handwritten medical notes/prescriptions; medical-record extraction is covered in DOC-007.
• Structured register extraction: turning tables of entries (names, dates, amounts) into records rather than running text.

How it works

1. scan / photo (300+ dpi, deskew, contrast)
2. layout analysis (regions, lines, reading order, tables)
3. text recognition (HTR model or VLM)
4. text + confidence per line
5. optional post-correction (LLM with glossary, spelling variants kept)
6. human review & correction (highlighted low-confidence lines)
7. export (ALTO/PAGE XML, TEI, searchable PDF, JSON) + entity tagging
8. search index / edition / database; corrections feed model retraining

• HTR engines are trained recogniser/segmenter models (CRNN/transformer-based, e.g. Kraken, TrOCR-style) trained per hand, period or script with transcribed sample pages [3][5].
• VLMs read the page image zero-shot and often need no training, but can silently "normalise" or invent text.
• Human-in-the-loop: mandatory for editions and legal use; sample-based QA for search-only use.

Data preparation

Data: page images of letters, registers, ledgers, forms and notes, plus a small set of ground-truth transcriptions. In an SMB these sit in paper folders, cellar archives, file shares with unlabelled scans or in e-mail attachments; in archives they are bound volumes and microfilm.

Typical gaps: low resolution or phone photos, skewed and cropped pages, mixed hands and periods, no page order or metadata, no transcribed samples, and personal data with retention limits or access restrictions.

Preparation steps: 1. Inventory and prioritise holdings (PREP-001, PREP-002); decide what is worth transcribing. 2. Digitise at 300+ dpi, uncompressed or lossless, with consistent naming (PREP-006). 3. Deskew, crop and enhance contrast; sort by hand, period and script. 4. Add minimal metadata (source, date, signature) and fix a transcription guideline with a glossary of names and places (PREP-015). 5. Transcribe 50-100 representative pages as ground truth for evaluation and fine-tuning (PREP-018, PREP-016). 6. Check legal basis and protection periods before processing personal data (PREP-025).

Tools: scanners or book cradles, ScanTailor, OCRmyPDF, Transkribus or eScriptorium for ground truth.

Effort share: roughly 50-60 percent of the project goes into scanning, sorting and ground truth; the recognition itself is quick. Related: DOC-014, DOC-018, GUIDE-003.

Implementation paths

Which path for whom

• SMB: Path A. Mature, EU-based transcription platforms exist at affordable per-page prices and the work is a standard, project-based task.
• Enterprise: Path B. Corporate archives and registers usually need the output pushed into DMS or case systems, which calls for API integration.
• Developer: Path C. Open-source HTR engines (e.g. Kraken) can be trained on your own hands and scripts and embedded freely.
• Private: Path A. Genealogists and hobbyists get the best results from ready-made apps without any setup.
• Public: Path C. Archives and libraries need sovereign operation and open formats (ALTO/PAGE XML) and often run community platforms on their own or German cloud infrastructure.

Path A — Buy (recommended)

• Transkribus (READ-COOP SCE, Austria — EU): the reference platform for historical documents; 300+ models, own-model training, layout/table/field recognition, API from the Scholar plan. Data stays on the cooperative's servers in Austria. Public pricing at time of research: free 50 credits/month; Scholar €99/year; Team €449/year; HTR costs 1 credit per page, API jobs half the credits [2]. On-premise is not advertised — ask the vendor.
• eScriptorium (EPHE/PSL Paris, open source, hostable): a transcription workspace using Kraken [3][4].
• General document-AI products (Azure Document Intelligence Read, Google Document AI, ABBYY, Adobe) for modern handwriting in forms; weaker on historical hands. Check EU region and DPA.
• Choose A when: a collection of up to some hundred thousand pages and no in-house ML team.

Path B — Build on cloud APIs

• Frontier multimodal models (Gemini, Claude, GPT families) with page images + a prompt requesting a diplomatic transcription; Gemini accepts PDFs up to 1,000 pages [8]. Mistral OCR (EU provider, batch API) for mixed print/handwriting [7]. Azure Document Intelligence prebuilt-read extracts handwritten text and flags handwritten lines with confidence [9].
• Pattern: segmentation by a layout tool, line- or page-level VLM call, confidence via self-consistency (run twice, compare) or logprobs, review UI. Use EU regions and zero-retention contracts for personal data.
• Choose B when: modern handwriting, many different hands, or no time to train per-hand models.

Path C — Self-host (open source)

• Kraken (Apache-2.0, trainable layout + recognition, ALTO/PAGE output) with eScriptorium [3][4].
• Open VLMs: Qwen3-VL (Apache-2.0; 2B–235B, 32 OCR languages incl. rare characters) [10]; olmOCR-2 (Apache-2.0, 7B FP8, ≥12 GB GPU) mainly for print and clean layouts [11]; Docling/Granite-Docling (MIT) for layout parsing [12].
• Hardware: HW2 (one 24–32 GB GPU) for training line recognisers and serving 7–8B VLMs; CPU-only is possible for Kraken inference at low volume.
• Choose C when: sensitive personal data (health, court, asylum files) must not leave the premises.

Path D — Custom training

Usually the standard route for historical material: fine-tune an HTR model on 25–100+ transcribed pages of the specific hand/period (the Transkribus and Kraken workflows are built for this). Fine-tuning a VLM is worth it only for large, homogeneous collections. Ground truth quality (consistent transcription rules) matters more than quantity.

State of the art (2026)

• Two worlds converge: specialised HTR (Transkribus, Kraken) gives controllable, coordinate-accurate, trainable output; multimodal LLMs give strong zero-shot results on modern handwriting and mixed documents. Benchmarks show no consistent winner; LLMs are best on English and modern hands [6].
• Best practice: hybrid — HTR/segmentation for geometry, VLM for hard lines or post-correction, confidence-based review.
• Limits and failure modes:
• LLMs hallucinate plausible words, modernise spelling and expand abbreviations unasked — fatal for diplomatic editions.
• Names, numbers and dates in registers are the most error-prone fields; always spot-check.
• Accuracy falls on non-English, rare scripts, bleed-through, damaged paper and dense marginalia.
• Reported error rates depend on the hand; always measure CER/WER on your own sample.
• Tables and mixed print/handwriting need dedicated layout handling.

Shared resources & combinations

• Same stack as the document family: ocr-docparse + vlm + llm-general serve DOC-001 (invoices), DOC-003 (forms), DOC-007 (medical records) and DOC-008 (mailroom classification) on one VLM server (HW2/HW3).
• Downstream: DOC-014 (parsing for RAG), KNOW-001 (enterprise search over transcribed archives), embedding-text + search-engine / vector index for semantic archive search, LANG-005 (translation of transcribed historical texts), entity extraction with a text-classifier/NER model.
• Shared data pipelines: scan pre-processing, review UI, ground-truth store and retraining loop are reusable for every capture use case.

Effort, cost & KPIs

• Path A: days to set up; cost is per page/credit and review labour.
• Path B: 1–2 months for a pipeline with review UI; API cost per page plus review effort.
• Path C/D: 2–4 months incl. ground-truth creation and a GPU workstation; training per collection is hours to days.
• KPIs: character/word error rate on a held-out sample, share of lines needing correction, correction minutes per page, search recall, pages per day, cost per page.

People & change

Involve early: management (scope, budget), the department lead or archivist as process owner, the end users who proofread, IT (storage, accounts), the data owner of the holdings and the data protection officer. Bring the DPO in before scanning starts if the material holds personal data such as civil registers or staff files.

Works council: usually not triggered. The system processes documents, not employee behaviour, so §87(1) no. 6 BetrVG applies only if reviewer productivity or error rates are logged per person. Inform the works council anyway under §90(1) no. 3 if workflows change; a staff council applies in the public sector. See GUIDE-004.

Data protection: check legal basis and retention; a DPIA is likely for large collections of sensitive personal data, and every cloud provider needs an AVV (GUIDE-005).

AI literacy: the AI Act (Art. 4) requires measures that support staff AI literacy; a short session on how HTR errs, why confidence scores matter and how VLMs can invent text is enough (GUIDE-006).

Communication and pilot: choose one volume or one hand as pilot with two or three volunteers and measure character error rate and correction time (GUIDE-009).

Typical concerns: fear of replacing specialists (they become reviewers and quality owners), trust in results (show the confidence highlighting) and editorial integrity (humans approve every edition text).

Risks, regulation & compliance

• EU AI Act: transcription itself is minimal risk (AI literacy duty, Art. 4). It becomes high-risk only when its output feeds an Annex III decision system (e.g. asylum/migration, benefits eligibility, law enforcement evidence); then the downstream system carries the obligations. Not a prohibited practice.
• GDPR: handwritten documents often contain special-category data (health, religion in church books); living persons require a legal basis; archive/research privileges (Art. 89) and national archive laws apply. Prefer EU hosting (Transkribus: Austria) or self-hosting.
• Authenticity & law: a machine transcription is not an authenticated copy; keep the original image as the source of truth. For tax-relevant documents GoBD rules on traceable conversion apply.
• Mitigations: diplomatic vs. normalised transcription policy, confidence thresholds, versioned corrections, clear labelling of AI-generated transcripts.

Getting started

1. Select 50 representative pages (best, typical, worst) and transcribe 10–20 by hand as ground truth.
2. Run them through Transkribus (public models), one VLM API and, if needed, Kraken; compute CER/WER.
3. Decide on transcription rules (abbreviations, spelling, line breaks) before training anything.
4. Fine-tune an HTR model on 25–100 pages or set up a VLM pipeline with confidence-based review.
5. Export to ALTO/PAGE XML or a search index; check privacy basis for personal data before publishing.

Sources

1. Transkribus (READ-COOP) — platform overview and stated figures — https://www.transkribus.org/
2. Transkribus — pricing, credits per page, API — https://www.transkribus.org/pricing
3. Kraken OCR/HTR engine (Apache-2.0) — https://github.com/mittagessen/kraken
4. eScriptorium documentation (EPHE/PSL) — https://escriptorium.readthedocs.io/en/latest/
5. TrOCR: Transformer-based OCR with pre-trained models — https://arxiv.org/abs/2109.10282
6. Crosilla, Klic, Colavizza — Benchmarking Large Language Models for Handwritten Text Recognition — https://arxiv.org/abs/2503.15195
7. Mistral AI — OCR documentation — https://docs.mistral.ai/capabilities/document_ai/basic_ocr/
8. Google — Gemini API document processing — https://ai.google.dev/gemini-api/docs/document-processing
9. Microsoft — Azure Document Intelligence Read model — https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/prebuilt/read
10. Qwen3-VL — https://github.com/QwenLM/Qwen3-VL
11. olmOCR (AI2) — https://github.com/allenai/olmocr
12. Docling — https://github.com/docling-project/docling
13. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj