Document anonymisation & redaction (GDPR)

Before a contract, court file, HR record, FOI/IFG response, DSAR bundle or training corpus can leave the organisation, personal and confidential details must be removed. Doing this by hand with black boxes is slow and error-prone, and badly applied redaction is a classic cause of leaks. AI detects names, addresses, IDs, health data and context-dependent identifiers, applies true (irreversible) redaction or consistent pseudonyms, and leaves the final judgement to a human reviewer.

Problem & value

• Today: staff or paralegals search documents visually, draw boxes in a PDF tool, and re-check by eye. Typical triggers: access-to-information and DSAR responses, publication of court decisions and administrative decisions, sharing contracts with third parties, handing documents to external AI tools or service providers, building test/training datasets, and records retention/reuse for research.
• AI value: consistent recall across thousands of pages, context-aware detection (an indirect identifier such as "the only female CFO of the Ansbach plant" is not caught by regex), reduced reviewer time, and an auditable log of what was removed and why.
• Key legal distinction: anonymised data is outside GDPR only if individuals are no longer identifiable by means reasonably likely to be used; pseudonymised data stays personal data and needs the additional information (key) kept separately. The EDPB Guidelines 01/2025 formalise this with the concept of a "pseudonymisation domain" in which attribution must be prevented [2].
• No reliable public figures on time saved are cited here; measure your own baseline (see KPIs).

Variants

• Public sector / courts: publication of decisions, FOI/IFG and transparency-portal releases, records for archives; consistent pseudonyms ("A.", "B.") so the text stays readable.
• Legal & eDiscovery: privilege and PII redaction before production (see DOC-021).
• Healthcare / research: de-identification of discharge letters and reports (see DOC-007); date shifting and generalisation instead of deletion.
• HR: blind CV and personnel-file redaction (the later decision step is a separate, possibly high-risk use case).
• Enterprise AI hygiene: masking PII in prompts, RAG corpora and logs before LLM calls (see SEC-017, DOC-014).
• Customer-service transcripts: redacting call and chat transcripts (text PII on ASR output).
• Private use: redacting personal documents before posting or sending them to a landlord, forum or authority; local tools preferred (see PERS-016).
• Images/video in documents: faces, signatures, licence plates inside scans (object detection; for video see VIS-030).

How it works

1. intake (PDF / scan / Office / email)
2. text + layout extraction (born-digital text layer or OCR; keep bounding boxes)
3. detection (rules/checksums for IBAN, tax-ID, phone; NER model; LLM/VLM for contextual and indirect identifiers; signature/face/stamp detector on page images)
4. policy engine (per purpose: remove / mask / generalise / replace with consistent pseudonym)
5. human review (highlighted candidates, accept/reject, add missed items)
6. true redaction (delete underlying text, images and metadata; rasterise or rebuild the page)
7. verification (re-extract text from output, re-run detectors, check metadata/attachments/OCR layer)
8. release + audit log; key vault for pseudonym mapping (if reversible)

• Detection ensemble: recall matters more than precision. Combine deterministic recognisers (Luhn, IBAN mod-97, Steuer-ID checksum), a NER model, and optionally an LLM pass for context ("my neighbour who works at the bakery on Hauptstrasse").
• Redaction is a document-engineering problem, not only an AI problem: a black rectangle over live text is not redaction. The output must lack the original text layer, hidden layers, comments, version history, embedded files and XMP/metadata.
• Human-in-the-loop: sampling or full review depending on risk; mandatory review for publication and for special-category data (Art. 9 GDPR).
• LLM caution: do not send the unredacted document to an external LLM just to find the PII; use local or EU-hosted detection for the first stage.

Data preparation

The documents to be anonymised are contracts, case files, HR and patient records, correspondence, tickets and exports. They typically live in the DMS, on file shares, in mailboxes and as paper scans. Typical gaps are image-only scans without text layer, mixed languages, handwriting, tables and forms with personal data in unexpected places, no list of what counts as confidential, and unclear access rights to the source folders.

Preparation steps: - Define what must be removed per purpose (names, IBAN, health data, trade secrets) as a rule set, see PREP-024 and PREP-019. - Run OCR on scans first (Tesseract, PaddleOCR, Docling), see PREP-006. - Pick 30-50 representative documents and have staff mark the correct redactions; this is the test set for recall, see PREP-018. - Add company-specific terms (customer names, product codes) as deny and allow lists, see PREP-015. - Check the legal basis for the redaction purpose, see PREP-025.

Rough share of effort: about 15 percent; the rule set and the test set matter more than technology. Tools: Presidio, GLiNER, spaCy. See also GUIDE-003 and DATA-023.

Implementation paths

Which path for whom

• SMB: Path C, because the content is by definition sensitive and Presidio, GLiNER and Docling run on modest hardware, and one redaction service can serve many use cases.
• Enterprise: Path C, because redaction sits in front of other AI pipelines and a shared internal service avoids per-page fees and sends no raw personal data out.
• Developer: Path C, because open-source detectors can be embedded and tuned directly.
• Public sector: Path C, because authorities must not send personal data to third-party clouds and need verifiable irreversible redaction.
• Private: Path A, because desktop PDF tools with redaction cover occasional use without any setup.

Path A — Buy (off-the-shelf)

• PDF/DMS tools: Adobe Acrobat (Redact tool, search-and-remove, "sanitize document") for small volumes; DMS/ECM vendors and eDiscovery platforms (e.g. Relativity Redact) with AI-assisted redaction. Many German DMS products offer redaction modules; verify EU hosting and whether irreversible removal is guaranteed.
• Privacy/DLP suites: Microsoft Purview, Google Sensitive Data Protection (classification and de-identification), plus specialised PII-redaction vendors (e.g. Private AI, Redactable). Check data residency, DPA and whether documents are used for vendor model training.
• Choose A when: low-to-medium volume, standard document types, no need to integrate into own pipeline. Always test with your own hard cases (scans, tables, handwriting, German addresses).

Path B — Build on cloud APIs

• Managed PII services: Azure AI Language PII detection (text, conversation, and native-document redaction for PDF/DOCX/TXT with JSON metadata; EU regions) [3]; Amazon Comprehend PII detection and redaction (English/Spanish only, so weak for German) [4]; Google Sensitive Data Protection (de-identification transforms). Check language coverage per service; German-language quality must be tested.
• OCR with geometry: Mistral OCR (EU provider; per-block bounding boxes and confidence scores in current OCR models) to locate spans on the page [5]; Azure Document Intelligence, Google Document AI.
• Contextual pass: an EU-hosted LLM (Mistral, Azure OpenAI/EU data zone, Claude or Gemini via EU regions) for indirect identifiers, with structured output (span, category, rationale).
• Orchestration: n8n/Power Automate/custom service; PDF redaction library (PyMuPDF apply_redactions, pikepdf, Apache PDFBox) to truly delete content.
• Choose B when: you need pipeline integration and good German quality and can accept a processor under a DPA.

Path C — Self-host (open source)

• Detection: Microsoft Presidio (analyzer, anonymizer, image redactor, structured-data module; spaCy/Stanza/ Transformers engines; Docker/Kubernetes) [1]; GLiNER zero-shot NER (Apache-2.0, CPU-capable) including PII variants such as gliner_multi_pii-v1 [6]; spaCy German models; fine-tuned BERT-class NER (text-classifier).
• Parsing/OCR: Docling (MIT, IBM Research, LF AI & Data) [7], PaddleOCR, Tesseract, olmOCR for scans.
• Contextual pass: open-weight LLM/VLM (Qwen3 family, Mistral models, Gemma) via vLLM or Ollama.
• Redaction engine: PyMuPDF/pikepdf/PDFBox, then verification by re-extraction.
• Hardware: HW1 for rule + GLiNER pipelines; HW2 (one 24–32 GB GPU) for OCR + a mid-size LLM; documents never leave the premises - the strongest answer for courts, hospitals and authorities.
• Ops effort: moderate; the main work is policy tuning, evaluation sets and the verification step.

Path D — Custom training

Worth it for domain-specific identifiers (clinical notes, court files, German dialect/abbreviation-heavy text) or very high volume. Fine-tune a NER/encoder model on a few thousand manually annotated pages; reviewer corrections from production are the natural training data. Synthetic PII-bearing text (see DATA-005) can bootstrap, but a real held-out test set is essential. Public annotation sets such as the ai4privacy collections or the Text Anonymization Benchmark (court decisions) help evaluation but are not German-administration-specific.

State of the art (2026)

• Layered detection (rules + NER + LLM) is the norm; zero-shot span models like GLiNER and small LLMs now match or exceed classic regex/NER on unseen entity types, at CPU or single-GPU cost [6].
• Layout-aware tooling: modern parsers return bounding boxes per block/word, so detected spans map directly to page coordinates for redaction [5][7].
• Indirect identifiers and re-identification risk remain the hard part: direct identifiers (names, IDs) are reasonably solved; quasi-identifiers (rare job, event, place + date combinations) need context, risk assessment and often generalisation. Research benchmarks for text anonymisation distinguish direct from quasi-identifiers for this reason.
• Failure modes: missed names in low-quality scans or handwriting; non-Latin or hyphenated names; false positives on company names and public figures; inconsistent pseudonyms across documents; LLMs paraphrasing instead of copying spans; redaction that leaves text in the layer, OCR layer, metadata or thumbnails.
• Never claim "anonymised" without a re-identification assessment; automated redaction is a risk-reduction tool, and a documented review step defines whether the result is anonymous or merely pseudonymised.

Shared resources & combinations

• Same stack (ocr-docparse + text-classifier + llm-general): DOC-014 (parsing for RAG: redact before indexing), DOC-003 forms, DOC-007 medical records, DOC-009 contract analysis; one OCR/VLM server (HW2/HW3) serves all.
• Natural combinations: DOC-021 (eDiscovery and DSAR review: redaction as final production step), SEC-017 (sensitive-data discovery, DLP: find where PII lives, then redact), SEC-018 (DSAR and records of processing), PERS-016 (private data-rights requests), VIS-030 (face and plate blurring for images), DATA-005 (synthetic data as alternative to redacted data).
• Shared building blocks: guardrails PII filter for LLM prompts, a key vault for pseudonym mappings, a review UI with accept/reject and audit log, a labelled evaluation set per document type.

Effort, cost & KPIs

• Path A: days to weeks; licence per user or per page.
• Path B: 1–3 months incl. review UI, verification and integration; API cost per page usually small versus reviewer time.
• Path C: 2–4 months; one GPU server or CPU cluster, plus maintenance of recognisers and policy.
• Path D: +2–3 months for annotation and training.
• KPIs: recall of direct identifiers on a held-out set (target near 100% before human review), precision, reviewer minutes per 100 pages, share of documents needing manual fix, leak incidents found in post-release audits, residual-text rate in verification (should be 0), turnaround time of DSAR/FOI requests.

People & change

Involve management and the data protection officer first: anonymisation is a GDPR measure, but the tool itself processes personal data, so legal basis, AVV for any external processor and possibly a DPIA must be clarified. IT and IT security set up the service and the audit log; department leads and end users (legal, HR, records, communications) define what must be removed and review the results.

Works council: usually no, because the tool works on documents, not on employee behaviour. It depends if the audit log is evaluated per person or HR files are processed in bulk; then §87(1) no. 6 BetrVG applies as the BAG treats objective suitability for monitoring as enough, and §90(1) no. 3 requires information in good time. In authorities the Personalrat is involved.

Support AI literacy (Art. 4 AI Act): staff must know that detection is probabilistic, that recall is never 100 percent and that every document intended for publication needs a human check, ideally a four-eyes review for high-risk releases. Pilot with one team and one document type, measure missed items, and publish the rule set. Typical concern: false confidence. Address it by showing recall numbers, keeping the reviewer in control, and never publishing automatically. Guides: GUIDE-005, GUIDE-006, GUIDE-008.

Risks, regulation & compliance

• EU AI Act: typically minimal risk (a privacy-protection tool, not in Annex III; AI literacy duty Art. 4 applies); not an Art. 5 practice [8]. Becomes part of a high-risk context only if embedded in, e.g., court or HR decision systems; the redaction step itself does not make them so.
• GDPR: processing for redaction is itself processing of personal data (Art. 6, and Art. 9 for special categories); data minimisation (Art. 5(1)(c)) and privacy by design (Art. 25) support the use case. Anonymisation requires that identification is not reasonably likely; pseudonymisation does not remove GDPR scope [2]. CJEU case law (EDPS v SRB, C-413/23 P, 2025) stresses that identifiability can depend on the recipient's means - check the judgment text before relying on it.
• Sector rules: IFG/FOI laws and court-publication rules; medical confidentiality (§203 StGB) and patient-data rules; trade secrets (GeschGehG); GoBD and retention duties conflict with deletion - keep the original in a restricted archive and release only the redacted copy.
• Risks and mitigations: missed identifiers (recall target, double review for publication); fake redaction (rebuild the page, verify output text, strip metadata); LLM data leakage (local/EU detection, DPA, no training on inputs); re-identification by linkage (generalise, risk assessment, restricted release); pseudonym key compromise (separate vault, access control, per-domain keys).

Getting started

1. Define the purposes and recipients (publication, third party, internal AI use) and write an entity policy per purpose.
2. Assemble 100–200 representative pages incl. scans and annotate the ground truth; compute recall/precision for a baseline (Presidio or Azure PII) and for a GLiNER/LLM variant.
3. Build the redaction output properly (delete the text layer, strip metadata) and add an automated verification step that re-extracts text from the result.
4. Add a review UI with confidence thresholds, sampling and an audit log; run in shadow mode next to the manual process.
5. Document the anonymisation vs pseudonymisation decision, the DPIA (if needed) and the key management.

Sources

1. Microsoft Presidio documentation (analyzer, anonymizer, image redactor, deployment) - https://presidio.dataprivacystack.org/ (accessed 2026-10)
2. EDPB, Guidelines 01/2025 on Pseudonymisation (pseudonymisation domain; public consultation Jan-Mar 2025) - https://www.edpb.europa.eu/system/files/2025-01/edpb_guidelines_202501_pseudonymisation_en.pdf and https://www.edpb.europa.eu/our-work-tools/documents/public-consultations/2025/guidelines-012025-pseudonymisation_en (accessed 2026-10)
3. Microsoft Learn, Azure Language PII detection overview (text, conversation, native-document PII) - https://learn.microsoft.com/en-us/azure/ai-services/language-service/personally-identifiable-information/overview (accessed 2026-10)
4. AWS, Amazon Comprehend: Detecting PII entities (English/Spanish) - https://docs.aws.amazon.com/comprehend/latest/dg/how-pii.html (accessed 2026-10)
5. Mistral AI docs, Document AI / OCR (mistral-ocr-latest, block bounding boxes, confidence scores) - https://docs.mistral.ai/capabilities/document_ai/basic_ocr/ (accessed 2026-10)
6. GLiNER (zero-shot NER, Apache-2.0, PII variants) - https://github.com/urchade/GLiNER (accessed 2026-10)
7. Docling (IBM Research / LF AI & Data, MIT) - https://github.com/docling-project/docling (accessed 2026-10)
8. EU AI Act Article 5, prohibited practices - https://artificialintelligenceact.eu/article/5/ (accessed 2026-10)