Official letters are written in administrative language, carry hard deadlines and often decide money, residence or benefits. A vision-language model can read a photographed letter, explain it in plain language (or the reader's mother tongue), extract the deadline and the legal remedy, and draft a reply for the person to check and send. It lowers the barrier for people who would otherwise miss a deadline, but it does not replace legal advice in contested cases.
Problem & value
- Today: people receive tax assessments, benefit decisions, fines, health-insurance and pension letters, landlord and utility letters. Wording is dense, the legal-remedy notice (Rechtsbehelfsbelehrung) is easy to overlook, and deadlines are short (in Germany typically one month for an objection against a notice, counted from its bekanntgabe). Help comes from relatives, social counselling, consumer centres, tenant or tax associations, or paid lawyers, often with waiting lists.
- AI value: immediate explanation at any hour, extraction of deadline, amount, reference number and sender, a checklist of what to prepare, and a draft in formal register. Particularly valuable for non-native speakers, older people, people with reading difficulties and anyone overwhelmed by paperwork.
- Evidence: no robust published figure on error reduction exists for this specific use; treat claims of time saved as unverified and measure on your own letters. Public bodies and EU institutions explicitly promote plain, understandable administrative language, which is the gap this closes [1][2].
Variants
- Consumer chat app: photo in, explanation and draft out (general assistants such as Claude, ChatGPT, Gemini, Le Chat).
- Dedicated bureaucracy apps: letter scanning with deadline tracking and templates (several German and Austrian start-ups; check current offers and their data hosting).
- Counselling-supported: social workers, migrant counselling or consumer centres use the tool to prepare cases faster (see PERS-015 for consumer-rights disputes).
- Multilingual: explanation in the user's language plus a reply in the official language (LANG-009).
- Easy-read: simplification for people with learning disabilities (LANG-010).
- Adjacent: tax notice and return checks (PERS-013), health insurer letters and reports (PERS-014).
How it works
- photo/scan/PDF of letter
- OCR + layout parsing (or direct VLM read), image quality check
- classification (sender type, document type: notice, fine, request for documents, reminder)
- extraction (reference no., date of notice, amount, deadline, remedy, authority, contact)
- explanation in plain language + target language, "what do I have to do" list
- deadline computation (date of notice + period, weekend rules)
- calendar reminder
- optional retrieval over official guidance / statute text (RAG with citations)
- draft reply (objection, request for extension, document submission, payment plan)
- user review
- user signs and sends (post, portal, e-mail)
- Key AI components: document VLM/OCR, an LLM with structured output for fields and deadlines, optional retrieval over statutes and authority guidance so claims carry a citation.
- Human-in-the-loop: deadlines are shown as proposals with the source passage highlighted; the user verifies the date and always sends the reply themselves. For large amounts, residence status or benefit cuts, route to a counselling service or lawyer.
Data preparation
The data is the household's own post: photographed or scanned letters, PDFs from online portals, and e-mails from authorities, usually stored in a drawer, a phone gallery, a mailbox or a folder on the computer. Typical gaps: skewed or shadowed photos, multi-page letters photographed in pieces, missing earlier correspondence and reference numbers, notices without a date of receipt (which matters for objection deadlines), and mixed personal data of several family members.
Preparation steps: - Scan with a document-scanner app (flat, well lit, all pages, 200-300 dpi) or use the portal PDF; check that the text is readable (PREP-006, DOC-014). - Keep a simple folder per authority or topic and file letters by date, so earlier correspondence can be attached (PREP-009). - Note the date of receipt and keep the envelope or the portal timestamp for deadlines. - Remove or mask data that is not needed, such as bank numbers or unrelated names, before upload to a cloud service (PREP-022, PREP-024). - Optionally index the archive with Paperless-ngx so the assistant can refer to earlier letters (DOC-018).
Rough share of effort: low, about 10-20 percent of the total; in practice a good photo is the main prerequisite, and a one-off scan session clears the backlog.
Implementation paths
Which path for whom
- private: A, because general assistants with image input (Claude, ChatGPT, Gemini, Le Chat) explain a letter in minutes; for very sensitive letters use an EU provider or the local option in PERS-005.
- developer: B, because a small app with structured output (sender, deadline, amount) and EU-region APIs adds deadline tracking and reminders; C is an option for those who want letters to stay on their own hardware.
Path A — Buy (off-the-shelf)
- General assistants with image input: Claude, ChatGPT, Gemini, Microsoft Copilot, Mistral Le Chat (EU provider, France). Fastest start, no integration.
- Dedicated apps and services: letter-scanning or document-inbox apps with deadline tracking (mostly start-ups; verify EU hosting and whether inputs are used for training).
- Public and non-profit help: citizen portals and legal-aid services with no AI, still the right route for contested cases. Choose A for occasional private use.
Path B — Build on cloud APIs
- Models: frontier multimodal APIs (Claude, Gemini, GPT, Mistral Medium/Large) with EU data residency options; Mistral OCR and Azure AI Document Intelligence for scan reading.
- Orchestration: small web/mobile app, structured-output schema (sender, deadline, amount, remedy), calendar integration (iCal), RAG over statute texts and authority FAQs with citations.
- Privacy design: no storage by default, redaction of unnecessary identifiers before the call (DOC-018), zero-retention API terms, processing agreement if offered as a service. Fits counselling organisations that build a shared tool.
Path C — Self-host (open source)
- Models: open-weight VLMs such as Qwen2.5-VL / Qwen3-VL, Gemma 3, Mistral Small 3.x (vision), Llama 3.x; OCR via Docling, PaddleOCR or olmOCR; serve with Ollama, llama.cpp or vLLM.
- Stack: local web UI (Open WebUI), n8n for reminders. Letters never leave the home server (see PERS-005).
- Hardware: HW1 for small quantised VLMs on a laptop or mini-PC; HW2 (one 16-32 GB GPU) for good quality German and long letters. Open models are weaker at legal reasoning in minority languages; check outputs carefully.
Path D — Custom training
Rarely worthwhile. A fine-tune on anonymised letter and reply pairs could help a counselling network with a recurring case type, but retrieval over official guidance and good prompts reach most of the benefit.
State of the art (2026)
- Reading: current VLMs read photographed letters, stamps and tables reliably; low-light photos, folded paper, handwriting and forms with check boxes are the weak spots.
- Explaining: strong at paraphrase and translation. Plain-language output is a solved task; correctness of legal consequences is not.
- Deadlines: the main failure mode. Models may miscompute the period, ignore the difference between date on the letter and date of deemed receipt (in Germany the fictional-receipt rule for postal notices, three days until end of 2024 and four days since 1 Jan 2025 under the Postrechtsmodernisierungsgesetz, e.g. § 41 Abs. 2 VwVfG; the date shifts to the next working day if it falls on a weekend or holiday), or miss that a deadline has already expired. Compute dates in code, not in the model.
- Hallucinated law: invented paragraph numbers, wrong remedy or confident advice about a different country's rules. Retrieval with citations and "country and authority" as explicit inputs reduce this.
- Agents: emerging assistants log into portals and submit forms; this raises liability and security issues and is not yet advisable for consequential filings.
- Legal-aid context: access to justice and plain-language initiatives treat such tools as triage support, not as a lawyer substitute.
Shared resources & combinations
- Same stack (ocr-docparse + vlm + llm-general): DOC-003 forms, DOC-018 redaction, PERS-013 tax, PERS-014 medical reports, PERS-001 bank letters and bills.
- Natural combinations: PERS-015 (legal self-help once an objection or dispute begins), PERS-016 (data-subject requests as the reply), PERS-009 (shared household deadlines), LANG-009/LANG-010 (translation, easy-read), PERS-005 (local assistant on own hardware).
Effort, cost & KPIs
- Path A: minutes; free tiers or a subscription of tens of euros per month for a premium assistant.
- Path B: 2-6 weeks for a prototype; per-letter cost is typically cents; the real cost is legal review of prompts and disclaimers.
- Path C: a weekend for a hobbyist setup; a few weeks to harden for a counselling service.
- KPIs: deadline extraction accuracy (target: near 100 percent, audited), share of letters correctly classified, user-reported understanding, replies sent in time, escalations to counsellors, and the rate of wrong legal statements found in sampling.
People & change
This is a private-person use case, so the stakeholders are the user and, where relevant, family members or a trusted helper who share the paperwork. Involve a counselling service, tax advisor or lawyer for contested or high-stakes letters; the assistant explains and drafts, it does not give legal advice, and the person always checks facts, deadlines and amounts before anything is sent. Never let the assistant submit objections or applications on its own.
Works council: no, since no employee system is involved. If a social-services provider, municipality or welfare association offers the service to clients, its data protection officer must be involved: letters contain special categories (health, benefits), so a legal-basis check, an AVV with every provider and usually a DPIA are required (GUIDE-005), and a works or staff council would be informed under §90(1) no. 3 and, if staff usage is logged, co-determine under §87(1) no. 6 BetrVG or the Personalrat rules (GUIDE-004).
AI literacy: a short introduction on what to upload, how to spot hallucinated deadlines and why an official remedy period always runs from the legal date of receipt, in support of Art. 4 AI Act (GUIDE-006). For an organisation, pilot with a few counsellors and a small client group before opening wider (GUIDE-009). Typical concerns are privacy of sensitive letters and wrong deadlines; address them with local or EU options (PERS-005), anonymised uploads, and a mandatory check of dates against the original.
Risks, regulation & compliance
- EU AI Act: typically minimal risk for a private assistant; Art. 4 AI literacy applies to organisational deployers. Chatbot disclosure (Art. 50) applies if offered publicly. Higher risk arises if a public authority uses AI to assess eligibility for benefits (Annex III point 5), which is the other side of the letter, not this use case [3].
- GDPR: letters contain special-category data (health, religion, benefits) and data of third parties. Prefer EU processing, no training on inputs, deletion, and local processing (Path C) for sensitive cases [4].
- Legal services law: in Germany the Legal Services Act (RDG) restricts individual legal advice by non-lawyers; a general explanation tool for the user's own case is generally tolerated, but commercial offers must watch the line. Not legal advice; check local rules.
- Risks and mitigations: missed deadlines (compute in code, always show source passage), wrong remedy (cite the remedy notice printed on the letter), phishing letters disguised as authority mail (add a scam check: sender, payment demand, urgency), over-trust (remind to consult counselling for large amounts). When in doubt, file a protective objection or ask the authority for an extension.
Getting started
- Pick 10 old, harmless letters; redact names and test explanation, deadline and draft quality.
- Always provide country, authority and the letter's date; ask the assistant to quote the passage for each deadline and remedy.
- Compute and diarise the deadline yourself (letter date plus period, plus receipt rules).
- Use the draft only as a base; check facts, add the reference number, send by a provable channel.
- For disputes or large sums, contact a consumer centre, social counselling or a lawyer.
Sources
- European Commission — Clear Writing (plain language in EU institutions) — https://commission.europa.eu/about/departments-and-executive-agencies/translation/clear-writing-europe_en
- Federal Government of Germany — Leichte Sprache and understandable administration (Bundesregierung) — https://www.bundesregierung.de/breg-de/leichte-sprache
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj
- Mistral AI — OCR documentation — https://docs.mistral.ai/capabilities/document_ai/
- Docling project — https://github.com/docling-project/docling
- Qwen-VL — https://github.com/QwenLM/Qwen2.5-VL
- German Fiscal Code (AO) §355, period for objection — https://www.gesetze-im-internet.de/ao_1977/__355.html
- German Legal Services Act (RDG) — https://www.gesetze-im-internet.de/rdg/