Personal tax return assistant

Filing an annual income tax return is tedious: finding the right form, remembering what is deductible and sorting a year of receipts. A tax assistant reads the documents, asks the questions a tax adviser would ask, fills the form and explains each entry in plain language. It also checks the assessment notice afterwards. The user stays responsible for the content and submits the return.

Problem & value

• Today: employees and pensioners in the EU either fill in forms or portals themselves, use a consumer tax app with a question-and-answer wizard, or pay a tax adviser or a Lohnsteuerhilfeverein (Germany). Common errors are forgotten deductions (work-related expenses, commuting, donations, craftsman and household services, insurance contributions), wrong allocation of amounts and unchecked notices.
• AI value: document capture replaces manual typing, a conversational interface replaces the form tree, and an LLM explains why an item belongs on which line. Reasoning models can check a notice against the submitted figures and flag deviations.
• Digitisation context: in Germany the tax authority already offers pre-filled data (Vorausgefüllte Steuererklärung / Belegabruf) in ELSTER, so much of the effort has shifted from entering data to checking and completing it [1]. Comparable pre-filled systems exist in e.g. Denmark, Sweden, Spain and Italy. No sourced figure on average time saved is claimed here; measure it in a pilot.

Variants

• Employee / pensioner (simple case): one salary statement, standard deductions, wizard-style.
• Families and homeowners: childcare, education, rental of a single flat, household services.
• Self-employed / freelancers: income-expense statement (EÜR), VAT return, receipt capture (see DOC-002).
• Expats and cross-border workers: double taxation treaties, foreign income — higher error risk, human adviser advised.
• Notice checker: upload the assessment notice, detect differences and draft an objection.
• Tax advisers (professional): the same pipeline as drafting tool — outside this private-use profile.

How it works

1. document intake (photo/PDF/email, authority pre-fill)
2. OCR + classification (salary statement, donation receipt, insurance, craftsman invoice ...)
3. extraction to schema (date, issuer, amount, category, tax ID if needed)
4. interview (LLM asks about circumstances: commute days, home office, children, moves)
5. rules + retrieval (tax-law knowledge base of the country/year, official form guides)
6. draft return (mapping to form fields/lines) + explanation per field + confidence
7. human review (user confirms every amount; unclear cases flagged)
8. submission by the user via official channel (ELSTER or national portal)
9. notice check (compare assessment with return)
10. objection draft if deviating

• Extraction: a vision-language model or document-AI service returns JSON per receipt.
• Tax logic: do not let the LLM calculate tax from memory. Use a deterministic calculation (tariff formulas, limits, flat rates per tax year) or the official calculator and let the LLM orchestrate, retrieve rules and explain. Rules must be versioned per tax year.
• Human-in-the-loop: the user checks and submits; any statement that sounds like binding advice is labelled as information. Complex cases are handed over to an adviser.

Data preparation

The data is the person's own paperwork: salary statement, insurance and donation receipts, childcare and craftsman invoices, last year's return and notice, and the pre-filled ELSTER data (Belegabruf). It usually lives in a paper folder, phone photos, e-mail attachments and bank apps. Typical gaps are blurry photos, receipts without clear dates or amounts, duplicates, documents named "scan_034.pdf" and missing proof for deductible items.

Preparation steps: - Create one folder per tax year and one sub-folder per category (income, insurance, donations, work, craftsmen). - Scan paper with a phone scanner app or a document scanner; let OCR run and keep the original. For a larger archive, Paperless-ngx on a home server tags and searches documents (see PREP-006 and DOC-014). - Rename files consistently (date, issuer, amount) or let the assistant propose names. - Retrieve pre-filled data from ELSTER before starting so the assistant works with authority data instead of guesses. - Remove documents of third parties (for example of other family members) that are not needed.

The effort is low: roughly 60 to 70 percent of the time goes into gathering and sorting documents, the AI part itself is quick. Doing it continuously through the year (photograph each receipt when it arrives) removes most of the effort next year. For readiness thinking see GUIDE-003.

Implementation paths

Which path for whom

• Private: A. Established German tax apps and the free ELSTER portal cover standard cases at low cost and with EU hosting; the person still checks and submits.
• Developer: B. Building a personal or community tool on EU-region APIs with OCR is simple and lets you control prompts and data flow, while the stack stays small.

Path A — Buy (recommended for most people)

• Consumer tax apps / software (Germany): Taxfix, WISO Steuer (Buhl), SteuerGo, Wundertax, Smartsteuer, Lohnsteuer-kompakt, Steuerbot-style assistants; many embed AI-guided questions and receipt scanning. Mostly German providers (EU) — check hosting and data processing terms.
• Official free channel: ELSTER (Germany's online tax portal, free) with pre-filled data; other countries have comparable portals (e.g. Skatteverket in Sweden, Agencia Tributaria in Spain).
• General assistants: frontier chat assistants can explain rules and check drafts but are not connected to the authority and are not tax-law certified.
• Choose A when: a standard case; compare 2–3 tools using last year's numbers and verify the refund estimate with the official portal.

Path B — Build on cloud APIs

• Models: EU-hosted or EU-provider models (Mistral, or frontier models in EU regions via Azure OpenAI, Amazon Bedrock, Google Vertex AI) with structured outputs for receipts.
• Document capture: Azure AI Document Intelligence (receipt/invoice models), Google Document AI (EU location), Mistral OCR.
• Integration: ELSTER has no general public API for private persons; Germany provides the ERiC library and certificates for software vendors, so a self-built tool prepares figures and the user enters or imports them in the official portal [2]. Check national rules before building.
• Choose B when: you build a product for a community, a Lohnsteuerhilfeverein or a fintech.

Path C — Self-host (open source)

• Models: Qwen3-class or Mistral open-weight models (e.g. Mistral Small), served via Ollama or vLLM; OCR with Docling, PaddleOCR or olmOCR; local vector search with SQLite/pgvector.
• Fits privacy-minded users: tax data stays on the home server (see PERS-005).
• Hardware: HW2 (one 24 GB GPU) or a laptop with 32 GB unified memory for ~8–14B quantised models.
• Effort: you maintain the rule base and calculation logic yourself — the biggest hidden cost.

Path D — Custom training

Rarely needed. Fine-tuning pays off only for a provider with thousands of anonymised returns and corrections, e.g. receipt categorisation specific to a country. Rules and tax logic are better handled by retrieval and deterministic code than by training.

State of the art (2026)

• Receipt and document understanding by VLMs works well on clean receipts; handwriting, thermal paper, foreign-language receipts and multi-page statements still cause errors.
• Reasoning models handle multi-step questions ("is my second flat a rental or a second home?") better but still cite outdated or foreign rules; retrieval over current official guides and year-specific rules is essential.
• Pre-filled returns shift the value to completeness checks: what is missing in the authority data and which deductions the user can still claim [1].
• Tool use: agents call a tax calculator tool instead of computing in text; this removes most arithmetic hallucinations.
• Limits and failure modes: hallucinated paragraphs and limits, confusing tax years, wrongly assuming that an item is deductible, over-optimistic refund estimates, silent mistakes in amounts read from receipts. Personal liability for false statements stays with the taxpayer (in Germany §§ 370, 378 AO apply to intent or gross negligence).

Shared resources & combinations

• Same stack (ocr-docparse + vlm + llm-general): DOC-002 receipts, DOC-017 bank statements and payslips, PERS-008 letters from authorities (the assessment notice is such a letter).
• Combines with: PERS-001 (categorised transactions give donations, insurance, subscriptions), PERS-005 (private local assistant), PERS-015 (objection and legal self-help).
• Hardware: the same HW2 VLM/LLM server serves other private document use cases.

Effort, cost & KPIs

• Path A: hours per return; consumer apps typically charge a per-return fee or subscription (check current pricing on vendor pages).
• Path B: several weeks for a prototype; costs mostly rule maintenance and legal review.
• Path C: a weekend for a hobby setup; permanent maintenance each tax year.
• KPIs: time to complete the return, share of receipts classified without correction, number of review warnings accepted, difference between estimated and assessed refund, number of notices with errors detected, number of objections successful.

People & change

This is a personal use case, so the relevant people are the user and, where applicable, the spouse or family whose data is in the return. Involve the partner early when a joint assessment is filed, and agree who submits. A tax adviser or Lohnsteuerhilfeverein is worth a short check for unusual cases (self-employment, foreign income, inheritance). No works council is involved because no employer system is used; if you use the tool on an employer device, check the company AI policy first (GUIDE-007).

Data protection: the data is personal and partly sensitive (religion, health costs). Choose providers with a data processing agreement and EU hosting, and do not paste data into free chatbots that train on inputs (GUIDE-005). A DPIA is not needed for private use.

AI literacy: know that the assistant can be wrong on rules and amounts. Treat each suggested deduction as a proposal, check the refund estimate against ELSTER and keep receipts for the authority's queries. The assistant must never submit on its own; the human reviews every figure and presses "send". Typical concerns are "is it safe with my data" (use a provider with clear terms, delete after filing) and "will it get me in trouble" (you remain responsible, so keep it conservative). See GUIDE-006 for basic AI skills.

Risks, regulation & compliance

• EU AI Act: typically minimal risk — private tax preparation is not an Annex III use case; AI literacy (Art. 4) applies to providers and deployers. A system used by tax authorities to assess natural persons or select audits is a different, regulated profile (see ANOM-018, PRED-028). Chatbot transparency (Art. 50) applies when users talk to an AI.
• GDPR: tax data reveals income, family, religion (church tax), health costs and donations — some Art. 9 special-category data. Need data processing agreement, EU hosting, deletion after filing, no training on user data without a legal basis. Tax secrecy rules bind authorities, not apps.
• Professional-law: in Germany, unauthorised tax advice for others is restricted by the Steuerberatungsgesetz; a self-service tool for one's own return or generic information is generally acceptable, but providers should verify their positioning with counsel.
• Mitigations: show sources and tax year for each rule, require explicit user confirmation per amount, keep an audit trail, disclaimers and advice for adviser help in complex cases, never submit on the user's behalf without confirmation.

Getting started

1. Gather last year's return, notice and this year's salary statement; enable the authority's pre-filled data retrieval.
2. Try the official portal first and note where you struggle (which questions, which deductions).
3. Test one consumer tool and one general assistant on last year's case; compare the refund with the notice.
4. Scan receipts continuously over the year (see DOC-002) into one folder with consistent names.
5. After filing, check the notice against the return within the objection period (one month in Germany).

Sources

1. ELSTER — German tax portal, pre-filled return and data retrieval — https://www.elster.de
2. ELSTER for developers (ERiC library and interfaces) — https://www.elster.de/elsterweb/infoseite/entwickler
3. German Federal Ministry of Finance — taxes and tax returns — https://www.bundesfinanzministerium.de
4. Microsoft — Azure AI Document Intelligence receipt model — https://learn.microsoft.com/azure/ai-services/document-intelligence/prebuilt/receipt
5. Docling project — https://github.com/docling-project/docling
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj