Social benefit fraud & error risk detection (AI Act high-risk safeguards)

Social-security, unemployment, housing and child-benefit agencies lose money to both fraud and honest error, and have too few caseworkers to check every claim. Risk models can rank which claims deserve a human look. But this is the use case where algorithmic selection has caused the most documented harm in Europe (Dutch childcare-benefit scandal, SyRI, Danish and Swedish systems). Done properly it is a high-risk AI system with bias audits, a fundamental-rights impact assessment, explainability and meaningful human review. Done carelessly it can slide into prohibited social scoring.

Problem & value

• Today: agencies rely on rules ("red flags"), tip-offs, data matching with employer, tax and residence registers, and random checks. Selection of cases for in-depth investigation is increasingly done by statistical models.
• AI value: better hit-rates per investigator hour, earlier detection of errors (wrong income, unreported change of household) that can be corrected cheaply and non-punitively, and consistent, documented selection criteria instead of ad-hoc suspicion.
• Evidence of harm, not just benefit: a self-learning risk model in the Dutch childcare-benefit system used nationality as a factor; tens of thousands of mostly low-income parents were wrongly accused and the government resigned in January 2021 [1][2]. The Hague district court declared the SyRI profiling legislation unlawful in 2020 (Art. 8 ECHR; reported in [2]). Amnesty (2024) criticised the Danish Udbetaling Danmark / ATP systems (up to 60 models; "Really Single" and "Model Abroad" use living-arrangement and foreign-ties proxies) [3]. Lighthouse Reports and Svenska Dagbladet (2024) reported that the Swedish Försäkringskassan model over-selected women, migrants and low earners; the agency disputes the findings [4].
• Honest framing: published figures on "fraud prevented" by such systems are mostly agency self-reports and are contested; treat any ROI claim without a randomised control group as unproven.

Variants

• Claim-level triage at intake (new application or change notification): flag inconsistent or incomplete data for a clarifying question, not an accusation.
• Post-payment review selection: rank paid cases for a periodic check; always combined with a random sample to measure the true error/fraud rate and detect model drift.
• Document and evidence checks (payslips, tenancy, bank statements): see ANOM-017 and ANOM-014; this entry covers the case-level decision layer.
• Network/organised-fraud analysis (shared addresses, bank accounts, employers, brokers): graph analytics on entities and schemes rather than on individual citizens' lifestyles; closest to ANOM-020.
• Error-prevention nudges: prefilled forms, reminders and plausibility checks that reduce unintended overpayments; usually lower risk and more effective than fraud scoring.
• Not in scope: tax and VAT fraud selection (ANOM-018); health-provider billing fraud (ANOM-007); insurance claims (ANOM-003); generic audit case selection framework (PRED-028).

How it works

1. claim / change notification + lawful register data
2. data minimisation & feature governance (no protected attributes, no proxies such as nationality, postcode, family status)
3. rules + supervised model (gradient boosting) and/or anomaly detection
4. score + reason codes + confidence
5. treated as a recommendation only
6. human caseworker review (can see reasons, can override, may ask the claimant first)
7. outcome label (correct / error / fraud / unclear) fed back
8. monitoring: per-group false-positive rates, drift, random control sample, complaints

• Models: tabular ML (gradient boosting such as XGBoost/LightGBM, explainable boosting machines) with SHAP-style reason codes; unsupervised anomaly detection for new patterns; graph analytics for shared-attribute clusters. LLMs are only suitable for document intake, summarising a case file for the reviewer, and drafting neutral clarification letters; they must not produce scores or accusations.
• Labels are the weak point: past investigations reflect where the agency already looked (selection bias), so supervised models learn the agency's own historic bias. Mitigation: a random audit sample as unbiased ground truth, and separating "error" from "intentional fraud" labels [4].
• Human-in-the-loop points: (1) feature approval by a governance board including legal and data-protection staff, (2) caseworker review before any investigation or benefit suspension, (3) claimant hearing and appeal, (4) periodic independent audit.

Data preparation

Data comes from benefit application systems, payment and case history, scanned evidence documents and register data accessed under an explicit legal basis. In an agency this lives in specialist procedures, file-based case records and paper dossiers. Typical gaps: scanned evidence without OCR, inconsistent case coding, no clean distinction between confirmed fraud, honest error and no finding, missing random-sample outcomes (only suspicious cases were ever checked, which biases labels), and strict access rights and purpose limits. Steps: (1) data inventory and use-case data mapping, see PREP-001 and PREP-002; (2) legal-basis and purpose check before any reuse, see PREP-025; (3) OCR of paper evidence, see PREP-006 and DOC-014; (4) define the category scheme (fraud, error, no finding) before labelling, see PREP-019, and harvest labels from past reviews, see PREP-020; (5) build a random control sample and test sets with caseworkers, see PREP-018; (6) pseudonymise development data, see PREP-022; (7) remove protected and proxy attributes where the law requires and document it. Tools: ETL pipelines, OCR (Tesseract, docTR), a feature store. Expect 50 to 60 percent of effort in data preparation. See GUIDE-003 for the readiness check.

Implementation paths

Which path for whom

• public (C): Benefit data is highly sensitive and the system is high-risk under the AI Act, so full control over models, logs and bias audits on sovereign infrastructure is needed; open-source ML on HW2 is sufficient.
• enterprise (C): Insurers, social-insurance carriers and large payers with comparable claims data need auditable, self-hosted models for the same reasons.
• developer (C): Vendors delivering to agencies must offer transparent, self-hostable scoring with explainability and audit tooling built in.

Path A — Buy (off-the-shelf)

Public-sector fraud and case-management suites exist (for example SAS fraud and compliance products, Pega case management, IBM and Quantexa-style entity-resolution platforms; German and EU integrators build on these or on custom code). Checks to demand: EU hosting and data sovereignty, white-box reason codes, per-group fairness reports, support for AI Act documentation (Arts. 9-15), no use of nationality or similar proxy features by default, and the right to inspect and test the model. Right choice only for agencies with a procurement route and an IT department able to govern a vendor; opaque "black-box" fraud scores are the pattern that failed in the cases above. Not a private-citizen or SMB product.

Path B — Build on cloud APIs

Suitable for the supporting tasks only: document parsing (Azure AI Document Intelligence, Mistral OCR, Google Document AI with EU regions), summaries of case files and neutral drafting with a frontier or EU-provider LLM (Mistral, Aleph Alpha-style providers, Claude/Gemini/GPT via EU regions) behind a guardrail layer. Scoring on citizen register data in a public cloud is possible only if the legal basis, a processing agreement and sovereignty requirements (BSI C5, SecNumCloud-style, national cloud policy) are met; many agencies will exclude it.

Path C — Self-host (open source)

Recommended default for authorities: tabular ML stays CPU-sized. - Modelling: scikit-learn, XGBoost/LightGBM, InterpretML (explainable boosting), SHAP; graph analysis with Neo4j Community / NetworkX / PyG. - Fairness and audit: Fairlearn (assessment and mitigation toolkit) [5], AIF360, Aequitas; Evidently or similar for drift. - Documents/LLM (optional): Docling or PaddleOCR with an open VLM/LLM (Qwen, Mistral, Llama families) via vLLM, in the agency's own data centre. - Governance: MLflow for model registry and lineage; immutable audit log of every score and every human decision. - Hardware: HW1 (CPU servers) for scoring and training; HW2/HW3 only if the optional document/LLM layer is self-hosted.

Path D — Custom training

This is where most real systems live: a model trained on the agency's own labelled outcomes. Needs thousands of verified outcomes plus an unbiased random-audit sample, a documented feature catalogue with legal justification per feature, and a train/validation scheme that avoids leakage from the investigation process. Budget more time for validation, fairness testing and legal sign-off than for modelling.

State of the art (2026)

• Technique: no frontier breakthrough; gradient boosting with explainability remains the practical standard on tabular claim data. The advances are governance-side: standard fairness toolkits, algorithm registers, impact-assessment templates, and (in the EU) a binding legal framework.
• Regulatory shift: the EU AI Act lists AI systems used by or for public authorities to evaluate eligibility for essential public assistance benefits and services, and to grant, reduce, revoke or reclaim them, as high-risk (Annex III, point 5(a)) [6]. The fraud-detection exception in point 5(b) applies to credit scoring for financial fraud, not to benefit eligibility. The Art. 6(3) "narrow task" derogation does not apply to systems that profile natural persons [7]. The Commission's Art. 5 guidelines (Feb 2025) address social scoring [8].
• Limits and failure modes: proxy discrimination (postcode, nationality, household composition), feedback loops (flagged groups get checked more, so more "fraud" is found there), conflating error with fraud, automation bias in caseworkers ("the system said so"), unexplained scores that make legal defence impossible, and function creep when data collected for one benefit is reused for scoring.
• Date caution: high-risk obligations were originally scheduled from 2 August 2026; the Digital Omnibus on AI (Reg. (EU) 2026/1744, in force since 27 Jul 2026) moved Annex III obligations to 2 December 2027.

Shared resources & combinations

• Same stack (tabular-ml + anomaly-detection + graph-ml): ANOM-003 insurance claims fraud, ANOM-007 healthcare billing fraud, ANOM-018 tax and VAT fraud, ANOM-020 procurement collusion and EU-funds fraud, PRED-028 risk-based case selection (shared selection framework with capacity budget and random control group).
• Evidence layer: ANOM-017 document tampering and ANOM-014 identity document checks; DOC-001-style document parsing for incoming evidence.
• Governance layer shared with: SEC-022 (AI inventory, risk classification, AI Act documentation) and SEC-018 (DPIA, records of processing, DSAR/access requests).
• Hardware: one CPU/HW1 analytics cluster plus the agency data warehouse serves all of these; add an HW2/HW3 GPU node only when LLM/VLM document handling is included.

Effort, cost & KPIs

• Path A: months for procurement plus 6-12 months to integrate; licence cost depends on volume (no reliable public figures).
• Path C/D: typically a team of data scientists, a legal/data-protection lead and domain investigators for 9-18 months to a governed production system; compute cost is small compared with staff and audit effort.
• KPIs: precision of flagged cases against a random control sample; share of findings that are error vs. fraud; per-group false-positive and flag-rate parity; time to decision; share of reviewer overrides; complaint and appeal outcomes; amounts recovered net of wrongful-suspension costs; model drift.

People & change

Involve management, the department lead and caseworkers (the end users) early, plus IT, the DPO and legal. The DPO and DPIA come first: special categories and vulnerable people are affected, so a DPIA (Art. 35 GDPR) and, for public bodies, a fundamental-rights impact assessment are needed because the system is high-risk (Annex III duties apply from 2 Dec 2027 under Regulation (EU) 2026/1744). Works council: yes. Caseworkers' handling of scored cases is logged, so the system is objectively suited to monitor behaviour or performance and §87(1) no. 6 BetrVG applies (staff council under BPersVG or state law in agencies); also §90(1) no. 3 for AI-planned work processes and §80(3) for an expert. Agree that scores are never used for individual performance review. AI literacy: support caseworkers with training on what a score means, automation bias and how to document an override (Art. 4 AI Act, GUIDE-006). Pilot in one office with a mixed sample including random control cases. Citizens' main concern is unfair suspicion: communicate openly what is scored, give affected people notice and appeal rights, test bias per group, and make sure no benefit is ever refused or reduced by the score alone. See GUIDE-004, GUIDE-005 and GUIDE-014.

Risks, regulation & compliance

• EU AI Act: high-risk (Annex III 5(a)) when the system evaluates eligibility or supports granting, reducing, revoking or reclaiming benefits. Duties include risk management, data governance, logging, transparency, human oversight, accuracy and robustness (Arts. 9-15) and, for public bodies, a fundamental-rights impact assessment before deployment (Art. 27) covering affected groups, specific risks, human oversight and complaint mechanisms [9]. The ban on social scoring (Art. 5(1)(c)) applies where scores from unrelated social behaviour or personal characteristics lead to unjustified or disproportionate detrimental treatment [10]; a design that builds lifestyle or "atypical living arrangement" profiles from merged registers is exposed to this prohibition, which is why Amnesty argues the Danish system should be banned [3]. Art. 5(1)(d) additionally bars criminal-risk prediction based solely on profiling. Where a deployment crosses these lines it cannot lawfully be put on the EU market; only the narrower, fact-based and human-supported variants described above are implementable.
• GDPR: special-category data (health, ethnicity, etc.) and proxies; need a specific legal basis (usually national social-security law), strict purpose limitation and data minimisation, DPIA (Art. 35), and Art. 22 limits on solely automated decisions. The CJEU's SCHUFA judgment (C-634/21) treated an upstream score that decisively shapes a decision as itself an automated decision, which applies directly to "recommendation only" scoring that caseworkers rubber-stamp.
• Fundamental rights: non-discrimination (Charter Art. 21), privacy (ECHR Art. 8, as applied in the SyRI judgment), good administration and effective remedy. National law adds duties (German SGB X on automated decisions, administrative-procedure rules, algorithm registers in the Netherlands).
• Mitigations: exclude protected and proxy features, publish the model's purpose and logic where possible, give claimants notice and a hearing, keep a random control sample, run external bias audits, and set a sunset/re-approval date for every model.

Getting started

1. Establish the legal basis and mandate in writing; map the data (SEC-018) and classify the system under the AI Act (SEC-022) before any modelling.
2. Start with errors and data quality (plausibility checks, prefilled forms) rather than fraud scoring; measure the true error rate with a random sample.
3. Define a feature whitelist with legal justification; ban nationality, ethnicity, religion, postcode and household-structure proxies and test for residual proxies with Fairlearn.
4. Build an explainable model (boosting + reason codes), run a pilot with caseworkers who may override, and keep a control group.
5. Complete the DPIA and FRIA, set up an audit log, complaint channel and independent review before scaling.

Sources

1. Amnesty International — Xenophobic machines: Dutch child benefit scandal — https://www.amnesty.org/en/latest/news/2021/10/xenophobic-machines-dutch-child-benefit-scandal/
2. Lighthouse Reports — The Algorithm Addiction (SyRI, childcare benefits, Overvecht profiling) — https://www.lighthousereports.com/investigation/the-algorithm-addiction/
3. Amnesty International — Denmark: AI-powered welfare system fuels mass surveillance (12 Nov 2024) — https://www.amnesty.org/en/latest/news/2024/11/denmark-ai-powered-welfare-system-fuels-mass-surveillance-and-risks-discriminating-against-marginalized-groups-report/
4. Lighthouse Reports / Svenska Dagbladet — Sweden's Suspicion Machine (27 Nov 2024) — https://www.lighthousereports.com/investigation/swedens-suspicion-machine/
5. Fairlearn project — https://fairlearn.org/
6. AI Act Annex III (point 5) — https://artificialintelligenceact.eu/annex/3/
7. AI Act Article 6 (classification, derogation, profiling) — https://artificialintelligenceact.eu/article/6/
8. European Commission — Guidelines on prohibited AI practices (4 Feb 2025) — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
9. AI Act Article 27 (fundamental rights impact assessment) — https://artificialintelligenceact.eu/article/27/
10. AI Act Article 5 (prohibited practices) — https://artificialintelligenceact.eu/article/5/
11. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj