Public-service client risk profiling (jobseeker integration, social-welfare triage)

Public employment services and welfare agencies have limited counselling capacity. Statistical profiling estimates who is likely to find work on their own, who risks long-term unemployment and where support need is highest, so that scarce programmes and caseworker time can be targeted. It is one of the most scrutinised AI uses in Europe: it is high-risk under the EU AI Act, has triggered data-protection and discrimination disputes, and is only defensible as transparent decision support with a human who actually decides.

Problem & value

• Today: caseworkers allocate counselling intensity, training and subsidies mostly by experience and rule-of-thumb; some agencies use statistical profiling tools. In child-welfare hotlines, screeners must decide quickly which referrals need investigation.
• AI value: consistent, data-based prioritisation; earlier intervention for clients at risk of long-term unemployment; more time for counselling instead of administration.
• Evidence is mixed and contested. Documented deployments include the Austrian AMS jobseeker classification (three groups by labour-market chances; criticised for negative weights on women, disabled people and people over 30, and for publishing only 2 of 96 models) [2], and the Allegheny County (USA) Family Screening Tool, used since 2016 to support call screeners with a risk score built from hundreds of data elements, "in conjunction with clinical judgement" [3]. Measured benefit figures differ per study and context; none are quoted here without a verified source.
• Cautionary cases: the Dutch SyRI system was struck down in 2020 under Article 8 ECHR [4]; the Dutch childcare-benefits scandal involved a model using dual nationality as a risk factor [5]; Amnesty alleges Denmark's welfare-fraud models may amount to social scoring [6].

Variants

• Jobseeker integration profiling: chance of re-employment within N months, used to choose service tier (e.g. intensive coaching vs. self-service). Often also uplift modelling: which programme helps whom — more defensible than pure "risk" labelling.
• Child and social-welfare referral triage: score to prioritise screening of referrals (Allegheny, Flemish VDAB-type tools in employment); highest stakes, strictest oversight.
• Caseworker assistant (LLM): summarising case files and suggesting next steps; no scoring of people — lower risk but still Annex III if it influences eligibility or benefits.
• Out of scope: fraud/inspection targeting (PRED-028), criminal recidivism (PRED-045), credit scoring (PRED-009). Social-benefit fraud risk scoring with broad data-matching is the pattern most exposed to the Art. 5 social-scoring ban (see below).

How it works

1. register data (employment, benefits, education, region) + labour-market context
2. data minimisation, legal-basis check, feature review (no proxies for protected traits)
3. model (gradient boosting / survival model / uplift model), calibrated probabilities
4. explanation (feature contributions, reason codes in plain language)
5. caseworker screen: score is advice; caseworker records agreement/override + reason
6. service allocation decided by human; client informed and can contest
7. monitoring: calibration, subgroup error rates, drift, override analysis, audit log

• Models: classical tabular ML is standard and preferable here — logistic regression or gradient-boosted trees with monotonic constraints, survival analysis for time-to-employment, and causal / uplift models for programme assignment. They are auditable and explainable (SHAP, reason codes).
• LLMs are only suitable for summarising notes and drafting explanations, not for producing scores.
• Human-in-the-loop: the human must have time, training and authority to deviate; overrides are logged. Automation bias is the central failure mode.

Data preparation

The core data are administrative register records: employment history, benefit episodes, qualifications, programme participation and outcomes, plus caseworker notes for the triage variant. They typically sit in specialised case-management systems, several legacy databases and, for older cases, paper files or scans. Typical gaps are inconsistent outcome definitions across years, missing or changed codes after legislation changes, duplicate client records, free-text notes without structure, and strict access rights per legal basis. Proxy variables for protected traits are a further trap.

Concrete steps: (1) inventory sources and owners and map them to the use case (PREP-001, PREP-002, PREP-004); (2) extract data via exports or APIs (PREP-005); (3) agree outcome and label definitions with policy owners and harvest labels from historical records (PREP-019, PREP-020); (4) pseudonymise training extracts and check purpose and legal basis (PREP-022, PREP-025); (5) build temporal validation sets with domain experts (PREP-018). Tools: SQL/ETL, pandas, dbt, Great Expectations for quality checks. Expect roughly 50 to 60 percent of total project effort to go into data work; see GUIDE-003.

Implementation paths

Which path for whom

• Public sector (agencies, ministries): Path C, because register data must not leave the agency and tabular models (XGBoost, survival analysis) run on open-source tools on sovereign infrastructure.
• Enterprise (large providers, integrators): Path C, because the model is core to the service, auditability and contractual audit rights matter, and no vendor black box is acceptable.
• Developer: Path C, because scikit-learn, Fairlearn and MLflow give full control over features, calibration and fairness checks.

Path A — Buy (off-the-shelf)

Few products fit: most agencies build with ministries, research institutes or integrators (for example statistical institutes, national PES in-house teams, public-sector IT providers). Generic case-management suites (e.g. from Oracle, SAP, Cúram/IBM, Adesso, Sopra Steria) can host scores but rarely ship validated models. Buying is sensible only if the vendor delivers full documentation, bias testing, EU hosting and contractual audit rights; vendor black boxes are a procurement risk under Art. 13 and 26 AI Act. I could not verify specific current product claims in this research round.

Path B — Build on cloud APIs

Managed ML platforms in EU regions (Azure ML, Google Vertex AI, AWS SageMaker, or EU clouds such as OVHcloud, STACKIT, IONOS) for training tabular models; fairness tooling such as Fairlearn and AIF360 (open source). Public bodies often must use sovereign or government clouds (e.g. German Verwaltungscloud, Austrian/Swiss government clouds). LLM APIs (EU-hosted, Mistral or others) only for note summarisation, with a data-processing agreement and no training on inputs.

Path C — Self-host (open source)

scikit-learn, XGBoost / LightGBM, lifelines or scikit-survival, EconML / CausalML for uplift, SHAP, Fairlearn, MLflow for registry and audit trail. Runs on CPU servers (HW1); no GPU needed. Self-hosting is the norm for register data that must not leave the agency. Ops effort: a small data science team plus a data-protection and legal function.

Path D — Custom training

Because the target population, legislation and labour market are national, models are almost always trained on the agency's own historical data (typically many years of register records with outcomes). Needs: outcome definitions agreed with policy owners, temporal validation, calibration, subgroup audits, a pilot with randomised or staggered rollout to measure real effect on integration.

State of the art (2026)

• Practice has moved from "risk labelling" to transparent decision support and uplift modelling (who benefits from which programme), driven by regulation and past controversies [2][4][5].
• Regulation is the defining development: AI systems used by public authorities to evaluate eligibility for essential public assistance benefits and services, and to grant, reduce, revoke or reclaim them, are listed in Annex III point 5(a) of the AI Act [1]. High-risk obligations for Annex III systems were scheduled to apply from August 2026; the Digital Omnibus on AI (reported as Regulation (EU) 2026/1744, in force July 2026) moved them to 2 December 2027 - verify against the Official Journal.
• Fairness reality: no single metric satisfies all fairness definitions; calibration and error balance across subgroups conflict. Proxies (postcode, citizenship, household structure) can encode protected traits, as Amnesty argues for Denmark's models [6].
• Limits: predicting individual life outcomes is inaccurate; scores reflect past labour-market discrimination; thresholds are often set by technical convenience rather than policy [2]; model drift after economic shocks.

Shared resources & combinations

• Same stack (tabular-ml, causal-inference): PRED-009 credit scoring, PRED-012 attrition, PRED-021 collections prioritisation, PRED-028 audit case selection — one MLOps and fairness-monitoring platform on HW1 CPU servers.
• Adjacent: REC-008 job recommendation (matching vacancies after profiling), PRED-045 for the criminal-justice analogue with equivalent oversight design.
• Building blocks: feature store for register data, explanation service, audit-log store, FRIA and documentation templates reusable across all public-sector high-risk systems.

Effort, cost & KPIs

• Path A: months, dominated by procurement and conformity review.
• Path B/C: 6–12 months including legal basis, data-protection impact assessment and FRIA.
• Path D: 12+ months incl. evaluation pilot; cost is mostly people (data science, legal, ethics, training).
• KPIs: calibration and AUC on a held-out later period, subgroup error-rate gaps, override rate and reasons, integration rate and duration vs. control, complaint and appeal rate, time per case, share of clients who receive an understandable explanation.

People & change

Involve management and the department lead from the start for mandate and policy goals; the data owners and IT in the discovery phase; the data protection officer and legal/compliance before any data is touched (legal basis, DPIA under Art. 35 GDPR is practically certain, AVV with every processor). The staff council (Personalrat under BPersVG or state law) must be involved early: the system evaluates people and logs caseworker agreement and override behaviour, so it is suited to monitor staff performance, and AI-supported work processes trigger information duties (analogous to §90(1) no. 3 BetrVG); an expert can be engaged (§80(3) BetrVG logic). Clients and their representatives need transparent information and a way to contest.

AI literacy: the AI Act (Art. 4) requires measures to support staff literacy; train caseworkers on score meaning, uncertainty and automation bias. Start with a pilot office or staggered rollout with a small champion group. Typical concerns: "the machine decides" (the caseworker decides, overrides are welcome and logged), discrimination (subgroup audits published), job fear (scores target attention, not staffing). Keep humans in control with mandatory review and authority to deviate. See GUIDE-004, GUIDE-005, GUIDE-008.

Risks, regulation & compliance

• EU AI Act: high-risk under Annex III 5(a) when used by or for public authorities to assess eligibility for or allocation of essential public benefits and services [1]. Obligations include risk management, data governance, logging, transparency, human oversight (Art. 14), and a fundamental rights impact assessment before deployment by public bodies (Art. 27); affected persons have a right to explanation of decisions (Art. 86). Whether a pure counselling-intensity tool is "essential public assistance" depends on the national service design — assess case by case.
• Prohibited variant: Art. 5(1)(c) bans scoring of people based on social behaviour or personality characteristics where it leads to detrimental treatment in unrelated contexts or unjustified or disproportionate treatment. Broad cross-agency data-matching that scores lifestyle or household patterns to select people for benefit-fraud investigation risks falling under it (the Commission has published guidelines on prohibited practices [7]); the SyRI judgement shows the ECHR limits [4]. Do not build such systems. Lawful variants: use only data relevant to the service purpose, directly tied to the decision context.
• GDPR: Art. 6(1)(e) with a specific national legal basis, Art. 22 limits on solely automated decisions (human decision with real discretion required), DPIA, purpose limitation, special-category data restrictions; Austria's data-protection authority reportedly found the AMS algorithm unlawful in 2020 [2], showing the need for an explicit statutory basis.
• Anti-discrimination law: direct use of sex, disability or age as negative weights is a legal risk [2].
• Mitigations: statutory basis, published model documentation, independent audits, client right to explanation and appeal, regular bias tests, caseworker training, sunset/review clause.

Getting started

1. Clarify the legal basis and the policy objective (support allocation, not sanction); involve the data-protection officer and an ethics/fairness board from day one.
2. Run the AI Act classification and a fundamental rights impact assessment.
3. Build a transparent baseline (logistic regression) on historic data with temporal validation and subgroup analysis before trying complex models.
4. Pilot as advisory display only, with logging of caseworker decisions and overrides.
5. Evaluate causal impact on integration outcomes with a staggered rollout, and publish results.

Sources

1. AI Act Annex III (artificialintelligenceact.eu) — https://artificialintelligenceact.eu/annex/3/
2. AlgorithmWatch — Austria's employment agency AMS rolls out discriminatory algorithm — https://algorithmwatch.org/en/austrias-employment-agency-ams-rolls-out-discriminatory-algorithm/
3. Allegheny County Analytics — Developing predictive risk models to support child maltreatment hotline screening decisions — https://analytics.alleghenycounty.us/index.php/2019/05/01/developing-predictive-risk-models-support-child-maltreatment-hotline-screening-decisions/
4. Wikipedia — SyRI (District Court of The Hague, 5 Feb 2020) — https://en.wikipedia.org/wiki/SyRI
5. Lighthouse Reports — The algorithm addiction — https://www.lighthousereports.com/investigation/the-algorithm-addiction/
6. Amnesty International — Denmark: AI-powered welfare system fuels mass surveillance (Nov 2024) — https://www.amnesty.org/en/latest/news/2024/11/denmark-ai-powered-welfare-system-fuels-mass-surveillance-and-risks-discriminating-against-marginalized-groups-report/
7. European Commission — Guidelines on prohibited AI practices — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj