Employee attrition prediction

Losing experienced staff is expensive, and most organisations only see the numbers after people resign. Attrition prediction uses HR data, pay data and engagement surveys to estimate where voluntary turnover will rise and why. Typical hotspots are a nursing ward, a software team, a warehouse shift or a branch. This lets HR and line managers fix causes such as pay gaps, overload, missing career moves or a poor manager, and plan hiring in time. In the EU, scoring individual employees is a high-risk AI use under the AI Act and needs co-determination in Germany. Good programmes therefore start with team-level forecasts and driver analysis and use individual scores only with strong safeguards, or not at all.

Problem & value

• Who: HR business partners, people-analytics teams, works councils, managers of high-turnover groups (care, hospitality, retail, logistics, IT) and finance for headcount planning.
• Today: turnover rates are reported after the fact in HRIS dashboards. Exit interviews are collected but rarely analysed systematically, and retention budgets (raises, counter-offers) are spent reactively when resignation letters arrive.
• Why AI helps: models combine many weak signals (time since last raise or promotion, pay position vs. market, manager changes, overtime trends, survey scores) into earlier and more reliable estimates than rules of thumb. LLMs make survey comments and exit interviews analysable at scale.
• Economics: Gallup estimates the cost of replacing an employee at one-half to two times the annual salary. In its research, 52% of voluntary leavers said their manager or organisation could have done something to prevent them from leaving [1]. These are directional US figures, not a business case.

Variants

• Aggregate attrition forecasting for workforce planning: expected leavers per unit, role and quarter as an input to hiring and capacity plans (links to PRED-011 and OPT-003). Low regulatory risk if no individual scores are produced.
• Driver / root-cause analysis: which factors explain turnover differences between teams (pay compression, shift patterns, commute, manager span). Used for policy, not for individuals.
• Individual "flight-risk" or "regrettable attrition" scores for key talent and succession planning. This is the classic HR-analytics variant and the one with the highest legal burden.
• Early-tenure attrition: leaving within the first 3–12 months. It is driven by onboarding quality and connects to AGENT-009.
• Healthcare & care: nurse and care-worker turnover per ward, combined with staffing forecasts (HLTH-008).
• Public sector: public employers face retirement waves and skills shortages. The same method applies, but personnel representation law (Personalvertretungsgesetze) replaces the BetrVG.
• Not covered here: customer churn (PRED-003), profiling of job seekers by employment services (PRED-029), communication surveillance for conduct risk (LANG-032) and performance-review assistants (LANG-036).

How it works

1. HRIS + payroll + survey + absence aggregates (pseudonymised)
2. point-in-time employee-month snapshots, label: voluntary exit within horizon
3. model: gradient boosting or survival model (time-to-exit)
4. calibration, fairness checks per protected group, minimum group sizes
5. aggregation to team / site / role level (default output)
6. driver analysis (SHAP), LLM topic analysis of survey and exit comments
7. retention measures piloted with control groups
8. measured effect
9. HR dashboard + headcount plan; individual view only for authorised HR roles

• Core model: LightGBM/XGBoost/CatBoost on monthly snapshots, or survival models (Cox, random survival forests, gradient-boosted survival) that predict when someone may leave and handle censoring of current employees correctly (scikit-survival, lifelines) [17].
• Small organisations: with a few hundred employees and a few dozen leavers per year, individual-level models are statistically weak. Tabular foundation models such as TabPFN-3.5 help on small tables [16], but the better choice is often a team-level forecast plus driver analysis.
• Text layer: an LLM classifies open survey answers and exit-interview notes into themes (pay, manager, workload, career, commute) and summarises them per unit, with minimum group sizes so no individual can be identified (shared with LANG-019 and LANG-013).
• Causal layer: what matters is not who will leave, but which measure actually reduces attrition. Measures such as pay adjustments, stay interviews or shift-pattern changes should be rolled out with comparison groups or difference-in-differences evaluation.
• Human-in-the-loop: HR decides on measures. Scores never trigger negative actions such as skipping someone for training or promotion. The works council approves the data scope and the outputs.

Data preparation

Which data: HR master data (tenure, role, pay band, location), exits with reasons, absences, promotions, engagement-survey results, and optionally aggregated workload data.

Where it lives in an SMB: HRIS and payroll, Excel lists kept by HR, applicant and personnel files (partly paper) in a DMS, survey tools, and time-tracking.

Typical gaps: small headcount and few exits (weak statistical basis), inconsistent exit-reason coding, personnel data in scattered spreadsheets, mixed history after reorganisations, strict access rights, and sensitive attributes that must not be used.

Steps: 1. Check whether you have enough leavers; below a few dozen per year, prefer aggregated team-level analysis. 2. Extract HR data into one table with consistent IDs (PREP-005, PREP-011). 3. Standardise exit reasons and harvest labels (PREP-019, PREP-020). 4. Remove protected and special-category attributes and decide on purpose and legal basis with the DPO and works council (PREP-025). 5. Pseudonymise before analysis and restrict access to HR analysts (PREP-022, PREP-023). 6. Set up a time-based test set (PREP-018).

Effort share: about 30-40 percent, plus coordination time for the agreements. See GUIDE-003.

Implementation paths

Which path for whom

• SMB (A): Companies with few staff lack the data for custom models, so the built-in people-analytics of established HRIS products is the realistic option; check that attrition scoring is even advisable at your size.
• Enterprise (C): HR data is highly sensitive and works-council scrutiny is high, so a controlled self-hosted model with an agreed works agreement is preferable to sending employee data to external APIs.
• Developer (C): Open-source tabular ML with explainability (scikit-learn, SHAP) on HW1, deployed inside the client's environment.
• Public (C): Staff-council rules and public-sector procurement favour a self-hosted or sovereign-cloud setup with strict purpose limitation.

Path A — Buy (recommended for most SMBs)

• People-analytics platforms: Visier Workforce AI (Canada; benchmarks, "prevent regrettable attrition", AI assistant and agents) [12]. One Model (US; forecasts for hiring, attrition and workforce cost, One AI) [13]. SAP SuccessFactors People Analytics (Germany) inside the SAP HCM suite.
• Engagement & listening suites: Workday Peakon Employee Voice (founded in Copenhagen; "understand attrition risks, identify the causes of turnover") [10]. Microsoft Viva Glint [14]. Leapsome (Berlin) and Culture Amp also link engagement surveys to turnover analytics. Workday's Sana agents now position retention as a manager-coaching topic [11].
• SMB reality: EU HRIS tools such as Personio (Munich) provide turnover reporting. For companies under ~250 employees, good reporting plus a regular pulse survey usually delivers more than a predictive model.
• Check: EU data residency, whether individual scores can be switched off, role-based access, minimum reporting group sizes, the vendor's AI Act documentation as a high-risk provider, and a works-council-ready description of the system.

Path B — Build on cloud APIs

• Managed ML in EU regions: Azure Machine Learning AutoML, SageMaker Autopilot/Canvas, Vertex AI or BigQuery ML, Databricks or Snowflake ML on a pseudonymised HR mart.
• Hosted tabular foundation model: Prior Labs (Freiburg, Germany) offers TabPFN via API and as a VPC deployment [16].
• LLM for comments: an EU-hosted model (Mistral La Plateforme, or Azure OpenAI / Claude on Bedrock / Gemini in EU regions) for theme extraction. Send only pseudonymised text, contractually exclude training on the data and strip names before the call.
• Integration: HRIS (SAP SuccessFactors, Workday, Personio) → warehouse → model → BI (Power BI, Tableau) with row-level security.

Path C — Self-host (open source)

• Stack: pandas/Polars, LightGBM/CatBoost, scikit-survival (GPL-3.0) [17] or lifelines, SHAP, Fairlearn for group-fairness metrics, MLflow, Evidently for drift, Apache Superset or Metabase for dashboards.
• Text: a small open-weight LLM (e.g. Mistral Small or Qwen3 class, ≤ 30B quantised) on HW2 for comment classification. The predictive model itself runs on HW1 (CPU).
• Why self-host: HR data is among the most sensitive data a company holds. On-prem keeps it in the HR network zone, which works councils and data protection officers often prefer.

Path D — Custom training

Worth it only for large employers (roughly 5,000+ employees) with several years of clean leaver histories, consistent reason codes (voluntary vs. involuntary vs. retirement) and a people-analytics team. "Custom" means your own features and survival models. Training sequence models on HR event histories rarely beats boosted trees at this data size. Do not train models on email, chat or calendar metadata to infer job-search behaviour. This is disproportionate monitoring under GDPR and co-determination rules, and LANG-032 covers the narrow lawful communication-surveillance cases.

State of the art (2026)

• Methods: boosted trees and survival models on point-in-time HR snapshots remain standard. Tabular foundation models (TabPFN-3.5: up to 1M rows and 20k features; weights under a non-commercial licence, commercial use via Prior Labs) are a strong no-tuning baseline for small HR datasets [16].
• Shift in products: vendors are moving from individual "flight-risk" flags to agentic HR analytics. Assistants and agents answer workforce questions, benchmark attrition against peers and coach managers on retention (Visier agents and MCP, Workday Sana) [11][12].
• Listening data: continuous pulse surveys with LLM comment analysis have replaced annual engagement surveys as the main leading indicator.
• Limits: labels are noisy (voluntary vs. "mutual agreement"), and events are rare in small units. Models drift after reorganisations or labour-market changes. Strong predictors such as age or tenure close to retirement are proxies for protected characteristics. Self-fulfilling effects are real: an employee flagged as "at risk" may be given fewer opportunities and then leave. Reported accuracy figures from vendors are usually not independently verified.

Shared resources & combinations

• Tabular ML stack (HW1): the same feature store, training and MLOps setup as PRED-003 customer churn, and DATA-009 AutoML. Point-in-time snapshot logic is identical.
• Workforce planning chain: PRED-012 expected leavers → PRED-011 demand forecast → OPT-003 shift planning / HLTH-008 hospital staffing → AGENT-008 / AGENT-033 recruiting.
• Retention measures: KNOW-012 skills mapping and internal mobility, EDU-005 learning paths, AGENT-009 onboarding (early-tenure attrition), DATA-025 pay-gap analysis (pay drivers), and KNOW-003 HR self-service.
• Text pipeline (llm-general, HW2 if self-hosted): shared with LANG-013 feedback analysis, LANG-019 qualitative coding and LANG-036 performance-review assistant.
• Boundaries: LANG-032 (communication surveillance) and PRED-029 (citizen profiling) are separate uses with different legal bases.

Effort, cost & KPIs

• Path A: 1–3 months, mostly HRIS integration, works-council agreement and DPIA. Cost is usually per-employee licences of the analytics or survey suite.
• Path B/C: 3–6 months for the first model, dashboard and governance. Ongoing effort is about 0.2–0.5 FTE of people-analytics capacity. Compute cost is negligible (CPU).
• Cost drivers: data cleaning (exit reasons, org history), legal and co-determination work, and the retention measures themselves (raises, career programmes).
• KPIs: regretted voluntary attrition rate, early-tenure (first-year) attrition, the effect of measures vs. comparison groups, and forecast error of leavers per unit (used in headcount plans). Technical KPIs: calibration and C-index / PR-AUC. Fairness KPIs: error-rate parity across gender, age band and disability status.

People & change

Involve early: management (purpose and limits), HR as process owner, the works council or staff council from the first idea, the DPO, legal/compliance, and IT security for access to HR data. Department leads join when results are shared. Employees are an affected group and should be informed transparently.

Works council: yes. HR analytics on employee data is suited to monitor behaviour and performance and requires co-determination under §87(1) no. 6 BetrVG. Inform the council in good time about AI-planned work processes (§90(1) no. 3), consider engaging an AI expert (§80(3)), and conclude a works agreement on purpose, data, access and no individual sanctions (GUIDE-004). Public sector: staff council under BPersVG or state law.

Data protection: a DPIA (Art. 35 GDPR) is very likely; employment data is covered by §26 BDSG and Art. 88 GDPR. Involve the DPO early (GUIDE-005).

AI literacy: Art. 4 AI Act requires measures that support literacy; train HR and managers on interpreting risk scores and on bias (GUIDE-006).

Pilot and communication: start with team-level, aggregated insights, not individual scores, and communicate the purpose openly (GUIDE-008, GUIDE-009).

Concerns: fear of surveillance and self-fulfilling labels is met by prohibiting use in dismissal or promotion decisions and keeping managers in a supportive role. Humans always decide.

Risks, regulation & compliance

• EU AI Act — high-risk: Annex III point 4(b) covers AI used to make decisions affecting promotion or termination and to monitor and evaluate the behaviour of persons in work relationships [2]. Individual attrition scores are profiling of natural persons, so the Art. 6(3) exemption for "preparatory" tasks does not apply [3]. Deployers must follow the instructions for use, assign competent human oversight, monitor the system, keep logs for at least six months and inform workers' representatives and affected workers before use (Art. 26) [5]. Annex III obligations apply from 2 December 2027 according to the current implementation timeline [6], after the Digital Omnibus simplification package [7]. Purely aggregate forecasts without per-person scores are generally outside Annex III, but document that assessment.
• Art. 5 prohibitions: inferring emotions of employees (for example from voice, video or typing behaviour) is prohibited in the workplace except for medical or safety reasons [4]. "Mood" or "stress" signals from such systems must not feed an attrition model.
• GDPR: Art. 88 lets member states set specific employment rules, with safeguards for transparency and workplace monitoring [8]. In Germany, §26 BDSG [9] is under strain since the CJEU ruled in C-34/21 (2023) that national rules merely repeating the GDPR do not qualify under Art. 88 [18]. Rely on a documented legal basis, usually Art. 6(1)(f) or a works agreement, plus a DPIA (large-scale profiling of employees). Art. 22 applies if scores produce significant effects without real human review. Exclude special-category data (health from absence reasons, union membership) and their proxies.
• Co-determination (Germany): §87(1) no. 6 BetrVG gives the works council a say over technical systems suited to monitor behaviour or performance [15]. For group-wide systems, the Federal Labour Court assigned this right to the central works council (1 ABR 20/21, M365) [19]. Conclude a works agreement covering purpose limitation, the ban on individual negative consequences, access roles and deletion.
• Other risks: discrimination (AGG; age and gender proxies), trust damage if employees learn they are scored, and data leaks of HR data. Mitigations: pseudonymisation, minimum group sizes (e.g. n ≥ 5–10), no exposure of individual scores to line managers, and transparency notices.

Getting started

1. Fix definitions: voluntary vs. involuntary exit, "regretted" leaver, horizon (6–12 months), and clean three or more years of leaver data with reasons.
2. Involve the works council and the DPO from day one. Agree on purpose, aggregate-first outputs and a DPIA before building anything.
3. Build a team-level attrition dashboard with driver analysis (pay position, promotion gap, manager changes, survey scores) and an LLM theme analysis of exit comments.
4. Pilot two or three retention measures in the hottest units with comparison groups and measure the effect after 6–12 months.
5. Decide only then whether individual risk bands are needed. If so, prepare high-risk AI Act documentation and oversight, fairness tests and a works agreement.

Sources

1. Gallup — This Fixable Problem Costs U.S. Businesses $1 Trillion (2019) — https://www.gallup.com/workplace/247391/fixable-problem-costs-businesses-trillion.aspx
2. EU AI Act — Annex III high-risk use cases (point 4) — https://artificialintelligenceact.eu/annex/3/
3. EU AI Act — Article 6 classification rules incl. profiling clause — https://artificialintelligenceact.eu/article/6/
4. EU AI Act — Article 5 prohibited practices (emotion recognition at work) — https://artificialintelligenceact.eu/article/5/
5. EU AI Act — Article 26 obligations of deployers — https://artificialintelligenceact.eu/article/26/
6. AI Act implementation timeline — https://artificialintelligenceact.eu/implementation-timeline/
7. European Commission — Digital Omnibus on AI regulation proposal — https://digital-strategy.ec.europa.eu/en/library/digital-omnibus-ai-regulation-proposal
8. GDPR Art. 88 — Processing in the context of employment — https://gdpr-info.eu/art-88-gdpr/
9. Bundesdatenschutzgesetz §26 — https://www.gesetze-im-internet.de/bdsg_2018/__26.html
10. Workday — Employee Voice / Peakon (employee retention) — https://www.workday.com/en-us/products/employee-voice.html
11. Workday — Talent management & Sana HR agents — https://www.workday.com/en-us/products/talent-management.html
12. Visier — Workforce AI — https://www.visier.com/
13. One Model — People analytics, One AI, attrition forecasting — https://www.onemodel.co/
14. Microsoft Learn — Introduction to Microsoft Viva Glint — https://learn.microsoft.com/en-us/viva/glint/introduction-viva-glint
15. Betriebsverfassungsgesetz §87 — https://www.gesetze-im-internet.de/betrvg/__87.html
16. Prior Labs — TabPFN repository (versions, limits, licences) — https://github.com/PriorLabs/TabPFN
17. scikit-survival — survival analysis on scikit-learn — https://github.com/sebp/scikit-survival
18. Court of Justice of the EU — Case C-34/21 (Hauptpersonalrat, employment data, Art. 88 GDPR) — https://curia.europa.eu/juris/liste.jsf?num=C-34/21
19. Bundesarbeitsgericht — 1 ABR 20/21 (8 March 2022), co-determination for Microsoft 365 — https://www.bundesarbeitsgericht.de/entscheidung/1-abr-20-21/