Pay transparency & gender pay-gap analysis (EU Pay Transparency Directive)

Directive (EU) 2023/970 makes pay structures auditable: employers must publish pay ranges, answer employee pay questions, report gender pay-gap indicators and fix unjustified gaps of 5% or more. The hard parts are statistical (which jobs are "of equal value", how much of a gap is explained) and organisational (hundreds of job descriptions, messy payroll, tight deadlines). ML and LLMs group jobs, run pay-equity regressions, simulate remediation and draft the documents; HR, works councils and counsel decide.

Problem & value

• Today: pay-gap figures are built in spreadsheets from payroll exports; job evaluation is manual (consultants, Hay/Mercer-style point methods) and often outdated; employee questions are answered ad hoc. The Directive turns this into a legal duty with burden of proof on the employer (Art. 18) [1].
• Key obligations (check national transposition law, deadline 7 June 2026): pay-range information for applicants and ban on asking pay history (Art. 5); employee right to information on own pay and average pay by sex for equal-value work, answered within two months (Art. 7); pay-gap reporting (Art. 9) — annually for 250+ employees, every three years for 150-249, first reports due 7 June 2027 (data year 2026); employers with 100-149 employees follow from 2031; a joint pay assessment with worker representatives when a gap of at least 5% in a category of workers is not justified by objective, gender-neutral criteria and not remedied within six months (Art. 10) [1].
• AI value: consistent job grouping across thousands of roles, reproducible statistics, scenario modelling instead of one-off audits, and faster, uniform answers to information requests. No peer-reviewed benefit figures exist for this new regime; do not rely on vendor ROI claims without a pilot.
• Context: the unadjusted EU gender pay gap is around 12-13% according to Eurostat [6].

Variants

• Annual Art. 9 reporting: indicators by sex (mean/median gap overall, in bonus pay, share receiving bonus, quartile distribution, gap by category of workers by ordinary basic salary and variable pay).
• Job architecture and equal-value grouping (Art. 4): cluster roles by skills, effort, responsibility and working conditions; also needed to name the comparator group in Art. 7 answers.
• Remediation planning: minimum-cost budget to close unexplained gaps, with promotion and hiring effects.
• Pay-range drafting for job ads: ranges derived from grade and market data (link LANG-004).
• Employee request assistant: draft answers, HR approves (link KNOW-003).
• Public sector / collective-agreement pay scales: gaps arise in grading and step placement rather than negotiation; analysis focuses on eligibility for supplements and progression.
• Small employers (<100): not reporting, but still bound by Art. 5 and 7 — a lightweight check suffices.

How it works

1. payroll + HRIS + job descriptions (secure extract, pseudonymised IDs)
2. data validation (FTE normalisation, hourly-equivalent pay, bonus/benefit mapping, outliers)
3. job grouping (embeddings + clustering of job descriptions
4. proposed equal-value groups; HR validates)
5. indicators (mean/median gaps, quartiles, bonus shares, by worker category)
6. explanatory modelling (regression: log pay ~ group, tenure, location, working time; unexplained gap with CI)
7. 5% flag per category + drivers (e.g. entry-offer differences, promotion speed, allowances)
8. remediation simulation (optimiser: minimum budget to close unexplained gaps under constraints)
9. drafts: Art. 9 report, joint assessment pack, Art. 7 answers, pay ranges
10. review by HR / works council / legal; versioned audit trail; export to authority format

• Statistics, not LLMs, produce the numbers. Gap indicators are deterministic SQL/pandas; the adjusted gap uses OLS/quantile regression or Oaxaca-Blinder decomposition. LLMs only draft text and help cluster or describe jobs.
• Job grouping: text embeddings of job descriptions plus a classifier against an existing grade structure; LLM generates a rationale per assignment against the four Directive criteria. A human confirms every group, because "equal value" is a legal judgement.
• Care with "explained": variables such as performance rating or negotiated starting pay can themselves carry bias. The tool must show gap with and without them, not hide gaps behind controls.
• Human-in-the-loop: HR owns grouping and thresholds; works council and counsel review the joint assessment.

Data preparation

The base is payroll and HRIS data (DATEV or SAP payroll, HR master data), job descriptions and grade tables, which in an SMB often sit in separate exports, Word files, collective agreement PDFs and Excel sheets kept by HR. Typical gaps: no consistent job architecture or grade per employee, outdated or free-text job titles, bonus and allowances booked under inconsistent pay codes, part-time and FTE not normalised, missing attributes (tenure, location, contract type), and strict access rights because the data is personal and sensitive.

Preparation steps: 1. Inventory all sources and owners (PREP-001, PREP-004) and extract payroll and HRIS data via export or API (PREP-005). 2. Map pay codes to base pay, bonus and allowances; normalise to hourly equivalents. 3. Tidy the Excel material into one machine-readable table (PREP-011) and write job descriptions in comparable form (PREP-014). 4. Define the category scheme for equal-value job groups before any clustering (PREP-019). 5. Pseudonymise IDs for analysis (PREP-022) and restrict access to a small HR group.

Tools: SQL/pandas, dbt or Excel Power Query; no AI is needed for most of this. Expect about 50-60% of total effort in data preparation, because the statistics are quick once the data is clean. See also GUIDE-003 and DATA-020.

Implementation paths

Which path for whom

• SMB (A): The reporting duty is standard and legally exacting; a payroll/HCM module, a specialist tool or the federal Logib-D self-test is cheaper and safer than building statistics yourself.
• Enterprise (B): Job architecture, several payroll systems and multi-country rules call for a warehouse-based pipeline with custom indicators, with LLMs only drafting text.
• Developer (C): Payroll data should stay in the HR enclave, and the stack (statsmodels, OR-Tools, small open-weight models) runs on HW2.
• Public (C): Pay data of public employees and procurement rules favour self-hosting or a sovereign German/EU cloud with an audit trail.

Path A — Buy (recommended for most employers)

• HCM suites with pay-transparency modules: SAP SuccessFactors (SAP, EU) with pay-transparency and pay-equity analytics, Workday, Oracle HCM [2].
• Specialists: Syndio (US, pay equity analytics), Figures (compensation benchmarking and transparency tooling, EU), Ravio (EU compensation benchmarking), Pave, Beqom, Mercer and WTW tools and consultancies.
• Public tooling: Switzerland's Logib and Germany's Logib-D (federal self-test tools) [5].
• Choose A when: payroll sits in a major HCM suite, 250+ employees, little in-house data science. Check EU hosting, data processing agreement, transparent methodology, export of Art. 9 indicators.

Path B — Build on cloud APIs

• Warehouse (Snowflake/BigQuery/Azure in EU regions) with SQL-defined indicators; Python (pandas, statsmodels) in notebooks, run in an EU tenant [3].
• LLMs (EU region): Mistral, or Claude/Gemini/GPT via EU-hosted endpoints, for job-description summarisation, rationale drafts and Art. 7 answer drafts. Send only pseudonymised or aggregated data.
• Embeddings via EU-hosted API for job clustering; orchestration with n8n or Power Automate.

Path C — Self-host (open source)

• Stats: Python with statsmodels, scikit-learn, DoWhy/EconML for causal sensitivity checks, SciPy/OR-Tools for remediation optimisation.
• Language: open-weight models (e.g. Mistral Small/Medium class, Qwen, Llama) via vLLM or Ollama; multilingual embedding models (e.g. BGE-M3, multilingual-e5) for job clustering.
• Hardware: HW2 suffices (payroll volumes are tiny; models are only for text). Highest ops effort is data security: payroll data should never leave the HR enclave.

Path D — Custom training

Rarely justified. A fine-tuned classifier mapping job descriptions to the company's grade system pays off only for very large, multi-country groups with thousands of roles and a few thousand labelled examples from past evaluations.

State of the art (2026)

• Statistical pay-equity analysis (regression, decomposition) is mature; the new element is EU-specific workflow: equal-value grouping, Art. 9 indicators, 5% flag, joint assessment and Art. 7 requests.
• Embeddings and LLMs reduce the manual effort of mapping roles to common frameworks; evaluation is agreement with expert job evaluators, not benchmark scores.
• Vendors (SAP, Workday, Syndio, Figures, Ravio) shipped or announced Directive-specific features; methods and exact scope vary, and national transposition differs — verify against your member state's law.
• Limits: small groups (privacy and statistical power; many member states have minimum-group rules); regression "explains away" bias if controls are tainted; job descriptions may not reflect actual duties; LLM-written rationales can sound convincing while being wrong; differing national definitions of pay components and delayed transposition.

Shared resources & combinations

• Same stack (tabular-ml + llm-general + workflow-automation): DATA-012 (CSRD social metrics incl. pay-gap data points), DATA-020 (regulatory reporting packaging), DATA-002 (narrative reports), DATA-024 (FP&A: remediation budget in planning).
• Text side: LANG-004 (pay-range wording in job ads), KNOW-003 (employee self-service answers).
• Data-quality side: ANOM-037 (payroll validation) supplies clean inputs.
• Caution: AGENT-008 (CV screening) is high-risk; keep it separate from pay analytics.

Effort, cost & KPIs

• Path A: weeks to a few months (data mapping and job grouping dominate); subscription per employee.
• Path B: 3-6 months incl. legal and works-council review; cloud cost is negligible next to people effort.
• Path C: similar effort, plus security hardening.
• KPIs: share of roles with validated equal-value group, unexplained gap per category (target below 5%), time to answer Art. 7 requests (limit: two months), data completeness, remediation cost vs. plan, reproducibility of indicators between runs.

People & change

Start with management and HR (process owner) to set the legal scope, then involve the data owners in payroll and IT early, since the extract is the hardest step. Bring in the data protection officer before any data is processed: employee pay data is personal, a DPIA is likely (systematic evaluation of employees), and every tool vendor needs an AVV (GUIDE-005). Works council: yes. Co-determination applies because analysis tools and job groupings touch individual pay and may be suited to monitor behaviour or performance (§87(1) no. 6 BetrVG); the council must be informed in good time about AI-supported work processes (§90(1) no. 3) and may engage an AI expert (§80(3)). The Directive also gives employee representatives a role in the joint pay assessment. In the public sector the staff council (Personalrat) takes this role. See GUIDE-004.

AI literacy: the AI Act (Art. 4) requires measures to support the literacy of HR staff who use the tool; train them on regression limits and bias in "explained" variables (GUIDE-006). Communicate openly what is analysed and why, aggregate results rather than individuals, and start with a pilot in one entity or pay group (GUIDE-008). Typical concern: "individual pay will be exposed"; answer with pseudonymisation, aggregate reporting and human sign-off on all groupings and answers to employees.

Risks, regulation & compliance

• EU AI Act: aggregate pay analytics is typically minimal risk. It becomes high-risk (Annex III point 4) if AI is used to decide individual pay, promotions or to evaluate performance; keep the system advisory and keep individual compensation decisions human. Art. 4 AI literacy applies [4].
• GDPR: payroll plus gender is personal data (Art. 6(1)(c) legal obligation for reporting); apply data minimisation, pseudonymisation, small-group suppression and access control; DPIA is advisable. Gender data in HRIS may be incomplete or non-binary — handle per national law. Employee works-council co-determination (e.g. Germany BetrVG) applies to tooling [7].
• Directive duties: burden of proof on employer, compensation for affected workers, penalties set nationally; transposition status varies, so check local law.
• Risks and mitigations: misgrouped jobs (human validation, documented criteria); false assurance from over-controlled models (show raw and adjusted gaps); leaks of individual pay (aggregation thresholds); hallucinated answers to employees (templates, HR sign-off); vendor methodology opacity (demand documentation).

Getting started

1. Check your member state's transposition law, employee count thresholds and reporting dates.
2. Inventory pay components (base, bonus, benefits) and build one clean, FTE-normalised payroll extract.
3. Run the raw indicators first (mean/median gap, quartiles); then pilot job grouping on one function with HR.
4. Add regression analysis, review the drivers with works council and legal, and document the method.
5. Set up the Art. 7 request workflow and pay-range templates before the first requests arrive.

Sources

1. Directive (EU) 2023/970 (Pay Transparency Directive) — https://eur-lex.europa.eu/eli/dir/2023/970/oj
2. SAP SuccessFactors compensation and pay-equity products — https://www.sap.com/products/hcm/compensation-management.html
3. statsmodels documentation (regression, Oaxaca-style decompositions) — https://www.statsmodels.org/stable/index.html
4. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. Logib (Swiss Federal Office for Gender Equality) — https://www.logib.admin.ch
6. Eurostat — gender pay gap in unadjusted form — https://ec.europa.eu/eurostat/databrowser/view/sdg_05_20/default/table
7. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj
8. Syndio — pay equity platform — https://synd.io