Securities settlement-fail prediction & post-trade exception management

Every securities trade that does not settle on its intended date ties up cash and inventory, creates penalty and funding costs and consumes operations staff time. Settlement-fail prediction scores each open instruction hours or days before settlement from counterparty, instrument, inventory, SSI and matching-status features, then ranks the exception queue so that scarce operations capacity goes to the trades that will actually fail, and fixes (SSI correction, borrow, partial settlement) happen before the cut-off. With the EU planning its move to a one-day settlement cycle (T+1, target 11 October 2027), the time window for manual repair shrinks sharply, which makes prioritisation a core operating capability.

Problem & value

• Today: operations teams at banks, brokers, asset managers and custodians work static daily break lists: unmatched or unaffirmed instructions, missing securities, missing cash, SSI mismatches. Prioritisation is by trade size or age, not by the likelihood of failing.
• Cost of a fail: under the EU settlement discipline regime of CSDR (Regulation (EU) 909/2014) late matching and settlement fails attract daily cash penalties charged by the CSD, on top of funding and operational cost. The mandatory buy-in element has been effectively removed by later reforms (verify the current state before relying on it).
• AI value: a calibrated probability per instruction lets teams (1) work the riskiest items first, (2) trigger automatic pre-emptive actions, (3) forecast penalty exposure and (4) identify structural root causes (a custodian chain, an instrument class, a market's cut-off) that justify process change.
• Why now: shorter settlement cycles remove the overnight repair time that T+2 allowed. Industry and regulator impact work on T+1 repeatedly points to affirmation, allocation, SSI quality, inventory/FX funding and automation as the success factors; these are exactly the features of the model. No quantified benefit is claimed here; measure it on your own history (see KPIs).

Variants

• Broker/dealer or investment bank: predicts fails on both client and street-side legs; links to securities lending/borrowing desk and repo funding.
• Custodian / global custodian: predicts fails per sub-custodian market and client; uses instruction-level status messages from many CSDs.
• Asset manager: focuses on allocation/affirmation timeliness and counterparty/broker quality ranking, especially for cross-time-zone T+1 (US/Asia funds trading EU securities).
• CSD / market infrastructure: system-wide settlement-efficiency analytics and participant dashboards (aggregated, not instruction-level advice to participants).
• Not a private-individual use case; retail investors only see the result as a settled order.
• Covered variants: predictive "partial-settlement or borrow" recommendation, and LLM-assisted exception triage (reading counterparty emails/chat to classify the cause), both as modules of the same pipeline.

How it works

1. trade capture / OMS / allocations
2. status ingestion (matching, affirmation, SSI, hold/release, partial, cash/credit) via ISO 20022 / SWIFT / CSD APIs
3. feature store (counterparty history, instrument liquidity & lendability, time-to-cut-off, SSI age & change flags, market calendar)
4. fail model (gradient boosting; probability + SHAP drivers; re-scored on every status event)
5. exception prioritiser (expected penalty + funding cost × probability, within SLA window)
6. action engine (rules + optional LLM agent: correct SSI, request borrow, chase counterparty, allow partial settlement)
7. human review for high-value / non-standard cases
8. feedback of outcome labels into training

• Core model: tabular supervised classification; label = instruction failed to settle on intended settlement date (or settled late by n days). Time-aware validation is essential (train on past periods, test on later ones).
• Calibration: operations needs probabilities, not scores; apply isotonic or Platt calibration and monitor drift.
• LLM layer (optional): classifies free-text exception reasons from emails/chat and drafts counterparty chasers. It never changes instructions; changes go through the existing workflow with four-eyes approval.
• Human-in-the-loop: operations staff own all settlement instructions and cancellations; the system recommends.

Data preparation

Needed are trade and allocation records, settlement instructions and status messages (ISO 20022 sese / SWIFT MT54x), standing settlement instructions, inventory and cash positions, and historical fails with penalties. They typically live in the back-office settlement system, the order-management system, the custodian and CSD portals, and partly in e-mails and Excel exception lists.

Typical gaps: fail causes recorded only as free text, inconsistent counterparty identifiers (LEI, BIC, internal IDs), static-data changes without history, status messages in different formats per CSD, e-mail chases not linked to the instruction, and access rights across front and back office.

Concrete steps: build a unified instruction table by joining messages and trades (PREP-005); harmonise counterparty identifiers and fail-reason codes (PREP-019); harvest labels (failed or settled, cause) from historic records (PREP-020); define KPIs such as fail rate and penalty exposure (PREP-012); parse chase e-mails with an LLM (DOC-014); keep a time-based hold-out set (PREP-018). Tools: SQL or a data warehouse, Python, message parsers. Expect roughly 30-40 % of effort. See GUIDE-003.

Implementation paths

Which path for whom

• enterprise: B - the model must integrate with own settlement, static-data and messaging systems and no product covers each firm's counterparties and CSD mix; cloud APIs for the language part and own tabular models keep effort moderate.
• developer: C - tabular models and open-source streaming and orchestration tools run on HW1, keeping trade data inside the own environment.

Path A — Buy (off-the-shelf)

• Post-trade and reconciliation platforms with ML/analytics modules: SmartStream, Broadridge, SS&C, Nasdaq Adenza (Calypso), Murex, Torstone and similar vendors offer settlement dashboards and predictive exception modules; market-infrastructure analytics from Euroclear, Clearstream (Deutsche Börse) and DTCC are EU-/globally relevant. Feature depth and product names change quickly, so verify per vendor.
• Choose A when: you already run one of these platforms and a predictive module is an upgrade; vendor models learn across clients but you must check data-sharing terms and EU hosting.

Path B — Build on cloud APIs (recommended for most firms with a data team)

• Data platform: lakehouse in an EU region (Azure, AWS Frankfurt, Google Cloud europe-west3 or EU-sovereign offerings) with streaming ingestion of status messages.
• Modelling: managed AutoML or notebooks with XGBoost/LightGBM; MLflow for tracking.
• LLM layer: frontier or EU LLM APIs (e.g. Mistral, Claude, GPT via EU regions) with structured output for exception-cause classification.
• Integration: push predictions into the existing break-management or workflow tool.

Path C — Self-host (open source)

• Stack: XGBoost (Apache-2.0), LightGBM (MIT), scikit-learn (BSD-3), SHAP (MIT), MLflow (Apache-2.0), Kafka/Flink or similar for events, PostgreSQL/ClickHouse as feature store.
• Hardware: the tabular model runs on CPU servers (HW1); a small open-weight LLM (HW2) is only needed for the optional free-text triage. Many banks run this on-premises because instruction data is commercially sensitive.
• Ops effort: a small MLOps team; model risk management (SR 11-7-style or internal MRM) applies in banks.

Path D — Custom training

This is the classic path: models are trained on the firm's own instruction history because fail patterns are firm-specific (counterparty mix, markets, workflows). Useful extras: survival or time-to-settle models, graph features over the counterparty/custodian chain, and cost-sensitive learning that weights large and penalty-heavy fails. Needs typically 1–3 years of instruction-level history with fail labels; rare-event imbalance is the main modelling issue.

State of the art (2026)

• Gradient-boosted trees remain the workhorse for structured post-trade data; deep and foundation models add little unless text or sequences matter.
• Event-driven re-scoring at each status change (matched, affirmed, SSI changed, borrow confirmed) beats a single overnight score.
• Explainability (SHAP) is used to convert a score into an operational action: "SSI changed 2 days ago", "counterparty affirmation usually late", "instrument hard to borrow".
• LLM agents are being piloted for exception triage and counterparty communication; reliable only with narrow tools and human approval.
• T+1 context: the US moved to T+1 in May 2024; the EU and UK are preparing for a coordinated move (EU target date 11 October 2027 as set out in the legislative amendment of CSDR; confirm the final legal text). Market infrastructures publish settlement-efficiency analytics and industry groups publish T+1 readiness guidance, which are useful feature and benchmark sources.
• Limits: fails are rare events with heavy class imbalance; external causes (market stress, CSD outages, corporate actions) are hard to predict; model drift after market-structure change (the T+1 switch itself) requires retraining on post-switch data; features based on counterparty behaviour can be gamed or become stale. Publicly benchmarked datasets do not exist; vendor claims are not independently comparable.
• Research gap in this file: web search quota was exhausted during writing, so vendor-specific measured results, DTCC/Euroclear ML publications and ESMA statistics were not re-verified. Treat them as leads to check, not as facts.

Shared resources & combinations

• Same tabular-ml + time-series stack: PRED-009 (credit/default scoring), PRED-016 (cash-flow and liquidity forecasting, which uses predicted fails to forecast funding needs), PRED-021 (payment behaviour).
• Adjacent post-trade/finance use cases: AGENT-006 (reconciliation and close agent), ANOM-012 (market abuse and trade surveillance; keep separate ownership), ANOM-006 (ledger anomalies), AGENT-002 (back-office workflow automation).
• Shared building blocks: event streaming (Kafka), feature store, MLflow model registry, workflow engine (n8n/Camunda or the bank's BPM), CPU or modest GPU nodes (HW1; HW2 if an LLM triage layer is added).
• Document side: DOC-008 / DOC-014 for parsing counterparty emails, confirmations and SSI documents feeding the exception text classifier.

Effort, cost & KPIs

• Path A: weeks to a few months of configuration and data feeds; licence per platform.
• Path B: roughly 3–6 months for data ingestion, labelling, model, integration and MRM validation.
• Path C/D: similar effort plus infrastructure; main cost is data engineering and model governance, not compute.
• KPIs: settlement efficiency rate (by value and volume), fail rate by market/counterparty/instrument, CSDR penalty cost, funding cost of fails, precision/recall and calibration of the fail model at the operational threshold, share of fails fixed pre-cut-off, manual touches per trade, ageing of breaks.

People & change

Involve management and the head of operations as owners, settlement and exception staff as daily users, IT for message feeds and integration, IT security and legal/compliance (DORA, outsourcing rules for cloud use), and the DPO for any personal data in e-mails or client names. Start with the exception team before designing the queue, since they know the real fail causes.

Works council: depends. The prioritised queue and activity logs can show who handled which exception and how fast, which makes the system objectively suited to monitor performance under §87(1) no. 6 BetrVG (BAG); a works agreement that excludes individual performance evaluation is advisable. Also inform it under §90(1) no. 3 and consider an expert under §80(3). Trade data is mostly not personal; a DPIA is normally not required if no staff scoring is done.

AI literacy: support operations staff with training on probabilities and automation bias (GUIDE-006). Pilot on one market or CSD in shadow mode, comparing predictions with actual fails. Typical concern: "the queue dictates my work" - staff can override priorities and no automatic action runs without approval at first. See GUIDE-004, GUIDE-008 and GUIDE-009.

Risks, regulation & compliance

• EU AI Act: minimal risk in the typical case. The model scores trades, not natural persons; Annex III lists credit scoring of natural persons and critical-infrastructure safety components, neither of which covers trade-level operations scoring [1]. Caveat: if counterparty-level scores profile natural persons (e.g. individual traders), Art. 6(3) says profiling is always high-risk [2]. AI literacy (Art. 4) applies.
• Financial regulation: CSDR settlement discipline (Reg. 909/2014 and amendments), T+1 transition rules, DORA (ICT risk, third-party and outsourcing register for cloud and vendor models), MiFID II best execution/record keeping, bank model-risk governance.
• GDPR: mostly institutional data (gdpr_personal_data: no); personal data can appear in client names or account references of individual clients – pseudonymise in features.
• Operational risks: false negatives on large trades; automated actions on stale data; over-reliance on model during market stress. Mitigation: thresholds, kill switch, monitoring for drift, human approval for cancellations/amendments, audit log of recommendation versus action.
• Data confidentiality: instruction data is commercially sensitive; prefer EU hosting or on-premises; check vendor pooled-learning clauses.

Getting started

1. Extract 12–24 months of instruction-level data with final settlement outcome and status-message timestamps; define the fail label (intended settlement date vs. actual).
2. Baseline the current process: fail rate, penalties, manual touches and which fixes work (SSI correction, borrow, partial).
3. Train a time-split LightGBM/XGBoost baseline with SHAP; calibrate; compare against the simple ageing/size ranking.
4. Run in shadow mode for 4–8 weeks, then pilot the ranked queue with one operations team and measure pre-cut-off fix rate.
5. Add event-driven re-scoring and automated low-risk actions; document under model risk and DORA controls ahead of the T+1 switch.

Sources

1. AI Act Annex III (high-risk areas) — https://artificialintelligenceact.eu/annex/3/
2. AI Act Article 6 (classification, Art. 6(3) derogation and profiling rule) — https://artificialintelligenceact.eu/article/6/
3. ECB — TARGET2-Securities (T2S; partial settlement, DvP in central bank money) — https://www.ecb.europa.eu/paym/target/t2s/html/index.en.html
4. XGBoost — https://github.com/dmlc/xgboost
5. LightGBM — https://github.com/microsoft/LightGBM
6. SHAP — https://github.com/shap/shap
7. MLflow — https://github.com/mlflow/mlflow
8. scikit-learn — https://github.com/scikit-learn/scikit-learn
9. DTCC — US Treasury clearing and T+1 context page — https://www.dtcc.com/ust1
10. Regulation (EU) 909/2014 (CSDR) — https://eur-lex.europa.eu/eli/reg/2014/909/oj (page content not retrievable in this session; cited for the legal basis only)