Market abuse & trade surveillance

Firms that execute or arrange trades must detect and report suspected insider dealing and market manipulation. Rules-based scenarios still produce most alerts, and most alerts are false positives. AI adds behavioural anomaly models on order and trade data, speech and text analytics on communications, and LLM assistants that summarise alerts and draft reports. The goal is fewer, better alerts and faster, better-documented investigations; the legal decision to report stays with people.

Problem & value

• Obligation: the EU Market Abuse Regulation (MAR, Regulation (EU) 596/2014) prohibits insider dealing (Art. 14) and market manipulation (Art. 12), and Art. 16 requires market operators, investment firms and persons professionally arranging transactions to maintain effective arrangements, systems and procedures to detect and report suspicious orders and transactions [1]. Delegated Regulation (EU) 2016/957 specifies these arrangements, including automated detection, human analysis of flagged activity and staff training [2].
• Today: a surveillance team tunes rule-based scenarios (thresholds on order-to-trade ratio, price impact, cancellation patterns), triages large alert queues, reads chats and listens to calls, then decides on a STOR. Alert fatigue and siloed data (trades vs. communications) are the classic pain points.
• AI value: behaviour-based models that adapt to the desk or instrument, joined-up views of trading plus communications, and automated alert enrichment/summaries. Vendors report lower alert volumes, but published figures are vendor-specific and not independently benchmarked; measure on your own data (see KPIs).

Variants

• Order-book manipulation (spoofing, layering, momentum ignition, quote stuffing): needs full-depth, time-stamped order data; sequence and graph models.
• Insider dealing: trades ahead of price-sensitive announcements, linked to insider lists, deal-team rosters and employee personal-account dealing.
• Wash trades / pump-and-dump / cross-product manipulation: includes commodities, energy (REMIT overlaps), FX benchmarks and OTC products.
• Communications surveillance (e-comms and voice): transcription, translation and semantic classification of chats and calls; heavy GDPR and works-council implications (see ANOM-016 for the screening analogue).
• Crypto-asset markets: MiCA introduces market-abuse rules for crypto-asset trading platforms; vendors such as Eventus and Solidus Labs serve this segment (see ANOM-023).
• Buy-side / asset managers and corporates: personal-account dealing, front-running and insider-list management at smaller scale.
• Regulator/exchange side: supervisory analytics at national authorities and trading venues (public audience).

How it works

1. order/trade feeds + market data + comms + HR/roster data
2. normalisation & time-sync (FIX, venue feeds, MiFID II clock sync, instrument reference data)
3. feature layer (order-book sequences, price impact, peer baselines, entity/relationship graph)
4. detection: rule scenarios + unsupervised anomaly models + supervised alert-scoring on past dispositions
5. comms analytics: ASR + diarization
6. NER / classifier / LLM triage, linked to trade alerts
7. alert enrichment (news, announcements, insider lists) and LLM-drafted narrative
8. analyst review (human-in-the-loop)
9. escalation to MLRO/compliance
10. STOR to competent authority
11. feedback loop: dispositions retrain scoring; model validation and audit trail

• Hybrid design: deterministic scenarios remain for regulator-explainable, high-stakes detections; ML ranks and suppresses noise; LLMs help investigators read and summarise, not decide. Behavox, for example, publicly describes agentic AI for open-ended investigation but deterministic, rules-based models for high-stakes decisions, validated under model-risk guidance [3].
• Human-in-the-loop: every STOR decision, escalation and alert closure rationale is made and recorded by a named compliance officer. Reporting to the authority (e.g. BaFin in Germany) must not be tipped off to the client (no tipping-off; see Art. 16 MAR and Delegated Regulation 2016/957).

Data preparation

Needed are order and execution events from venues and OMS/EMS, market and reference data, e-communications and recorded voice, insider and restricted lists, personal-account dealing and trader rosters. In a financial firm these live in trading systems, data warehouses, archiving and recording platforms, e-mail and chat archives. Typical gaps: inconsistent timestamps and clock sync across venues, missing instrument identifiers, order lifecycle events split across systems, communications not linked to traders or trades, voice recordings in proprietary formats, and no labelled history of confirmed abuse.

Preparation steps: normalise order and trade feeds into one event model with synchronised timestamps and ISINs; link communications to people and orders (PREP-007); transcribe voice with a speech-to-text model such as Whisper (DATA-007) and translate multilingual chat; reconcile reference data and trader rosters (PREP-011); build an evaluation set from past alerts and closed cases with compliance experts (PREP-018, PREP-020); pseudonymise data used for model development (PREP-022) and enforce strict access rights (PREP-023). Expect 50-60 % of the effort in data work. See GUIDE-003.

Implementation paths

Which path for whom

• Enterprise: Path A, because certified surveillance suites from established vendors cover MAR/MiFID scenarios, venue connectivity and audit trails that regulators expect.
• Developer: Path C, because open-source detection and speech-to-text on own hardware suits building or evaluating surveillance components with sensitive trading and communication data.
• Public: Path C, because supervisors and public bodies need sovereign hosting, full control of logic and no dependence on non-EU services for market data.

Path A — Buy (default for banks, brokers, venues)

• Specialist trade-surveillance platforms: Eventus Validus (multi-asset trade surveillance, risk controls and algo monitoring; its "Frank AI" assistant for plain-English queries and alert review; on-premises deployment offered; strong in crypto venues) [4]; Nasdaq Trade Surveillance (formerly SMARTS, widely used by exchanges and regulators); NICE Actimize (market-abuse and conduct modules); Behavox Polaris (trade) and Quantum (communications surveillance, 50+ languages, ISO/IEC 42001-certified AI management system per the vendor) [3]; Global Relay and Smarsh (e-comms capture and surveillance); Solidus Labs (crypto). Data-residency and hosting need contractual verification; EU-headquartered specialists exist and should be included in any RFP, but EU origin must be checked per vendor.
• Choose A when: regulated firm with MAR/MiFID II duties, no in-house quant surveillance team. Regulators expect documented model validation and tuning, which vendors support with packaged evidence.

Path B — Build on cloud APIs

• Use for the assistive layer, not the detection core: frontier LLMs on EU regions (Claude via AWS Bedrock/Google Vertex EU regions, Mistral La Plateforme, Azure OpenAI EU Data Zone) summarise alerts, translate chats and draft STOR narratives from a structured evidence bundle. Speech-to-text via cloud ASR for trader calls where retention law allows.
• Orchestration: streaming ingestion (Kafka/Flink), a feature store, an alert-case system (existing case manager or custom) and an agent layer with read-only tools into the alert store.
• Choose B when: you already own a surveillance engine and want to modernise investigator tooling; keep personal data pseudonymised before it leaves your perimeter wherever possible.

Path C — Self-host (open source)

• Detection: Python with scikit-learn, XGBoost/LightGBM, PyOD (isolation forest, autoencoders), PyTorch for sequence models over order-book events; Neo4j or Memgraph for entity/relationship links; Kafka + Flink or Spark Structured Streaming; ClickHouse/kdb+-style columnar store for tick data (kdb+ is commercial).
• Comms: Whisper-class ASR or NVIDIA Parakeet, pyannote for diarization, open-weight LLMs (Llama, Qwen, Mistral families) via vLLM for triage and summaries.
• Hardware: HW3 for the LLM/ASR layer; detection workloads are CPU/columnar-store heavy. Strict on-prem is common because order and communication data are highly sensitive.
• Effort: a dedicated quant/data-engineering team; usually justified only for large banks, venues or regulators.

Path D — Custom training

Train supervised alert-scoring models on historical analyst dispositions (closed-as-false-positive vs. escalated vs. STOR). Genuine positives are very rare, so labels are scarce and biased towards past rules; combine with unsupervised models, synthetic scenario injection for testing and careful back-testing. Fine-tuning a domain classifier for jargon, code words and multilingual chat is realistic once a few thousand labelled messages exist.

State of the art (2026)

• Vendor trend: AI assistants inside surveillance suites (natural-language alert queries, alert summaries, noise reduction) are now a standard feature, e.g. Eventus Frank AI [4], and communications platforms emphasise multilingual semantic models and agentic investigation [3].
• Research: sequence and graph-based models on limit-order-book data for spoofing/layering detection are active areas; public data is limited (LOBSTER and similar research datasets), and real labelled manipulation cases are scarce. Treat published accuracy figures with caution; synthetic or simulated manipulation inflates results.
• Convergence of trade and comms: linking a trade alert to the surrounding chat/voice is the main driver of better precision.
• Limits: low base rates, concept drift as manipulators adapt, label scarcity, latency demands on full order books, multilingual slang, and LLM hallucination in narratives. Generated summaries must cite underlying events.

Shared resources & combinations

• Shared building blocks: streaming ingestion, anomaly-detection and graph-ML stacks shared with ANOM-002 (AML monitoring), ANOM-001 (payment fraud), ANOM-023 (crypto monitoring); ASR + diarization + LLM triage shared with call-recording and meeting use cases; entity resolution/link analysis with KNOW-026.
• Combinations: ANOM-016 (sanctions and PEP screening) for client onboarding context; AGENT-018 (continuous controls monitoring) for the control framework; KNOW-023 (financial research) for news context in alerts; PRED-035 (settlement-fail prediction) shares post-trade data; ANOM-006 shares audit-trail techniques.
• Hardware: HW3 serves LLM + ASR; detection runs on CPU/columnar stores; HW2 suffices for a small buy-side comms pilot.

Effort, cost & KPIs

• Path A: 3-9 months implementation incl. data feeds, scenario calibration and model validation; licence cost scales with order volume and asset classes.
• Path B: 2-4 months for an assistive investigator layer on top of an existing engine.
• Path C/D: 12+ months and a specialised team.
• Cost drivers: data feeds and storage of full order-book history, communications capture and transcription, model validation, analyst headcount.
• KPIs: false-positive rate and alert-to-case conversion, STOR conversion per scenario, time to disposition, alert backlog age, coverage (venues, instruments, comms channels), model back-test results on injected scenarios, regulator findings.

People & change

Involve management and the head of compliance as sponsors from the start; the MLRO or surveillance team are the daily users and must co-define scenarios and thresholds. IT and information security handle venue feeds, recording archives and access control; data owners of trading and communication data must agree to reuse. Legal covers MAR duties, retention and regulator expectations.

Works council: yes. Surveillance of traders' orders, chats and calls is objectively suited to monitor behaviour and performance (§87(1) no. 6 BetrVG), so a works agreement on purpose, access, retention and prohibition of use for performance rating is needed; §90(1) no. 3 requires early information and §80(3) allows an AI expert. In the public sector the Personalrat applies.

Data protection: the DPO must be involved early. Employee communications and personal-account dealing make a DPIA (Art. 35 GDPR) likely; sign an AVV (Art. 28 GDPR) with every processor and check the legal basis in the regulatory duties.

AI literacy: train analysts on model limits, false positives and documenting alert decisions (Art. 4 AI Act supports this). Run new models in parallel with the existing system, tune on a pilot desk, and keep humans in control: AI prioritises alerts, analysts decide and file reports. Guides: GUIDE-004, GUIDE-005, GUIDE-011.

Risks, regulation & compliance

• MAR / MiFID II: Art. 16 MAR duty and Delegated Regulation 2016/957 requirements (automated systems, human analysis, training, record-keeping) [1][2]. Trade and order recordkeeping and communications recording under MiFID II apply.
• EU AI Act: market surveillance for firms is not an Annex III high-risk category; typical deployment is minimal risk with AI literacy (Art. 4). Check carefully if surveillance includes employee monitoring or profiling that feeds employment decisions (Annex III, employment), or any emotion recognition from voice in the workplace, which Art. 5 prohibits; do not implement voice emotion/stress inference [5].
• GDPR and labour law: communications and voice contain personal data of employees and clients; lawful basis (legal obligation, Art. 6(1)(c)), purpose limitation, DPIA, works-council agreements (Betriebsrat, BetrVG § 87) and retention limits apply. Avoid automated decisions with legal effect on individuals (Art. 22).
• Model risk and DORA: model validation, explainability for supervisors, ICT third-party risk and exit plans for cloud vendors.
• Risks: missed abuse (false negatives), tipping-off, biased alerting on specific desks or nationalities, LLM leakage of confidential inside information to external APIs. Mitigations: on-prem or EU-region processing, redaction, citation-required summaries, periodic scenario back-testing, documented human sign-off.

Getting started

1. Map your MAR scope: venues, instruments, order flow, comms channels and who files STORs; document gaps against Art. 16 MAR and Delegated Regulation 2016/957.
2. Baseline today's alerts: volume per scenario, false-positive share, time per alert, STORs filed.
3. Shortlist 2-3 platforms (include an EU-based one) and run a proof of concept on 3-6 months of your own data, with injected test scenarios.
4. Add an LLM alert-summary assistant in a sandbox with strict evidence citation and data-minimisation, then measure analyst time saved.
5. Agree the data-protection and works-council framework for communications surveillance before enabling voice or chat analytics.

Sources

1. Regulation (EU) No 596/2014 (Market Abuse Regulation), EUR-Lex — https://eur-lex.europa.eu/eli/reg/2014/596/oj
2. Commission Delegated Regulation (EU) 2016/957 (arrangements, systems and procedures to detect and report suspicious orders/transactions), EUR-Lex — https://eur-lex.europa.eu/eli/reg_del/2016/957/oj
3. Behavox — platform overview (Quantum, Polaris, ISO/IEC 42001, agentic vs. deterministic AI) — https://www.behavox.com/
4. Eventus — Validus trade surveillance and Frank AI — https://www.eventus.com/
5. Regulation (EU) 2024/1689 (AI Act), EUR-Lex — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
6. ESMA — Q&A on the Market Abuse Regulation — https://www.esma.europa.eu/sites/default/files/library/esma70-145-111_qa_on_mar.pdf