Portfolio optimisation & algorithmic trading

Investors must decide how to split capital across assets and how to execute trades at low cost under risk, liquidity and regulatory constraints. Classical optimisation (mean-variance, risk parity, CVaR) combined with ML return and risk forecasts and execution algorithms automates both steps. LLMs now add text signals and research/code assistance, but the core remains quantitative, heavily tested and tightly supervised: in the EU, algorithmic trading is among the most regulated AI-like activities.

Problem & value

• Today: portfolio managers combine analyst views, factor models and spreadsheet or vendor optimisers; traders choose execution algorithms from broker menus. Rebalancing, tax/cost awareness and constraint handling (mandate limits, ESG exclusions, liquidity) are largely manual or rule-based.
• AI value: better covariance estimation and forecasts, systematic rebalancing with transaction-cost awareness, faster research cycles, lower implementation shortfall, consistent constraint compliance.
• Honest caveat: return prediction is a low signal-to-noise problem; most published backtest gains disappear after costs, overfitting control and capacity limits. Value is more reliable in risk control, cost-aware execution and operational efficiency than in "alpha from AI".

Variants

• Strategic/tactical asset allocation for pension funds, insurers and family offices (monthly to quarterly rebalancing, liability-aware constraints, Solvency II / IORP considerations).
• Systematic equity/multi-asset funds: factor and ML signal portfolios with turnover and risk budgets.
• Smart execution: VWAP/TWAP/implementation-shortfall algorithms, increasingly ML-tuned or RL-driven.
• Treasury and corporate hedging: FX/interest-rate hedge optimisation (no speculation).
• Robo-advice for retail: model portfolios per risk profile (suitability duties apply, see Risks).
• High-frequency / market making: specialist, latency-driven; out of reach for most SMBs.
• Private use: self-directed investors with rebalancing scripts or broker robo-services; no AI Act issue, but retail trading bots carry high loss risk.

How it works

1. data ingestion (prices, order book, fundamentals, news)
2. feature engineering + signals (factors, ML forecasts, LLM text sentiment)
3. risk model (covariance shrinkage, factor model, regime detection)
4. optimiser (mean-variance / CVaR / risk parity / Black-Litterman; constraints, turnover, costs)
5. pre-trade compliance & risk limits (mandate, concentration, ESG, liquidity)
6. execution algorithm (order slicing, venue selection) via OMS/EMS
7. post-trade analysis (TCA, attribution)
8. monitoring, kill switch, model-risk review

• Forecasting: gradient boosting, sequence models and time-series foundation models (e.g. Chronos, TimesFM) produce return/volatility inputs; robust treatment of non-stationarity is essential.
• Optimisation: convex solvers (quadratic/conic) remain the workhorse because they are auditable and respect hard constraints; ML supplies the inputs, not the final weights, in most regulated settings.
• Reinforcement learning: used for execution and some allocation research; production use is limited by simulator realism and explainability.
• LLMs: extract signals from filings, earnings calls and news; generate backtest code; summarise attribution. They should not place orders without deterministic risk gates.
• Human-in-the-loop: portfolio manager approves mandates, model changes and overrides; compliance owns limits; a senior manager can trigger the kill switch.

Data preparation

• Which data: prices, order books and volumes, fundamental and macro data, reference data and corporate actions, news and filings, own positions, orders, fills and mandate limits.
• Where it lives in an SMB: data-vendor terminals and feeds, the portfolio or order management system, custodian and broker reports, Excel sheets of analysts, mandate documents as PDFs.
• Typical gaps: survivorship bias, missing corporate-action adjustments, inconsistent security identifiers, stale or erroneous prices, limits only described in PDFs, licence terms restricting reuse of vendor data.
• Steps: (1) build a single security master and clean price history; (2) record mandates and limits in machine-readable form; (3) adjust for splits, dividends and delistings; (4) store point-in-time data to avoid look-ahead; (5) extract text from filings and news, see PREP-001 and PREP-011 for tidy tables; (6) check reuse rights of data licences. For internal order and fill history use PREP-005 for exports. Tools: Postgres or Parquet with DuckDB, Great Expectations for validation.
• Share of effort: roughly 50-60 percent of an in-house build, much less with Path A. See GUIDE-003.

Implementation paths

Which path for whom

• Enterprise: B. Portfolio construction is core business; combine licensed market data, cloud compute and your own solver logic so strategy and controls stay in-house without building everything.
• Developer: C. PyPortfolioOpt, Riskfolio-Lib, Qlib and NautilusTrader give transparent, auditable research and backtesting.
• SMB: A. A small asset manager, bank or treasury without a quant team gets tested portfolio, risk and execution tools plus RTS 6 documentation support from established vendors.

Path A — Buy (recommended for most firms)

• Portfolio construction & risk: MSCI Barra / RiskMetrics, Axioma (SimCorp/Deutsche Börse group, EU), BlackRock Aladdin, Bloomberg PORT/AIM, FactSet, Northern Trust/Charles River, SimCorp One (EU).
• Execution: broker algo suites, EMS vendors such as FlexTrade, Bloomberg EMSX, Virtu, and EU providers and exchange-native tools; check best-execution reporting.
• Robo-advice platforms: e.g. EU providers like Scalable Capital and white-label stacks, plus Wealth-tech vendors for banks.
• Choose A when: you are a small asset manager, bank or treasury without a quant team; vendor handles RTS 6 documentation support and data licences.

Path B — Build on cloud APIs

• Data: LSEG, Bloomberg B-PIPE, Refinitiv, EU providers such as Deutsche Börse market data.
• Compute: cloud time-series and tabular ML (Azure ML, Vertex AI, SageMaker, EU regions); frontier LLMs (Claude, Gemini, GPT, Mistral) via EU-region endpoints for text signals and research assistance.
• Brokerage/execution APIs: FIX connectivity via your broker/EMS; Interactive Brokers API for small accounts. Keep order logic deterministic and wrapped with pre-trade controls.
• Solvers: Gurobi, MOSEK, FICO Xpress, or open-source CVXPY with commercial backends.

Path C — Self-host (open source)

• Portfolio optimisation: PyPortfolioOpt, Riskfolio-Lib, cvxpy, scikit-learn, skfolio.
• Backtesting/research: Microsoft Qlib, vectorbt, backtrader, NautilusTrader (event-driven, live-ready), QuantLib for pricing.
• RL: FinRL, Stable-Baselines3, with market simulators.
• Models: open time-series models (Chronos, TimesFM, Lag-Llama), FinBERT-class sentiment, open LLMs (Qwen, Llama, Mistral) via vLLM for text signals.
• Hardware: HW1-HW2 suffices for daily/weekly strategies; GPU only for text/LLM and deep models. Latency-sensitive trading needs co-location, not GPUs.

Path D — Custom training

Worthwhile for proprietary signals and execution models using your own order/fill history. Requires long, clean, survivorship-bias-free data, walk-forward validation, purged cross-validation and capacity analysis. Expect a dedicated quant/ML team and a model-risk function.

State of the art (2026)

• Optimisation: shrinkage covariance estimators, hierarchical risk parity, robust and distributionally robust optimisation, and explicit transaction-cost and market-impact terms are standard practice.
• ML signals: gradient-boosted and deep models give modest, regime-dependent gains; ensemble and strictly out-of-sample evaluation are expected. Time-series foundation models are being benchmarked but have not displaced tuned domain models.
• LLMs and agents: widely used for research summarisation, filings/earnings-call analysis and coding assistance; autonomous LLM trading agents remain experimental and are not accepted for unsupervised use in regulated firms.
• Supervision: ESMA published a supervisory briefing on algorithmic trading and AI in February 2026, pointing national authorities to testing, pre-trade controls, kill functionality and governance of ML-based algorithms [3].
• Limits and failure modes: overfitting and backtest leakage, regime change, crowded trades, feedback loops and flash-crash dynamics, data errors, model drift, and herding when many firms use similar models.

Shared resources & combinations

• Same stack (time-series + tabular-ml + optimization-solver): OPT-004 dynamic pricing, OPT-039 renewable portfolio marketing and algorithmic power trading, PRED-008 price forecasting, PRED-035 settlement-fail prediction.
• Controls and surveillance: ANOM-012 market abuse and trade surveillance should monitor your own algorithms; PRED-009 credit scoring informs credit-risk constraints.
• Text signals: LANG-017 (competitive and market intelligence), LANG-037 (ESG, SFDR portfolio analytics).
• Infrastructure: one data lake and feature store plus a solver service supports many strategies.

Effort, cost & KPIs

• Path A: weeks to months for onboarding; licences are typically the main cost, with data fees on top.
• Path B: 3-9 months for a production pipeline including OMS/EMS integration and controls.
• Path C/D: 6-18 months with a quant team; market data and compliance/testing are the main cost drivers, not compute.
• KPIs: net risk-adjusted return (Sharpe/information ratio after costs), tracking error, turnover, implementation shortfall vs. benchmark, constraint breaches, model drift, kill-switch test results, TCA.

People & change

• Involve early: management (risk appetite, mandate), portfolio managers and traders (users), compliance and legal (MiFID II, RTS 6, MAR), IT and IT security (DORA, access to order systems), the DPO for client data in robo-advice.
• Works council: depends. Trading and attribution systems that log individual decisions and performance are objectively suited to monitor employees, so §87(1) no. 6 BetrVG typically applies where a works council exists; inform it under §90(1) no. 3 and agree how trader performance data may be used. Public institutions: staff council.
• Data protection: market data is non-personal; client profiles and personal accounts are personal data. Conduct a DPIA for robo-advice with automated decisions and conclude AVV with each data or cloud provider.
• AI literacy: support portfolio managers and risk staff in understanding model limits, overfitting and drift (Art. 4 AI Act), see GUIDE-006.
• Pilot and communication: run paper trading in parallel with a small team before any live capital. Typical concerns are accountability and fear of black-box blame; address them with documented model validation, hard limits, a tested kill switch and clear sign-off by the portfolio manager.
• Guides: GUIDE-005, GUIDE-009, GUIDE-014.

Risks, regulation & compliance

• EU AI Act: a typical trading or portfolio-optimisation system is not listed in Annex III and is minimal risk, with AI literacy duties (Art. 4). Creditworthiness scoring of natural persons is high-risk, which matters if credit models feed retail products. No Art. 5 practice is involved, but manipulative or subliminal techniques toward retail clients would be.
• MiFID II Art. 17 and Delegated Regulation (EU) 2017/589 (RTS 6): firms engaged in algorithmic trading need governance, pre-launch testing, pre-trade limits, real-time monitoring, kill functionality, annual self-assessment and notification to the competent authority; trading venues and direct electronic access add obligations [2][3][4].
• MAR: algorithms must not create spoofing, layering or manipulative patterns; link with ANOM-012.
• Suitability and best execution: robo-advice must meet MiFID II suitability duties; AIFMD/UCITS mandates constrain the optimiser. DORA applies to ICT risk and third-party providers of financial entities.
• GDPR: mostly non-personal data; client profiles for robo-advice and personal trading accounts are personal data; automated decisions with significant effects need Art. 22 safeguards.
• Mitigations: independent model validation, walk-forward testing, hard position and order limits, kill switch, full audit log, and no direct LLM-to-order path.

Getting started

1. Define mandate, constraints, benchmark and rebalancing frequency; start with a daily or weekly strategy.
2. Build a clean data layer and an honest backtest (costs, slippage, survivorship bias, walk-forward).
3. Prototype optimisation with PyPortfolioOpt or Riskfolio-Lib against your current allocation process.
4. Run paper trading in parallel and add pre-trade limits and a tested kill switch before any live capital.
5. Document governance for RTS 6 and DORA, and involve compliance and model-risk early.

Sources

1. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
2. Directive 2014/65/EU (MiFID II) - https://eur-lex.europa.eu/eli/dir/2014/65/oj
3. Deloitte - ESMA algorithmic trading supervisory briefing - https://www.deloitte.com/uk/en/blogs/ecrs/mind-the-gaps-why-esma-s-algorithmic-trading-supervisory-briefing-is-a-turning-point-for-firms.html
4. European Commission - RTS 6 (Delegated Regulation (EU) 2017/589) - https://ec.europa.eu/finance/securities/docs/isd/mifid/rts/160719-rts-6_en.pdf
5. PyPortfolioOpt - https://github.com/robertmartin8/PyPortfolioOpt
6. Riskfolio-Lib - https://github.com/dcajasn/Riskfolio-Lib
7. Microsoft Qlib - https://github.com/microsoft/qlib
8. FinRL - https://github.com/AI4Finance-Foundation/FinRL
9. NautilusTrader - https://github.com/nautechsystems/nautilus_trader