Trains cannot swerve and need hundreds of metres to stop, so human drivers only see far-enough obstacles by luck. Automated train operation (ATO) lets software drive within the signalling system, while AI perception (camera, LiDAR, radar) finds people, vehicles, animals and fallen objects on the track early enough to brake. Railways pursue it for capacity, energy-efficient driving, driver shortages and safer depots; the hard part is proving that a learned perception function is safe enough for certification.
Problem & value
- Today: conventional mainline trains are driven by a person who also acts as the obstacle sensor. Depot moves (shunting, stabling, washing, coupling) are manual and slow. Metros with CBTC are often already highly automated, but they rely on closed, fenced infrastructure and do not need to see the track ahead.
- Why AI helps: classic ATO controls speed against signalling data, but cannot detect something not in the data. ADAS-type perception adds the missing "eyes": detection, classification and track-relevance assessment at long range, in rain, fog, night and tunnels.
Variants
- Depot and yard automation (GoA4-like, low speed, closed area): remote or driverless shunting, stabling and train preparation; the earliest commercially viable variant (depot automation programmes by Siemens Mobility and DB [2][3]).
- Mainline ATO GoA2 (driver supervises): the train accelerates and brakes automatically, the driver closes doors and watches the track. ETCS-based ATO over ETCS is specified in the EU TSI CCS [4].
- Mainline GoA3/GoA4 (driver-less) with remote supervision: the German AutomatedTrain project and the Hamburg S-Bahn ATO project (Siemens Mobility/DB) are reference programmes; obstacle detection is the key enabler [1][3].
- Urban rail/metro (CBTC, GoA3-4): obstacle detection mainly at platforms, level crossings and open sections.
How it works
- sensors (camera RGB/thermal, LiDAR, radar) + track map + signalling state
- sensor fusion + track/rail detection (where is my track, including switches and curves?)
- object detection & classification (person, vehicle, animal, debris) + tracking
- relevance check (is it in the clearance gauge on my path? distance, closing speed)
- safety supervision (independent, certified brake-curve and speed limits)
- ATO driving command or warning
- train control & brakes
- handover to driver / remote operator control centre; event logging
- Safety: AI perception sits beside a certified conventional chain (ETCS/CBTC, brake supervision); it warns or restricts speed.
- Perception: CNN/transformer detectors and segmentation models on camera images, LiDAR clustering and radar tracks; thermal cameras for night and fog. Rail-specific models for rail line extraction keep detection restricted to the relevant corridor. Long range (several hundred metres to over a kilometre) is the main technical challenge because objects shrink to a few pixels.
- Human-in-the-loop: driver on GoA2; remote operator (teleoperation or supervision) on GoA3/4; fallback to a minimal-risk condition (stop) and manual handling of unknown objects.
Data preparation
Rail perception needs far more than business data. Sources: on-board and trackside camera, lidar and radar recordings; train and event data from the vehicle bus and diagnostics; track and infrastructure data (route atlas, signalling plans, GIS); incident and obstacle reports in the operator's safety management system; and driver reports, often on paper or in mail. Typical gaps: recordings are unlabeled and dominated by uneventful runs, rare obstacles and bad weather are missing, sensor data from different vehicle series is inconsistent, time synchronisation between sensors is poor, and recordings contain persons (passengers, staff, trespassers) and need access rights.
Preparation steps: define the operational design domain and data collection campaigns per route and weather; build a labeling scheme for obstacle classes (PREP-019); label with domain experts and keep a test set per season and route (PREP-018); digitise incident reports (PREP-006); blur or discard persons and define retention (PREP-022); add synthetic edge cases from simulation (ROBO-010). Tools: CVAT, Label Studio, ROS 2 bag tooling. Data work is the largest cost block, roughly 50 percent or more, and must be documented for the safety case (GUIDE-003).
Implementation paths
Which path for whom
- Enterprise: A, because railway operators need certified, type-approved systems from established rail suppliers; a homemade safety function would need a multi-year safety case.
- Developer: C, because perception research and prototypes rely on open detectors, datasets and simulation before any certified product.
- Public: A, because public transport authorities and state-owned operators procure certified products through tenders with safety acceptance by the authority.
Path A: Buy (recommended)
Rail operators buy from railway system suppliers; SMBs (private railways, industrial and port rail) buy depot or trackside solutions. - Rolling-stock/signalling suppliers (EU marked): Siemens Mobility (DE, ATO over ETCS, automated trains, Hamburg S-Bahn), Alstom (FR, ATO, autonomous train programmes with SNCF), Hitachi Rail, Thales (FR, signalling/CBTC and ATO), Stadler (CH), Wabtec/Progress Rail for freight. - Obstacle-detection specialists: Rail Vision (IL, on-board AI detection, mainline pilots), Hitachi Rail, LiDAR/radar suppliers such as Sick and Bosch, Teledyne FLIR (thermal). - Choose A when: you operate trains; certification, homologation and liability sit with the supplier and the infrastructure manager.
Path B: Build on cloud APIs
Safety-critical inference cannot depend on a cloud link. The cloud is useful for offline work: data lake and annotation (CVAT, Labelbox), VLM-assisted pre-labelling of rare events (e.g. Gemini, Claude or Qwen-VL on EU regions), model training on EU GPU clouds, fleet analytics and event triage.
Path C: Self-host (open source)
- Models/tools: YOLO-family and RT-DETR detectors, SAM-family segmentation, open datasets such as RailSem19, OSDaR23 (multi-sensor rail dataset published by DB/Fusion Perception) and FRSign for signal recognition [5].
- Stack: ROS 2, PyTorch, ONNX Runtime, TensorRT on NVIDIA Jetson/DRIVE class modules, Autoware-style perception modules adapted to rail, CARLA/Gazebo-type simulators for scenario tests (see ROBO-010).
- Useful for research, prototypes and shadow-mode evaluation; not a certified safety function by itself.
Path D: Custom training
Almost always required for production: train and validate detectors on your own routes, rolling stock and sensor placement. Needs tens of thousands of annotated frames covering seasons, weather, night, tunnels, vegetation and rare obstacle classes, plus synthetic and staged data (people on tracks, objects at distance). Statistical evidence of detection performance and a defined operational design domain (ODD) are part of the safety case.
State of the art (2026)
- Programme level: Europe's Rail and national programmes (DB/Siemens ATO, AutomatedTrain, SNCF/Alstom) moved from lab to demonstrator/pilot operation; GoA2 ATO over ETCS is the nearest-term commercial step; fully driverless mainline operation remains limited to pilots and closed or low-speed settings [1][3][4].
- Perception: multimodal fusion (camera + LiDAR + radar + thermal) with transformer detectors and foundation-model-assisted labelling; open multi-sensor datasets (OSDaR23) enable benchmarking [5].
- Standards: CENELEC EN 50126 (RAMS lifecycle), EN 50128 (software), EN 50129 (safety electronics), EN 50716 (consolidated software standard replacing 50128/50657), IEC 62267 (automated urban guided transport). How to certify machine-learning components is still open; the ISO/PAS 8800 and ISO/TR 5469 work from road/functional safety is used as guidance, and rail-specific guidance is emerging.
- Limits: long-range small objects, occlusion in curves, darkness/glare/snow, false alarms that cause harsh braking and disruption, rare classes (animals, debris, vandalism), sensor soiling, and distribution shift between routes. No public evidence that AI alone achieves safety integrity level 4.
Shared resources & combinations
- Same stack as road autonomy (ROBO-005): detection, tracking and fusion models, annotation tools and simulation; rail needs longer range, fixed track geometry and different certification.
- ROBO-016 (AI safety functions, person detection) shares the safety-case methodology; ROBO-010 (simulation/digital twins) for scenario testing; ROBO-006/VIS-003 drones for track inspection.
- OPT-020 and PRED-006 reuse the same train and event data.
Effort, cost & KPIs
- Path A: years-long programmes; for depots 6-18 months from purchase to operation, mainline much longer.
- Path C/D: a perception prototype takes months; a certified function needs a multi-year safety case, data collection campaigns and homologation. Cost drivers: sensors, data collection and annotation, independent safety assessment (ISA), route-specific validation.
- KPIs: detection range and recall per obstacle class and weather, false-alarm rate per 1,000 km, emergency-braking events, availability and punctuality, interventions per hour (handover rate), hazardous-event rate vs. safety target, energy per km.
People & change
Involve management, the head of operations and the safety organisation from day one, together with the supplier and the national authority (EBA in Germany) for acceptance. IT-security covers remote operation and NIS2; legal/compliance covers liability and approval; train drivers and dispatchers are the key end users.
Works council: yes. On-board cameras, driver monitoring and event logs are objectively suited to monitor behaviour or performance, so §87(1) no. 6 BetrVG applies (public operators: Personalrat under BPersVG or state law); §90(1) no. 3 requires early information about changed work processes and §80(3) allows an expert (GUIDE-004). Involve the DPO and run a DPIA for recordings of persons (GUIDE-005).
Support AI literacy (Art. 4 AI Act) with training for drivers and dispatchers on system limits, handover and degraded modes (GUIDE-006). Typical concerns: deskilling, job loss, liability after an incident. Address them with a clear role description (driver or attendant remains responsible for supervision at lower automation grades), a staged rollout on a depot or siding first, a shadow-mode phase and transparent incident reporting (GUIDE-008, GUIDE-009). Humans stay in control through defined handover rules and the ability to stop the train.
Risks, regulation & compliance
- EU AI Act: Annex III point 2 (critical infrastructure) names road traffic, not rail, so rail is mainly covered through Annex I Section B (Directive (EU) 2016/797): the AI Act applies there only via Art. 2(2) and requirements taken into account in future delegated/implementing acts under that sectoral law. Treat as high-risk-equivalent and plan for risk management, data governance, logging, human oversight (risk management, data governance, logging, human oversight). Not legal advice; classification depends on the concrete function [6]. No Art. 5 prohibited practice is inherent, but passenger-facing camera analytics (e.g. behaviour or emotion analysis, biometric ID) must be kept out of scope or checked against Art. 5.
- Rail law: Railway Safety Directive (EU) 2016/798, Interoperability Directive (EU) 2016/797, TSI CCS, CSM on risk evaluation (Regulation (EU) 402/2013), national authority (e.g. EBA in Germany) approval, ERA authorisation for placing in service; new vehicles also trigger the EU Machinery/ Radio rules where relevant.
- GDPR: on-board and trackside cameras record people (passengers, staff, trespassers, bystanders). Use purpose limitation, DPIA, blur/discard non-event footage, short retention, and avoid identification. Works-council involvement for staff-visible recordings (in DE).
- Risks: false negatives (collision), false positives (disruption), ODD creep, cyber-attacks on remote operation (NIS2, CRA), liability and driver deskilling. Mitigations: independent safety supervision, defined ODD, staged GoA rollout, shadow mode, secure remote links, driver training.
Getting started
- Define the use case and ODD: depot, siding, mainline GoA2 or trackside crossing; speed, lines, weather.
- Start in shadow mode: record sensor data, run detection offline and compare with driver reports.
- Evaluate public datasets (OSDaR23, RailSem19) and 1-2 vendors against your own recorded routes.
- Involve the safety assessor and national authority early; plan the safety case per EN 50126.
- Pilot in a closed area (depot) before extending to the mainline; define handover and fallback.
Sources
- Europe's Rail Joint Undertaking, programme overview (ATO, automation, innovation pillars) — https://rail-research.europa.eu/
- Siemens Mobility, automated and autonomous rail solutions — https://www.mobility.siemens.com/global/en/portfolio/rail/automation.html
- Deutsche Bahn, AI and digitalisation in rail operation — https://digitale-schiene-deutschland.de/en
- European Union Agency for Railways (ERA), TSI CCS and ATO over ETCS specification — https://www.era.europa.eu/domains/technical-specifications-interoperability_en
- OSDaR23 and RailSem19 rail datasets — https://data.fid-move.de/dataset/osdar23 and https://wilddash.cc/railsem19
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Directive (EU) 2016/798 on railway safety — https://eur-lex.europa.eu/eli/dir/2016/798/oj
- Rail Vision, on-board obstacle detection systems — https://railvision.io/