Autonomous & assisted ship navigation (collision avoidance, remote-controlled vessels)

Most collisions and near-misses at sea involve human factors: fatigue, a thin watch, targets that radar misses (kayaks, buoys, small craft) and crowded waters. AI-based situational awareness fuses camera, thermal, radar, LiDAR and AIS data, predicts other vessels' trajectories and advises the officer of the watch, or in higher autonomy levels executes COLREG-compliant manoeuvres. Deployments range from bridge decision support on merchant ships to shore-controlled inland vessels and autonomous short-sea ferries.

Problem & value

• Today: the officer of the watch combines visual lookout, radar/ARPA and AIS. Radar clutter, small non-AIS targets, night, fog and watch fatigue limit detection. Inland waterways add dense traffic, bridges and locks; crewing for 24/7 operation is costly and scarce.
• AI value: earlier detection of small/unreported targets, fewer false alarms than raw radar guard zones, objective risk scoring, recorded evidence for incident analysis, and (with remote operation) one shore operator supervising several vessels or relieving on-board crews.
• Evidence: vendor-reported figures exist (e.g. Orca AI states its systems run on 1,000+ commercial vessels; Seafar operates remote-controlled inland vessels in Belgium and Germany). Treat claimed alert-reduction or fuel/safety gains as vendor claims until independently verified [1][4][5].

Variants

• Bridge decision support (SAE-like "advisory"): sensors + edge unit give alerts and suggested manoeuvres; the crew stays fully responsible. Largest installed base.
• Assisted / auto-collision-avoidance: system proposes and, with approval, executes track-pilot changes (e.g. Avikus HiNAS autonomous navigation assist on large commercial ships).
• Remote-controlled inland vessels: shore control centre (SCC) with video, radar and latency-managed control; crew reduced or absent on board (Seafar; CCNR rules for inland navigation).
• Autonomous ferries and short-sea ships: e.g. Yara Birkeland (Kongsberg), Norwegian and Finnish ferry pilots with sensor fusion and automated docking.
• Workboats, leisure and sailing: camera-based collision warning for yachts and small commercial craft.
• Naval and security use: out of scope here (defence level only); see ANOM-032 for maritime domain awareness.

How it works

1. sensors (radar, AIS, visible + thermal camera, LiDAR, GNSS/IMU, ECDIS)
2. time sync + calibration
3. detection & classification (object detection on camera/thermal, radar clutter filtering)
4. multi-sensor tracking & fusion (AIS + radar + vision tracks)
5. trajectory prediction of targets (CPA/TCPA, intent model)
6. COLREG situation assessment (give-way / stand-on, encounter type)
7. manoeuvre planning (rule-based + optimisation / learned planner, own-ship dynamics, depth/charts)
8. human-in-the-loop: advice
9. approval
10. autopilot command
11. safety layer (independent monitor, minimum-risk condition, fallback to manual)
12. logging (voyage-data-recorder-style evidence) Remote variant: video + radar + status over redundant links
13. SCC operator
14. commands
15. on-board safety layer

• Perception: CNN/transformer detectors trained on maritime imagery; thermal cameras extend night/fog performance; radar remains primary for range. Small, non-AIS targets are the key AI gain.
• Prediction and planning: kinematic and learned trajectory predictors; planners such as velocity-obstacle or model-predictive methods encode COLREG rules 5-19. Learned (RL) planners are mostly trained and tested in simulation.
• Human-in-the-loop: the master remains legally responsible; at advisory level the system never acts alone. In remote operation, a deterministic on-board safety layer must handle link loss.

Data preparation

The data is not business documents but sensor streams: radar/ARPA tracks, AIS, visible and thermal video, LiDAR, GNSS/IMU and ECDIS charts. It lives in the voyage data recorder, on the bridge systems and in vendor edge units, rarely in ERP or the DMS. Typical gaps are unsynchronised clocks, missing sensor calibration, no labelled small-target examples (kayaks, buoys, debris), few recordings of rare weather and night conditions, and crew faces on bridge cameras.

Preparation steps: - Record 3-6 months of representative voyages in shadow mode with synchronised time stamps (PREP-005 covers getting data out of closed systems). - Calibrate and document sensor mounting; keep chart and own-ship dynamics data versioned. - Label a test and evaluation set with experienced watch officers (PREP-018, PREP-019), including near-misses and difficult light. - Mask or crop crew and passenger faces before storing video (PREP-022) and define retention (PREP-024, PREP-025). - Build COLREG scenario sets for simulation.

With Path A the vendor brings pretrained models, but you still must provide installation data, shadow-mode recordings and evaluation labels. For Path C or D, data preparation and labelling is the largest share of the effort, roughly 50-60 percent.

Implementation paths

Which path for whom

• Enterprise (shipping companies, ferry and inland operators): Path A, because class-approved bridge systems (Orca AI, Avikus, Seafar) exist and flag-state and class approval are far easier with certified products.
• Developer (maritime tech, research): Path C, because open perception and planning stacks plus simulation (ROBO-010) allow own R&D and sensor-fusion IP on edge hardware.
• Public (port authorities, waterway administrations, public ferries): Path A with certified products, because procurement rules and safety approval favour proven, type-approved systems.

Path A — Buy (recommended)

• Bridge/edge AI situational awareness: Orca AI (SeaPod, 1,000+ vessels), Avikus (HiNAS Control / Navigation Assist), Kongsberg (autonomy and bridge systems; EU/Norway), Wärtsilä (Finland, EU, navigation and integrated bridge), Furuno, Raymarine/Teledyne FLIR (maritime thermal cameras), Sea Machines (US, autonomy kits), Greenroom Robotics (Australia).
• Remote operation: Seafar (Belgium, EU; SCC for inland vessels), Kongsberg, Rolls-Royce Commercial Marine ship intelligence (part of Kongsberg since 2019), Zeabuz (Norway, autonomous urban ferries), ABB Marine (Finland/Switzerland, Dynamic Positioning and Ability marine pilot).
• Choose A when: a fleet operator or shipyard needs a class-approved, supported product. Check class-society notation (DNV, Bureau Veritas, Lloyd's Register, RINA), flag-state acceptance, cybersecurity (IACS UR E26/E27) and EU data-hosting for recorded video.

Path B — Build on cloud APIs

• Cloud is for the shore side only: fleet analytics, offline model improvement, voyage replay and incident review. Vision-language models (Gemini, Claude, Mistral multimodal) can label recorded scenes or summarise near-miss events; they must not be in the real-time control loop.
• Integration: NMEA 0183/2000, IEC 61162-450 and AIS decoders (e.g. open pyais), satellite or 5G links, EU-region storage for recorded footage.

Path C — Self-host (open source)

• Detection/tracking: open maritime datasets (SeaShips, Singapore Maritime Dataset, MODD, MaSTr1325, LaRS) with open-weight detectors (YOLO-family, RT-DETR) and segmentation, trackers (ByteTrack, OC-SORT).
• Autonomy stack: ROS 2 with Nav2, MOOS-IvP (marine autonomy), OpenCPN for chart display, Gazebo/Unity maritime simulators for tests.
• Hardware: HW1 on-board (Jetson Orin-class, industrial IP-rated edge computer, TensorRT); HW2-3 on shore for training and replay. Suitable for research, small unmanned surface vessels and pilots, not for certified SOLAS vessels.

Path D — Custom training

Needed for niche sensors, specific waterways or ship types. Requires thousands of annotated maritime frames across night, glare, fog and sea states, plus labelled radar and AIS encounters. Synthetic data and simulation (ROBO-010) fill rare near-miss cases; real recorded encounters remain the key asset.

State of the art (2026)

• Advisory AI is commercial: camera/thermal/radar fusion with small-target detection and encounter prediction is deployed at scale; reported benefit is mainly earlier awareness and reduced alarm fatigue [1].
• Regulation is catching up: the IMO has been developing a goal-based Maritime Autonomous Surface Ships (MASS) Code, adopted as a non-mandatory instrument at MSC 111 (May 2026, in effect from 1 July 2026), with a mandatory version under SOLAS targeted for adoption around 2030 and entry into force around 2032 (IMO roadmap) [2]. Check the current status before relying on it.
• Inland waters lead: CCNR (Rhine) and EU-level work (CESNI/ES-TRIN standards) enable remote-controlled and automated inland navigation; Seafar and national pilots in Belgium, the Netherlands and Germany [3][5].
• Research direction: COLREG-aware learned planners, trajectory prediction with transformers, and formal verification of the safety layer.
• Limits: poor-weather vision, sun glare and wave clutter; false positives from birds, spray or floating debris; ambiguous multi-vessel encounters where COLREGs are not clear; non-AIS vessels that ignore the rules; sensor failure and GNSS jamming/spoofing; remote-link latency and cyber attacks. Fully autonomous deep-sea commercial navigation is not yet routine.

Shared resources & combinations

• Same perception stack: ROBO-005 (road driving), ROBO-006 (drones), ROBO-019 (rail obstacle detection), ROBO-020 (delivery robots): object detection, tracking, sensor fusion, edge inference.
• Safety patterns: ROBO-016 (independent AI safety functions and certification patterns).
• Simulation: ROBO-010 for COLREG scenario generation and sim-to-real testing.
• Adjacent: OPT-026 (weather/voyage routing supplies the plan, this case executes it locally), ANOM-032 (AIS spoofing and dark-vessel detection), SPEECH-020 (VHF radio transcription for bridge-to-bridge communication).

Effort, cost & KPIs

• Path A: weeks to months for retrofit (cameras, edge unit, integration); cost per vessel scaled by sensor set. Remote operation: quarters to years incl. SCC, class approval and exemptions.
• Path C: 6-12+ months for pilots on small unmanned vessels; cost is mostly integration and sea trials.
• KPIs: small-target detection range and recall by weather/light, false-alarm rate per watch hour, CPA at closest approach, near-miss count, operator workload, remote-link availability and latency, minimum-risk-condition success rate, disengagement/override rate.

People & change

Start with management and the fleet or operations lead (business case, target autonomy level), then involve IT security early (NIS2, IACS UR E26/E27, remote links), legal/compliance (class, flag state, liability) and suppliers (sensor and system vendors). Masters and watch officers are the real end users: involve them in the shadow-mode pilot from day one.

Works council: yes, as a rule. Bridge cameras, recorders and remote-operator logs are objectively suited to monitor crew behaviour and performance, so co-determination under §87(1) no. 6 BetrVG applies; §90(1) no. 3 requires timely information on AI-supported work processes, and §80(3) allows an AI expert. For seafarers on German-flag ships a Bordvertretung or Seebetriebsrat may be responsible; public operators involve the Personalrat. Agree on purpose, no performance evaluation of individuals and short retention.

DPO and DPIA: crew and passengers are visible on video, so involve the DPO and check whether a DPIA (Art. 35 GDPR) is needed; sign an AVV (Art. 28 GDPR) with cloud-connected vendors.

AI literacy: support the development of officers' and remote operators' AI literacy (Art. 4 AI Act), including known failure modes and override training. Communicate that the master stays responsible, run the pilot on one vessel with volunteers, and track override rates openly. Guides: GUIDE-004, GUIDE-008, GUIDE-009.

Risks, regulation & compliance

• EU AI Act (Reg. 2024/1689): classification depends on the context. Sea-going ship equipment sits under the Marine Equipment Directive 2014/90/EU and international conventions; these fall into Annex I Section B, where only limited AI Act provisions (Art. 2(2)) apply to the later delegated acts. AI as a safety component of products under Annex I Section A is high-risk. Critical-infrastructure management (Annex III, point 2) can cover road traffic and critical digital infrastructure but not ships as such. A typical advisory bridge system is therefore sector-regulated first; check the legal position of each deployment [6].
• Art. 5: no prohibited practice by default. Crew monitoring with biometric or emotion recognition (fatigue detection through face analysis) would risk Art. 5(1)(f) in a workplace setting; use non-biometric alternatives.
• GDPR: on-board cameras and remote-operation video can capture crew and passengers; define purpose, minimise recording, works-council agreement, retention limits, EU hosting.
• Maritime rules: SOLAS, COLREG 1972, STCW (human responsibility, watchkeeping), IMO MASS Code (non-mandatory, adopted May 2026; mandatory version planned), classification-society rules for autonomy notations, flag-state approval, CCNR / ES-TRIN for inland vessels, IACS UR E26/E27 and NIS2 for cyber security.
• Liability and risk: responsibility of the master and the operator, remote-operator training, sensor degradation, cyber attacks, over-reliance. Mitigations: independent safety layer, redundant sensors and links, advisory-first deployment, extensive simulation and sea trials, recorded evidence.

Getting started

1. Define the target level: alerts only, advice, supervised execution or remote control, and the waterway (open sea, coast, inland).
2. Record sensor data and AIS from a representative fleet for 3-6 months in shadow mode and measure missed small targets and false alarms.
3. Shortlist 2-3 Path A vendors with class approval, and run a pilot on one vessel with crew feedback.
4. Run a hazard analysis (STPA / HAZID) and agree with class and flag state on the approval path.
5. Build a scenario library in simulation (ROBO-010) of COLREG encounters before moving to higher autonomy.

Sources

1. Orca AI — product and company information — https://www.orca-ai.io
2. IMO — Autonomous shipping (MASS) — https://www.imo.org/en/mediacentre/hottopics/pages/autonomous-shipping.aspx
3. CCNR — Automation and remote navigation on inland waterways — https://www.ccr-zkr.org/
4. Avikus — HiNAS — https://www.avikus.ai
5. Seafar — remote-controlled inland vessels — https://www.seafar.eu
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Kongsberg Maritime — Yara Birkeland autonomous vessel — https://www.kongsberg.com/maritime/
8. Directive 2014/90/EU on marine equipment — https://eur-lex.europa.eu/eli/dir/2014/90/oj