AI-assisted robotic surgery (autonomous camera control & surgical sub-task automation)

In laparoscopic surgery a human assistant holds the camera and retracts tissue while the surgeon operates, and fatigue or miscommunication affects consistency. Surgical robots increasingly add AI that keeps the view centred on the instruments, tracks tools, limits forces and, in research, performs supervised sub-tasks such as suturing. Camera and instrument assistance is in clinical use; autonomous manipulation of tissue is still emerging and mostly preclinical.

Problem & value

• Today: a bedside assistant positions the laparoscope and retracts organs on verbal instructions; teleoperated systems (e.g. da Vinci) have the surgeon switch between instruments and camera by pedal. Staffing, ergonomics and variability between assistants limit consistency.
• AI value: hands-free, stable and surgeon-centred view; fewer camera interruptions; reduced dependence on a second assistant; guardrails (no-go zones, force limits) that reduce tissue damage risk; structured data on every procedure that supports training and quality benchmarking.
• Evidence status: published clinical series for camera/co-manipulation systems are mostly feasibility and safety studies; comparative outcome benefits over conventional laparoscopy or standard robotic surgery are not established in large trials. Full autonomy has been shown on ex vivo tissue (Johns Hopkins SRT-H, 2025) and not as routine human practice [3][4].
• This is not video analytics: unlike HLTH-022 (analysing surgical video), the AI here physically acts on or near the patient.

Variants

• Level 1-2 assistance (marketed): autonomous scope positioning that follows instruments, tool tracking, tremor filtering, motion scaling, haptic or force-limit guardrails.
• Level 3 task autonomy (emerging): supervised execution of defined sub-tasks, e.g. tissue retraction, suturing or knot-tying, needle handling; surgeon approves each step and can take over.
• Procedure-level autonomy (research): autonomous steps of an entire procedure (ex vivo cholecystectomy, animal anastomosis studies such as STAR); no routine human use.
• Specialities: general/abdominal, urology, gynaecology, orthopaedics (bone-cutting robots with haptic boundaries), spine and neuro (navigation-guided trajectories), ENT.

How it works

1. laparoscopic video + robot kinematics + force sensing
2. perception (instrument detection/tracking, tissue segmentation, depth/pose estimation)
3. scene and phase understanding (what step of the procedure, where is the target)
4. planning (camera pose, retraction trajectory, suture path; constraints and no-go zones)
5. safety layer (workspace limits, force/speed caps, uncertainty check, watchdog)
6. motion execution (real-time controller on the robot)
7. surgeon supervision (confirm, override, take manual control at any time)
8. logging for post-hoc review and regulatory post-market surveillance

• Perception: CNN/transformer detectors and segmenters trained on annotated surgical video; stereo or monocular depth for 3D positioning.
• Control: classical visual servoing for camera following; imitation learning or reinforcement learning for manipulation sub-tasks, often trained in simulation and fine-tuned on tissue.
• Hierarchical autonomy: SRT-H uses a high-level language-conditioned policy that plans steps and a low-level policy that executes motions, with the ability to recover from errors [4].
• Human-in-the-loop: the surgeon stays responsible, approves autonomous steps, and can override instantly (hard-wired stop independent of the AI). Inference runs on-device in the OR for latency and safety; deterministic safety code is separate from learned components.

Data preparation

The relevant data are laparoscopic video, robot kinematics and force logs, and preoperative CT/MRI. In a hospital they sit in the robot's own storage, the PACS/VNA, the OR video recorder and the hospital information system. Buyers of a certified system mostly need little preparation of their own; the manufacturer trains the models. Hospitals still need to arrange export and retention of procedure logs for quality registries.

Typical gaps for developers and research partners: video without annotations, inconsistent phase and instrument labels between centres, missing synchronisation of video and kinematics, identifiable patients and staff in recordings, unclear consent for secondary use, and access rights spread across departments.

Preparation steps: inventory sources and consent status, de-identify video (blur or crop faces and room views, remove metadata, pseudonymise case IDs), synchronise time stamps, define a label scheme with surgeons before annotation, build a held-out test set from other centres, and use public datasets (CholecT50, Endoscapes) for first experiments. Annotation by clinical experts is the largest cost, and data work typically takes well over half of a developer project. See PREP-001 (readiness assessment), PREP-019 (category schemes), PREP-018 (evaluation sets), PREP-022 (pseudonymisation) and PREP-025 (purpose and legal-basis check). GUIDE-003 gives a data readiness check.

Implementation paths

Which path for whom

• Enterprise (hospital): Path A, because only a CE-marked, MDR-certified system may be used on patients and no hospital can certify its own autonomous system.
• Developer (manufacturer or research group): Path C, because open research platforms (dVRK, ROS 2, Isaac, public datasets) allow prototypes on phantoms and ex vivo tissue; clinical use later requires a regulated manufacturer.

Path A: Buy (the realistic route for hospitals)

• Scope/camera assistance: Moon Surgical Maestro with ScoPilot (AI-driven autonomous laparoscope positioning, works with standard laparoscopic instruments; Maestro is CE-marked under the MDR (2023) and FDA-cleared, ScoPilot FDA-cleared in March 2025, so check the CE status of ScoPilot; French company, EU) [1].
• Teleoperated platforms with AI/data features: Intuitive da Vinci 5 (force feedback, analytics), Medtronic Hugo, CMR Surgical Versius (UK), Distalmotion Dexter (CH), Senhance (now Karl Storz, ex-Asensus; camera eye-tracking control), Zimmer Biomet ROSA and Stryker Mako for orthopaedics, Brainlab/Medtronic navigation for spine and neuro. Procurement is capital equipment with service contracts, training and proctoring.
• Choose A: always for clinical use. No hospital can deploy an uncertified autonomous system; a purchase decision rests on clinical evidence, surgeon training, OR workflow and total cost.

Path B: Build on cloud APIs

• Not suitable for the control loop (latency, certification). Cloud is fine for offline annotation, analytics and simulation in EU regions under a DPA with de-identified OR video.

Path C: Self-host (open source, research)

• Research platforms: dVRK (da Vinci Research Kit) for research use of retired da Vinci hardware; ROS 2 with MoveIt; NVIDIA Isaac for Healthcare / Isaac Sim and Holoscan for real-time sensor pipelines; ORBIT-Surgical and SurRoL simulators; open datasets such as CholecT50, Endoscapes and SAR-RARP50 for perception [5].
• Models: open segmentation and tracking models (SAM-family, YOLO-family) fine-tuned on surgical video; diffusion/transformer imitation policies (ACT, Diffusion Policy) as baselines for sub-tasks.
• Hardware: HW3-class GPU (or an embedded Jetson/Holoscan device in the robot) plus a robot arm or dVRK. Research use only on phantoms and ex vivo tissue; clinical use needs a regulated manufacturer.

Path D: Custom training

Core of what manufacturers do. Requires large annotated surgical video corpora with ethics and consent frameworks, simulation for safe policy learning, ex vivo and animal validation, and a clinical investigation under MDR. Data are the moat; multi-centre collection with federated or de-identified sharing is common. Typical programmes run for years.

State of the art (2026)

• Camera and co-manipulation autonomy is commercial: AI-driven scope positioning that follows the surgeon's instruments exists as a CE-marked product [1].
• Task autonomy research: STAR (2022) performed autonomous intestinal anastomosis in pigs; SRT-H (Science Robotics, 2025) executed key steps of cholecystectomy autonomously on ex vivo porcine gallbladders using imitation learning with a language-conditioned hierarchy [4]. These are controlled laboratory results, not clinical equivalence.
• Vision-language-action (VLA) models are being adapted for surgery (see ROBO-004), and simulation plus synthetic data are used to reduce animal and cadaver work (ROBO-010).
• Limits and failure modes: anatomical variability, bleeding and smoke that obscure the view, tissue deformation, distribution shift between hospitals, rare events, and unclear fault attribution. Learned policies lack formal safety guarantees, so deterministic safety envelopes and instant override remain mandatory. Autonomy levels (L0-L5 frameworks proposed in the literature) help structure claims [2].

Shared resources & combinations

• HLTH-022: the same perception stack (phase recognition, instrument segmentation, operative-report drafts) feeds the robot's scene understanding; combine for closed-loop assistance plus documentation.
• HLTH-004: intra-operative decision support and risk alerts; HLTH-021: auto-contouring and planning share segmentation models and trajectory planning; HLTH-024: AI-guided ultrasound for intra-operative imaging; HLTH-032: neural and myoelectric interfaces share closed-loop control concerns.
• ROBO-004 / ROBO-010 / ROBO-003: VLA policies, sim-to-real training and demonstration learning transfer from general robotics; ROBO-016: safety-function concepts (speed and force limiting).

Effort, cost & KPIs

• Path A (hospital): capital cost of a robot plus per-procedure consumables, service contract, surgeon and OR-team training; months for adoption, quarters to reach stable throughput.
• Path C/D (developer or manufacturer): multi-year, multi-million budgets for data, hardware, clinical investigation and notified-body certification; an academic prototype is feasible in months.
• KPIs: operative and console time, number of manual camera corrections, conversion to open surgery, complication rates (e.g. Clavien-Dindo), blood loss, length of stay, intervention rate (how often the surgeon overrides), assistant staffing needs, learning curve, and system malfunction reports.

People & change

Hospital: involve management (investment, liability), the surgical department lead and the OR team early, then IT, medical-device safety officer and IT security (MDR, MDCG 2019-16, NIS2), legal/compliance (liability, consent wording) and the data protection officer. Health data from video and logs require a DPIA (Art. 35 GDPR) and processor agreements (Art. 28 GDPR) with the manufacturer; see GUIDE-005.

Works council: yes. Robot and OR logs record the actions and performance of surgeons and staff, so the system is objectively suited to monitor behaviour or performance and triggers co-determination under §87(1) no. 6 BetrVG; §90(1) no. 3 requires early information when work processes change, and an AI expert may be engaged under §80(3). In public hospitals the staff council (Personalrat) applies. Agree on a works agreement that limits use of logs for performance evaluation (GUIDE-004).

AI literacy and training: Art. 4 AI Act obliges deployers to take measures that support staff AI literacy; combine this with manufacturer training and proctoring (GUIDE-006). Start with a small pilot team of volunteer surgeons, track intervention rate openly, and address concerns (deskilling, liability, job loss for assistants) in dialogue (GUIDE-008). The surgeon always stays responsible and can override at any time; patients are informed about the level of automation.

Risks, regulation & compliance

• EU MDR (2017/745): surgical robots are active therapeutic devices, typically class IIb or III; require notified-body conformity assessment, clinical evaluation or investigation, ISO 13485, IEC 60601-1 and 80601-2-77 (robotically assisted surgical equipment), IEC 62304, ISO 14971 and IEC 62366. Novel autonomous functions may need clinical investigation [6].
• EU AI Act (2024/1689): a medical device requiring third-party conformity assessment, with AI as a safety component, is high-risk under Art. 6(1) and Annex I (MDR). Obligations (risk management, data governance, logging, human oversight Art. 14, accuracy and robustness) apply with the extended timeline for Annex I products; check the current Digital Omnibus status. Not an Art. 5 prohibited practice [7].
• GDPR: surgical video and robot logs are health data (Art. 9); need legal basis, DPIA, retention limits, de-identification for training, and controls for vendor access and cloud transfer.
• Product liability and safety: revised Product Liability Directive (EU 2024/2853) covers software and AI; unclear responsibility between surgeon, hospital and manufacturer for autonomous steps must be settled in contracts and consent forms; patients must be informed about the level of automation. Cybersecurity (MDCG 2019-16, NIS2 for hospitals) is mandatory.
• Mitigations: staged autonomy with supervised approval, independent safety controller, hard stop, prospective registries, simulation-based validation, post-market surveillance, and incident reporting under MDR vigilance.

Getting started

1. Define the goal: hospital evaluating assistance systems (Path A) or research team building a perception/control prototype (Path C).
2. Hospitals: run a structured health-technology assessment on CE-marked camera/co-manipulation systems; review the clinical evidence and the vendor's MDR certificate and AI Act roadmap.
3. Plan training, OR workflow, consent wording and a data protection impact assessment.
4. Researchers: start with dVRK or a simulator (ORBIT-Surgical/Isaac) and public surgical datasets; reproduce a single sub-task (e.g. camera following or suture-needle handling) on phantoms.
5. Track intervention rate and complications from day one and report incidents through MDR vigilance.

Sources

1. Moon Surgical, Maestro system with ScoPilot — https://www.moonsurgical.com
2. Yang et al., Medical robotics: regulatory, ethical, and legal considerations for increasing levels of autonomy, Science Robotics 2017 — https://www.science.org/doi/10.1126/scirobotics.aam8638
3. Saeidi et al., Autonomous robotic laparoscopic surgery for intestinal anastomosis (STAR), Science Robotics 2022 — https://www.science.org/doi/10.1126/scirobotics.abj2908
4. Kim et al., SRT-H: A hierarchical framework for autonomous surgery via language-conditioned imitation learning, Science Robotics 2025 — https://www.science.org/doi/10.1126/scirobotics.adt5254
5. dVRK, da Vinci Research Kit — https://github.com/jhu-dvrk/sawIntuitiveResearchKit
6. Regulation (EU) 2017/745 (MDR) — https://eur-lex.europa.eu/eli/reg/2017/745/oj
7. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
8. Directive (EU) 2024/2853 (Product Liability) — https://eur-lex.europa.eu/eli/dir/2024/2853/oj
9. Surgical Robotics Technology — Moon Surgical receives FDA clearance for ScoPilot on Maestro (Mar 2025) — https://www.surgicalroboticstechnology.com/news/moon-surgical-receives-fda-clearance-for-scopilot-on-maestro/
10. The Robot Report — Moon Surgical picks up CE mark for updated Maestro (Sep 2023) — https://www.therobotreport.com/moon-surgical-picks-up-ce-mark-for-updated-maestro-surgical-robot/