Real-time AI for endoscopy & surgical video

During colonoscopy, gastroscopy and minimally invasive surgery the video stream is the main data source, and clinicians must interpret it live under fatigue and time pressure. Real-time computer vision highlights suspected polyps, suggests their histology, recognises the surgical phase, instruments and critical anatomy, and afterwards turns the recording into a structured report and a quality or training asset. The best-evidenced benefit is a higher adenoma detection rate (ADR) in screening colonoscopy; surgical video analytics is earlier in evidence but growing fast.

Problem & value

• Today: endoscopists spot lesions unaided; detection varies strongly between operators, and missed adenomas are a main cause of interval colorectal cancers. Surgeons dictate operative reports from memory; video is rarely reviewed because manual review takes hours. Variation in technique and OR turnover is hard to measure.
• AI value: a second observer that never tires (CADe), decision support on lesion type (CADx), objective documentation of key safety steps, faster reporting, and searchable video for training and quality.
• Evidence: multiple randomised controlled trials and meta-analyses of CADe colonoscopy report a significant ADR increase over standard colonoscopy, with the effect smaller or less certain in experienced endoscopists and for advanced adenomas; concerns include deskilling and extra false-positive resections [1][2]. ESGE has published a position statement on AI in GI endoscopy [3]. For surgery, phase recognition on benchmarks such as Cholec80 is technically strong, but evidence on patient outcomes is still limited.

Variants

• Colonoscopy CADe (detection): overlay boxes plus sound on suspected polyps; the most mature variant.
• Colonoscopy CADx (characterisation): optical diagnosis (adenoma vs hyperplastic) supporting "resect and discard" or "diagnose and leave" strategies; needs strict validation.
• Upper GI and others: Barrett's neoplasia, early gastric cancer, capsule endoscopy review, bronchoscopy, cystoscopy.
• Surgical video analytics: phase and instrument recognition, critical view of safety in laparoscopic cholecystectomy, bleeding or smoke detection, OR workflow and turnover analytics.
• Operative-report generation: timeline and structured report drafted from video, with surgeon sign-off.
• Frontier: robotic surgery guidance and sub-task automation: anatomy overlays, camera guidance and research systems that autonomously perform steps such as suturing in ex-vivo settings (e.g. SRT-H, Johns Hopkins). The robot platform is a medical device in itself; autonomous surgery in humans is not a market product (see HLTH-037).

How it works

1. endoscope / laparoscope / robot camera
2. video capture (SDI/HDMI frame grabber or vendor video interface, < ~100 ms budget)
3. frame preprocessing (deinterlace, crop, quality gate: blur, bubbles, bile, smoke)
4. on-device inference (detector/segmenter/classifier, temporal model for phases)
5. overlay renderer (boxes, labels, confidence) back to the monitor + optional audio cue
6. human decision (endoscopist/surgeon decides, resects, documents)
7. session recording + metadata (EHR, DICOM, PACS/VNA)
8. offline pipeline: phase timeline, key-event clips, report draft (LLM/VLM), QA dashboards

• Detection and CADx: convolutional or transformer detectors (YOLO/RT-DETR-class) with temporal smoothing and tracking so a polyp is flagged once and stays tracked; a separate classifier handles optical diagnosis.
• Surgical phase recognition: CNN or vision-transformer features followed by a temporal model (TCN, Transformer, e.g. TeCNO/Trans-SVNet-style), trained on frame-level annotated videos.
• Operative reports: phase and event timeline plus structured data feed an LLM that drafts text. The surgeon reviews and signs; nothing is filed unreviewed.
• Human in the loop: the clinician stays fully responsible; AI output is advisory and can be switched off.

Data preparation

The relevant data is live video from the endoscopy tower or OR camera, plus reports in the hospital information system, PACS or an endoscopy documentation system. For the real-time function itself no data preparation is needed, because the certified product ships with its model. Preparation matters for three things: local validation, report drafting and any retraining or research.

Typical gaps: video recorded in different formats and resolutions, no structured link between video, findings and histology, recordings stored on individual workstations or USB drives, missing consent for secondary use, and unclear retention periods.

Preparation steps: (1) decide whether recordings are stored at all and for how long; (2) define a standard export (video plus structured report plus histology result) from the documentation system; (3) build a local validation set of 50 to 100 reviewed procedures with specialist endoscopists, see PREP-018; (4) pseudonymise and cut off patient overlays on frames before any external use, see PREP-022; (5) clean up storage and access rights for video archives, see PREP-009 and PREP-023.

Rough share of effort: about 15 to 25 percent for a pure Path A rollout, clearly higher if the clinic wants own research or retraining. See GUIDE-003 for the readiness check.

Implementation paths

Which path for whom

• Enterprise (hospital groups, private clinics, endoscopy centres): Path A, because CE-marked polyp detection (CADe) and OR video modules integrate with the existing endoscopy tower and carry the MDR responsibility.
• Developer: Path C, because open video models and datasets (for example Cholec80, SAGES-style critical-view datasets) support research and prototypes, but not clinical use without own certification.
• Public (university and municipal hospitals): Path A on-premises or in an EU-hosted setup, because latency, procurement rules and patient data favour certified edge appliances.

Path A — Buy (recommended)

Because real-time use is a regulated medical-device function, buying a CE-marked product is the default. - Colonoscopy CADe/CADx: Medtronic GI Genius, Olympus ENDO-AID (with Odin Vision cloud tools), Fujifilm CAD EYE, Pentax Medical (with Satisfai Health), EndoBRAIN (Cybernet). Pick one that works with your existing scope and processor generation. - Surgical video platforms: Medtronic Touch Surgery, Theator, Caresyntax, Activ Surgical, (Caresyntax is Berlin-based, EU). Check EU hosting, video retention controls and operative-report export. - Choose A when: you are a hospital or endoscopy practice. Procurement is a hospital-level decision.

Path B — Build on cloud APIs

• Real-time inference over the cloud is rarely suitable (latency, patient video leaving the OR, MDR scope).
• Suitable for offline work: video-to-report drafting, de-identified review, search and teaching clips using EU-region multimodal LLMs and speech models (e.g. Gemini, Claude, Mistral in EU regions) under a data processing agreement.
• Pair with VIS-030 style anonymisation (face and room cues) before any video leaves the hospital.

Path C — Self-host (open source)

• Models and code: MMDetection/Ultralytics-style detectors, nnU-Net or SAM-family models for segmentation, PyTorch Video models, public datasets (Cholec80, CholecT50, EndoVis, HyperKvasir, SUN colonoscopy database, PolypGen, Kvasir-SEG).
• Runtime: NVIDIA Holoscan / Clara Holoscan and TensorRT for low-latency streaming, ONNX Runtime, OpenVINO on edge GPUs.
• Hardware: an edge GPU (Jetson Orin / RTX-class, HW1-HW2) in the tower for inference; offline report drafting on HW2-HW3.
• Caveat: research use only. Clinical use of self-built software makes you a manufacturer (see Risks).

Path D — Custom training

Worth it for med-tech manufacturers and research hospitals building a device or a new indication. Needs thousands of expert-annotated procedures across multiple centres, scope vendors and image processors, plus prospective validation. Annotation cost and domain shift are the main hurdles; foundation models pretrained on large unlabelled endoscopy video reduce the labelled data needed.

State of the art (2026)

• Colonoscopy CADe is commercially established with CE marking and FDA clearance for several systems; RCT and meta-analytic evidence supports ADR gains, while effects on interval cancer and long-term outcomes are still open [1][2].
• CADx is moving toward use in leave-in-situ strategies for small distal polyps, but requires high negative predictive value and guidelines are cautious [3].
• Surgical AI focuses on foundation models for surgical video, critical-view-of-safety assessment and automated video-to-report tools; vendors market operating-room analytics, with peer-reviewed outcome data still sparse.
• Autonomy: research robots demonstrate learned autonomous sub-tasks (e.g. ex-vivo suturing and cholecystectomy steps), not clinical products.
• Limits and failure modes: false-positive alerts and alert fatigue, poor performance on bubbles, residual stool, smoke or blood, domain shift between scope generations, automation bias and deskilling, flat-lesion misses, and phase models failing on rare anatomy or complications.

Shared resources & combinations

• Same vision capabilities (detection, segmentation, tracking) as HLTH-001/002 for still-image analysis, but streaming and edge runtime are specific here.
• HLTH-011 for discharge letters, HLTH-021 for contouring, HLTH-037 for robotic surgery, HLTH-004 for decision support.
• VIS-030 for anonymising recorded video before research use or cloud processing.

Effort, cost & KPIs

• Path A: weeks to months for procurement, integration with the tower and PACS/VNA, training and change management; licence is usually per room or per procedure.
• Path B/C: months for offline analytics; a regulated real-time device takes years and a quality system.
• Path D: multi-year, multi-centre programme with clinical studies and notified-body review.
• KPIs: ADR, polyp detection rate, adenomas per colonoscopy, false-positive resection rate, withdrawal time, CADx negative predictive value, report turnaround, share of cholecystectomies with documented critical view, OR turnover time, clinician acceptance.

People & change

Involve management and the medical director for budget and liability. The head of endoscopy or surgery owns the process; endoscopists, surgeons, nurses and OR managers are the daily users and should join the vendor demo and the pilot design. IT and IT security cover the video network and edge hardware; the data protection officer (DPO) joins before any recording is stored.

Works council: depends. Systems that log which physician used the tool, how often they accepted or ignored prompts, or that record OR video of staff are suited to monitor behaviour or performance and trigger co-determination under §87(1) no. 6 BetrVG (objective suitability is enough per BAG). §90(1) no. 3 requires information about AI-supported work processes. In public hospitals the Personalrat applies. Agree in writing that detection statistics are not used for individual performance evaluation.

Data protection: patient video is health data; a DPIA is likely, plus an AVV (Art. 28 GDPR) for cloud or support access. Staff are also visible in OR video.

AI literacy: support staff literacy under Art. 4 AI Act, with training on false positives, missed lesions and automation bias. Communicate that AI is a second pair of eyes and the physician decides. Start with one endoscopy room as pilot group.

Guides: GUIDE-004, GUIDE-005, GUIDE-008.

Risks, regulation & compliance

• MDR (EU 2017/745): real-time diagnostic and guidance software is typically class IIa or IIb (rule 11) and needs CE marking via a notified body. Self-built tools used on patients make the hospital a manufacturer unless the in-house exemption (Art. 5(5)) is met.
• EU AI Act: a medical-device AI that needs third-party conformity assessment is high-risk (Art. 6(1), Annex I); obligations for such products apply from 2 August 2028 (deferred from August 2027 by the Digital Omnibus on AI, Reg. (EU) 2026/1744). Operative-report drafting alone is lower risk but still handles health data. No Art. 5 prohibition applies; avoid using the same video to monitor staff performance in ways that become employee surveillance.
• GDPR: procedure video is health data (Art. 9) and can identify patients or staff; needs a legal basis, DPIA, access control, retention limits and anonymisation for secondary use. Check the European Health Data Space rules for secondary use.
• Clinical risks: automation bias, deskilling, over-treatment from false positives; mitigate through training, audit of AI-on versus AI-off performance, and keeping the clinician's judgement final.
• Liability and consent: inform patients that AI assistance is used; clarify responsibility in the contract.

Getting started

1. Audit the endoscopy fleet: scope and processor models, video interfaces, and PACS/VNA readiness.
2. Pilot one CE-marked CADe system in a few rooms with a baseline ADR audit and an AI-on/AI-off comparison.
3. Run a DPIA for video recording, set retention rules and decide on pseudonymisation.
4. For surgery, start with an offline video-review platform for quality and training before any live guidance.
5. Define KPIs and review monthly with the clinical lead, including false-positive and overtreatment rates.

Sources

1. Hassan C. et al. — Computer-aided detection for colonoscopy: systematic review and meta-analysis (Gut / Clinical Gastroenterology and Hepatology literature) — https://pubmed.ncbi.nlm.nih.gov/?term=computer-aided+detection+colonoscopy+adenoma+detection+rate+meta-analysis
2. Mori Y., Bretthauer M. et al. — Artificial intelligence for colonoscopy: evidence and caveats — https://pubmed.ncbi.nlm.nih.gov/?term=artificial+intelligence+colonoscopy+randomized+trial
3. ESGE — Artificial intelligence in gastrointestinal endoscopy: position statement — https://www.esge.com/
4. Medtronic — GI Genius intelligent endoscopy module — https://www.medtronic.com/covidien/en-us/products/gastrointestinal-artificial-intelligence/gi-genius-intelligent-endoscopy-module.html
5. Cholec80 / CAMMA surgical data science group (Strasbourg) — https://camma.unistra.fr/datasets/
6. NVIDIA Holoscan for medical devices — https://developer.nvidia.com/holoscan-sdk
7. Regulation (EU) 2017/745 (MDR) — https://eur-lex.europa.eu/eli/reg/2017/745/oj
8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
9. Kim J.W. et al. — Surgical Robot Transformer (SRT-H), autonomous surgical sub-tasks — https://arxiv.org/abs/2505.10251