Remote electronic monitoring (REM) of fishing vessels with AI review of catch & discard footage

Human observers on fishing vessels are expensive, cover only a tiny share of trips and are not always welcome onboard. Remote electronic monitoring (REM) installs tamper-evident cameras and sensors instead, but the resulting video is enormous and reviewing it manually is the bottleneck. AI detects hauling events, identifies and counts species and flags possible discards, so that reviewers only inspect flagged segments. The revised EU Fisheries Control Regulation makes REM mandatory for defined high-risk vessels, turning this from pilot into compliance infrastructure.

Problem & value

• Today: compliance with the EU landing obligation (discard ban) is checked via port inspections, at-sea patrols, logbook cross-checks and a small number of human observer trips. Coverage is low and observer data is costly. Pilot REM programmes (e.g. Denmark, Netherlands, Germany, Ireland, Spain) produced video that human analysts review manually, typically a sample of trips or hauls.
• AI value: automated event detection (haul start/stop, catch on conveyor, discard chute activity) cuts the video that humans must watch; species classification supports logbook reconciliation; consistent review criteria across analysts; faster feedback to skippers and authorities.
• Regulatory driver: Regulation (EU) 2023/2842 amends Control Regulation (EC) No 1224/2009 and introduces mandatory REM (camera-based) for vessels in categories with a high risk of non-compliance with the landing obligation, plus a phased framework for other vessels. EFCA published technical guidance on REM specifications and data handling [1][2]. Exact vessel categories and deadlines should be verified in the regulation text and national implementing rules.
• No verified quantitative accuracy or savings figure is quoted here; published pilots report varying accuracy by species and camera set-up, so measure on your own fleet footage.

Variants

• Authority-run review centre: national fisheries control agency ingests data from many vessels.
• Fleet / producer organisation self-monitoring: operators use REM for certification (MSC), due diligence and bycatch documentation, sharing results with authorities.
• Fishery-specific: trawl (conveyor/sorting), purse seine, longline (hook and bycatch events, protected species), gillnet/pot (haul and bycatch counts).
• Bycatch and protected species: sea birds, marine mammals, turtles (e.g. under the Habitats and Birds Directives and MSFD), overlapping with VIS-032.
• Outside the EU: US, Canada, Australia, New Zealand, Pacific tuna fleets run larger REM programmes.

How it works

1. cameras + sensors on vessel (GPS, hydraulics, drum rotation)
2. onboard recorder (tamper-evident, encrypted storage, event-triggered recording)
3. offload at port / satellite or 4G upload of selected clips
4. AI pipeline: haul detection (sensor + video)
5. fish/discard detection and tracking
6. species classification and counting
7. event log
8. rules: compare video-derived events with e-logbook, flag anomalies (unreported discards, camera obstruction, gear events)
9. human reviewer checks flagged segments (+ random audit sample)
10. compliance report
11. authority / operator case handling

• Detection and tracking: YOLO-class or transformer detectors with multi-object tracking follow individual fish across conveyor or deck; counting happens per track, not per frame.
• Species identification: fine-grained image classifiers, often hierarchical (group, then species), because fish are occluded, wet, piled up and under variable lighting; a long tail of rare species needs open-set handling ("unknown" goes to humans).
• Sensor fusion: hydraulic and winch sensors define hauls cheaply and reliably; AI is used inside those time windows.
• Camera integrity: models detect blocked, dirty, moved or switched-off cameras.
• Human-in-the-loop: reviewers confirm every potential infringement; AI output alone is an indicator, not evidence of an offence. Random audits of unflagged video measure missed events.
• Edge vs. shore: edge inference on the recorder (e.g. NVIDIA Jetson-class) reduces upload; heavy species models run ashore.

Data preparation

The main data is multi-camera deck and processing-line video plus sensor streams (GPS, hydraulics, drum rotation), e-logbook declarations and VMS/AIS tracks. Video sits on tamper-evident recorders and is offloaded at port or by satellite/4G; logbooks live in national control systems or the operator's fleet software; species reference images and quota rules are usually scattered PDFs and expert knowledge.

Typical gaps: dirty or misaligned cameras, unsynchronised timestamps between video and sensors, no species labels, a long tail of rare species, and strict access rights because crew members are visible on the footage.

Preparation steps: synchronise clocks and map sensor events to haul windows; build a species catalogue and label a representative sample per fishery with fisheries experts (CVAT or Label Studio); keep a held-out set of fully human-reviewed trips for evaluation (see PREP-018 and PREP-020 for harvesting labels from logbooks); define retention, masking of faces and role-based access before any AI processing (PREP-022, PREP-025); inventory data sources and gaps (PREP-001, PREP-002, PREP-005).

Rough share of effort: high, typically more than half of a Path B/C project is labelling, validation and data governance; for Path A it shrinks to onboarding, camera placement and access rules.

Implementation paths

Which path for whom

• Public (control authorities): C, because footage of crews is sensitive, sovereignty and procurement rules favour open, auditable detectors and review tools hosted in an EU or German government cloud; certified products (A) remain an option where the authority prescribes them.
• Enterprise (fleet operators): A, because certified REM hardware with tamper evidence, EFCA-conformant review software and service is a standard offer that operators cannot sensibly rebuild.
• Developer: C, because open detectors and trackers on HW2 edge/shore hardware plus own labelled data give full control over species models per fishery.

Path A — Buy (recommended for fleet operators)

• REM system and analytics providers: Anchor Lab (Denmark, EU), Satlink (Spain, EU), Marine Instruments (Spain, EU) offer onboard systems with software for review; Archipelago Marine Research (Canada) and Saltwater Inc. (US) are established non-EU REM providers. Several are adding or piloting AI-assisted review; verify current feature status in demos.
• Check: compliance with EFCA specifications, tamper evidence, encryption, GDPR privacy features (masking, access logging), integration with national control systems and e-logbook.
• Choose A when: you need certified hardware plus software and a review workflow quickly.

Path B — Build on cloud APIs

• Use cloud object storage in EU regions for clips; managed training/inference (Azure ML, Google Vertex AI, AWS SageMaker, EU sovereign clouds such as OVHcloud, Scaleway, STACKIT) for detectors.
• Multimodal LLMs/VLMs (Gemini, Claude, GPT, Mistral) can generate clip summaries or answer questions about a segment ("was anything discarded?"), but are not reliable enough for species counting; use them for triage and reviewer assistance only.
• Good fit for authority analytics layers that sit on top of vendor-provided REM data.

Path C — Self-host (open source)

• Detection/segmentation/tracking: Ultralytics YOLO (AGPL, check licence terms), RT-DETR, SAM 2 for segmentation-assisted labelling, ByteTrack / BoT-SORT trackers; DINOv2 or CLIP-type backbones for species classifiers. Labelling with CVAT or Label Studio.
• Serving: TensorRT/ONNX Runtime, NVIDIA DeepStream for video pipelines; open VLMs such as the Qwen-VL family via vLLM for clip captioning.
• Hardware: HW1 edge devices on vessels for event detection; HW2 (one 24 GB GPU) shore-side for review pipelines of a small fleet; HW3 for national-scale processing.
• Effort: needs ML engineers and fisheries scientists; ongoing re-labelling.

Path D — Custom training

Almost always required in some form: species and fishery-specific models need labelled footage from the target fishery and camera set-up. Data needs: thousands of annotated hauls or many tens of thousands of labelled fish images per target species group, drawn from multiple vessels, seasons and light conditions. Prefer fine-tuning a pretrained backbone; share annotated data via authorities or research bodies (e.g. ICES working groups) where legally possible.

State of the art (2026)

• Commercial REM analytics have moved from research prototypes towards operational assistance: event detection, discard detection and species counting are being validated in national programmes, with human review remaining standard.
• Foundation models (SAM 2, DINOv2, open VLMs) reduce labelling effort and help with open-set "unknown species" handling.
• Standardisation: EFCA technical specifications and ICES guidance on EM data are the reference for data quality and review protocols [2][3].
• Limits and failure modes: occlusion and piles of fish, spray and condensation on lenses, night work and glare, similar-looking species, discards hidden below deck or overboard outside camera view, camera tampering, domain shift between vessels and gears, and unrepresentative training data for rare protected species. Models can look accurate overall but fail on exactly the rare discards that matter.

Shared resources & combinations

• Same stack (object-detection + tracking + image-classification + edge-runtime): VIS-032 species identification and camera-trap monitoring, VIS-014 livestock monitoring, VIS-006 food grading, VIS-031 natural-language video search for review assistance.
• Complementary monitoring: ANOM-032 (dark vessel and AIS-spoofing detection) covers vessels without onboard cameras and spots suspicious tracks; combine to target inspections.
• Data pipelines: e-logbook, VMS and landings data feed reconciliation; the video annotation and edge deployment stack is shared with other camera analytics.

Effort, cost & KPIs

• Path A: weeks to months per fleet for installation and onboarding; costs are hardware per vessel plus review service or licence. Not quantified here; request quotes.
• Path B/C: 6-12+ months for a validated species/discard pipeline per fishery, mostly labelling and validation.
• KPIs: share of video needing human review (reduction vs. full review), recall on discard events (measured on audited unflagged samples), species-count error vs. observer or landing data, logbook-vs-video discrepancy rate, review time per trip, camera uptime and tamper-event rate, time from trip end to feedback.

People & change

Involve management early for budget and the legal mandate, the department lead (fleet or control unit) as process owner, IT for recorder, upload and integration with logbook systems, and legal/compliance for fisheries control law and evidence rules. The data protection officer must be involved before the pilot: continuous video of crews is personal data, a DPIA (Art. 35 GDPR) is very likely, and every vendor needs an AVV (Art. 28 GDPR).

Works council: yes. Cameras and analytics that capture crew behaviour are objectively suited to monitor employees, so co-determination under §87(1) no. 6 BetrVG applies; §90(1) no. 3 requires timely information, and an expert may be engaged under §80(3). For public bodies the Personalrat applies under BPersVG or state law. Agree in a works agreement on purpose limitation (fisheries compliance, not performance control), masking, retention and access.

Support AI literacy of reviewers and skippers (AI Act Art. 4) with short trainings on what the system detects and where it errs. Communicate openly why cameras are installed, start with a voluntary pilot group of vessels, and address fears of surveillance with transparency about who sees what. AI output is only an indicator: a human reviewer confirms every potential infringement.

See GUIDE-004, GUIDE-005 and GUIDE-008.

Risks, regulation & compliance

• EU AI Act: typically minimal risk: it is not a listed Annex III case, and control of fishing is not safety management of critical infrastructure. Reassess if the system evaluates or profiles crew behaviour or is combined with biometric functions. Art. 5 caution: do not use emotion recognition on crew (prohibited in workplaces) or biometric categorisation. AI literacy (Art. 4) applies to operators and reviewers. Public authorities should document human oversight.
• GDPR: continuous filming of crew is personal data processing and a workplace surveillance issue. Regulation (EU) 2023/2842 and the EFCA guidance require data protection safeguards; apply purpose limitation (catch handling, not crew performance), camera placement away from private areas, privacy masking or blurring (see VIS-030), limited retention, access logging, controller roles between authority and operator, and a DPIA. National labour law and works councils or seafarer representatives may be involved.
• Evidence: AI flags are not proof; infringement decisions need human-verified, chain-of-custody secured video under national procedural law.
• Other risks: fishers' acceptance and perceived surveillance, vendor lock-in, data ownership disputes, cybersecurity of recorders, mis-attribution of discards to wrong vessel/haul.
• Mitigations: transparent rules agreed with the sector, audit sampling, model validation per fishery, open data formats, encryption and tamper-evident logs.

Getting started

1. Check whether your vessels or fleet fall under the mandatory REM categories in Reg. (EU) 2023/2842 and national implementing rules.
2. Read the EFCA REM technical guidance and define camera layout, sensors and data retention.
3. Run a pilot on a few vessels with a certified provider; record at least a full season of hauls.
4. Build a labelled validation set with fisheries scientists and measure recall on discards.
5. Complete the DPIA, agree with crew representatives, and set up a human review and audit process.

Sources

1. Regulation (EU) 2023/2842 amending the Fisheries Control Regulation — https://eur-lex.europa.eu/eli/reg/2023/2842/oj
2. EFCA (European Fisheries Control Agency) — https://www.efca.europa.eu/
3. ICES — Electronic monitoring and data standards, working groups — https://www.ices.dk/
4. European Fisheries Control Agency (EFCA) — https://www.efca.europa.eu
5. Satlink — Remote electronic monitoring — https://www.satlink.es
6. Satlink — Electronic monitoring — https://www.satlink.es/
7. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
8. Ultralytics YOLO — https://github.com/ultralytics/ultralytics