Sites record thousands of camera-hours that nobody watches; finding one event means scrubbing footage, and every new detection need (a blocked exit, a missing guard rail) classically means labelling data and training a detector. Video agents combine a vision-language model (VLM), an LLM and a retrieval index: users type what they are looking for, define alerts as sentences and receive summaries with clip evidence. The gain is speed of investigation and flexibility; the price is GPU cost, false positives and a heavy privacy and works-council workload.
Problem & value
- Today: CCTV and operations cameras are used reactively. After an incident, staff scrub footage manually. Rule-based video analytics (VIS-010, VIS-007) detect only the classes they were trained for; each new rule is a project with labelled data.
- AI value: (1) search archived and live video with free text; (2) open-vocabulary alerts defined in minutes by a safety or operations manager; (3) automatic summaries of shifts, incidents or SOP steps; (4) question answering ("how often was bay 3 blocked last week?").
Variants
- Forensic search: natural-language retrieval over recorded video after an incident.
- Live open-vocabulary alerting: text-defined rules evaluated on sampled frames or clips.
- Shift / incident reports: periodic narrative summaries per camera group or zone.
- SOP-compliance review: compare observed steps with a written procedure (see KNOW-027, VIS-034).
How it works
- cameras (RTSP/ONVIF) / VMS export
- ingest & chunking (clips of a few seconds–minutes, frame sampling, optional motion gating)
- fast detector/tracker (YOLO-class) for cheap pre-filtering and object tracks
- VLM captioning & event description per chunk (+ timestamps, camera, zone)
- indexing: caption text embeddings + visual/multimodal embeddings in a vector DB, metadata in SQL/search
- query / alert layer: LLM agent translates question or alert rule
- retrieval
- VLM re-check on candidate clips
- verification step (second VLM pass or rule on tracks)
- alert / summary with clip citations
- human review
- ticket, chat, MES/SCADA event, report
- Two-stage design is the key to cost: a cheap always-on filter (motion, detector, tracker) feeds the expensive VLM only with candidate segments. Running a large VLM on every frame of hundreds of cameras is not economical.
- Video RAG: captions and embeddings of segments let an LLM answer questions and cite clips; long-video summarisation uses hierarchical (clip → hour → shift) summarisation.
- Human-in-the-loop: alerts above defined severity go to an operator who confirms or dismisses; dismissals are logged as evaluation data. Any consequence for a person (disciplinary action, access decision) must be decided by a human who reviews the footage, never by the generated text.
Data preparation
Data is live camera streams and archived recordings from VMS/NVR systems, plus context documents such as SOPs, zone names and shift plans. In an SMB the footage sits on NVR disks with short retention, often from mixed camera brands and without consistent time stamps or zone labels.
Typical gaps: cameras not on a common VMS, poor image quality and positioning, no naming of zones and cameras, unclear retention, and access rights that let many people view everything. Preparation steps: connect cameras via RTSP/ONVIF to one VMS, synchronise clocks, name cameras and zones, write the alert rules in plain language with examples, and convert SOPs into short searchable text. Create a test set of known events (for example a forklift in a walkway) to measure hit rate and false alarms before go-live. Mask or exclude areas with high privacy sensitivity such as changing rooms.
Effort share: about 25-35 percent, mainly camera and zone set-up and evaluation. Related groundwork: PREP-005 (getting data out of systems), PREP-010 (metadata and tagging), PREP-018 (evaluation sets) and PREP-024 (classification rules).
Implementation paths
Which path for whom
- enterprise: C - sites generate large video volumes, footage is sensitive and a shared internal GPU platform amortises the cost across sites and use cases.
- developer: B - cloud video-language APIs allow fast prototypes and product integration before committing to GPU hardware.
- public: C - video of public spaces and facilities should stay on sovereign or self-operated infrastructure, also because of procurement rules.
- smb: A - search and summary add-ons of established VMS vendors fit the standard camera setup, and self-hosting needs HW4 hardware that rarely pays off for a few cameras.
Path A — Buy (off-the-shelf)
- Video-AI platforms: NVIDIA Metropolis VSS ecosystem partners and integrators [1]; Twelve Labs (video search/understanding API and platform) [2]; Verkada, Genetec, Milestone Systems (EU, Denmark), Axis Communications (EU, Sweden) and Hanwha Vision, which add natural-language search or LLM-assisted investigation to their VMS and camera lines; Spot AI and Coram AI for prompt-defined alerts; Cloudian/Hitachi-type video-lake offerings for archives. Check feature availability and EU data location per product — the feature set changes quickly.
- Check: on-prem or EU-hosted processing, retention and deletion controls, export of audit logs, DPIA support documentation, works-council-friendly role and zone restrictions.
Path B — Build on cloud APIs
- Video-capable models: Google Gemini (native video input, EU regions via Vertex AI) [3], Twelve Labs Marengo/Pegasus [2], frontier multimodal models via frame sampling (Claude, GPT), Mistral (EU provider; vision models on frame sets). Prefer EU regions and zero-retention terms.
- Orchestration: event-driven pipeline (ffmpeg or GStreamer chunking, queue, worker) with a vector DB (pgvector, Qdrant — EU-built, OpenSearch); agent layer with MCP tools for tickets and CMMS.
- Choose B when: footage may leave the site (low sensitivity, no persons identifiable) and you need custom logic quickly.
Path C — Self-host (open source)
- Models: open-weight video-capable VLMs such as Qwen3-VL / Qwen2.5-VL [4], InternVL, LLaVA-Video, SmolVLM / Cosmos-Reason class models from NVIDIA [1]; Whisper for audio where legal.
- Stack: NVIDIA VSS blueprint (DeepStream, NIM microservices, Milvus/Elasticsearch, Neo4j) [1], or a custom stack with vLLM serving, Frigate or go2rtc for stream handling, Qdrant/pgvector.
- Hardware: HW3 for a handful of cameras with gated VLM calls; HW4 (multi-GPU node) for dozens to hundreds of streams or larger VLMs; edge boxes (HW1, Jetson-class) for the pre-filter stage only.
- Choose C when: footage shows employees or the public and must stay on-premises.
Path D — Custom training
Usually unnecessary at the start, which is the point of the pattern. Fine-tune a small VLM or train a dedicated detector (VIS-001, VIS-010 style) once an alert type is business-critical, has high volume and the VLM is too slow, costly or inaccurate. Dismissed and confirmed alerts form the training set (hundreds to thousands of clips).
State of the art (2026)
- Video-native VLMs accept long clips with timestamps and do temporal grounding ("when did the pallet fall?"); benchmarks such as Video-MME, MLVU and LongVideoBench track progress, and open-weight models are close to proprietary ones on many of them [4].
- Verification pattern: a cheap detector proposes, the VLM confirms, which lowers alert noise.
- Limits and failure modes: hallucinated events or actors; weak counting and fine-grained spatial relations ("behind the line" vs "in front of it"); sensitivity to camera angle, low light, occlusion; prompt wording changes alert behaviour; latency of seconds to tens of seconds rules out hard real-time safety functions; sampling rate can miss brief events. Never use for safety-critical interlocks — use certified sensors and PLC safety functions for that.
Shared resources & combinations
- Shares vlm, detection, tracking and streaming with VIS-010, VIS-007, VIS-008, VIS-012, VIS-048; procedures via KNOW-027, VIS-034; privacy layer VIS-030.
- Archive search: KNOW-016 (multimodal archives), VIS-025 (video tagging and highlights).
- Do not confuse with: VIS-016 (face recognition), which is a distinct, much more restricted use.
Effort, cost & KPIs
- Path A: weeks for pilot on a few cameras; licences per camera or per hour of analysed video.
- Path B: 2–4 months for a production pipeline; cost driven by VLM calls per analysed minute.
- Path C: 3–6 months incl. GPU infrastructure and MLOps; hardware plus operations.
- KPIs: precision and recall of alerts on a labelled test set of your own footage, false alerts per camera per day, time-to-find an event versus manual scrubbing, share of alerts confirmed by operators, summary accuracy (spot-checked), end-to-end latency, GPU cost per camera-day.
People & change
Involve management for purpose limitation, the department lead (security, logistics, production) as owner of the alert rules, end-users who will receive alerts, IT and information security for streams, GPU hosts and access rights, and the data protection officer from day one. A DPIA (Art. 35 GDPR) is likely: systematic video processing at scale. Sign an AVV (Art. 28 GDPR) with any cloud or VMS provider.
Works council: yes in practice. Video analytics is objectively suited to monitor behaviour and performance, so co-determination under section 87(1) no. 6 BetrVG applies, and the council must be informed in good time about AI-planned work processes (section 90(1) no. 3 BetrVG). Consider an expert under section 80(3) BetrVG and conclude a works agreement that excludes performance evaluation of individuals. Public bodies involve the staff council.
AI literacy: support staff with training on how alerts arise and what they cannot prove (Art. 4 AI Act). Communicate purpose, areas and retention openly. Pilot at one site or one hazard, such as forklift safety. Keep humans in control: alerts are suggestions that a person verifies; no automatic sanctions or decisions about individuals.
Guides: GUIDE-004, GUIDE-005, GUIDE-008.
Risks, regulation & compliance
- GDPR: video of identifiable people is personal data. Needs a lawful basis (typically legitimate interest, Art. 6(1)(f), after balancing test), purpose limitation, minimisation, short retention, signage, access control and a DPIA (Art. 35, systematic monitoring of publicly accessible areas). The EDPB Guidelines 3/2019 on video devices apply [5]. Free-text search increases the risk of purpose creep (searching for a named or described person); restrict query roles and log all queries.
- Employee monitoring: in Germany, works-council co-determination (§87(1) no. 6 BetrVG) and §26 BDSG limit performance and behaviour monitoring; covert or blanket evaluation of staff is typically unlawful.
- EU AI Act [6]: classification depends on use. Pure operations and safety analytics: usually minimal or limited risk. Art. 5 prohibitions: emotion inference in workplaces and education, biometric categorisation by sensitive attributes, and untargeted facial-image scraping are prohibited; real-time remote biometric identification in public spaces for law enforcement is prohibited except in narrow cases. Do not let a text rule such as "angry person" or "suspicious behaviour" turn the system into emotion recognition or profiling. Searching for a person by face or biometric traits is biometric identification (high-risk Annex III or prohibited) — keep this out of scope or evaluate separately (VIS-016). Use in critical-infrastructure safety components, worker management or law enforcement can be high-risk. Art. 4 AI literacy applies; Art. 50 transparency applies to chat interfaces.
Getting started
- Pick one narrow, non-personal question (e.g. blocked fire exit, pallet in aisle) and 3–5 cameras.
- Involve the data protection officer and works council first; complete the DPIA and signage.
- Collect two weeks of footage; label 100–200 positive and negative events as a test set.
- Prototype with a hosted video-capable model, then compare with an open-weight model on the same set.
- Add the cheap pre-filter, measure false alerts and cost per camera-day, then decide buy, build or host.
Sources
- NVIDIA — Metropolis Video Search and Summarization (VSS) blueprint — https://docs.nvidia.com/vss/latest/index.html
- Twelve Labs — API documentation (Marengo, Pegasus) — https://docs.twelvelabs.io/
- Google Cloud — Gemini video understanding (Vertex AI) — https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding
- Qwen team — Qwen3-VL repository — https://github.com/QwenLM/Qwen3-VL
- EDPB — Guidelines 3/2019 on processing of personal data through video devices — https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Video-MME benchmark — https://video-mme.github.io/