Smart video surveillance & intrusion detection

Conventional CCTV records everything and alarms on pixel motion, so operators drown in false alarms from shadows, rain, insects and animals. Video analytics classifies what moved (person, vehicle, animal), checks it against zones, direction and schedule, and only escalates plausible intrusions with a snapshot. Typical deployments cut false alarms drastically and let one operator cover many sites. In Europe the technical part is easy; the legal design (GDPR, works councils, AI Act Art. 5 limits) is the hard part.

Problem & value

• Today: passive recording for after-the-fact evidence, plus motion detection or PIR sensors that trigger constantly. Alarm receiving centres (ARC) and guard services pay for every false alarm verification; police in many countries de-prioritise unverified alarms.
• AI value: fewer false alarms through person/vehicle classification, earlier detection (before the fence is crossed), verified alarms with video evidence, faster forensic search, remote guarding instead of patrol visits.
• Benefit figures: vendors report large false-alarm reductions, but figures are vendor-reported and site-dependent; measure on your own cameras during a pilot rather than trusting brochure numbers [1][2].

Variants

• SMB / retail / home: cloud-managed cameras with built-in person/vehicle detection and phone notifications (private homes: see household use below).
• Perimeter security: logistics yards, solar and wind parks, substations, data centres, construction sites with thermal cameras and radar fusion; remote video verification by an ARC.
• Private use: doorbell and garden cameras; the household exemption does not cover recording public space or neighbours' property.

How it works

1. IP cameras (RTSP/ONVIF, visible + thermal)
2. edge decode + motion pre-filter (camera or NVR)
3. object detector (person / vehicle / animal) + multi-object tracker
4. rule engine (zone, tripwire, loitering, direction, time schedule, arming state)
5. secondary check (classifier or VLM on the snapshot: "is this a person climbing the fence?")
6. alert with snapshot + clip
7. operator / ARC (human verification)
8. response (audio talk-down, guard dispatch, police call by a human decision)
9. event index for forensic search; retention + automatic deletion

• Detection and tracking: YOLO-class or transformer detectors (RT-DETR) running on the camera SoC, NVR or edge GPU; tracking (ByteTrack-style) gives identity within a scene and enables line-crossing and dwell rules.
• VLM verification: an optional second stage sends only candidate alerts to a small vision-language model to suppress remaining false alarms and write a short description; see VIS-031.
• Human-in-the-loop: a trained operator verifies every escalation to guards or police; no automated enforcement. Operators need a tuning workflow for zones and sensitivity.

Data preparation

The main inputs are live video streams (RTSP/ONVIF) from existing IP cameras and the site configuration: zones, tripwires, schedules and arming states. These live in the video management system or NVR, in installer documentation and often only in the head of the security officer. Typical gaps are old analogue or low-resolution cameras, unclear camera positions, missing site plans, unmanaged camera passwords and wide fields of view that include public roads or neighbouring land.

Preparation steps: inventory cameras with position, resolution and network access; define and document zones and arming schedules; secure accounts and network segments; mask private areas and neighbours' property; collect 50 to 100 real false-alarm clips (wind, animals, headlights) as a test set (PREP-018); set retention and automatic deletion rules (PREP-024, PREP-025). No training data is usually needed for standard detectors.

Rough share of effort: low, about 10 to 20 percent. The main work is camera placement, lighting and tuning of zones, not data cleaning.

Implementation paths

Which path for whom

• SMB: Path A, because camera makers and alarm-centre providers offer mature, EU-hostable analytics at per-camera prices and the intrusion process is standard.
• Enterprise: Path C, because multi-site video is sensitive, volume makes per-camera cloud fees high and detection on own edge hardware (HW1) keeps footage on site.
• Developer: Path C, because open detectors, trackers and ONVIF tooling run on HW1 and allow a tailored rule engine.
• Public: Path C, because sovereignty, procurement rules and strict limits on video of people favour self-operated, on-premises systems.

Path A — Buy (recommended for most SMBs and sites)

• Camera-native analytics: Axis (EU, Sweden; AXIS Object Analytics, edge-based, ONVIF), Bosch Security (EU, Germany; Intelligent Video Analytics), Hanwha Vision, Hikvision/Dahua (note: significant political and procurement restrictions on these vendors in several countries; check national guidance and public-procurement rules), Avigilon/Motorola (Alta, Unity), Verkada, Eagle Eye Networks.
• VMS platforms: Milestone XProtect (EU, Denmark), Genetec Security Center, Axis Camera Station, Network Optix.
• Specialist perimeter and remote-guarding AI: Sighthound, Ipsotek, Davantis (EU, Spain), integrated offerings from security service providers, plus German remote-video-verification ARCs working with VdS-certified systems.
• Choose A when: fewer than a few hundred cameras, standard zones, and an ARC or in-house operator exists. Check: on-camera analytics (data stays local), EU hosting for any cloud storage, ONVIF Profile M metadata support, VdS/EN 62676 / EN 50131 conformity where insurers require it, and a data processing agreement.

Path B — Build on cloud APIs

• Video analytics APIs: Google Cloud Video Intelligence (object tracking, EU location), AWS Rekognition Video / Kinesis Video Streams (Frankfurt, Ireland regions), Azure AI Vision and Azure Video Indexer. VLM APIs (Gemini, Claude, GPT, Mistral) for verification of single snapshots.

Path C — Self-host (open source)

• Detection and tracking: Ultralytics YOLO (AGPL-3.0, check licence for commercial use), RT-DETR, YOLOX or RF-DETR (Apache-2.0), Roboflow Supervision for zones and tracking, NVIDIA DeepStream / Metropolis microservices.
• NVR and event layer: Frigate NVR (open source, Coral / Hailo / OpenVINO / TensorRT detectors, MQTT/Home Assistant integration), Shinobi, ZoneMinder; MediaMTX or go2rtc for stream handling.
• VLM verification: open-weight vision-language models such as Qwen2.5-VL / Qwen3-VL, InternVL or SmolVLM served via vLLM or Ollama.
• Hardware: HW1 handles roughly a handful to a few dozen streams per edge device (Coral, Hailo-8, Jetson Orin) depending on resolution and frame rate; HW2 for dozens of 1080p streams plus a VLM. Ops effort: low for Frigate-class setups, moderate at 100+ cameras.

Path D — Custom training

Worth it for unusual scenes: thermal-only cameras, fisheye or aerial views, night scenes at long range, or site-specific nuisance classes (wind-blown foliage, wildlife, reflections). Needs a few thousand labelled frames per site type; fine-tune a small detector and use hard-negative mining from false alarms logged by operators. Synthetic data helps with rare intrusion poses, but validate only on real footage.

State of the art (2026)

• Edge-first analytics is the norm: object detection on camera SoCs (ARTPEC/Ambarella-class) and NPUs, with ONVIF metadata streams feeding VMS rules. Person/vehicle/animal classification is commodity.
• Open-vocabulary detectors and VLMs (Grounding DINO-class, YOLO-World, Qwen-VL) allow text-defined rules ("person on the roof", "open gate") and cheap second-stage verification of alerts; they are too costly to run on every frame, so use them as a verifier.
• Limits and failure modes: low-light and weather, occlusion, camera tampering or spray painting, tiny distant targets, adversarial clothing or patches, domain shift after camera moves, and alert fatigue. Behaviour classifiers ("suspicious behaviour", "aggression detection") are scientifically weak and legally risky; prefer geometric rules plus human review. Always test recall for the scenarios that matter (night, crawling, partial occlusion), not only precision.

Shared resources & combinations

• Same stack (object-detection + tracking + edge-runtime + streaming): VIS-007 PPE and workplace safety, VIS-008 footfall, VIS-012 traffic, VIS-044 counter-UAS, VIS-013 wildfire cameras. One edge GPU box or NVR can run several analytics.
• Natural combinations: VIS-031 (natural-language search and VLM alerts over the same camera feeds), VIS-030 (automatic anonymisation of exported clips and for privacy masking), VIS-011 (gate and plate logic, separate legal basis), VIS-016 (access control; separate, stricter legal regime).

Effort, cost & KPIs

• Path A: days to a few weeks; camera-licence or per-channel subscription pricing; mostly installation and zone tuning.
• Path B: 1-3 months for a verified-alert pipeline with integration.
• Path C: 1-3 months for a Frigate/DeepStream-based setup; edge hardware plus maintenance.
• KPIs: false alarms per camera per week, recall on staged intrusion tests, time to verified alert, share of alerts closed without guard dispatch, operator minutes per alert, uptime and tamper-detection rate.

People & change

Involve management for the security concept, IT and IT security for network, camera hardening and integration, and the DPO from the first plan because video of people is personal data. Legal or compliance clarifies signage, retention and the rules of §4 BDSG for video surveillance. Employees on site are affected daily, so involve them before cameras go live.

Works council: yes in practice. Video analytics is a technical system suited to monitor employees, so co-determination under §87(1) no. 6 BetrVG applies; the works council must also be informed under §90(1) no. 3 and may engage an AI expert under §80(3). Conclude a works agreement that excludes performance monitoring and limits access and retention. In public bodies, involve the staff council (Personalrat). DPO: a DPIA is likely, plus an AVV with any alarm-centre or cloud provider (GUIDE-004, GUIDE-005).

AI literacy: support operators with training on false alarms, zone tuning and escalation rules (GUIDE-006). Start with one site as pilot (GUIDE-009). Typical concerns are surveillance and false accusations; address them with transparent signage, strict purposes and a human operator who verifies every alert before guards or police are called (GUIDE-008).

Risks, regulation & compliance

• EU AI Act: classification depends on function. Plain object detection for intrusion on private premises is generally minimal risk. Safety components in critical infrastructure management (Annex III 2) can be high-risk. Art. 5 prohibitions to avoid: real-time remote biometric identification in publicly accessible spaces for law enforcement (narrow exceptions only with authorisation), untargeted scraping of facial images for databases, emotion recognition in workplaces and education, biometric categorisation inferring sensitive attributes, and individual criminal-risk prediction based on profiling. Do not add face matching, emotion or "aggression/suspicion" scoring to a general-purpose surveillance system; many such features sit in Art. 5 or high-risk Annex III biometrics [4].
• GDPR: video of identifiable people is personal data. You need a documented legitimate interest (Art. 6(1)(f)) or public-task basis, a balancing test, signage and layered information, data minimisation (privacy masks, narrow fields of view), short retention (commonly 72 hours unless an incident occurs, as regulators recommend), access control and logging, and a DPIA where monitoring is systematic or large-scale (public areas, employees). The EDPB video-device guidelines are the reference [5].
• Workplace: in Germany, works council co-determination (BetrVG sec. 87(1) no. 6) and BDSG sec. 26 apply (sec. 4 BDSG on video surveillance is of doubtful applicability to private bodies); covert monitoring is only allowed in narrow circumstances. Do not use analytics to evaluate employee performance.

Getting started

1. Define the threat and the legal basis first: what must be detected, where, at what hours; write the DPIA and signage text before buying cameras.
2. Survey existing cameras for ONVIF and on-camera analytics support; fix lighting, mounting height and field of view before adding AI.
3. Pilot 5-10 cameras with two candidate products (one Path A, optionally Frigate on an edge box) for four weeks; log every false alarm and run staged intrusion tests.
4. Set arming schedules, zones, retention and operator escalation rules; train operators and involve the works council early.
5. Review KPIs monthly and keep a model/rule change log for audits.

Sources

1. Axis Communications - AXIS Object Analytics documentation - https://help.axis.com/en-us/axis-object-analytics
2. Milestone Systems - XProtect VMS documentation - https://doc.milestonesys.com/
3. Frigate NVR documentation - https://docs.frigate.video/
4. Regulation (EU) 2024/1689 (AI Act), Art. 5 and Annex III - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. EDPB - Guidelines 3/2019 on processing of personal data through video devices - https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en
6. Roboflow Supervision - https://github.com/roboflow/supervision
7. ONVIF - Profile M (analytics metadata) - https://www.onvif.org/profiles/profile-m/