Audio event detection & acoustic monitoring

Microphones are cheap, work in the dark and around corners, and hear things cameras and vibration sensors miss: breaking glass, alarms, a failing bearing, a fall, a shout. Audio classification models turn continuous sound into discrete, timestamped events that trigger alerts or maintenance tickets. The technology is mature; the hard parts are false alarms in real acoustic environments and, above all, data protection: a microphone that can pick up speech is a surveillance device in the eyes of the law.

Problem & value

• Today: security staff watch camera walls; maintenance relies on scheduled rounds or an experienced technician "listening" to a machine; alarm sounds are only detected by humans present; cities measure noise with a few sound-level meters that report decibels, not causes.
• AI value: 24/7 coverage of sound-only events, early warning of mechanical faults (acoustic signatures change before vibration or temperature limits are hit), sound-source attribution in noise mapping, and verification of alarms before dispatching staff.
• Evidence base: sound event detection is a long-running research field with annual benchmark challenges (DCASE); AudioSet-trained models classify 527 sound classes, with published top results around 0.50 mAP, which shows that open-world sound tagging is far from solved and that site-specific tuning is needed [1][2][3]. No vendor ROI figures are quoted here.

Variants

• Building & facility security: glass break, alarm-sound verification, impact detection (works as audio add-on in existing IP cameras).
• Industrial acoustic condition monitoring: anomalous-sound detection for pumps, fans, gearboxes, conveyors → PRED-006 / ANOM-005. Often unsupervised: only "normal" sound is available.
• Smart-city & environmental noise: noise-source classification, event-based noise mapping per Environmental Noise Directive 2002/49/EC, traffic/industrial noise complaints.
• Care & ambient assisted living: fall impact, call-for-help, smoke-alarm and doorbell recognition, cough detection; privacy-preserving sensors that never store speech.
• Public safety: gunshot or aggression detection in public space: sensitive, see risks.
• Accessibility / private use: phone or watch features that notify deaf and hard-of-hearing users about alarms, doorbells, baby crying, sirens (OS-level sound recognition).
• Wildlife and ecology: see SPEECH-015. Emergency calls: see SPEECH-016.

How it works

1. microphone / sensor (edge)
2. audio capture (16 kHz mono, 1–10 s sliding windows) + level gating
3. log-mel spectrogram
4. audio model (tagging: multi-label, or detection: frame-level onset/offset; or anomaly score)
5. post-processing (thresholds per class, smoothing, debouncing, site calibration)
6. event store (timestamp, class, score, optional short clip)
7. routing (VMS / alarm panel / CMMS ticket / dashboard / phone push)
8. human verification & feedback labels

• Models: pretrained audio transformers or CNNs fine-tuned on AudioSet-style classes (AST, PaSST, BEATs, PANNs, CLAP-style audio-text models for zero-shot classes). Frame-level detection needs strongly labelled data. Industrial anomalies typically use autoencoders or embedding-distance methods trained on normal sound only.
• Edge first: inference on the sensor or a gateway; only event metadata (or a short clip after a trigger) leaves the site. This is the main privacy-by-design lever.
• Human in the loop: an operator confirms safety-relevant alerts before dispatch; technicians confirm acoustic fault alerts and label the cause, which becomes training data.

Data preparation

Data needed: continuous audio from IP cameras, dedicated acoustic sensors or machine-mounted microphones, plus a few labelled examples of the target events (alarm, glass break, bearing squeal). In an SMB this audio lives nowhere yet: it must be captured, usually via cameras already connected to a VMS, and event labels sit in maintenance logs, CMMS tickets or the heads of technicians.

Typical gaps: no baseline recordings of normal background noise, no labels, microphones with poor placement or clipping, and unclear rules on whether speech may be recorded at all. Retention and access rights for audio are rarely defined.

Preparation steps: 1. Record one to four weeks of normal-condition audio per site and mark known incidents (use PREP-020 to harvest labels from maintenance tickets and alarm logs). 2. Build a small test set with technicians (PREP-018) and define event categories before configuring alerts (PREP-019). 3. Configure the system to store only short event clips and no continuous speech; apply PREP-022 where clips may contain voices. 4. Clarify purpose and legal basis (PREP-025).

Effort share: roughly 30 to 40 percent of the project, mostly site-specific baselines and threshold tuning. Capturing operator feedback on false alarms (PREP-021) keeps this effort low after go-live.

Implementation paths

Which path for whom

• smb: Path A, because VMS and camera vendors (e.g. Axis) ship mature audio analytics that are added by licence or firmware update.
• enterprise: Path B, because machine-sound and site-specific events need integration with CMMS, SIEM and own sensors that no single product covers.
• developer: Path C, because open audio models (e.g. PANNs, AST, BEATs) run on HW1 edge devices and can be tuned to a site.
• public: Path C, because audio in public or care spaces is sensitive and procurement favours sovereign, on-premise processing.

Path A: Buy (recommended for most SMBs)

• Security/video ecosystem: audio analytics in VMS and camera platforms such as Axis (audio analytics in cameras and "AXIS Audio Analytics"), Bosch, Hanwha, Genetec, Milestone integrations; alarm-verification products using sound classes. Check EU availability and whether audio is processed on the device.
• Specialist vendors: Sound Intelligence (NL, EU: aggression/scream/gunshot detection inside security systems), Audio Analytic (UK: embedded sound recognition licensed to device makers), Sensory, Soundtrace-type municipal noise-monitoring providers, and industrial acoustic-condition vendors (e.g. Siemens, SKF, Brüel & Kjær / HBK, Cetim-style sensor-plus-analytics offers). Verify current offerings before procurement.
• Consumer/OS level: Apple Sound Recognition and Android Sound Notifications for accessibility on phones.
• Choose A when: the sound classes are standard (glass, alarm, siren, scream) and a video or alarm platform already exists.

Path B: Build on cloud APIs

• Audio tagging via multimodal models: Gemini (audio input), Google's audio models on Vertex AI (EU location), and OpenAI-class audio-input models can label clips; useful for offline analysis of archives, not for continuous streams (latency, cost, data export).
• Pipeline: Azure IoT / AWS IoT Greengrass / Google Distributed Cloud edge for capture, cloud for fine-tuning and fleet management; Amazon Lookout for Equipment-style services have been discontinued or changed in the past, so check lifecycle before depending on one.
• Choose B when: you need custom classes with few engineers and can keep raw audio at the edge, sending only embeddings or event clips.

Path C: Self-host (open source)

• Models: AST (MIT/BSD-style via Hugging Face), BEATs (Microsoft, unilm), PANNs, PaSST, CLAP (LAION / Microsoft), YAMNet (TensorFlow Hub, Apache-2.0), BirdNET for birds (see SPEECH-015). Check each checkpoint licence: some are research-only.
• Tooling: PyTorch/torchaudio, Hugging Face Transformers (audio-classification pipeline), ONNX Runtime, TensorFlow Lite, sound-event-detection baselines from the DCASE challenge (DESED), MQTT + Node-RED or n8n for alerting, Grafana for dashboards.
• Hardware: HW1 suffices for inference: Raspberry Pi 5, Jetson Orin Nano or industrial gateways run quantised small audio models in real time for several channels. Training or fine-tuning fits on one workstation GPU (HW2).

Path D: Custom training

Worth it for site-specific sounds (your machine type, your alarm tones, your building's acoustics). Needs hundreds of labelled event clips per class plus many hours of background audio to estimate false-alarm rates; for industrial anomaly detection only normal operating sound is needed. Fine-tune a pretrained audio encoder rather than training from scratch; augment with room impulse responses and noise mixing.

State of the art (2026)

• Pretrained audio encoders (AST, BEATs, PaSST) pretrained on AudioSet dominate the DCASE sound-event-detection and tagging tasks; strongly labelled AudioSet subsets improve frame-level timing [2][3].
• Audio-language models (CLAP-family and large audio-language models) enable zero-shot and text-queryable sound classes and audio captioning; accuracy on rare or safety-critical classes is still below dedicated, tuned detectors.
• Anomalous sound detection for machines is an established DCASE task, with embedding-based methods that work from normal data only [1].
• Limits and failure modes: domain shift (a model tuned on AudioSet fails in a reverberant warehouse), overlapping sources, low signal-to-noise, class confusion (shout vs. cheering), and adversarial or accidental triggers. False-alarm rate per day per microphone, not accuracy, is the decisive metric. Aggression or "emotion" detection from voice is scientifically contested and legally sensitive.

Shared resources & combinations

• Same audio stack (audio-classification, edge-runtime): SPEECH-015 bioacoustics, SPEECH-008 enhancement as pre-processing, SPEECH-023 voice-chat moderation, SPEECH-016 emergency-call analysis.
• Downstream: PRED-006 (predictive maintenance), ANOM-005 (process anomaly detection), VIS-004 (asset monitoring) for multimodal fusion with camera and vibration data.
• Hardware: one HW1 edge fleet plus one shared HW2 training workstation serves all audio-classification use cases.

Effort, cost & KPIs

• Path A: weeks; licence per camera/channel or per sensor, plus integration into VMS/alarm panel.
• Path B: 2–4 months; costs driven by data egress, labelling and fleet management.
• Path C/D: 3–6 months incl. data collection campaign, field pilot and threshold tuning; sensor hardware and installation often dominate cost.
• KPIs: precision and recall per class, false alarms per microphone per day, detection latency, share of alerts confirmed by humans, mean time to detect a fault, avoided unplanned downtime, operator workload.

People & change

Involve management and the department lead (security, maintenance) first to fix the purpose: protecting property and machines, not observing staff. IT and IT security follow for network, microphone placement and alert integration; the DPO should assess the setup before any microphone is installed, since audio can capture conversations. A DPIA is likely where microphones cover workplaces or care settings, and an AVV is needed with every cloud analytics vendor.

Works council: yes. Microphones that run continuously in workplaces are objectively suited to monitor behaviour and performance, which triggers co-determination under §87(1) no. 6 BetrVG (BAG: objective suitability suffices). Inform it early under §90(1) no. 3; the §80(3) expert option is available. Public sector: staff council. Agree in a works agreement that events only, not speech, are stored, and that no individual performance evaluation takes place.

Communicate openly with signage and an explanation of what is and is not recorded. Start a pilot in one hall or building with a few end users and technicians who review alerts. Typical concerns are surveillance and false alarms: answer them with the works agreement and a tuned alert threshold. Operators always confirm an alert before action is taken. AI literacy measures should support staff in understanding the limits of detection (Art. 4 AI Act).

See GUIDE-004, GUIDE-005 and GUIDE-008.

Risks, regulation & compliance

• EU AI Act: classification depends on use. Machine-sound monitoring and accessibility alerts: minimal risk. Safety components of critical infrastructure (Annex III(2)) can be high-risk. Real-time remote biometric identification and emotion recognition in workplaces and education are prohibited under Art. 5: inferring emotion or stress from employee voices is not allowed in those settings, so aggression or "emotion" detection must not be used on staff. Use by law enforcement (e.g. gunshot detection feeding police action) may fall under Annex III(6) high-risk rules; this file does not cover operational policing detail.
• GDPR: sound that can contain speech is personal data even if the system only wants events. Prefer sensors that cannot reconstruct intelligible speech (on-device inference, band-limited or spectral-only processing, no raw audio storage), a documented legitimate interest or legal basis, DPIA (Art. 35, systematic monitoring of public areas), signage, and works-council co-determination in Germany (§87 BetrVG). Recording of non-public speech can also be criminal (§201 StGB, Germany), so a legal review is mandatory.
• ePrivacy / telecom secrecy may apply when audio is transmitted; municipal noise monitoring needs a legal basis in national law.
• Safety: acoustic detection must not be the sole safety function; missed events are expected.
• Mitigations: edge processing, short retention for evidence clips, access logging, purpose limitation, regular false-alarm audits, human confirmation before action.

Getting started

1. Define the exact events and the decision each triggers; reject use cases that need speech content or emotion inference.
2. Do a legal and DPIA check early (works council, signage, retention) before installing microphones.
3. Record 1–2 weeks of background audio on site, with the real sensors, to measure the baseline.
4. Test a pretrained model (AST/BEATs/YAMNet) and one commercial product against that data; compare false alarms per day.
5. Pilot with human-confirmed alerts, log feedback, then tune thresholds or fine-tune.

Sources

1. DCASE Community — Detection and Classification of Acoustic Scenes and Events challenges — https://dcase.community/
2. Gong et al. — AST: Audio Spectrogram Transformer — https://arxiv.org/abs/2104.01778
3. Chen et al. — BEATs: Audio Pre-Training with Acoustic Tokenizers — https://arxiv.org/abs/2212.09058
4. Google Research — AudioSet ontology and dataset — https://research.google.com/audioset/
5. Hugging Face — Audio classification task documentation — https://huggingface.co/docs/transformers/tasks/audio_classification
6. Regulation (EU) 2024/1689 (AI Act), Art. 5 and Annex III — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. EDPB — Guidelines 3/2019 on processing of personal data through video devices (principles apply to audio monitoring) — https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en
8. Directive 2002/49/EC on environmental noise — https://eur-lex.europa.eu/eli/dir/2002/49/oj