Voice cloning needs seconds of audio and real-time face swapping runs on a laptop, so "I recognised the voice" is no longer proof of identity. Deepfake detection scores audio, video and images for synthetic or manipulated content in calls, video meetings and onboarding. Because detectors are probabilistic and attackers adapt, it works only as one layer next to hard process controls (call-back, four-eyes, payment limits).
Problem & value
- Today: fraudsters impersonate executives, suppliers or relatives by phone and video to trigger payments, bank-detail changes, password resets or data release. The reference case is the 2024 Arup incident: an employee in Hong Kong paid roughly USD 25 million after a video call in which the CFO and colleagues were deepfakes [1]. Contact centres see cloned voices used against voice authentication and agents; HR sees fake candidates in remote interviews.
- AI value: an automated second opinion on media that humans cannot judge reliably, applied at scale and in real time, plus evidence for investigations.
- Honest benefit framing: vendor accuracy claims are rarely independently verified and depend on the codec, language and generator. Independent in-the-wild evaluation found open detectors performing substantially worse than on academic benchmarks [5]. Value comes from combining detection with process controls, not from a single score.
Variants
- Executive/CEO-fraud defence (finance, SMB and enterprise): meeting and phone screening for payment approvers, plus mandatory call-back rules.
- Contact-centre voice security (banking, insurance, telecom): synthetic-speech and liveness scoring on inbound calls, combined with caller-device and network signals.
- Remote onboarding / KYC: injection-attack and face-swap detection in selfie/video identification (overlaps ANOM-014).
- Hiring: detecting impersonated candidates in video interviews.
- Media/newsroom and public sector: forensic verification of user-generated content (file-based).
- Private use: family "emergency call" scams; practical defence is a shared safe word and call-back, not software. See SEC-024 for awareness training.
How it works
- media source (phone/SIP, meeting bot or plug-in, upload, KYC SDK)
- capture & preprocessing (resample, codec normalisation, face/voice segmentation)
- detectors: audio anti-spoofing | video face-forgery | injection/virtual-camera check | metadata & provenance (C2PA)
- fusion with context (new caller device, unusual request, executive named, urgency)
- risk score + explanation (which cues, which time window)
- policy engine: allow | warn user | force out-of-band verification | block / escalate to SOC
- case record, evidence retention, feedback labels
- Audio: anti-spoofing models (e.g. AASIST-style graph-attention networks, SSL front-ends such as wav2vec 2.0) trained on corpora like ASVspoof [6]; performance drops on telephone codecs and unseen generators.
- Video/image: artefact and temporal-consistency detectors, increasingly VLM/foundation-model based; plus detection of virtual cameras and stream injection, which often matters more than the face itself in KYC.
- Provenance: C2PA content credentials and AI-Act-mandated machine-readable marking help for files from cooperating generators but are absent in live attacker-controlled streams [3][4].
- Human-in-the-loop: a high score never auto-accuses a person; it triggers verification. Payment release stays with humans under four-eyes control.
Data preparation
The system needs audio and video streams, call metadata and the business context for risk scoring. Audio comes from the telephone system or contact centre, video from Teams/Zoom/Webex, and context from the ERP (supplier master data, payment requests) and the executive directory.
Typical gaps: calls are not recorded or only with poor codecs, no list of who may approve what, master-data changes without documented call-back numbers, no reference voice samples, and no labelled examples of real or fake calls. Recordings are personal data and often subject to consent and retention limits.
Preparation steps: (1) record which channels exist and what recordings are legally allowed (PREP-025); (2) clean up payment and master-data approval rules and verified call-back numbers in the ERP (PREP-005, PREP-012); (3) build a test set of genuine and synthetic samples with the security team (PREP-018); (4) capture analyst feedback on false alarms as labels (PREP-021); (5) define retention and pseudonymise stored evidence (PREP-022).
Tools: the telephony and meeting platform APIs, an internal wiki for the verification procedure. Rough share of effort: about 20 to 30 percent, mainly process clean-up and consent rules rather than technical data wrangling.
Implementation paths
Which path for whom
- enterprise: B, because detection must be integrated into own telephony, meeting and KYC flows and combined with payment-release rules, which off-the-shelf tools only partly cover.
- smb: A, because the realistic first step is a mature detection add-on for meetings or a contact-centre product combined with simple callback and four-eyes rules; building detectors is out of reach.
- developer: B, because detector APIs and open models (e.g. anti-spoofing research models) can be embedded into products and onboarding flows.
Path A — Buy (recommended)
- Multimodal platforms: Reality Defender (API, meeting and contact-centre integrations), GetReal Security (forensics), Sensity AI, Hive, Resemble AI Detect.
- Voice/contact-centre specialists: Pindrop (synthetic-speech and liveness scoring with device and network signals), Daon, ID R&D / Mitek (voice and face liveness), Nuance/Microsoft heritage products in banking.
- EU vendors and EU-hostable options: Netherlands-based DuckDuckGoose; Idiap-derived and other European identity vendors such as Veridas (ES), IDnow (DE), Signicat (NO) and Onfido/Entrust for KYC injection-attack detection. Check EU data residency and processor terms.
- Choose A when: you lack ML staff and need meeting/telephony integrations. Pilot with your own recorded calls and codecs; ask for third-party test results (e.g. iBeta/ISO 30107-3 for presentation attacks) rather than marketing accuracy.
Path B — Build on cloud APIs
- Detection is a poor fit for general LLM APIs (frontier multimodal models are unreliable at judging authenticity); use them for explanation and triage text only.
- Practical build: telephony/meeting recording hooks (SIP recording, Teams/Zoom compliance recording APIs) → vendor detection API (e.g. Reality Defender, Resemble Detect) → workflow engine (n8n, Power Automate) → ticket/Teams alert. Provenance checks via the open-source C2PA SDK [3].
Path C — Self-host (open source)
- Audio: AASIST and related ASVspoof baselines, SpeechBrain, wav2vec 2.0 / XLS-R fine-tunes [6].
- Video: FaceForensics++-style detectors, DeepfakeBench model zoo [7]; C2PA reference tools.
- Hardware: HW2 handles real-time scoring of dozens of audio streams; HW1 suffices for batch audio.
- Ops effort is high: models degrade as new generators appear and need regular re-evaluation. Self-hosting is justified mainly by data-sovereignty needs (call recordings, biometric data).
Path D — Custom training
Worth it for large contact centres or banks with their own codecs, languages (German dialects, Turkish, Polish) and attack telemetry. Needs thousands of bona fide calls plus synthetic samples from many current generators (open and commercial TTS/voice-conversion) created under legal and ethical controls; refresh quarterly.
State of the art (2026)
- Generators now produce real-time, low-latency voice conversion and face swaps; detection is an arms race with no stable benchmark lead. Academic benchmarks (ASVspoof 5, DeepfakeBench) overstate real-world performance; in-the-wild evaluation shows large drops [5][6][7].
- Trend: shift from "is this file fake?" to liveness and channel integrity (virtual-camera and injection detection, device attestation) and to provenance (C2PA, watermarking) on the generator side.
- Trend: detection embedded in meeting platforms and contact-centre suites rather than standalone.
- Limits: false positives on poor connections, compressed audio, accents and accessibility tools (voice prosthetics, real-time translation, noise filters); false negatives on unseen generators; adversarial evasion once the detector is known. Scores are not evidence of guilt.
Shared resources & combinations
- Shared capabilities: audio-classification and face-biometrics stacks with ANOM-014 (identity document and liveness) and CONV-002 (voice agents, where bot-versus-human disclosure matters).
- Combine with: SEC-002 phishing (the email that precedes the call), SEC-005 UEBA (anomalous payment or login behaviour), ANOM-009 and ANOM-001 (payment fraud controls), SEC-001 (SOC triage of alerts), SEC-021 (impersonation outside your perimeter), SEC-024 (simulated vishing and awareness training), ANOM-015 (marking and provenance of AI-generated content).
Effort, cost & KPIs
- Path A: days to weeks for pilot; pricing usually per minute/seat/API call, quoted on request.
- Path B: 1-3 months incl. telephony integration.
- Path C/D: 3-6+ months and ongoing model maintenance.
- KPIs: detection rate and false-positive rate on your own channels, time to alert, share of high-value payment requests with completed call-back, attempts blocked, incident losses, reviewer workload.
People & change
Management and finance leads own the payment-release rules; the information security officer and IT integrate and run detection; the DPO assesses voice and face analysis; HR and the works council are involved because employees' calls are analysed. Inform customers and suppliers where their voices are checked.
Works council: yes. Analysing employees' calls and meetings is a system suited to monitor behaviour, so §87(1) no. 6 BetrVG applies; §90(1) no. 3 requires early information. Agree scope, retention and a ban on using scores for performance evaluation (GUIDE-004). Public sector: staff council.
Data protection: voice and face are likely biometric data when used for identification, so a DPIA (Art. 35 GDPR) is likely, plus an AVV with the vendor (GUIDE-005). Check whether the case falls under AI Act transparency or biometric rules (GUIDE-014).
AI literacy: AI Act Art. 4 asks employers to support staff AI literacy; run short awareness sessions with a live voice-clone demo for finance, assistants and the contact centre (GUIDE-006). Pilot with the finance team. Typical concerns: "will I be accused by a machine?" and false alarms. A high score never accuses a person; it triggers out-of-band verification, and payments stay under four-eyes control (GUIDE-008).
Risks, regulation & compliance
- EU AI Act: classification depends on use. Pure fraud/security detection is generally minimal or limited risk. Remote biometric verification and identification can be high-risk (Annex III) and biometric categorisation or emotion inference is restricted; Art. 5 prohibitions apply to emotion recognition in the workplace and to certain biometric uses, so do not extend detectors to infer emotion, honesty or stress from voice or face. Providers of generative systems must mark synthetic output and deployers must disclose deepfakes (Art. 50) [2].
- GDPR: voice and face data analysed to identify or verify persons can be biometric data under Art. 9; recordings of employees raise works-council and Art. 88 / national employment-law issues (in Germany, co-determination under BetrVG). Needs a legal basis, DPIA, transparency to participants, minimisation and short retention. Do not store voiceprints beyond need.
- Sector rules: NIS2 and DORA risk-management duties for finance; BaFin/EBA guidance on remote onboarding; Europol and national agencies publish deepfake threat assessments [8].
- Risks: over-reliance on scores, false accusations, discrimination against atypical voices, vendor lock-in, detector evasion. Mitigations: layered controls, human review, bias testing, regular red-team with permitted synthetic samples, procedural rules independent of detection.
Getting started
- Define high-risk requests (payments above a threshold, bank-detail changes, credential resets) and require out-of-band call-back to a known number plus four-eyes approval, regardless of AI.
- Run a tabletop exercise with a deepfake-CFO scenario; brief finance, assistants and IT service desk.
- Pilot two Path A vendors on recorded internal samples and your telephony codecs; request test reports.
- Complete a DPIA and consult the works council before screening employees or meetings.
- Define escalation playbook and feed alerts into SEC-001 triage; review KPIs quarterly.
Sources
- CNN — Arup revealed as victim of USD 25 million deepfake scam — https://edition.cnn.com/2024/05/16/tech/arup-deepfake-scam-loss-hong-kong-intl-hnk
- Regulation (EU) 2024/1689 (AI Act), Art. 5 and Art. 50 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- C2PA — Content Credentials specification and SDK — https://c2pa.org/
- European Commission — Code of practice on marking and labelling AI-generated content — https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content
- Chandra et al. — Deepfake-Eval-2024: in-the-wild deepfake benchmark — https://arxiv.org/abs/2503.02857
- ASVspoof 5 paper and AASIST baseline — https://arxiv.org/abs/2502.08857 ; https://github.com/clovaai/aasist
- DeepfakeBench — https://github.com/SCLBD/DeepfakeBench
- Europol — Facing reality? Law enforcement and the challenge of deepfakes — https://www.europol.europa.eu/publications-events/publications/facing-reality-law-enforcement-and-challenge-of-deepfakes
- Reality Defender — https://www.realitydefender.com/ ; Pindrop — https://www.pindrop.com/
- BSI (Germany) — Künstliche Intelligenz (topic page) — https://www.bsi.bund.de/DE/Themen/Unternehmen-und-Organisationen/Informationen-und-Empfehlungen/Kuenstliche-Intelligenz/kuenstliche-intelligenz_node.html