Voices can be used to confirm a claimed identity (verification, 1:1), to find an unknown speaker among enrolled people (identification, 1:N), or simply to separate "speaker A" from "speaker B" in a recording (diarization, no identity). The three have very different legal weight: a voiceprint is biometric data under GDPR, 1:N identification can fall under the EU AI Act's high-risk or even prohibited categories, while diarization without identity is usually low-risk. Since voice cloning became cheap, voice alone is no longer a safe authentication factor.
Problem & value
- Today: call-centre agents ask security questions (date of birth, PIN, passwords), which is slow, guessable and annoying; media and meeting teams label speakers by hand; investigators and archives attribute recordings manually.
- AI value: passive authentication during the first seconds of natural conversation shortens handling time and reduces knowledge-based-authentication fraud; diarization makes transcripts usable (SPEECH-001, SPEECH-002); speaker search helps archives and broadcasters.
- Threat shift: realistic voice cloning and replay attacks mean voice-only authentication is now treated as a risk by banks and regulators; deepfake defence is covered in SEC-011.
Variants
- Voice authentication (1:1): banking, telecom, insurance, healthcare hotlines; opt-in enrolment.
- Speaker diarization (no identity): meetings, call analytics, podcasts, court and council recordings (SPEECH-001, SPEECH-021).
- Speaker identification within a closed set: broadcaster and archive tagging of known voices, smart-speaker household profiles, in-car driver profiles (CONV-014).
- Fraudster watchlists: contact centres match calls against voices of known fraudsters (consent and legal-basis critical, see Risks).
- Public-security uses (policy level only): forensic voice comparison by experts and law-enforcement voice search are Annex III high-risk contexts; real-time use in public spaces is restricted by Art. 5. Not detailed further here.
How it works
- audio (call / stream / file)
- VAD + quality check (SNR, duration, codec, channel)
- [diarization: segment + cluster "who spoke when"]
- speaker embedding (x-vector / ECAPA-TDNN / ResNet / WavLM-based)
- comparison: cosine / PLDA score vs. enrolled template (1:1) or gallery (1:N)
- anti-spoofing / liveness (replay, TTS, voice conversion detection)
- decision policy (threshold, step-up to OTP/app, human review)
- audit log, retention and deletion
- Embeddings: a neural network maps a few seconds of speech to a fixed vector; same speaker yields close vectors. Text-independent models work on free conversation; text-dependent ones use a passphrase.
- Diarization: neural segmentation plus clustering (pyannote-style) or end-to-end models; output is anonymous labels unless linked to enrolled profiles.
- Anti-spoofing: separate countermeasure model (ASVspoof-style) trained on synthetic and replayed speech.
- Human-in-the-loop: low-confidence or high-value transactions go to step-up authentication or an agent; no adverse decision (account lock) from voice score alone; experts, not scores, produce forensic opinions.
Data preparation
Data needed: short voice samples for enrolment (typically 10 to 30 seconds of speech per person), call recordings from the telephony platform or contact-centre system, and a consent record per enrolled person. For diarization only (who spoke when, without identity) no enrolment data is needed. Recordings typically live in the telephony or recording system, sometimes in mail or file shares as meeting audio.
Typical gaps: no documented consent or legal basis for voiceprints, narrow-band telephone audio with poor quality, no labelled impostor or fraud examples, and no clear retention rules for embeddings. Existing recordings were often made for quality assurance, not biometrics, so reuse needs a purpose check.
Preparation steps: 1. Check purpose and legal basis before any enrolment (PREP-025); voiceprints are special-category data. 2. Define which data may go into which tool (PREP-024). 3. Pseudonymise exports for tests and evaluation (PREP-022). 4. Build an evaluation set with genuine and impostor trials across devices, noise levels and age groups (PREP-018). 5. Collect corrections from agents on wrong matches (PREP-021).
Effort share: low, about 10 to 20 percent. The effort lies mostly in the legal clearance and consent process, not in data cleaning. See also GUIDE-003 for a data readiness check.
Implementation paths
Which path for whom
- enterprise: Path A, because mature voice-biometrics and fraud-detection suites exist for call centres and this is a commodity security function.
- developer: Path C, because open toolkits such as SpeechBrain and pyannote cover speaker embeddings and diarization on HW1 or HW2 hardware.
- public: Path C, because voiceprints are biometric data and sovereign, on-premise processing eases procurement and legal approval.
Path A — Buy (recommended)
- Voice biometrics suites: Nuance Gatekeeper (Microsoft; Microsoft stopped selling Nuance hosted and on-premise enterprise licences in August 2024 and no Gatekeeper end-of-support date is published, so verify availability), Pindrop (fraud, deepfake detection), NICE, Daon, ValidSoft (UK), Phonexia (Czech, EU; widely used by public bodies), Speechmatics (diarization).
- Choose A when: regulated authentication at scale, telephony integration, certified anti-spoofing and support are needed. Require EU hosting, a DPIA template, template deletion API and documented false-accept/false-reject operating points.
Path B — Build on cloud APIs
- Diarization and speaker-attributed ASR: Azure AI Speech (conversation transcription with speaker separation, EU regions), Google Cloud Speech-to-Text diarization, AWS Transcribe speaker partitioning (Frankfurt), AssemblyAI, Deepgram, Speechmatics, Gladia (EU/French).
- Identity-level matching: use of cloud "speaker recognition" APIs has been scaled back by several hyperscalers; prefer a specialised vendor (Path A) for 1:1 or 1:N matching.
Path C — Self-host (open source)
- Toolkits and models: pyannote.audio (diarization, speaker embeddings), SpeechBrain (ECAPA-TDNN verification recipes), NVIDIA NeMo (TitaNet, Sortformer diarization), WeSpeaker, 3D-Speaker, Kaldi/k2 legacy recipes; ReDimNet-type compact models. Check licences (some models are gated or non-commercial).
- Anti-spoofing: models and baselines from the ASVspoof challenge community; AASIST-family.
- Hardware: HW1 (CPU or small edge device) for embedding extraction in real time; HW2 for large-scale batch diarization with ASR (e.g. Whisper plus pyannote).
Path D — Custom training
Worth it for narrowband telephony in specific languages, dialect mixes, noisy environments (industrial, vehicle) or to calibrate anti-spoofing to your attack landscape. Needs thousands of speakers with multiple sessions and channels; public corpora (VoxCeleb, CN-Celeb) carry licence and consent questions, so use for research with legal review.
State of the art (2026)
- Embedding models (ECAPA-TDNN, ResNet variants, self-supervised WavLM front-ends) are the standard; large-scale benchmarks (VoxCeleb speaker recognition challenges, NIST SRE) report equal error rates in the low single digits or below on clean, long audio, but telephony, short utterances, emotion, illness, aging and language mismatch degrade results considerably.
- Diarization improved with end-to-end and LLM-assisted pipelines, yet overlapping speech, many speakers and short turns remain error sources (diarization error rate is the key metric).
- Spoofing arms race: zero-shot voice cloning from seconds of audio defeats many text-independent systems unless countermeasures are used; NIST, ASVspoof and industry bodies treat synthetic speech as the main threat.
- Limits: a score is a probability, not proof; scores are not comparable across channels without calibration; templates are not revocable like passwords, so leaks are permanent.
Shared resources & combinations
- Same stack (asr + diarization): SPEECH-001 meeting transcription, SPEECH-002 call analytics, SPEECH-016 emergency call analysis, SPEECH-021 verbatim records.
- Counter-measures and fraud: SEC-011 deepfake detection, ANOM-022 APP-scam and mule detection, SPEECH-007 / SPEECH-019 (voice cloning and conversion are the attack and consent counterparts).
- Related identity modality: VIS-016 face recognition and biometric access control; CONV-002 voicebots needing caller authentication.
Effort, cost & KPIs
- Path A: 2-6 months for a bank-grade rollout (enrolment campaign, telephony integration, DPIA, fallback).
- Path B: 1-3 months for diarization and speaker-attributed transcripts.
- Path C: 1-3 months for a diarization service; 3-6 months with 1:1 verification and anti-spoofing hardening.
- KPIs: false-accept rate and false-reject rate at the chosen threshold, equal error rate, spoof detection rate, enrolment rate and opt-out rate, authentication time saved, fraud losses, diarization error rate, fallback rate.
People & change
Start with management and legal/compliance to decide whether voice biometrics is needed at all, because lower-risk alternatives (PINs, call-back) may suffice. Involve the DPO from the very beginning: voiceprints are biometric data, so a DPIA (Art. 35 GDPR) is almost certainly required, together with an AVV (Art. 28 GDPR) for each processor and an explicit consent or other Art. 9 basis. IT security assesses spoofing and deepfake risks; IT handles integration with telephony.
Works council: depends. If used only for customer authentication, co-determination is less likely. If agents are enrolled or call analytics identify employee speakers, it is a technical system suited to monitor behaviour and performance (§87(1) no. 6 BetrVG), so involvement is necessary; §90(1) no. 3 also requires early information. Public sector: staff council.
Customers must be informed and offered a non-biometric alternative. Run a pilot with a small customer group that opted in. Concerns are privacy and false rejections: address them with opt-in, short retention of embeddings and a human fallback process. Never let the match decide alone on high-stakes actions. AI literacy training should support agents in recognising limits and spoofing (Art. 4 AI Act).
See GUIDE-005, GUIDE-004 and GUIDE-013.
Risks, regulation & compliance
- EU AI Act (Regulation 2024/1689):
- Art. 5 prohibitions to flag clearly: biometric categorisation that infers race, political opinions, trade-union membership, religion, sex life or sexual orientation from biometric data (including voice); real-time remote biometric identification in publicly accessible spaces for law enforcement, except narrow listed exceptions with authorisation; emotion recognition in workplaces and education. Do not add these functions to a speaker product.
- Annex III(1): remote biometric identification systems and biometric categorisation are high-risk; biometric verification whose sole purpose is confirming that a person is who they claim to be is excluded from the remote-identification category. Therefore 1:1 authentication is typically not Annex III but 1:N identification and watchlist matching usually is. Hence classification is "depends".
- Diarization without identity: typically minimal risk. Art. 50 transparency may apply to synthetic-voice parts.
- GDPR: voiceprints used to uniquely identify a person are special-category biometric data (Art. 9); needs explicit consent or another Art. 9(2) ground, a DPIA (Art. 35), data minimisation, short retention, and a non-biometric alternative so consent is freely given. Recording callers needs transparency; watchlists of fraudsters need a solid legal basis and national DPA guidance (e.g. CNIL, German DSK).
- Security: template theft, replay, synthetic voices, insider access; encrypt templates, bind to device or channel, rate-limit attempts, combine with another factor. PSD2 strong customer authentication does not accept voice as the sole possession/inherence proof in many setups; clarify with your supervisor.
- Mitigations: opt-in, alternative path, per-group testing, human review for adverse outcomes, deletion on request, logging of every match, regular red-teaming with cloned voices.
Getting started
- Define the goal: diarization only, 1:1 authentication, or 1:N matching, and run a legal screening (AI Act and GDPR) before any technical work.
- For diarization, test pyannote.audio or a cloud API on 20 of your own recordings and measure diarization error.
- For authentication, shortlist two vendors, request EU hosting, anti-spoofing evidence and a DPIA kit; pilot opt-in with 500-1,000 volunteers.
- Calibrate the threshold on your channel mix and add step-up authentication; never single-factor for high-value actions.
- Red-team with voice-cloned samples before go-live and re-test every release.
Sources
- Regulation (EU) 2024/1689 (AI Act), Art. 5 and Annex III — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2016/679 (GDPR), Art. 4(14), 9, 35 — https://eur-lex.europa.eu/eli/reg/2016/679/oj
- pyannote.audio — https://github.com/pyannote/pyannote-audio
- SpeechBrain — https://github.com/speechbrain/speechbrain
- NVIDIA NeMo speaker tasks documentation — https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/speaker_recognition/intro.html
- VoxCeleb speaker recognition challenge — https://mm.kaist.ac.kr/datasets/voxceleb/
- ASVspoof challenge — https://github.com/asvspoof-challenge
- Phonexia speaker identification — https://www.phonexia.com/
- Microsoft Azure AI Speech, conversation transcription — https://learn.microsoft.com/azure/ai-services/speech-service/conversation-transcription