Intro & scope
Use cases where audio is the primary modality: speech to text (pattern SPEECH.TRANSCRIBE), text or voice to synthetic speech (SPEECH.SYNTH), and non-transcription audio analysis or processing (SPEECH.AUDIO). Text-only chatbots and voice agents live in CONV; text translation in LANG; video understanding in VIS.
Scope boundaries between neighbours: - SPEECH-001 (general meetings) vs SPEECH-021 (court/parliament): 021 is the official-record case with legal-grade accuracy and human correction (high risk); 001 is convenience notes. - SPEECH-003 (live translation, any setting, cascade or S2S) vs SPEECH-019 (accent/voice conversion on call-centre lines) vs SPEECH-007 (offline dubbing): 003 is interpreting for communication, 019 is agent-voice transformation on live phone lines, 007 is post-production localisation. - SPEECH-004 (captioning) is text output for accessibility; SPEECH-003 adds translation/voice output. - SPEECH-009 (non-speech events) vs SPEECH-015 (wildlife) vs SPEECH-023 (voice chat moderation): different domains and risk profiles. - SPEECH-010 (voice biometrics) is a shared component for 002, 016, 022; biometric identification is high-risk under the AI Act, 1:1 verification is generally outside Annex III.
Overview
| ID | Title | Pattern | Maturity | Path | HW | AI Act |
|---|---|---|---|---|---|---|
| SPEECH-001 | Meeting transcription, summaries & action items | TRANSCRIBE | mainstream | A | HW2 | minimal |
| SPEECH-002 | Contact-centre call analytics & QA | TRANSCRIBE | mainstream | A | HW3 | depends |
| SPEECH-003 | Real-time speech translation & interpreting | TRANSCRIBE | early-adopter | A | HW2 | limited |
| SPEECH-004 | Live captioning & subtitling | TRANSCRIBE | mainstream | A | HW2 | minimal |
| SPEECH-005 | Professional dictation | TRANSCRIBE | mainstream | A | HW2 | minimal |
| SPEECH-006 | Text-to-speech narration | SYNTH | mainstream | A | HW1 | limited |
| SPEECH-007 | Voice cloning & AI dubbing | SYNTH | early-adopter | A | HW3 | limited |
| SPEECH-008 | Audio enhancement & podcast editing | AUDIO | mainstream | A | HW1 | minimal |
| SPEECH-009 | Audio event detection & acoustic monitoring | AUDIO | early-adopter | A | HW1 | depends |
| SPEECH-010 | Speaker identification & voice biometrics | AUDIO | mainstream | A | HW1 | depends |
| SPEECH-011 | Hands-free voice control for industrial work | TRANSCRIBE | mainstream | A | HW1 | depends |
| SPEECH-012 | Pronunciation & speech-therapy coaching | TRANSCRIBE | early-adopter | B | HW2 | depends |
| SPEECH-013 | Music information retrieval | AUDIO | mainstream | C | HW2 | minimal |
| SPEECH-014 | Automatic audio description of video | SYNTH | early-adopter | B | HW3 | limited |
| SPEECH-015 | Bioacoustics & wildlife monitoring | AUDIO | early-adopter | C | HW1 | minimal |
| SPEECH-016 | Emergency call (112) analysis | TRANSCRIBE | early-adopter | A | HW3 | high |
| SPEECH-017 | AI hearing aids & hearables | AUDIO | early-adopter | A | HW1 | high |
| SPEECH-018 | Assistive speech: voice banking, atypical ASR, AAC | SYNTH | early-adopter | A | HW1 | limited |
| SPEECH-019 | Real-time accent & voice conversion | SYNTH | early-adopter | A | HW3 | limited |
| SPEECH-020 | Mission-critical radio transcription | TRANSCRIBE | early-adopter | A | HW2 | depends |
| SPEECH-021 | Verbatim records of hearings & sessions | TRANSCRIBE | early-adopter | C | HW3 | high |
| SPEECH-022 | Spoken language & dialect identification | AUDIO | early-adopter | C | HW1 | high |
| SPEECH-023 | Real-time voice chat moderation | AUDIO | early-adopter | A | HW3 | depends |
Shared stack
One audio gateway, one ASR service, one TTS service, reused across families (CONV voice agents, LANG translation, DOC, VIS video).
Common building blocks: audio ingest (WebRTC/SIP/RTP, VAD, resampling to 16 kHz); ASR (Whisper-class or Parakeet/Canary-class, streaming variant for live cases); diarization (pyannote-class); text post-processing (LLM for summaries, punctuation, glossary correction); TTS (open multilingual model or EU-hosted API); an audio classifier (PANNs/BEATs-class) for event detection; a vector DB for transcript search; human-review UI; audit log with retention rules.
HW1 (CPU / edge / single small GPU, <= 8 GB): SPEECH-006, 008, 009, 010, 011, 015, 017, 018, 022
- whisper.cpp / faster-whisper small-medium (int8), Silero VAD, ECAPA-TDNN speaker embeddings, small audio classifier (ONNX), Piper or Kokoro-class TTS, small 1-4B LLM for commands.
- Runs on a laptop, mini-PC or Jetson-class edge device; hearables use dedicated DSP/NPU firmware.
HW2 (one 16-24 GB GPU): SPEECH-001, 003, 004, 005, 012, 013, 020
- faster-whisper large-v3-class or streaming ASR, pyannote diarization, 7-14B LLM (vLLM) for summaries and structuring, NLLB/MADLAD or LLM for translation, XTTS/F5-class TTS, Qdrant/pgvector for search.
- Serves tens of concurrent streams or batch jobs; live captioning with 1-3 s latency.
HW3 (multi-GPU / 48-80 GB, or dedicated cluster): SPEECH-002, 007, 014, 016, 019, 021, 023
- Batched large ASR plus diarization on one GPU pool, 30-70B LLM or VLM for analytics/description, streaming speech-to-speech or voice-conversion models, lip-sync/video models for dubbing, autoscaling with Kubernetes, HA and failover for 24/7 operation (emergency, moderation).
- Use EU-region hosting or on-premises for call, court and emergency audio.
Cross-cutting rules
- Voice is personal data; biometric templates are special-category data (GDPR Art. 9). Keep voiceprints in a separate store with explicit consent and deletion.
- Synthetic or cloned voices need AI Act Art. 50 disclosure (watermark/label); cloning requires documented consent of the speaker.
- Recording consent and works-council agreements for call/meeting audio; define retention and redaction (PII masking) before the transcript reaches the LLM.
- Always keep a human in the loop for high-risk uses (016, 017, 021, 022).