Localising a video into ten languages used to mean ten studio sessions, translators and voice actors. AI dubbing chains transcription, speaker separation, context-aware translation and expressive speech synthesis (optionally cloning the original speaker's voice and adjusting lip movement) so one master becomes many language versions in hours. It fits training, marketing, product and support content well; premium drama and anything involving a person's voice needs consent, review and disclosure.
Problem & value
- Today: subtitles are cheap but many viewers (training audiences, older people, mobile users) prefer audio in their own language. Classic dubbing requires translators, adapters, casting, studio time and mixing, so most corporate and e-learning content stays in one or two languages.
- AI value: faster turnaround, per-language marginal cost close to compute plus review, consistent brand voice across languages, and easy updates (re-render one changed sentence instead of re-booking a studio).
- Evidence: vendors report large time and cost reductions, but I found no independent, peer-reviewed benchmark of cost savings; treat vendor figures as marketing and measure a pilot yourself [1][2].
Variants
- Corporate / e-learning: training videos, onboarding, compliance courses, town halls (usually single speaker, clean audio).
- Marketing & social: product videos, ads, influencer content, with brand glossary and fast turnaround.
- Media & streaming: series, documentaries, news clips; hybrid with professional adapters and actors, often with union or actor-consent agreements on voice use.
- Education & public sector: lectures and public information in minority and migrant languages (see SPEECH-004 for captions).
- Live / real-time interpreting: a different latency profile, see SPEECH-003.
- Voice licensing / digital voice doubles: actors or executives license an official synthetic voice.
How it works
- master video
- audio extraction + stem separation (dialogue vs music/effects)
- ASR with word timestamps + speaker diarization
- transcript review (optional human)
- context-aware translation (LLM + glossary, length-controlled for timing)
- speech synthesis in target language (voice clone or stock voice, emotion/prosody transfer)
- timing fit (time-stretch, rephrase, pause control) + remix with music/effects stems
- optional visual lip-sync
- human QA (native speaker)
- AI-label + provenance metadata
- publish
- ASR + diarization: Whisper-class or commercial models give timestamped text per speaker (see SPEECH-001).
- Translation: LLMs outperform sentence-level MT on context, tone and terminology; prompts must also respect syllable/duration budgets so the dubbed line fits the original timing.
- Voice cloning: zero-shot or few-shot TTS reproduces timbre from seconds to minutes of reference audio; cross-lingual cloning keeps the identity while speaking another language. Speech-to-speech models transfer the original prosody and emotion directly.
- Lip-sync: video models re-render mouth movement to match new audio (HeyGen, Synthesia-style tools, open-source Wav2Lip derivatives).
- Human-in-the-loop: transcript/translation approval, native-speaker listening check, and mandatory consent verification before any voice is cloned.
Data preparation
Dubbing needs the video or audio master, ideally with separate dialogue, music and effects stems, a glossary with brand and product terms, a pronunciation lexicon and a documented consent and voice sample for each cloned speaker. In an SMB these live on file shares, in video tools, YouTube or Vimeo accounts and in the LMS; stems are often missing and consent is rarely recorded.
Typical gaps: only a mixed audio track (music under speech), poor recording quality, existing subtitles with errors, no glossary, unclear rights to voices and footage, and several language versions of the same video.
Preparation steps: 1. Inventory videos by value and language need, then select a first batch (GUIDE-001). 2. Obtain clean masters or stems; clean the audio first (SPEECH-008). 3. Transcribe and correct the source transcript (SPEECH-001), since transcription errors propagate into every language. 4. Compile glossary and lexicon with terminology owners (PREP-015, PREP-012). 5. Collect written consent and licence terms for each voice (PREP-024, PREP-025). 6. Define a native reviewer per target language and a small test clip.
Rough share of total effort: about 20 percent, mainly stems, glossary and consent. See GUIDE-003.
Implementation paths
Which path for whom
- smb: Path A, because mature dubbing platforms with voice cloning, translation and subtitles exist at SMB prices; choose one with EU hosting and clear consent handling.
- enterprise: Path B, because brand terminology, review workflows and a consent register must be integrated into own media and e-learning pipelines via cloud APIs.
- developer: Path B, because speech, translation and voice APIs can be orchestrated into custom localisation pipelines; self-hosting needs HW3 and is rarely worth it.
Path A — Buy (recommended for most SMBs)
- Dubbing platforms: ElevenLabs Dubbing (Studio and API), HeyGen Video Translate, Rask AI, Dubverse, Deepdub (enterprise/media), Synthesia (avatar videos with translation). Check: speaker count handling, glossary support, editable transcript, export of stems/SRT, data retention and training opt-out.
- EU-relevant options: DeepL Voice (DE, interpreting/meetings rather than dubbing), Acapela Group (FR/BE, synthetic voices), Dubformer (DE, media dubbing). Verify current EU hosting and sub-processors in the DPA.
- Choose A when: under a few hundred hours per year, no strict on-premises rule, and consent handling can be done by contract.
Path B — Build on cloud APIs
- ASR: Azure AI Speech, Google Cloud Speech-to-Text, Deepgram, AssemblyAI, Mistral Voxtral (EU provider), OpenAI transcription models.
- Translation: frontier LLMs (Claude, Gemini, GPT, Mistral) with glossary prompts; DeepL API (DE) for terminology-controlled MT.
- Synthesis and cloning: ElevenLabs API (dubbing endpoint plus multilingual TTS), Azure AI Speech custom neural voice (gated access and consent verification required by Microsoft), Google Cloud TTS, Cartesia, OpenAI TTS.
- Orchestration: queue + workflow engine (n8n, Temporal) with a review UI; store jobs and consent in EU region.
- Choose B when: you need glossaries, CMS/LMS integration, batch processing or custom review.
Path C — Self-host (open source)
- ASR/diarization: Whisper large-v3 / faster-whisper / WhisperX, pyannote.audio, NVIDIA Parakeet and Canary.
- Translation: open LLMs (Qwen, Llama, Mistral, Gemma), NLLB-200, MADLAD-400, SeamlessM4T.
- TTS / voice cloning: open-weight models such as XTTS-v2 (Coqui licence is non-commercial, check), F5-TTS, Fish Speech, Kokoro, Chatterbox, Orpheus, StyleTTS 2, CosyVoice; Meta Seamless Expressive for expression transfer. Licences vary: several are research-only, so check commercial terms.
- Stems and lip-sync: Demucs for separation; LatentSync or Wav2Lip derivatives for lip-sync (research-grade).
- Hardware: HW2 for ASR, translation and TTS in small batches; HW3 for several models in parallel and lip-sync.
Path D — Custom training
Worthwhile for a brand voice in a low-resource language, a licensed actor voice or media-grade quality. Needs roughly 30 minutes to several hours of clean, consented studio audio per voice, a pronunciation lexicon and a listener evaluation panel. Fine-tune a TTS model; do not train on voices without documented written consent.
State of the art (2026)
- End-to-end speech-to-speech and expressive TTS now preserve emotion, pace and speaker identity across languages far better than 2023-era cascades; commercial dubbing tools handle multiple speakers and background music routinely.
- Quality limits: timing mismatch between languages (German and Finnish often run longer than English), mispronounced names and terms, flat emotion, speaker-confusion errors in overlapping speech, hallucinated words in ASR on noisy audio.
- Cloning needs little audio: short samples suffice for convincing clones, which is exactly why consent checks and provenance matter and why voice-based authentication is no longer safe (see SPEECH-010).
- Provenance: C2PA content credentials, signed metadata and imperceptible audio watermarks are being adopted; the EU transparency Code of Practice expects at least two machine-readable marking techniques [3][4].
Shared resources & combinations
- Shared stack: asr, diarization, translation and tts building blocks are shared with SPEECH-001 (meetings), SPEECH-003 (live translation), SPEECH-004 (captions, usually delivered together with the dub) and SPEECH-006 (narration). One GPU server (HW3) can host ASR + TTS for all of them.
- Combines with: GEN-003 (avatar presenter videos localised per language), GEN-002 (video generation), VIS-025 (video understanding for scene and speaker cues), SPEECH-018 (voice banking uses the same consent practice).
- Risk neighbours: SPEECH-010 (voice biometrics and spoofing) and SPEECH-019 (real-time voice conversion).
Effort, cost & KPIs
- Path A: days to set up; pricing is typically per minute of output per language or by subscription.
- Path B: 1-3 months incl. glossary, review UI, LMS/CMS integration and consent register.
- Path C: 3-6 months with evaluation; GPU server plus ML-ops.
- KPIs: cost and turnaround per finished minute per language, review edit rate, MOS / listener preference versus subtitles or human dub, speaker similarity, term-accuracy, completion rate and watch time in the new language, complaints about voice or translation, share of assets with valid consent and labels.
People & change
Involve management (brand and budget decisions), the department lead for marketing or training, legal/compliance (voice rights, personality rights, Art. 50 AI Act labelling), the DPO (voice is personal data and possibly biometric when used to identify) and native-speaking reviewers. IT is needed only for platform access. Agree with speakers individually before cloning their voice.
Works council: normally no, since the system does not monitor employees. It depends if employees' voices are cloned or the tool is used in internal training where staff voice is replaced; then involve the council under §90(1) no. 3 BetrVG (information on AI-planned work processes) and clarify consent and reuse limits. Public sector: Personalrat.
Data protection: AVV (Art. 28 GDPR) with the provider, explicit consent for voice cloning, retention limits for voice samples; a DPIA is likely if many staff voices are cloned.
Transparency: label synthetic voices and dubbed content (Art. 50 AI Act, in force since 2 Aug 2026).
AI literacy: short training on consent, review and labelling (GUIDE-006).
Pilot: one short video in two languages, reviewed by native speakers. Typical concerns: loss of voice talent, misuse of cloned voices and translation quality. Address them with consent records, a human review before release and watermarking. See GUIDE-008, GUIDE-014 and GUIDE-010.
Risks, regulation & compliance
- EU AI Act: limited risk. Art. 50(2) obliges providers of generative systems to mark synthetic audio/video in a machine-readable and detectable way; Art. 50(4) obliges deployers to disclose deepfakes (audio or video resembling real persons that could be mistaken for authentic), with lighter duties for evidently artistic or creative work. Obligations apply from 2 Aug 2026 (transition for earlier systems to Dec 2026 per Commission material); a Code of Practice was finalised in 2026 [3][4]. A dub that uses a real person's voice in a language they never spoke should be labelled. Using cloned voices to deceive, e.g. impersonation fraud, is unlawful; manipulating people through subliminal or exploitative techniques falls under Art. 5 prohibitions, not under ordinary localisation.
- GDPR: a voice is personal data and can be biometric data if used to identify a person (Art. 9 only then). Cloning requires a lawful basis, in practice explicit, informed, revocable consent (and for employees, a separate agreement since consent at work is rarely free). Define scope (languages, media, duration), compensation, deletion on exit and rules for the voice model after contract end. Run a DPIA for large-scale cloning or voice archives.
- Personality and copyright law: voice and likeness rights vary by member state (Germany: allgemeines Persoenlichkeitsrecht); collective agreements for voice actors and performers' rights apply to media productions.
- Mitigations: consent register, native-speaker QA, visible and machine-readable labels, vendor opt-out from model training, EU data residency, regular review of mistranslations in regulated content (safety instructions, medical, legal).
Getting started
- Pick one content type (e.g. 10 training videos) and two target languages; capture baseline cost and time per language.
- Obtain written consent from speakers and define voice-use scope; add a standard disclosure line and AI label.
- Pilot 2-3 Path A tools on the same clips, with a glossary, and let native speakers score them blind against subtitles.
- Decide the review depth (e.g. full review for external, spot check for internal) and set up export of SRT plus audio stems.
- Measure viewer engagement and review effort; scale to Path B only if integration or volume demands it.
Sources
- ElevenLabs - Dubbing documentation - https://elevenlabs.io/docs/capabilities/dubbing
- HeyGen - Video Translation - https://www.heygen.com/translate
- European Commission - Code of Practice on transparency of AI-generated content (FAQ on signing) - https://digital-strategy.ec.europa.eu/en/faqs/signing-code-practice-transparency-ai-generated-content
- AI Act Service Desk / artificialintelligenceact.eu - Article 50 transparency rules - https://artificialintelligenceact.eu/transparency-rules-article-50/
- Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Microsoft - Azure AI Speech custom neural voice, limited access - https://learn.microsoft.com/azure/ai-services/speech-service/custom-neural-voice
- OpenAI Whisper - https://github.com/openai/whisper
- pyannote.audio - https://github.com/pyannote/pyannote-audio
- C2PA - Content Credentials specification - https://c2pa.org/specifications/
- EDPB - Guidelines on virtual voice assistants (voice data as personal data) - https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-022021-virtual-voice-assistants_en