Catalog / Speech & Audio / SPEECH-003

Real-time speech translation & interpreting

Translate spoken language live into another language as subtitles or synthetic speech, for meetings, events, customer counters and conversations.

Parts needed to run it yourselfLanguage modelSpeechPrediction & time seriesVision

Multilingual meetings, events and service counters are limited by the languages people in the room speak. Streaming speech recognition, machine translation and speech synthesis (or a single end-to-end speech-to-speech model) now translate speech with a delay of a few seconds, as subtitles or as a synthetic voice. This is good enough for internal meetings, training, service desks and travel, but it is an aid, not a replacement for qualified interpreters where errors carry legal, medical or diplomatic consequences.

Problem & value

  • Today: human simultaneous interpreting needs booths, two interpreters per language per session and advance booking, so it is reserved for large events and institutions. Elsewhere teams fall back on a shared lingua franca (usually English), ad-hoc consecutive interpreting by colleagues, or phone translation apps in turn-taking mode that break the flow of conversation.
  • AI value: low-cost coverage of many language pairs on demand, for every participant on their own device, with a transcript as by-product. It makes meetings with non-native speakers more inclusive and lets small organisations serve customers, patients or citizens in languages they could never staff.
  • Evidence: no vendor-independent cost study is cited; treat vendor claims as marketing and measure on your own audio.

Variants

  • Meeting interpreter: translated captions or a selectable translated audio channel in video conferencing (Teams, Zoom, Webex, Google Meet).
  • Event / conference: audience scans a QR code and follows translated captions or audio on a phone; speaker audio comes from the mixing desk.
  • Service counter / reception / clinic: two-way face-to-face translation on a tablet (authorities, hospitals, hotels, retail). Higher stakes: see risks.
  • Call-centre / telephony: live translation inside a phone call between agent and customer (see CONV-002).
  • Private use: travel translation in earbuds or phones. Sign language is separate: see LANG-030.

How it works

  1. audio capture (mic / meeting bus / telephony)
  2. voice activity detection + audio cleanup
  3. streaming ASR (+ optional language identification, diarization)
  4. segmentation / stabilisation of partial hypotheses
  5. machine translation (NMT or LLM, glossary + context aware)
  6. output: captions | TTS (optionally voice-preserving) | bilingual transcript
  • Cascade vs. end-to-end: the classic cascade (ASR → MT → TTS) is modular, lets you plug in glossaries and logs intermediate text. End-to-end speech-to-speech models translate directly from audio to audio, which reduces latency and keeps prosody, but give less control and weaker auditing.
  • Simultaneous policy: the system must decide when to emit a translation before the sentence is finished. Waiting longer improves quality (word order, verbs at the end in German or Japanese) but increases lag. Typical systems accept a lag of a few seconds; the trade-off is tunable.
  • Human-in-the-loop: a moderator can pause or correct the glossary; for critical settings a human interpreter supervises or the AI output is labelled "indicative only". Output must tell listeners that speech is machine-translated.

Data preparation

Live translation needs little training data: the input is a live audio stream in good quality, the output text or synthetic speech. What matters is context. In an SMB this context sits in glossaries, product lists, offers, manuals and customer communication (ERP, DMS, mail, file shares), mostly in different languages and different spellings. Typical gaps: no uniform terminology across languages, abbreviations and product names, poor audio from room microphones, unclear speaker rights, and no list of language pairs actually needed.

Preparation steps: identify the two or three most used language pairs and settings; collect a bilingual glossary of terms, products, names and forbidden translations from existing translated documents and the translation memory (PREP-015, PREP-016); clean it of duplicates and conflicting variants (PREP-012); prepare the technical setup (headsets, conference microphone, separate audio channel per speaker); define a test set of 20 to 30 typical sentences per pair with human reference translations to compare tools (PREP-018). For sensitive meetings clarify which data may go to a cloud service (PREP-024).

Effort share: low, about 10 to 15 percent, mostly glossary and audio setup. See also GUIDE-003.

Implementation paths

Which path for whom

  • smb: Path A. Translation features of meeting platforms and EU-hosted interpreting products are mature and cheap enough for occasional multilingual meetings.
  • enterprise: Path B. Cloud speech and translation APIs integrate into own event, meeting and support platforms and scale with usage.
  • developer: Path B. Streaming speech and translation APIs are the standard building block for products.
  • public: Path C. Sensitive talks and sovereignty requirements favour open speech and translation models on German or EU infrastructure.
  • private: Path C. Local translation apps and open models work offline, for example when travelling.

Path A — Buy (off-the-shelf)

  • Meeting platforms: Microsoft Teams (Interpreter in meetings, translated captions), Zoom (translated captions, AI Companion), Google Meet (speech translation) and Webex (real-time translation) include translation for selected language pairs, often only in higher licence tiers.
  • Specialised vendors: DeepL Voice (EU, Germany) for meetings and face-to-face; KUDO AI, Interprefy, Wordly, Verbit/Interactions and Pocketalk/Timekettle devices for events and travel. Interpreter-marketplace vendors combine AI with human interpreters (hybrid).
  • Choose A when: meetings or events are the use case and the vendor offers EU data residency, a data processing agreement and no training on your audio.

Path B — Build on cloud APIs

  • Speech translation APIs: Azure AI Speech (speech translation incl. streaming and neural voices, EU regions), Google Cloud Speech-to-Text + Translation + Text-to-Speech, AWS Transcribe + Translate + Polly (Frankfurt/Ireland), plus speech-to-speech and realtime audio endpoints of frontier-model providers (OpenAI Realtime, Gemini Live). Check that the realtime endpoint is available in an EU region before sending personal data.
  • EU components: DeepL API for text translation, Mistral models (Voxtral for speech understanding) for the LLM and ASR stage, speechmatics (UK, with EU deployment options), and ElevenLabs or Azure voices for synthesis (check data residency).
  • Orchestration: WebRTC gateway (e.g. LiveKit) plus segmentation, glossary injection and logging.
  • Choose B when: translation must be embedded in your own product or workflow.

Path C — Self-host (open source)

  • End-to-end: Meta Seamless (SeamlessM4T v2 and SeamlessStreaming; research licence, check terms for commercial use) and Kyutai Hibiki (streaming French-English speech-to-speech, open).
  • Cascade: Whisper / faster-whisper or NVIDIA Parakeet / Canary for ASR (Canary also translates speech), NLLB-200, MADLAD-400 or an open LLM (Qwen, Mistral, Gemma) for translation, Piper, Kokoro or Coqui-style TTS. Streaming wrappers: whisper_streaming / WhisperLiveKit, Pipecat for pipelines.
  • Hardware: HW2 handles one to a few concurrent streams; HW3 for many parallel sessions or large LLM translation. On-device (HW1, phone NPUs) works for travel use and offline needs with smaller models and lower quality.
  • Choose C when: audio must not leave the premises (works council, health, authorities, board meetings).

Path D — Custom training

Fine-tune ASR and MT on in-domain audio and terminology or for low-resource languages and dialects. Needs tens to hundreds of hours of transcribed, translated audio; glossary injection is usually cheaper and sufficient.

State of the art (2026)

  • Quality: high-resource pairs (English, German, French, Spanish, Chinese, Japanese) are usable for general business speech; low-resource languages and dialects lag far behind. Research systems (Seamless, Hibiki) show end-to-end streaming with voice preservation approaching human-interpreter naturalness in controlled evaluations, but results on noisy real meetings remain weaker.
  • Latency: streaming cascades reach captions within a few seconds; human interpreters also lag a few seconds, but AI quality varies far more.
  • Trends: voice-preserving translation, LLM-based context-aware translation, runtime glossaries, built-in meeting-platform features.
  • Failure modes: dropped or hallucinated sentences, mistranslated numbers and negations, names and acronyms, overlapping speech, accents, code-switching, sarcasm and idioms, domain jargon. Whisper-style models can invent text on silence. Fluent but wrong output is the main danger because listeners cannot check it.

Shared resources & combinations

Effort, cost & KPIs

  • Path A: days; per-user licence or per-hour/event pricing; add-on fees for language packs are common.
  • Path B: 1-3 months for a product-grade pipeline; cost drivers are streaming audio minutes, TTS and number of target languages (one translation stream per target language).
  • Path C: 2-4 months; GPU server, monitoring and ongoing model updates.
  • KPIs: end-to-end lag (seconds), word error rate of ASR and translation quality (COMET/chrF on a sample of your own recordings, plus human rating), accuracy of numbers and named terms, share of listeners who report understanding, and incident count (critical mistranslations).

People & change

Involve the department lead and end users (who actually hold multilingual meetings) early, plus IT for audio and platform integration and the data protection officer for processor contracts and treatment of voice data (GUIDE-005). Management decides on the quality boundary: machine interpreting for internal and routine talks, human interpreters for contracts, negotiations and legal or medical content. Customers and partners should be told that machine translation is used (Art. 50 AI Act transparency where applicable, GUIDE-014).

Works council: depends. If only the audio is translated and nothing stored per person, co-determination may not be triggered; once the platform logs who says what or stores transcripts, §87(1) no. 6 BetrVG is likely applicable, and §90(1) no. 3 requires informing the council about new AI-supported work processes. Public sector: Personalrat.

AI literacy: support staff in speaking clearly, short sentences and checking critical numbers and names in writing (GUIDE-006, GUIDE-007). Communicate that the tool supplements professional interpreters and does not replace them where accuracy is critical, which addresses job fears of language staff (GUIDE-008). Start with a pilot in one team and one language pair, collect error examples, and set go/no-go criteria (GUIDE-009). Humans stay in control: participants can switch to human interpreting at any time.

Risks, regulation & compliance

  • EU AI Act: typically limited/minimal risk. Art. 50 transparency applies where synthetic speech or voice-cloned output could be mistaken for the real speaker: disclose machine translation and synthetic voice. Art. 4 AI literacy applies to staff. The use becomes high-risk if used in Annex III contexts (e.g. asylum or migration procedures, law enforcement, administration of justice, employment decisions); there, machine translation must not be the sole basis for decisions. Emotion recognition from voice in workplaces or schools is prohibited under Art. 5 and must not be bundled into translation products.
  • GDPR: voice is personal data (and may reveal special categories, e.g. in health contexts). Needs legal basis, information to all participants, DPA, EU hosting or valid transfer mechanism, retention limits for audio and transcripts, and a DPIA for systematic use. Works-council rules (Germany: BetrVG) apply in the workplace.
  • Sector rules: in court, medical consent, police and asylum procedures, qualified interpreters are generally required; AI translation only as an auxiliary tool.
  • Mitigations: label output as machine-translated, provide a human fallback, show source captions beside translation, glossaries for critical terms, pilot with domain experts, no training on customer audio, logging with short retention.

Getting started

  1. Define pairs, settings and stakes: internal meetings and service desks are fine to start; exclude legal, medical and asylum settings.
  2. Test the translation features of the platform you already use (Teams, Zoom, Meet) with real participants.
  3. Record 1-2 hours of your own typical audio, including jargon and accents, and compare 2-3 tools on the KPIs above, with native-speaker review.
  4. Sign the DPA, check data residency, inform participants and add the machine-translation notice.
  5. Build a glossary, define a human escalation path and review incidents monthly.

Sources

  1. Regulation (EU) 2024/1689 (AI Act), incl. Art. 4, 5, 50 and Annex III — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  2. Meta — Seamless Communication (SeamlessM4T v2, SeamlessStreaming) paper — https://arxiv.org/abs/2312.05187
  3. Kyutai — Hibiki, high-fidelity simultaneous speech-to-speech translation — https://arxiv.org/abs/2502.03382
  4. Microsoft — Azure AI Speech: speech translation overview — https://learn.microsoft.com/azure/ai-services/speech-service/speech-translation
  5. DeepL Voice — https://www.deepl.com/en/products/voice
  6. OpenAI Whisper — https://github.com/openai/whisper
  7. Microsoft — Interpreter in Microsoft Teams meetings and calls — https://support.microsoft.com/topic/c7efe2bb-535d-42ab-a5c4-d2d91619b46d
  8. European Data Protection Board — guidelines and opinions on GDPR and AI models — https://www.edpb.europa.eu/our-work-tools/general-guidance/guidelines-recommendations-best-practices_en