Live captioning & subtitling

Deaf and hard-of-hearing people, non-native speakers and viewers in silent environments depend on captions, yet human stenographers or respeakers are scarce and expensive. Streaming speech recognition now produces captions with a delay of roughly one to a few seconds, in dozens of languages and with optional live translation. Human captioners remain the quality benchmark for high-stakes events; AI makes captions affordable for everything else.

Problem & value

• Today: live captions come from professional stenographers (CART), respeakers (a human repeats speech to a trained recogniser) or nothing at all. Capacity is limited, booking takes days and cost is high, so most meetings, lectures, town halls and local events have no captions.
• AI value: on-demand captions for every session, translated captions for multilingual audiences, searchable transcripts as by-product, and a hybrid model where a human editor corrects the AI stream for important events.
• Legal driver: the European Accessibility Act (Directive (EU) 2019/882) applies since 28 June 2025 to consumer-facing services such as e-commerce, e-books and audiovisual media access services. The Audiovisual Media Services Directive requires media providers to make services progressively accessible, and public bodies must meet the Web Accessibility Directive (EU) 2016/2102 for their websites and apps. EN 301 549 is the harmonised standard; it requires captions for live synchronised media where applicable [1][2][3].

Variants

• Meetings and webinars: built-in captions in Teams, Zoom, Google Meet, Webex; third-party caption overlays for hybrid events.
• Education: lecture captioning in universities (accessibility offices, lecture-capture systems); see EDU-001 for related teaching tools.
• Conferences and live events: on-site displays and per-attendee phone view via QR code, with translated streams (overlap with SPEECH-003).
• Broadcast and streaming media: live subtitles for news, sport and public service broadcasters, delivered as DVB subtitles, EBU-TT-D or CEA-608/708; mostly respeaking plus AI.
• Public sector: council sessions, citizen meetings and ministry press events (see SPEECH-021 for verbatim official records, which have stricter accuracy duties).
• Private use: phone-based live captions for conversations (Android Live Caption and Live Transcribe, iOS Live Captions) for people with hearing loss.

How it works

1. audio source (mic / mixer / RTMP-SRT / meeting platform)
2. audio capture + voice activity detection (chunks of ~100-250 ms)
3. streaming ASR (partial hypotheses, then finalised segments)
4. custom vocabulary, punctuation, casing, number formatting
5. [optional] machine translation per target language
6. caption formatting (line length, ≤2 lines, reading speed, timing)
7. [optional] human editor / captioner correction queue
8. delivery (overlay, WebVTT/HLS, SDI embedder, event display)
9. archive + post-event cleanup into final subtitles

• Streaming ASR: the model emits partial text that is revised as more audio arrives, then commits a final segment. Lower latency means more revisions (flicker) and more errors; most products let you trade delay for stability.
• Human-in-the-loop: for legal, medical, broadcast or board-level events a trained captioner monitors the stream, injects corrections and speaker labels, and can take over; the audience must be told whether captions are automatic.

Data preparation

The inputs are live or recorded audio and video plus, for better quality, glossaries and speaker names. In an SMB the material lives in video platforms, event tools, the website CMS, file shares and with agencies; audio quality is the biggest factor. Typical gaps: poor microphones and room acoustics, background music, mixed languages, unknown names and technical terms, missing subtitle style rules (line length, reading speed) and no archive of already corrected subtitles.

Preparation steps: improve audio at the source (headset or lavalier microphones, separate channels); provide a glossary with names, products and abbreviations before each event (PREP-015); decide on style rules for subtitles and collect good examples of corrected files such as SRT or VTT (PREP-016); convert legacy video and audio formats into standard formats (PREP-007); store finished subtitles together with the media and add metadata for search (PREP-010). For personal data in recordings (speakers, audience questions) clarify consent and retention (PREP-025). Build a small test set of recordings from your own events to compare tools (PREP-018).

Effort share: low, about 10 to 20 percent, mainly audio setup and glossaries. See also GUIDE-003.

Implementation paths

Which path for whom

• smb: Path A. Captioning in meeting and video platforms and mature subtitle tools is ready to use, EU-hosted options exist, and the process is standard.
• enterprise: Path B. Streaming ASR APIs integrate into own event, intranet and video platforms and allow glossaries and several languages at scale.
• developer: Path C. Open ASR such as Whisper or similar streaming models can be embedded without per-minute cost and give full control over latency and vocabulary.
• public: Path C. Accessibility duties and procurement rules favour open models on sovereign EU infrastructure with transparent processing.
• private: Path C. Local apps with Whisper create subtitles for own videos for free and offline.

Path A — Buy (recommended for most organisations)

• Platform-native captions: Microsoft Teams live captions and translated captions, Zoom, Google Meet, Webex. Cheapest start, limited vocabulary control and often no EU-only guarantee for translation.
• Event and caption specialists: Verbit (incl. VITAC), Ai-Media (Lexi, hybrid with human captioners), Wordly, Interprefy; European options include Speechmatics (UK), Gladia (FR) and Mistral-based services. Verify hosting region and sub-processors for each.
• Choose A when: you need captions for recurring meetings or events and have no developers. Check data residency, DPA, whether recordings are used for training, custom vocabulary, languages and export formats.

Path B — Build on cloud APIs

• Streaming ASR APIs: Azure AI Speech (real-time transcription with phrase lists, EU regions such as West Europe), Google Cloud Speech-to-Text v2 (Chirp models, EU location), Amazon Transcribe streaming (Frankfurt, Ireland), Deepgram (streaming Nova models, EU endpoint), AssemblyAI streaming, Speechmatics real-time API (strong European language coverage, EU deployment), Gladia (France, EU provider), Mistral Voxtral (EU provider, transcription API) [5][6].
• Translation: DeepL API (Germany, EU), Azure Translator, Google Cloud Translation; or the same vendor's built-in live translation.
• Delivery: WebSocket/WebRTC to a caption web page; LiveKit or Daily for conferencing; RTMP/SRT ingest into OBS, vMix or hardware encoders with caption insertion (CEA-708 or WebVTT in HLS).
• Choose B when: you embed captions in your own product, need custom workflows or want per-language streams and archiving under your control.

Path C — Self-host (open source)

• Models: OpenAI Whisper (large-v3, turbo) is offline by design; use streaming wrappers such as whisper_streaming / WhisperLiveKit (SimulStreaming), faster-whisper or whisper.cpp. Natively streaming open models: NVIDIA Parakeet and Canary (NeMo, FastConformer/RNN-T), Kyutai STT (low-latency streaming, French/English), Vosk and Sherpa-ONNX for small devices, and Mistral Voxtral open weights [7][8].
• Translation: Opus-MT / MarianMT, NLLB-200, or an open LLM for polishing.
• Stack: WebSocket server (e.g. WhisperLive, Sherpa-ONNX server), VAD (Silero), caption formatter, OBS plugin or web overlay; Kubernetes optional.
• Hardware: HW1 for small models on CPU or NPU (one or two speakers, lower accuracy), HW2 (one 16-32 GB GPU) serves several parallel streams with a Whisper-large-class or Parakeet model. Ops effort: moderate, mainly latency tuning, monitoring and fallback.
• Choose C when: audio must not leave premises (works councils, board meetings, confidential hearings) or volumes make per-minute pricing expensive.

Path D — Custom training

Fine-tune only for strong dialects, domain jargon or low-resource languages; needs tens to hundreds of hours of transcribed target audio. Try vocabulary boosting and context biasing first.

State of the art (2026)

• Streaming ASR quality for major European languages in clean audio is close to human transcription on standard benchmarks; latency of 300 ms to 1 s for partial results is typical in commercial APIs. Open streaming models (Parakeet, Kyutai STT, streaming Whisper variants) have narrowed the gap to cloud services.
• Limits and failure modes: overlapping speakers, accents and dialects (Swiss German, Bavarian, Scots), far-field microphones, music, domain terms, numbers and proper names; delayed or flickering captions; missing speaker changes; wrong text shown as fact. Reported word error rates from vendors rarely reflect your venue, so test on your own recordings.

Shared resources & combinations

• Same stack (asr + streaming + diarization): SPEECH-001 meeting transcription and summaries, SPEECH-002 call analytics, SPEECH-003 real-time translation, SPEECH-021 verbatim records, SPEECH-011 voice control. One GPU ASR server (HW2) can serve live captions and batch jobs.
• Downstream: post-event cleanup into final subtitles, dubbing and localisation (SPEECH-007), audio description for blind users (SPEECH-014).
• Shared infrastructure: conferencing platform integrations, translation service, glossary and vocabulary management.

Effort, cost & KPIs

• Effort: A hours to days (per-hour pricing, human-assisted tiers cost several times more); B 1-3 months; C 2-4 months incl. GPU server and on-call.
• KPIs: word error rate and NER-style accuracy on own test set, caption delay (end of speech to display), flicker rate, reading speed (words per minute), coverage (share of events captioned), user satisfaction among deaf and hard-of-hearing users, correction effort for final subtitles.

People & change

Involve the department lead (events, communication, training), end users such as moderators and editors, IT for streaming and platform integration, and the data protection officer for processors and voice data (GUIDE-005). Legal or compliance clarify accessibility duties (for example accessibility rules for websites and events) and what quality counts as sufficient. People who use captions, such as deaf and hard-of-hearing colleagues or customers, should be asked directly and included in the pilot.

Works council: depends. Pure live captions of events and public videos usually do not monitor employees; if internal meetings are captioned and transcripts are stored with speaker names, §87(1) no. 6 BetrVG may apply, and §90(1) no. 3 requires informing the council about AI-supported work processes. Public sector: Personalrat (GUIDE-004).

AI literacy: support moderators in speaking clearly and in checking subtitles of important content before publication (GUIDE-006). Communicate that captions are an accessibility gain for everyone and do not replace professional live captioners where accuracy is critical, which also addresses job concerns of subtitlers (GUIDE-008). Pilot with one recurring event, measure error rate and feedback, then extend (GUIDE-009). Humans stay in control: recorded content is reviewed before publishing, live errors are corrected on request and disclosed as machine-generated.

Risks, regulation & compliance

• EU AI Act: minimal risk for captioning itself; AI literacy (Art. 4) applies. Transparency duties under Art. 50 mainly concern synthetic content, not transcripts, but labelling captions as automatic is good practice. Caution: if the same audio is also used for emotion recognition of employees or students, that is prohibited under Art. 5(1)(f); keep captioning free of such analytics.
• GDPR: speech is personal data; audiences, speakers and bystanders must be informed. Prepare a DPA with the vendor, EU data residency, no vendor training on your audio, minimal retention and a lawful basis (often legitimate interest or public task; employee settings may need works council agreement). Public streams and recordings need a clear information notice.
• Accessibility law: an unreliable auto-caption may not satisfy legal obligations or accommodation duties; some deaf users and public bodies require human-verified captions for important content. Document how quality is ensured and keep a fallback.
• Mitigations: vocabulary lists, human monitor, backup channel, "automatic captions" notice, correction before publishing on-demand versions.

Getting started

1. Define who needs captions, which events, languages and which quality level (AI only vs. human-assisted) is acceptable; involve deaf and hard-of-hearing users early.
2. Record 2-3 hours of real, representative audio incl. hard parts and build a small test set with reference transcripts.
3. Try the captions built into your meeting platform first, then compare two Path A or B services on your test set for accuracy, delay and cost; check EU hosting and DPA.
4. Prepare vocabulary lists (names, products, terms) and a rehearsal checklist: microphones, network, display, fallback.
5. Run a pilot at a real event with a human monitor, measure KPIs and collect feedback before rolling out.

Sources

1. Directive (EU) 2019/882 (European Accessibility Act) — https://eur-lex.europa.eu/eli/dir/2019/882/oj
2. European Commission — European Accessibility Act overview — https://commission.europa.eu/strategy-and-policy/policies/justice-and-fundamental-rights/disability/union-equality-strategy-rights-persons-disabilities-2021-2030/european-accessibility-act_en
3. ETSI EN 301 549 — Accessibility requirements for ICT products and services — https://www.etsi.org/deliver/etsi_en/301500_301599/301549/
4. Ofcom — Measuring live subtitling quality — https://www.ofcom.org.uk/tv-radio-and-on-demand/information-for-industry/guidance/subtitling
5. Microsoft — Azure AI Speech real-time speech to text — https://learn.microsoft.com/azure/ai-services/speech-service/get-started-speech-to-text
6. Mistral AI — Voxtral — https://mistral.ai/news/voxtral
7. NVIDIA NeMo ASR models (Parakeet, Canary) — https://github.com/NVIDIA/NeMo
8. Whisper — Robust Speech Recognition via Large-Scale Weak Supervision — https://github.com/openai/whisper
9. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj