Automatic audio description of video for blind people

Blind and low-vision people cannot access what is only shown on screen: actions, faces, scene changes, slides, charts, on-screen text. Audio description (AD) adds a spoken narration in the natural pauses. Until recently it was produced by trained describers in a studio at high cost per hour, so only a small share of video had it. Vision-language models plus neural text-to-speech now draft the script and voice in minutes; a human editor reviews it where quality matters.

Problem & value

• Today: AD is written by professional describers, recorded by a narrator and mixed in. It is mostly available for broadcast drama, cinema and some streaming titles. Corporate, education, public-sector and social video, and almost all live content, have none.
• Legal pull: the European Accessibility Act (Directive (EU) 2019/882) has applied since 28 June 2025 to e-commerce, consumer audiovisual media services and related services; the Audiovisual Media Services Directive (2010/13/EU, as amended) requires media service providers to make services continuously and progressively more accessible; WCAG 2.2 success criteria 1.2.3 and 1.2.5 require audio description for prerecorded video at levels A/AA, and public-sector bodies must meet them under the Web Accessibility Directive (EU) 2016/2102 [1][2][3].
• AI value: drafting is cheap and fast, which makes coverage of the long tail (lectures, training, product and municipal videos, archives) feasible. Quality for fiction is not yet equal to a skilled describer, so the realistic model is AI draft plus human edit.
• Scale of need: the World Health Organization estimates at least 2.2 billion people have a near or distance vision impairment [4].

Variants

• Broadcaster / streaming: AI-assisted authoring tool for describers; human sign-off before release.
• Education and corporate video: lectures, onboarding and product videos where much of the information is on slides or screen recordings; often "text-to-speech narration of on-screen content".
• Public sector: council meetings, information videos, campaign clips to meet BITV 2.0 / EN 301 549.
• Social and private use: a viewer-side assistant that describes any video on demand (browser extension or phone app), with no publisher involvement.
• Live content: event streams and sport; experimental, latency and error rate are the problem.

How it works

1. video (+ audio, subtitles)
2. shot/scene segmentation + speech/silence detection (find gaps in dialogue)
3. frame sampling / video VLM
4. visual facts per shot (who, what, where, on-screen text)
5. LLM writes concise description per gap, using style guide, character names, context memory
6. length fit: words ≈ available seconds × speaking rate; else shorten or flag "extended AD"
7. human review (edit, approve, fix names/sensitive content)
8. neural TTS (voice, speed)
9. mix with original audio (ducking) or deliver as separate track
10. player: selectable AD track (HLS/DASH alternate audio) or WebVTT "descriptions" track

• Visual understanding: a VLM or video-capable multimodal model describes keyframes or short clips; OCR adds exact on-screen text; the dialogue transcript (ASR) avoids describing what is already heard.
• Consistency: a running "story so far" and a character list prevent "a man" turning into five different people. Identifying real people by face must not be done (see risks); use script, credits or user-supplied cast lists instead.
• Timing: standard AD fits into pauses; extended AD pauses the video (possible in web players, not in linear broadcast).
• Human-in-the-loop: mandatory for fiction, news and anything regulated; sampling-based QA for low-risk internal video.

Data preparation

The raw material is video with audio, plus optional scripts, subtitles, shot lists and a style guide. In an SMB, broadcaster or public body these sit in a media asset system, on file shares, in a CMS or in e-mail threads from agencies. Typical gaps are missing subtitles or scripts, inconsistent character names and terminology, mixed codecs, no glossary, and unclear usage rights for third-party footage.

Steps: 1. Inventory videos by priority, for example public-facing and legally required first (PREP-001). 2. Transcode to a standard format with ffmpeg and extract audio and keyframes. 3. Create or collect transcripts and subtitles with ASR (see SPEECH-004 for captioning). 4. Build a glossary of names, places and terms (PREP-015). 5. Collect good human-written audio descriptions as style examples (PREP-016).

Effort is low: no model training is needed, so preparation is perhaps 10 to 20 percent, mostly glossary, style guide and rights clearance.

Implementation paths

Which path for whom

• enterprise: Path B, because vision-language and TTS APIs on EU regions deliver broadcast-grade drafts quickly and a human editor finishes them.
• public: Path B on an EU or German sovereign cloud, because public bodies need sovereignty and procurement compliance and volume is moderate; Path A only for certified products.
• developer: Path B, because orchestration around APIs is the product and HW3 self-hosting is rarely worth it.
• private: Path A, because built-in accessibility features and ready apps need no setup.

Path A — Buy (off-the-shelf)

• AD production services with AI drafting: US vendors such as Verbit and 3Play Media offer AI-assisted AD with human review; European subtitling/localisation houses and broadcaster accessibility departments offer human or hybrid AD. Check current offers and EU data location.
• Video platforms: some video hosting and LMS products add AI-generated AD tracks or are adding them; verify EU data location and track format.
• Choose A when: you publish a manageable volume and want a managed human-reviewed service (a service-level agreement on quality) rather than building tooling.

Path B — Build on cloud APIs (recommended default)

• Video understanding: Google Gemini (native video input, Vertex AI EU regions), OpenAI GPT vision-capable models, Anthropic Claude on frames (via AWS Bedrock / Vertex EU regions), Mistral multimodal models (EU provider); structured output gives time-coded JSON.
• TTS: Azure AI Speech neural voices (EU regions, SSML for rate/pause), Google Cloud TTS, Amazon Polly, ElevenLabs (EU data residency option), plus EU providers such as Acapela (FR/BE) and ReadSpeaker (SE).
• Plumbing: FFmpeg for shot detection, ducking and mixing; PySceneDetect; WebVTT output; n8n or a queue-based pipeline; review UI (simple timeline editor) is the main custom work.

Path C — Self-host (open source)

• VLMs: Qwen2.5-VL / Qwen3-VL, InternVL, Gemma 3 (vision), Llama vision variants and LLaVA-Video-class models, served with vLLM; PySceneDetect and Whisper / faster-whisper for the dialogue gaps.
• TTS: Piper, Kokoro, Coqui-derived models, F5-TTS or Orpheus-class open voices; check licences for commercial use and for any voice-cloning component.
• Hardware: HW3 (one 48 GB+ GPU) for a 30B-class VLM at acceptable throughput; HW2 works with ~7–8B VLMs and sparse frame sampling at lower quality. Offline batch is fine: no real-time need.

Path D — Custom training

Worth it for a broadcaster or archive with thousands of hours of professionally written AD: fine-tune a VLM on (clip, description, timing) pairs to learn house style, brevity and what describers leave out. Needs rights to the training material and consent from describers; licensed AD datasets are scarce, so expect to build a dataset from your own back catalogue.

State of the art (2026)

• Video-native multimodal models (Gemini family, Qwen-VL, GPT and Claude with frame input) produce fluent scene descriptions; research on movie AD (e.g. the MAD dataset and AutoAD papers from Oxford VGG) showed that character-aware, context-aware prompting is what makes descriptions useful rather than generic captions [5].
• Gap-fitting and "describe only what matters for the plot" remain hard: models over-describe, describe what is already audible, or miss narratively important details (a glance, a prop).
• Neural TTS is natural enough for long listening; user-configurable speed and verbosity are a real advantage of synthetic AD over fixed human recordings.
• Limits and failure modes: hallucinated objects and actions, wrong character identity, missed text in motion, cultural misreadings, and flat handling of emotional or sensitive scenes (violence, nudity). Blind users report preferring neutral, factual descriptions over guessed emotions or intent.

Shared resources & combinations

• Same stack: VLM video understanding is shared with VIS-025 (video tagging) and KNOW-016 (multimodal archive search); TTS with SPEECH-006 (narration) and SPEECH-007 (dubbing, same timing-fit problem); ASR and gap detection with SPEECH-004 (captioning).
• Natural bundle: accessibility package of captions (SPEECH-004), AD, accessible documents (DOC-026) and sign-language provision where available.
• Hardware: a single HW3 GPU server can host VLM and TTS for several of these use cases.

Effort, cost & KPIs

• Path A: days to weeks to onboard; priced per video minute, with human-reviewed tiers costing several times more than AI-only.
• Path B: 1–3 months for pipeline plus review tool; per-minute API cost for VLM and TTS is typically small compared with studio production.
• Path C: 3–4 months; GPU server plus ML-ops.
• KPIs: share of catalogue with AD, editor time per finished minute (vs. from-scratch baseline), edit rate of AI sentences, factual error rate in sampled QA, user ratings from blind testers, playback usage of the AD track.

People & change

Involve management (accessibility goals, budget), the department lead for media or communication, the editors who review descriptions, IT (pipeline, storage) and legal or compliance (accessibility law, rights in footage). Involve blind and low-vision users early: they test descriptions, set verbosity and voice preferences and tell you what is useful. Associations of blind people are good partners for a pilot group.

Works council: usually no, since the system processes content, not staff behaviour (§87(1) no. 6 BetrVG). Inform the council anyway if editorial workflows change (§90(1) no. 3 BetrVG). In public bodies the staff council (Personalrat) is informed under the applicable law. A DPO is needed when people appear in footage; check legal basis and AVV with API providers, and a DPIA only for sensitive material.

AI literacy: support editors with short training on typical errors (hallucinated details, wrong names, missed on-screen text), in line with AI Act Art. 4. Be transparent: label synthetic descriptions and voices (Art. 50). Keep humans in control through a review step before publication. Typical concern is job loss for describers; position AI as a first draft that frees them for quality work.

Guides: GUIDE-008, GUIDE-006, GUIDE-010.

Risks, regulation & compliance

• EU AI Act: typically limited risk. Art. 50 requires machine-readable marking and disclosure of synthetic audio ("deepfake" rules apply to realistic voice that resembles real persons; a clearly synthetic narrator voice is low risk), with an exception for assistive standard editing but not for fully generated content. Not high risk in itself. Art. 5 caution: do not add emotion recognition of people in workplace/education settings or biometric identification of people from the footage; describe visible behaviour only, not inferred emotions or identities [6].
• Accessibility law: EAA, AVMSD, Web Accessibility Directive/BITV 2.0 and EN 301 549 set the duty; AI-only output with unchecked errors may not meet the "equivalent" standard for regulated media; document your QA.
• GDPR: videos show identifiable people; describing a person's appearance can reveal special category data (health, ethnicity, religion). Use EU regions or self-hosting, a data processing agreement, and avoid retaining frames at the provider; do not name unnamed individuals.
• Copyright: AD for third-party material needs rights; voice actors' contracts matter if a voice is cloned (see SPEECH-007).
• Mitigations: human review for high-stakes content, neutral style guide, user feedback channel, test with blind users, label AD as automatically generated.

Getting started

1. Pick 10 representative videos (talking head, slides, drama, screen recording) and count how many have information that is only visual.
2. Prototype Path B on those: Gemini or another video-capable model for time-coded descriptions, a neural TTS voice, FFmpeg for ducking and mixing.
3. Have two to three blind or low-vision users and an accessibility specialist rate usefulness, verbosity and errors; adapt the prompt and style guide.
4. Add a simple review step and a selectable AD track in your player (WebVTT or alternate audio).
5. Define which content may go out AI-only and which needs human sign-off, and log it.

Sources

1. Directive (EU) 2019/882 (European Accessibility Act) — https://eur-lex.europa.eu/eli/dir/2019/882/oj
2. Directive 2010/13/EU (Audiovisual Media Services Directive, consolidated) — https://eur-lex.europa.eu/eli/dir/2010/13/oj
3. W3C WCAG 2.2, SC 1.2.5 Audio Description (Prerecorded) — https://www.w3.org/WAI/WCAG22/Understanding/audio-description-prerecorded.html
4. WHO — Blindness and vision impairment fact sheet — https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment
5. Han et al., AutoAD: Movie Description in Context (CVPR 2023) — https://arxiv.org/abs/2303.16899
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Google Cloud — Gemini video understanding — https://ai.google.dev/gemini-api/docs/video-understanding
8. Azure AI Speech — text to speech and SSML — https://learn.microsoft.com/azure/ai-services/speech-service/text-to-speech
9. Qwen2.5-VL — https://github.com/QwenLM/Qwen2.5-VL