Catalog / Personal & Everyday AI / PERS-007

Assistive AI for blind & low-vision people (scene description)

Describe photos, live camera views, documents and screens in speech or text so blind and low-vision people can navigate daily life with more independence.

Parts needed to run it yourselfDocument readingLanguage modelSpeechVision

Blind and low-vision people have long relied on screen readers, OCR apps and sighted helpers. Vision-language models now let a phone or a pair of smart glasses describe a scene, read a label, explain a chart and answer follow-up questions in natural language, within seconds and at any hour. The result is more independence in daily tasks, but with real limits: descriptions can be wrong, and they must never be the sole safeguard for mobility, medication or other safety-critical decisions.

Problem & value

  • Who: the WHO estimates that at least 2.2 billion people worldwide have a near or distance vision impairment [5]. In the EU, millions of people are blind or have low vision; the European Blind Union is the main advocacy body [6].
  • Today: a mix of screen readers (VoiceOver, TalkBack, NVDA, JAWS), OCR apps, white cane and guide dog, and sighted help from family, volunteers or paid services. Images without alt text, unlabeled packaging, paper letters and unfamiliar rooms remain recurring barriers.
  • AI value: on-demand description of arbitrary images and scenes, conversational follow-up ("is the expiry date visible?", "what colour is this shirt?"), reading of printed and handwritten text, and automatic alt text across apps. Volunteer-based services already show the demand: Be My Eyes reports a very large community of blind users and sighted volunteers, and added an AI visual assistant, Be My AI, on top of it [1].
  • Regulatory pull (EU): the European Accessibility Act (Directive 2019/882) applies from 28 June 2025 to e-commerce, banking services, e-books and more [3].

Variants

  • Smartphone assistants: photo or live-camera Q&A (Be My AI, Microsoft Seeing AI, Google Lookout, Envision) [1][2].
  • OS-level, glasses, education: VoiceOver/TalkBack image descriptions, smart glasses (see DOC-026 for alt text), chart and slide description.
  • Smart glasses and wearables: hands-free description through camera glasses with voice (e.g. Envision glasses, Ray-Ban Meta with accessibility features) — see PERS-017.
  • Human + AI hybrid: AI first, one-tap escalation to a trained volunteer or professional agent (Aira, Be My Eyes) — see CONV-027.

How it works

  1. camera frame / image / screenshot / PDF + spoken question
  2. capture & optional on-device pre-processing (blur check, text detection, face blurring)
  3. ASR (speech
  4. text) for the question
  5. VLM (+ OCR for dense text) with a system prompt tuned for blind users (spatial layout, hazards, no guessing)
  6. answer generation with uncertainty statements ("I cannot read the date clearly")
  7. TTS or braille display; follow-up turns keep image context
  8. optional: escalate to human volunteer / save description to personal memory
  • Key components: a multimodal model that handles images and, in realtime variants, audio streams; a dedicated OCR/document parser for dense print; low-latency speech-to-speech for conversation.
  • Human-in-the-loop: the user is the final judge; for medication, hazards, money and legal documents the app should prompt to confirm with a person or a second source. Human escalation is the safety net.

Data preparation

This use case needs almost no training data: the inputs are live camera frames, photos, screenshots and documents captured by the user in the moment. Preparation therefore concerns the setup, not archives. What matters is image quality (lighting, framing, blur), clear prompts in the user's language and a way to capture steadily, for example a chest-mounted phone, glasses or a document stand.

Typical gaps: poor framing because the user cannot see the viewfinder, text and numbers read wrongly, missing context about the user's home and routines, and sensitive scenes (faces, mail, medication) leaving the device. For organisations that build their own service (libraries, associations, public bodies): - Collect a small test set of real situations (labels, menus, mail, public transport displays) together with blind users and keep it as an evaluation set (PREP-018, PREP-016). - Define which content may be sent to the cloud and which must stay on the device, and apply face blurring or pseudonymisation where possible (PREP-022, PREP-024). - For document reading, scan and OCR frequent forms in advance (PREP-006, DOC-026). - Write a short glossary of local place names and personal contacts so the assistant uses the right terms (PREP-015).

Rough share of effort: low, about 10-20 percent; most effort goes into testing with blind users and accessible design.

Implementation paths

Which path for whom

  • private: A, because tested apps (Be My Eyes, Seeing AI, Envision, Lookout) are free or cheap, work with screen readers and are validated by blind users.
  • developer: B, because frontier vision-language APIs with EU region give the best description quality for custom or embedded accessible apps; add C where offline use matters.
  • public: B, on sovereign EU or German cloud, because authorities, libraries and associations need controllable data flows and accessibility obligations under procurement rules; an off-the-shelf app (A) only with a clear data-processing statement.
  • Apps: Be My Eyes / Be My AI (Denmark-founded; free for users) [1], Microsoft Seeing AI (free) [2], Google Lookout (Android), Envision app and Envision glasses (Netherlands, EU), Aira (human agents plus AI), Apple VoiceOver image descriptions and Magnifier detection modes, Ray-Ban Meta glasses with voice-driven scene description.
  • When right: almost always for individuals; products are free or low-cost and tested with blind users. Check data location, whether images are stored or used for training, and whether the app works offline.

Path B — Build on cloud APIs

  • Models: Claude, Gemini, GPT and Mistral (EU provider, Pixtral/vision models) via API with image input; Azure AI Vision (image captioning, dense captions, OCR) in EU regions; Google Cloud Vision and Document AI; AWS Rekognition and Textract in Frankfurt.
  • Realtime voice: realtime/live APIs of frontier providers for speech-to-speech; fallback chain ASR → VLM → TTS (e.g. Azure Speech, Google Cloud Speech, Whisper-class models, Mistral Voxtral).

Path C — Self-host (open source)

  • Models: open-weight VLMs such as Qwen2.5-VL / Qwen3-VL, Gemma 3 (multimodal), Llama 3.2 Vision, SmolVLM, PaliGemma and Florence-2 for captioning and OCR; Whisper or faster-whisper for ASR; Piper, Kokoro or Coqui-class models for TTS; Tesseract or PaddleOCR for print.
  • Hardware: HW1 suffices for compact models (2–4B VLM on a recent phone or laptop NPU) with slower, shorter answers; HW2 gives near-realtime quality for a 7–8B VLM on a home server.

Path D — Custom training

Worthwhile only for specific needs: indoor navigation for a site, a local language or dialect, or domain vocabulary (medication packs, transit signage). Starting points are public datasets built from blind users' photos, such as VizWiz [4], plus the organisation's own consented images. Fine-tune a small VLM with LoRA and evaluate with blind testers, not just benchmarks.

State of the art (2026)

  • General VLMs describe everyday scenes, read text in many languages and answer follow-up questions well; realtime voice conversation with a live camera is now a mainstream feature in several assistants. Smart glasses make this hands-free.
  • Benchmarks: VizWiz-based tasks (VQA, captioning, grounding) are the most realistic tests because photos are taken by blind users and are often blurred, badly framed or poorly lit [4].
  • Limits and failure modes:
  • Hallucination: confident but wrong details (text, colours, counts, expiry dates, medication names).
  • Spatial errors: left/right, distances and depth are often unreliable; models are not mobility aids and must not replace cane, guide dog or orientation training.
  • Bias: descriptions of people can be stereotyped, and providers may deliberately limit identity or sensitive-attribute statements, which can frustrate users who want such information.

Shared resources & combinations

Shares vlm + ocr + asr + tts with CONV-027, DOC-026, PERS-006, PERS-008, PERS-017, PERS-018; complements LANG-030, SPEECH-004, SPEECH-006. One HW2 VLM server can serve several of them.

Effort, cost & KPIs

  • Path A: minutes to install; mostly free apps; glasses hardware costs several hundred euros and some services charge subscriptions. Allow time for training and orientation.
  • Path B: 1–3 months for a custom app including accessibility testing with blind users; API cost per image is small, but realtime video sessions raise costs.
  • Path C: days for a hobbyist setup; 1–2 months for a robust association server; ongoing maintenance.
  • KPIs: task completion rate in tests with blind users, perceived independence and trust (survey), description accuracy on critical attributes (text, numbers, hazards), latency, escalation rate to humans, error reports per 1000 sessions.

People & change

The most important stakeholder is the blind or low-vision person: involve users from the first sketch, test with them, and let them decide when a description is trustworthy. For an organisation (public library, association, employer providing workplace assistance) also involve the department lead and IT for devices and accounts, legal and compliance for accessibility duties, and the data protection officer: images of surroundings contain third parties and often health-related context, so a legal-basis check and, for a service at scale, a DPIA are advisable, plus an AVV with every cloud provider (GUIDE-005).

Works council: depends. Private use needs none. When an employer provides the tool as workplace accommodation, camera and usage logs can be suited to monitor behaviour, so §87(1) no. 6 BetrVG may apply and the works council or Personalrat should be informed in good time (§90(1) no. 3) and, with the severely disabled representation, involved early (GUIDE-004).

AI literacy: offer short trainings on limits (hallucinated details, misread numbers, no replacement for a cane or guide dog in traffic) to support the Art. 4 AI Act duty (GUIDE-006). Start with a small pilot group of 5-10 volunteers and name peer trainers (GUIDE-008). Typical concerns are privacy, over-reliance and errors in safety-critical situations; address them with clear "do not rely on this for hazards" guidance, an easy way to call a human volunteer, and local processing where feasible.

Risks, regulation & compliance

  • EU AI Act: typically limited or minimal risk. Transparency duties (Art. 50) apply if the system interacts with people or generates synthetic content — users must know they are talking to AI. A product marketed as a medical device or as a navigation safety system could fall under MDR and the high-risk category (Annex I), so claims must be worded carefully. Art. 5 caution: remote biometric identification and emotion inference from faces are prohibited or restricted contexts; assistants must not identify strangers by face or infer emotions in workplace or education settings. AI literacy (Art. 4) applies to providers' staff [7].
  • GDPR: camera images show bystanders, home interiors, documents and health information; vision impairment is itself health data (Art. 9). Use data minimisation, no storage or training use without consent, EU regions and a DPA. Household exemption covers private use, but providers are controllers. Face recognition of known persons needs explicit consent and should be on-device.
  • Accessibility law: EAA (Directive 2019/882) and EN 301 549 set requirements for product and service providers; assistants should themselves follow WCAG 2.2 [3].

Getting started

  1. Try two free apps (Be My Eyes with Be My AI, Seeing AI or Lookout) on typical tasks: mail, packaging, clothing, screenshots.
  2. Switch on built-in OS features (VoiceOver image descriptions, TalkBack with Gemini) and agree with the user what they trust and what needs a second source.
  3. Check privacy settings: image storage, training use, region, offline mode; avoid sending bank or medical documents to services without clear terms.
  4. For organisations: pilot a custom assistant (Path B) with 5–10 blind testers early, measure the KPIs above and keep human escalation.
  5. For tech-savvy users: set up a local VLM (Path C) for private documents and offline use.

Sources

  1. Be My Eyes — Be My AI — https://www.bemyeyes.com/
  2. Microsoft — Seeing AI — https://www.microsoft.com/en-us/ai/seeing-ai
  3. Directive (EU) 2019/882 (European Accessibility Act) — https://eur-lex.europa.eu/eli/dir/2019/882/oj
  4. VizWiz — datasets and challenges from blind users — https://vizwiz.org/
  5. WHO — Blindness and vision impairment fact sheet — https://www.who.int/news-room/fact-sheets/detail/blindness-and-visual-impairment
  6. European Blind Union — https://www.euroblind.org/
  7. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  8. Google — Lookout — https://play.google.com/store/apps/details?id=com.google.android.apps.accessibility.reveal
  9. Qwen-VL open-weight models — https://github.com/QwenLM/Qwen2.5-VL
  10. UploadVR — Ray-Ban Meta glasses get Meta AI detailed responses for blind and low-vision users — https://uploadvr.com/ray-ban-meta-glasses-meta-ai-detailed-responses-blind-low-vision