Assistive speech: voice banking, atypical-speech ASR & AAC

People who lose speech to ALS/MND, laryngeal cancer, stroke or Parkinson's, or who were born with dysarthria (cerebral palsy), have long had to choose between silence and a generic robotic voice. Voice banking and neural voice cloning now rebuild a recognisably personal voice from minutes of recordings, atypical-speech recognition lets people be understood by people and devices, and LLM-based prediction speeds up AAC (augmentative and alternative communication). The result is communication that keeps identity, not just words.

Problem & value

• Today: progressive conditions erode speech over months to years. Standard care is referral to a speech-language therapist, an AAC assessment and a communication device (tablet or eye-tracker) with a stock synthetic voice. Mainstream voice assistants and ASR fail on dysarthric speech, shutting these users out of smart-home control and dictation.
• AI value: (1) a voice that sounds like the person, preserved before or reconstructed after loss; (2) speech recognition personalised to the individual; (3) faster, more natural AAC through text prediction and context-aware phrase suggestions.
• Why timing matters: recordings must be made while speech is still intelligible; therapists and clinics should offer banking at diagnosis, not later.

Variants

• Voice banking for progressive disease (ALS/MND, Parkinson's, head-and-neck cancer before laryngectomy): record a script before speech declines; produce a personal voice for AAC use.
• Voice reconstruction from degraded or historical audio: after speech is already lost, build a voice from voicemails, videos or a relative's recordings; quality and ethics need extra care.
• Atypical-speech ASR (dysarthria, cerebral palsy, stuttering, post-stroke): recogniser adapted to one user or a population for transcription, device control and live captioning of what they say.
• AAC integration: symbol/text grids, eye-gaze and switch access on dedicated devices (Tobii Dynavox and similar) or tablets, with personal voice output and AI sentence prediction.

How it works

1. recording (guided script, 15 min–hours; or archive audio)
2. quality check + denoising + alignment to text
3. voice model creation (few-shot cloning / fine-tune of a TTS model)
4. deployment on AAC device (on-device preferred) or cloud voice AAC input (eye-gaze / switch / touch / residual speech)
5. text prediction (small LLM, personal phrase history, context such as place/person)
6. TTS in the personal voice
7. loudspeaker / video call Atypical speech: speech
8. personalised ASR (adapted acoustic model / fine-tuned Whisper-class)
9. text, command, caption
10. low-confidence fallback: ask user to confirm or switch to AAC

• Atypical ASR: adapting a general model with few user-specific examples, or training on pooled corpora of disordered speech. Within-user adaptation outperforms generic models; accuracy varies with severity and day-to-day fatigue.
• AAC prediction: small LLMs propose complete utterances from partial input; these must be editable and never speak on the user's behalf without their selection.
• Human-in-the-loop: therapist assesses, selects script and device, tunes vocabulary and monitors fit; the user always chooses what is spoken.

Data preparation

The central data are the person's own voice recordings (for voice banking) or recordings of atypical speech with transcripts (for adapted ASR). They live on the user's phone, in therapy documentation, clinic file shares or research servers, often unlabelled and scattered.

Typical gaps: few and inconsistent recordings, poor microphones, missing or wrong transcripts, no consent record, and recordings made after the voice has already deteriorated. - Recording protocol: quiet room, same microphone, prompt lists from the tool; record early while speech is still clear. - Transcripts: correct them with the speaker or caregiver; build a gold-standard test set (PREP-018, PREP-016). - Personal vocabulary: names, places, therapy terms (PREP-015). - Protection: voice is biometric and health data; pseudonymise research sets (PREP-022) and document purpose and legal basis (PREP-025). - Tools: the voice-banking apps themselves, Audacity or similar for trimming, Whisper for draft transcripts.

For private users the effort is low (hours of recording); for developers and clinics building adapted models it is medium and dominated by transcript correction and consent handling.

Data preparation

The central data are the person's own voice recordings (for voice banking) or recordings of atypical speech with transcripts (for adapted ASR). They live on the user's phone, in therapy documentation, clinic file shares or research servers, often unlabelled and scattered.

Typical gaps: few and inconsistent recordings, poor microphones, missing or wrong transcripts, no consent record, and recordings made after the voice has already deteriorated. - Recording protocol: quiet room, same microphone, prompt lists from the tool; record early while speech is still clear. - Transcripts: correct them with the speaker or caregiver; build a gold-standard test set (PREP-018, PREP-016). - Personal vocabulary: names, places, therapy terms (PREP-015). - Protection: voice is biometric and health data; pseudonymise research sets (PREP-022) and document purpose and legal basis (PREP-025). - Tools: the voice-banking apps themselves, Audacity or similar for trimming, Whisper for draft transcripts.

For private users the effort is low (hours of recording); for developers and clinics building adapted models it is medium and dominated by transcript correction and consent handling.

Data preparation

The central data are the person's own voice recordings (for voice banking) or recordings of atypical speech with transcripts (for adapted ASR). They live on the user's phone, in therapy documentation, clinic file shares or research servers, often unlabelled and scattered.

Typical gaps: few and inconsistent recordings, poor microphones, missing or wrong transcripts, no consent record, and recordings made after the voice has already deteriorated. - Recording protocol: quiet room, same microphone, prompt lists from the tool; record early while speech is still clear. - Transcripts: correct them with the speaker or caregiver; build a gold-standard test set (PREP-018, PREP-016). - Personal vocabulary: names, places, therapy terms (PREP-015). - Protection: voice is biometric and health data; pseudonymise research sets (PREP-022) and document purpose and legal basis (PREP-025). - Tools: the voice-banking apps themselves, Audacity or similar for trimming, Whisper for draft transcripts.

For private users the effort is low (hours of recording); for developers and clinics building adapted models it is medium and dominated by transcript correction and consent handling.

Implementation paths

Which path for whom

• Private: Path A. Established voice-banking and AAC apps work out of the box and are often covered by therapists and insurers.
• Enterprise: Path A. Clinics, rehab providers and care organisations need supported, certified tools and therapist workflows rather than own builds.
• Developer: Path C. Fine-tuning open ASR models such as Whisper on atypical speech runs on HW1 and keeps sensitive voice data local.
• Public: Path C. Public bodies and non-profits handle very sensitive health and voice data; self-hosted open source on sovereign infrastructure avoids procurement and transfer issues.

Which path for whom

• Private: Path A. Established voice-banking and AAC apps work out of the box and are often covered by therapists and insurers.
• Enterprise: Path A. Clinics, rehab providers and care organisations need supported, certified tools and therapist workflows rather than own builds.
• Developer: Path C. Fine-tuning open ASR models such as Whisper on atypical speech runs on HW1 and keeps sensitive voice data local.
• Public: Path C. Public bodies and non-profits handle very sensitive health and voice data; self-hosted open source on sovereign infrastructure avoids procurement and transfer issues.

Which path for whom

• Private: Path A. Established voice-banking and AAC apps work out of the box and are often covered by therapists and insurers.
• Enterprise: Path A. Clinics, rehab providers and care organisations need supported, certified tools and therapist workflows rather than own builds.
• Developer: Path C. Fine-tuning open ASR models such as Whisper on atypical speech runs on HW1 and keeps sensitive voice data local.
• Public: Path C. Public bodies and non-profits handle very sensitive health and voice data; self-hosted open source on sovereign infrastructure avoids procurement and transfer issues.

Path A — Buy (recommended)

• Voice banking and personal voices: Acapela "my-own-voice" (Belgium/France, EU), bundled with Tobii Dynavox AAC solutions; ElevenLabs Impact Program (free voice cloning for eligible people with ALS/MND and similar conditions) and Professional Voice Cloning; Apple Personal Voice + Live Speech (iOS/macOS, on-device, about 15 min of prompts); Cereproc (UK) voices; ModelTalker/MND Association banking tools in the UK; SpeakUnique (UK).
• Atypical-speech ASR: Voiceitt (app and integrations for dysarthric speech, including enterprise and care settings); Android accessibility tools (Google's Project Relate research app may no longer be available; verify); Apple Voice Control and the Listen for Atypical Speech feature (introduced with iOS 18).
• AAC devices and apps: Tobii Dynavox (SE), Smartbox Grid, Control Bionics, Liberator, Proloquo2Go (AssistiveWare, NL), Snap Core First, LAMP Words for Life; eye-tracking via Tobii.

Path B — Build on cloud APIs

• TTS/cloning APIs: ElevenLabs (instant and professional cloning), Azure AI Speech Custom Neural Voice (restricted access, EU regions), Google Cloud TTS custom voice, Cartesia, Mistral Voxtral (EU provider, open weights and API for speech understanding).
• ASR/personalisation: Azure custom speech, Google Chirp / Speech-to-Text with adaptation, Deepgram, AssemblyAI, OpenAI Whisper-class APIs; custom phrase lists help but do not fix severe dysarthria.
• Sentence prediction: any LLM API with a personalised prompt/memory; prefer EU regions and no training on user data.

Path C — Self-host (open source)

• TTS with cloning: open-weight models such as XTTS-v2 (Coqui, licence restrictions), F5-TTS, Piper (small, fast, good on Raspberry Pi-class hardware), StyleTTS 2, Kokoro, OpenVoice; check each licence for permitted use.
• ASR: Whisper (large-v3/turbo) fine-tuned with LoRA on personal or pooled disordered-speech data; faster-whisper or whisper.cpp for on-device inference; Voxtral open weights.
• AAC stack: open-source AAC software such as Cboard (web), AsTeRICS Grid (EU-funded, eye-gaze and switch support) and Open AAC symbol sets.
• Hardware: HW1 for on-device TTS/ASR; HW2 only for voice training. Self-hosting suits privacy-sensitive clinics and research groups; production use needs a therapist and device partner.

Path D — Custom training

Worth it for severe or rare dysarthria, non-English or dialect speakers, and clinical research. Needs the user's own recordings (tens of minutes to hours), clean transcripts and consented use. Pooled datasets such as the University of Illinois Speech Accessibility Project (SAP) provide disordered-speech data for research under a data-use agreement; model improvements from SAP have been reported by participating tech companies. Expect a clinical collaboration and IRB/ethics review.

State of the art (2026)

• Few-shot cloning yields convincing voices from minutes of clean audio; the gap is expressiveness, prosody control and robustness to noisy source recordings.
• On-device personal voices are standard on phones (Apple Personal Voice), removing cloud dependence.
• Atypical ASR: foundation models fine-tuned on disordered-speech corpora reduce word error rate considerably for mild-to-moderate dysarthria; severe cases, unintelligible speech and open vocabulary remain unsolved. Personal adaptation beats generic models.
• Limits: voice degrades as disease progresses so early banking matters; cloning can't recover emotional range; ASR drifts with fatigue and disease progression; synthetic voices can be misused.

Shared resources & combinations

• Shares tts/asr/edge-runtime with SPEECH-007, SPEECH-006, SPEECH-012, SPEECH-017; adjacent: PERS-007, CONV-016, HLTH-019 (keep vocal-biomarker use separate), DOC-026.
• A single on-device runtime (whisper.cpp / Piper / small LLM, HW1) serves several of these.

Effort, cost & KPIs

• Path A: days to weeks to bank a voice; AAC device and software assessment takes weeks; costs vary from free (some phone features, ElevenLabs Impact Program for eligible users) to device prices in the thousands of euros, usually funded as a medical aid.
• Path B: 1–3 months; Path C/D: 3–9 months incl. clinical validation.
• KPIs: intelligibility rating by listeners, word error rate on the user's speech, words per minute in AAC, voice similarity as rated by the user and family, device abandonment rate, time from diagnosis to banked voice, user and caregiver satisfaction.

People & change

Involve the end users themselves and their customers side (family, caregivers, speech-language therapists) from the start, together with the department lead (therapy or clinic lead) and management. The DPO and legal-compliance must confirm consent, AVV and storage location for voice data before recording; a DPIA is likely for health and biometric data at scale (GUIDE-005).

Works council: no, because the users are patients or clients. If staff such as support workers use the tool and activity is logged, §87(1) no. 6 BetrVG may apply; in public bodies the Personalrat.

AI literacy: train therapists and caregivers on limits (errors, synthetic voice misuse) as support measures under Art. 4 AI Act (GUIDE-006). Communicate that the person keeps full control of their voice and can delete it. Start with a small pilot group, ideally a few motivated users with therapist support (GUIDE-008). Typical concerns: loss of identity, misuse of the cloned voice, dependence on the app; address them with local storage, clear consent and a fallback to manual AAC.

People & change

Involve the end users themselves and their customers side (family, caregivers, speech-language therapists) from the start, together with the department lead (therapy or clinic lead) and management. The DPO and legal-compliance must confirm consent, AVV and storage location for voice data before recording; a DPIA is likely for health and biometric data at scale (GUIDE-005).

Works council: no, because the users are patients or clients. If staff such as support workers use the tool and activity is logged, §87(1) no. 6 BetrVG may apply; in public bodies the Personalrat.

AI literacy: train therapists and caregivers on limits (errors, synthetic voice misuse) as support measures under Art. 4 AI Act (GUIDE-006). Communicate that the person keeps full control of their voice and can delete it. Start with a small pilot group, ideally a few motivated users with therapist support (GUIDE-008). Typical concerns: loss of identity, misuse of the cloned voice, dependence on the app; address them with local storage, clear consent and a fallback to manual AAC.

People & change

Involve the end users themselves and their customers side (family, caregivers, speech-language therapists) from the start, together with the department lead (therapy or clinic lead) and management. The DPO and legal-compliance must confirm consent, AVV and storage location for voice data before recording; a DPIA is likely for health and biometric data at scale (GUIDE-005).

Works council: no, because the users are patients or clients. If staff such as support workers use the tool and activity is logged, §87(1) no. 6 BetrVG may apply; in public bodies the Personalrat.

AI literacy: train therapists and caregivers on limits (errors, synthetic voice misuse) as support measures under Art. 4 AI Act (GUIDE-006). Communicate that the person keeps full control of their voice and can delete it. Start with a small pilot group, ideally a few motivated users with therapist support (GUIDE-008). Typical concerns: loss of identity, misuse of the cloned voice, dependence on the app; address them with local storage, clear consent and a fallback to manual AAC.

Risks, regulation & compliance

• EU AI Act: assistive TTS/ASR is generally limited or minimal risk. Synthetic speech triggers transparency duties under Art. 50 (disclose AI-generated audio to listeners where not obvious); personal use by the voice owner is largely a low-risk situation. Emotion recognition is not part of this case (Art. 5 bans it at work and school; avoid adding it). Medical-purpose products may be regulated as medical devices (MDR), which can trigger Annex I high-risk treatment under the AI Act — vendor responsibility.
• GDPR: voice recordings are personal data and, as a voiceprint usable for identification, may be biometric/special-category (Art. 9) data, plus health data. Explicit consent, data minimisation, deletion of the voice model on request, EU processing and DPIA. Legal capacity: for children or people with cognitive impairment, involve guardians while respecting the person's own wishes.
• Voice identity and misuse: the cloned voice could be used for fraud or deepfakes; restrict export, watermark outputs where possible, and control access. Posthumous use of a voice should be agreed in advance (advance directive or family agreement).
• Funding: in Germany, AAC devices are Hilfsmittel (SGB V); prescription by a doctor, request to the health insurer, and listing in the Hilfsmittelverzeichnis; other EU countries have different schemes.
• Mitigations: user-owned voice models, local storage, therapist oversight, clear fallback to non-AI AAC, expectation management on recognition accuracy.

Getting started

1. Offer voice banking at diagnosis: involve a speech-language therapist, record the first session in a quiet room with a good microphone, and keep original audio securely.
2. Try the free options first (phone personal voice, an eligible voice programme) while assessing a dedicated AAC device.
3. For recognition, pilot an atypical-speech app with the user for two weeks and compare to the baseline.
4. Document consent, retention and deletion rules for recordings and voice models; run a DPIA for service providers.
5. Start the funding application early; devices can take weeks to approve.

Sources

1. Acapela Group — my-own-voice voice banking — https://www.acapela-group.com/
2. Tobii Dynavox — AAC solutions and voice banking — https://www.tobiidynavox.com/
3. ElevenLabs — Impact Program — https://elevenlabs.io/impact-program
4. Apple — Personal Voice and Live Speech (iOS support) — https://support.apple.com/guide/iphone/create-a-personal-voice-iph0bf3f2f5e/ios
5. Voiceitt — speech recognition for non-standard speech — https://voiceitt.com/
6. University of Illinois — Speech Accessibility Project — https://speechaccessibilityproject.beckman.illinois.edu/
7. Google — Project Relate — https://sites.research.google/relate/
8. AsTeRICS Grid — open-source AAC — https://github.com/asterics/AsTeRICS-Grid
9. Piper TTS — https://github.com/rhasspy/piper
10. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
11. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj