Pronunciation & speech-therapy coaching

Language learners rarely get enough individual speaking feedback, and speech-language therapy is limited by therapist capacity and by the short time spent in sessions. AI coaching listens to every utterance, compares it with a target at phoneme level, tells the learner what went wrong and generates the next exercise. Teachers and therapists keep the diagnosis and the plan; the AI provides high-volume practice and measurement between sessions.

Problem & value

• Today: pronunciation teaching in classes is mostly whole-group repetition with sparse individual correction. Speech therapy (e.g. for articulation disorders, stuttering, post-stroke aphasia/dysarthria) relies on weekly appointments plus home exercises that are rarely checked, and waiting lists are long in many EU countries.
• AI value: immediate, private, always-available feedback (learners are often less anxious speaking to a machine); objective, repeatable scoring; more practice repetitions; session data that lets professionals focus appointments on what is not working.
• Evidence: vendor claims about learning gains are common but mostly unaudited. Microsoft states that its pronunciation scores depend on ASR accuracy and agreement with human raters, and advises customers to test on their own data and tune thresholds per audience [1][2]. Treat any benefit figure as to-be-verified in your own pilot.

Variants

• Second-language pronunciation (consumer and school): apps and classroom tools scoring read-aloud and free speech (English, German as a second language, etc.).
• Exam preparation and assessment: speaking practice for CEFR-aligned tests; scoring for high-stakes decisions is a different regulatory class (see risks).
• Corporate / call-centre accent and clarity training: coaching, not conversion (live voice conversion is SPEECH-019).
• Speech-language therapy support: articulation, fluency and voice exercises prescribed by a therapist, with home practice and logging; includes children (e.g. sound-production games) and adults after stroke.
• Reading fluency for children: read-aloud tutors checking decoding and rate (EDU-004 adjacent).
• Private use: self-study apps for travel or integration language courses.

How it works

1. microphone capture (app / browser / classroom device)
2. voice activity detection + noise check
3. forced alignment / phoneme recognition against reference text (or ASR for free speech)
4. scoring: phoneme, word, sentence level (accuracy, fluency, completeness, prosody)
5. error analysis (substitution, omission, insertion; L1-specific patterns)
6. feedback generation (LLM text + TTS model audio, visual articulation hints)
7. exercise selection (adaptive, therapist-defined targets)
8. progress store
9. teacher / therapist dashboard (human review)

• Scoring: either a pronunciation-assessment service on top of ASR, or open phoneme-recognition models (wav2vec 2.0/HuBERT/WavLM fine-tuned with CTC) used for mispronunciation detection and diagnosis against a canonical phoneme sequence [3]. Whisper-class ASR transcribes well but tends to "autocorrect" mispronunciations, so it is poor as the only scorer.
• Feedback: an LLM turns scores into short, age-appropriate advice; TTS plays the target sound or sentence (SPEECH-006). Keep explanations grounded in the detected errors to avoid invented advice.
• Human-in-the-loop: in therapy, the clinician sets targets, reviews recordings flagged by the system and decides on progression. AI scores are a measurement aid, not a diagnosis.

Data preparation

Data needed: reference texts and exercise material, learner or patient recordings, native-speaker reference audio, and for adaptation labelled speech with phoneme-level ratings by trained raters. In an SMB (language school, speech therapy practice) exercise material lives in textbooks, worksheets, PDFs and file shares; learner progress sits in paper notes, spreadsheets or practice software.

Typical gaps: material only on paper, no consistent level scheme, few labelled recordings of children or accented speech, no consent forms for recordings, and therapy notes mixed with administrative data.

Preparation steps: 1. Digitise worksheets and word lists (PREP-006) and convert them into structured exercise tables (PREP-011). 2. Define levels and error categories before scoring (PREP-019). 3. Collect rated example recordings with consent as a test set (PREP-018, PREP-016). 4. Pseudonymise recordings and progress data (PREP-022); check purpose and consent, especially for minors (PREP-025). 5. Capture teacher or therapist corrections of scores (PREP-021).

Effort share: about 30 percent for a product-based setup, higher if a custom scoring model needs rated speech data. Health-related data needs strict separation from general learner data.

Implementation paths

Which path for whom

• smb: Path A, because mature learning and assessment products exist for standard languages and a language school or practice needs a product, not a platform.
• enterprise: Path B, because corporate language training or branded tools need integration with LMS and HR systems via assessment APIs.
• developer: Path C, because open phoneme recognisers (wav2vec 2.0, WavLM) allow on-device scoring and tuning for specific languages or age groups.
• private: Path A, because consumer apps work out of the box; choose apps that process on-device where possible.
• public: Path C, because audio of children and patients is highly sensitive and schools and clinics benefit from on-premise or sovereign processing.

Path A — Buy (off-the-shelf)

• Language learning: consumer apps (e.g. ELSA Speak, Duolingo, Babbel, Speechling) and EU-based platforms; school/enterprise tools with teacher dashboards. Check data location, minors' consent, and whether recordings are used to train models.
• Assessment engines as SDK/SaaS: Speechace, Pearson/Versant-type engines, SpeechSuper; Azure AI Speech pronunciation assessment (see Path B).
• Speech therapy: regulated digital therapeutics and clinic tools are mostly national products (look for EU/DiGA-listed offerings in Germany, e.g. for aphasia training); verify MDR status before clinical use.
• Choose A when: you need a learner or patient product, not a platform; the use is standard (English, major languages).

Path B — Build on cloud APIs

• Pronunciation assessment: Azure AI Speech returns accuracy, fluency, completeness and prosody scores with phoneme and word detail for a reference text, with EU regions and a continuous mode for audio over 30 seconds [1][2].
• ASR for free speech and transcripts: Azure, Google Cloud Speech-to-Text (EU location), Deepgram, AssemblyAI, or EU providers such as Speechmatics (UK) and Mistral's Voxtral.
• Feedback and exercise generation: any frontier LLM with EU data residency; TTS from Azure, ElevenLabs or Google.
• Orchestration: web app with WebRTC/WebAudio capture, a rules layer for exercise sequencing, per-user progress store in an EU region.
• Choose B when: you build a branded product or school tool and can accept cloud audio processing under a DPA.

Path C — Self-host (open source)

• Models: wav2vec 2.0 / WavLM / HuBERT phoneme recognisers (fine-tuned), Whisper or faster-whisper for transcripts, NVIDIA NeMo/Parakeet and Canary, Montreal Forced Aligner or WhisperX for alignment, Piper or Coqui-style open TTS for model audio.
• Why: audio of children and patients stays on-premise or on-device; offline classroom or clinic use.
• Hardware: HW1 for on-device scoring of short utterances; HW2 (one 16-24 GB GPU) for a multi-user school or clinic server.
• Ops effort: moderate; the hard part is calibrating scores per language and age group.

Path D — Custom training

Worth it for under-served languages, children's speech, accented L2 speech or disordered speech, where generic models fail. Needs hundreds of hours of labelled speech with phoneme-level annotations by trained raters, plus consent-based collection. Open corpora such as L2-ARCTIC and speechocean762 help for English; for disordered speech, clinical data is scarce and highly sensitive.

State of the art (2026)

• Self-supervised speech models (wav2vec 2.0 family) fine-tuned for phoneme recognition are the standard for mispronunciation detection; end-to-end multimodal LLMs that listen to audio are increasingly used to give explanations, but their numeric scoring is less reliable and less reproducible than dedicated scorers.
• Prosody and fluency scoring is improving, but agreement with human raters varies by language, speaker age and accent.
• Limits and failure modes: children's and elderly or disordered speech are under-represented in training data and produce high error rates; ASR "forgives" errors; native-accent bias penalises legitimate regional variation; noisy rooms and cheap microphones distort scores; scores can mislead users into over-trusting a number. Microsoft's own limitations note stresses testing with your population and choosing thresholds accordingly [2].
• Therapy: evidence for fully automated therapy is limited; the credible model is clinician-led therapy with AI-assisted home practice and measurement.

Shared resources & combinations

• Same stack (asr + tts + llm-general): SPEECH-001 (transcription), SPEECH-006 (TTS), SPEECH-004 (live captions); one self-hosted Whisper/NeMo server on HW2 can serve several.
• Close relatives: EDU-011 (conversation partner for language learning), EDU-004 (AI tutor), EDU-005 (adaptive learning paths), EDU-003 (formative feedback).
• Assistive speech: SPEECH-018 (atypical-speech ASR, AAC) shares disordered-speech data and models; do not mix datasets without clear consent.

Effort, cost & KPIs

• Path A: days to weeks; per-user or per-seat licences.
• Path B: 2-4 months for a product with UI, progress store and privacy work; API cost scales with audio minutes.
• Path C/D: 4-9 months incl. data collection and rater calibration; GPU server plus ML maintenance.
• KPIs: agreement of AI scores with human raters (correlation/kappa per subgroup), practice minutes per learner per week, retention, measured improvement on independent human-rated tests, therapist time saved, rate of flagged recordings overturned by humans.

People & change

Involve management and the department lead (head of school, therapy practice) first to define what the tool does and does not do: practice support, not diagnosis. Teachers and therapists as end users should shape exercises and test the scores. The DPO is central: voice of therapy patients is health data (Art. 9 GDPR) and children need parental consent, so a DPIA is advisable, plus an AVV with each vendor. Legal/compliance checks MDR status if any therapeutic claim is made. IT handles devices and accounts.

Works council: no, in the typical case, because the learners or patients are the users, not employees being monitored. It becomes relevant when employees are assessed (corporate language training with reporting to HR): then §87(1) no. 6 BetrVG applies, as the system could monitor performance. Public schools: staff council where staff are affected.

Communicate openly to learners and parents what is recorded, how long and who sees it. Pilot with a small class or patient group that consented, and have teachers compare AI scores with their own. Typical concerns are bias for accents and children and anxiety about scoring; address them with transparent score explanations and alternative human assessment. The teacher or therapist always decides on progression and treatment. Support staff AI literacy per Art. 4 AI Act.

See GUIDE-005, GUIDE-006 and GUIDE-008.

Risks, regulation & compliance

• EU AI Act: classification depends on use. Pure self-study practice is minimal risk. AI used to assess learning outcomes or steer admission/placement in education is high-risk (Annex III, point 3), which includes speaking tests that determine access or progression. Art. 5(1)(f) prohibits emotion recognition in education and workplaces: do not infer learner emotions, stress or confidence from voice, except for medical or safety reasons. Art. 4 AI literacy applies to staff [4].
• Medical devices: software that diagnoses or treats a speech disorder may be a medical device under the MDR (Regulation 2017/745), requiring CE marking and clinical evidence; general pronunciation training for learners is not. State the intended purpose carefully.
• GDPR: voice recordings are personal data; speech patterns of therapy patients are health data (Art. 9) requiring explicit consent or another Art. 9 basis; a DPIA is advisable. Children: parental consent, minimal retention, no secondary use of recordings for model training without separate consent. Avoid voice-print creation, which moves into biometric data (SPEECH-010) [5].
• Fairness: accent, age, and disability bias; offer human review and alternative assessment.
• Mitigations: on-device or EU processing, short retention, transparent score explanations, therapist oversight, subgroup testing before rollout.

Getting started

1. Define purpose: practice aid, formal assessment or therapy support; this decides the regulatory class.
2. Record 100-200 utterances from your actual target group (with consent), rated by two humans.
3. Compare 2-3 scorers (Azure pronunciation assessment, a fine-tuned wav2vec 2.0 model, a vendor SDK) against the human ratings per subgroup.
4. Pilot with one class or clinic for 6-8 weeks; no emotion inference, no automatic grades.
5. Set retention, consent and DPIA before scaling; review KPIs monthly.

Sources

1. Microsoft Learn: Use pronunciation assessment (Azure AI Speech) — https://learn.microsoft.com/azure/ai-services/speech-service/how-to-pronunciation-assessment
2. Microsoft Learn: Characteristics and limitations of Pronunciation Assessment — https://learn.microsoft.com/en-us/legal/cognitive-services/speech-service/pronunciation-assessment/characteristics-and-limitations-pronunciation-assessment
3. Baevski et al., wav2vec 2.0 (arXiv 2006.11477) — https://arxiv.org/abs/2006.11477
4. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj
6. Regulation (EU) 2017/745 (MDR) — https://eur-lex.europa.eu/eli/reg/2017/745/oj
7. OpenAI Whisper — https://github.com/openai/whisper