Language learning conversation partner

The scarcest resource in language learning is speaking practice: tutors are expensive, classmates are few, and many learners are afraid of making mistakes in front of people. Voice-capable LLMs now hold fluent, patient conversations in dozens of languages, adapt to the learner's level and give feedback on grammar, vocabulary and (more roughly) pronunciation. Used as a complement to teachers, they multiply speaking time per learner at low marginal cost.

Problem & value

• Today: classroom time gives each learner only minutes of speaking per lesson; private tutors and conversation platforms cost per hour; apps offered mostly scripted drills. Learners often stall at the "intermediate plateau" because they rarely speak freely.
• AI value: unlimited, judgement-free practice at any time; scenario role-plays tailored to the learner's job or exam; instant corrections; consistent level control (e.g. stay at CEFR B1); teachers receive summaries of recurring errors rather than correcting everything by hand.
• Evidence: studies of LLM chatbots for language learning (systematic reviews, 2023-2025) report mostly positive effects on speaking confidence and willingness to communicate, but evidence on durable proficiency gains is still thin and many studies are short and small [6]. No reliable universal benefit figure exists; measure locally.

Variants

• Consumer apps (private): subscription app with conversation mode and character role-plays.
• Schools and universities: teacher-configured scenarios inside the LMS, with safeguards for minors and teacher dashboards; exam preparation (Goethe, DELF, IELTS-style speaking tasks).
• Corporate and public sector: job-specific language training (customer service in English, German for care workers, onboarding of migrants and refugees, integration courses).
• Specialised: pronunciation coaching, accent reduction, heritage and minority languages (low-resource languages remain weaker), sign languages are out of scope (see LANG-030).
• Embodied: avatar or robot conversation partners (see CONV-018).

How it works

1. learner speech/text
2. ASR (streaming, accent-tolerant, optional phoneme-level scores)
3. dialogue manager: system prompt with scenario, CEFR level, learner profile, error memory
4. LLM reply (simplified to target level, stays in character, recasts errors)
5. guardrails (age-appropriate, off-topic, safety, PII)
6. TTS (natural target-language voice) | or one speech-to-speech model end to end
7. feedback layer: error log, corrections, vocabulary cards, session summary
8. progress store (teacher dashboard, spaced repetition hand-off)

• Pipeline vs. speech-to-speech: the classic cascade (ASR → LLM → TTS) offers control, logging and per-component choice; native speech-to-speech models cut latency and keep prosody but give less visibility into what was heard, which matters for pronunciation feedback.
• Correction strategy: pedagogically better to recast gently during talk and give detailed corrections afterwards; the prompt encodes the teacher's policy.
• Human-in-the-loop: teachers define scenarios, goals and correction style, review dashboards and samples of transcripts; summative assessment stays with humans (see EDU-012).

Data preparation

A conversation partner needs little data of its own: the target level (CEFR), the topics and vocabulary of the course, role-play situations, and optionally corrections rules for typical learner errors. In an organisation this material sits in course books and PDFs, LMS content, company glossaries and onboarding documents; for work-related language training it is useful to add specialist terms from product sheets and e-mails. Typical gaps: no defined level per learner, vocabulary lists scattered across files, scanned course material, and recordings of learners that contain personal data.

Preparation steps: define level and learning goals per group, assemble a glossary and phrase list per topic (PREP-015), collect example dialogues as gold standards (PREP-016), digitise paper material via OCR (PREP-006), and decide what happens to audio: ideally it is not stored, or only in pseudonymised form (PREP-022, PREP-025). A small test set of learner utterances with expert corrections helps to check feedback quality (PREP-018).

Tools: spreadsheet or tidy tables for vocabulary (PREP-011), Whisper for transcripts. Rough share of effort: low, about 10-15 percent; most products work after choosing level and topic.

Implementation paths

Which path for whom

• private: A - mature language apps with voice conversation cost a few euros per month; local apps are an option for privacy-minded learners.
• smb: A - company language courses are a standard process, and hosted EU products fit SMB prices without own data.
• enterprise: A - language training is a commodity function; a vendor licence with SSO and an AVV is cheaper than building.
• developer: C - an open stack (Whisper, a small LLM, open TTS) on HW2 suits apps that need cost control and voice data kept inside.
• public: B - schools, adult education and integration courses should use APIs on sovereign EU cloud because learners are often minors or vulnerable.

Path A — Buy (recommended for private users, SMBs and enterprises)

• Consumer and institutional apps: Duolingo (Max tier with video call and role-play), Speak, Praktika, ELSA (pronunciation), Babbel, Busuu, Memrise, Langotalk-style tutors; EU-based: Babbel (DE), Busuu (UK/Spain-linked, now Chegg-owned), Lingoda (DE, live teachers plus AI tools).
• Education platforms: Microsoft Reading/Speaking Coach in Teams Education, Google Gemini and Khanmigo-type tutors with language modes, Moodle/LMS plugins with AI chat.
• Choose A when: individual learners, or a school wanting a ready product. Check: data location, child-safety mode, DPA, whether voice recordings are used for model training, teacher controls.

Path B — Build on cloud APIs

• Speech-to-speech: OpenAI Realtime API (gpt-realtime), Google Gemini Live API, both via WebRTC/WebSocket; EU data-residency options exist for some providers, verify per contract.
• Cascade: ASR (Azure Speech incl. pronunciation assessment, Deepgram, Speechmatics (UK), Mistral Voxtral (EU), OpenAI/Whisper-class transcription), LLM (Claude, Gemini, GPT, Mistral), TTS (ElevenLabs, Azure Neural TTS, Google Cloud TTS).
• Orchestration: LiveKit Agents or Pipecat for realtime voice, plus a learner-state store.
• Choose B when: you need your own pedagogy, branding or LMS integration (e.g. a university speaking lab).

Path C — Self-host (open source)

• ASR: Whisper large-v3 / faster-whisper, NVIDIA Parakeet/Canary, Voxtral open weights.
• LLM: Qwen, Llama, Mistral or Gemma-class models (8-30B) for dialogue; weaker for rare languages.
• TTS: Kokoro, XTTS/Coqui-derived, Piper, Orpheus, Kyutai (FR) models; Kyutai Moshi/Unmute for full-duplex speech-to-speech.
• Orchestration: LiveKit (open source) or Pipecat.
• Hardware: HW2 serves a classroom-sized load with a cascade of small models; HW3 for several concurrent users on larger LLMs. Choose C when voice data of minors must stay on premises.

Path D — Custom training

Worthwhile for low-resource or minority languages, for pronunciation scoring tuned to specific L1 backgrounds, or for exam-aligned feedback. Needs transcribed learner speech with expert error annotation (learner corpora such as those used for CEFR research) and consent for reuse.

State of the art (2026)

• Realtime speech-to-speech models deliver sub-second turn-taking, interruption handling and expressive voices; frontier LLMs cover dozens of languages with near-native fluency in major ones.
• Pronunciation feedback: general ASR is tolerant of accents, which hides errors; dedicated pronunciation assessment (Azure, ELSA, phoneme-level models) is still more reliable for scoring.
• Personalisation: persistent memory of learner errors and vocabulary, linked with spaced repetition (EDU-016) and adaptive paths (EDU-005).
• Limits and failure modes: over-politeness and failing to correct; wrong or invented grammar explanations; dialect and regional-variety bias; weaker quality in low-resource languages; drifting above the target CEFR level; learners drifting into English; overreliance and reduced effort; latency or ASR errors that make learners doubt themselves.

Shared resources & combinations

• Same stack (asr + tts + llm-general): CONV-002 voice agents, CONV-018 digital humans, SPEECH-003 speech translation, EDU-010 role-play simulations (same engine, different scenarios), EDU-004 tutor (text mode).
• Natural combinations: EDU-005 adaptive paths, EDU-016 flashcards from conversation errors, EDU-012 formal speaking proficiency assessment, EDU-001 content generation.
• One voice stack on HW2/HW3 can serve all of these.

Effort, cost & KPIs

• Path A: days; per-user subscription or institutional licence.
• Path B: 1-3 months for a pilot with LMS integration; cost driven by minutes of realtime audio, which is notably more expensive than text.
• Path C: 2-4 months; GPU server plus voice-latency tuning.
• KPIs: speaking minutes per learner per week, retention/active days, error-rate trend per error category, learner confidence (survey), proficiency gain on an independent test, teacher time saved, cost per speaking hour.

People & change

Involve the learners as the daily users from the start, the department lead or training manager as process owner, HR for budget and learning goals, IT for accounts, and the data protection officer because speech recordings and learner progress are personal data (AVV with the provider, DPIA likely if minors or large-scale voice processing are involved, GUIDE-005). Management only needs to approve the tool in the AI policy (GUIDE-007).

Works council: depends. If a company course platform logs per-person usage or progress visible to superiors, it is suited to monitor behaviour and section 87(1) no. 6 BetrVG applies; a purely private, anonymous use by learners does not trigger it. Section 90(1) no. 3 requires information when work processes are planned with AI. Schools and public bodies involve the staff council (Personalrat). Agree that progress data is not used for performance reviews (GUIDE-004).

AI literacy under AI Act Art. 4: support users with a short introduction on strengths (patient, always available) and limits (wrong corrections, invented idioms, accent bias in speech recognition) (GUIDE-006). Start with a pilot group of motivated learners and a course teacher who sees the aggregated results, then widen (GUIDE-008, GUIDE-009). Typical concerns: replacing teachers and being recorded. Address them by positioning the tool as practice between lessons, with the teacher setting goals and judging progress.

Risks, regulation & compliance

• EU AI Act: chat and voice partners must disclose that the user talks to an AI (Art. 50(1)); synthetic voices need machine-readable marking (Art. 50(2)). Art. 5(1)(f) prohibits emotion recognition in educational institutions (except medical or safety reasons): do not infer frustration or engagement from voice or face. Using the tool to assess or evaluate learning outcomes, steer learning or decide on admission is high-risk under Annex III(3); keep the partner formative and keep graded speaking exams human-reviewed. Art. 4 AI literacy applies to deploying staff [5].
• GDPR: voice recordings are personal data and can be biometric if used for identification (avoid voiceprints). Minors: parental consent/age rules (Art. 8), data minimisation, no profiling for advertising, DPIA likely for school-wide deployment. Prefer no retention of audio, transcripts only, and no vendor training on learner data [4].
• Child safety: age-appropriate filters, no emotional-dependency design, escalation path for disclosures of harm; see also CONV-024.
• Quality: teacher review of sampled transcripts; mark AI explanations as possibly wrong.
• Mitigations: DPA with EU hosting, transparency notice, opt-out for audio, bias testing across accents.

Getting started

1. Define goal, target languages and CEFR levels; pick 5 scenarios with a teacher.
2. Pilot 2-3 Path A tools with 10-20 learners for 4 weeks; log speaking minutes and feedback.
3. Check privacy: audio retention, training use, child-mode, DPA, EU hosting.
4. If you need control, prototype a Path B cascade with LiveKit and a prompt encoding the correction policy; compare with speech-to-speech.
5. Evaluate against an independent speaking test before scaling.

Sources

1. OpenAI — Realtime API documentation — https://platform.openai.com/docs/guides/realtime
2. Microsoft — Azure AI Speech pronunciation assessment — https://learn.microsoft.com/azure/ai-services/speech-service/how-to-pronunciation-assessment
3. Council of Europe — CEFR Companion Volume — https://www.coe.int/en/web/common-european-framework-reference-languages
4. European Data Protection Board — guidelines and GDPR resources — https://www.edpb.europa.eu/our-work-tools/general-guidance/guidelines-recommendations-best-practices_en
5. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
6. Systematic review of chatbots in language learning — https://www.sciencedirect.com/search?qs=chatbot%20language%20learning%20systematic%20review
7. LiveKit Agents — https://github.com/livekit/agents
8. Pipecat — https://github.com/pipecat-ai/pipecat
9. Kyutai (Moshi, Unmute) — https://github.com/kyutai-labs/moshi
10. Whisper — https://github.com/openai/whisper