Classroom observation by a mentor is rare, subjective and expensive, so most teachers get little feedback on how they actually talk with students. Speech AI turns a recorded lesson into a diarised transcript and classifies pedagogical discourse moves: who talks how much, which questions are asked, how long the teacher waits, and whether student ideas are taken up. The teacher gets a private, formative report and coaching prompts. In the EU the system must stay limited to discourse metrics: emotion recognition in education is prohibited under AI Act Art. 5.
Problem & value
- Today: feedback comes from occasional peer or principal observations and practicum supervisors' notes, scored with rubrics such as CLASS. Coverage is thin, observers differ and teachers rarely see data on their own talk patterns.
- AI value: every recorded lesson can be analysed cheaply and repeatably. Teachers can self-reflect on their own schedule, and mentors can focus on the moments that matter instead of watching whole lessons.
- Evidence: a randomized controlled trial in a large online course found that automated feedback on teachers' uptake of student ideas changed teacher behaviour (Demszky et al.) [1][2]. Effects on student outcomes in regular school classrooms are less established; treat vendor claims as hypotheses and measure locally.
Variants
- In-service self-reflection: teacher records own lessons, sees private dashboard (TeachFX-style).
- Teacher-training practicum: trainees upload video, get AI feedback before the supervisor's review (Edthena-style).
- Instructional coaching programmes: coaches see aggregated trends for consenting teachers.
- University teaching and corporate training: lecture and workshop talk-time and questioning analysis.
- Research: large-scale coding of discourse for education studies (e.g. NCTE transcripts) [2].
- Not in scope / prohibited: inferring student or teacher emotion, engagement or attention from voice or face; ranking teachers for pay or dismissal (see Risks).
How it works
- classroom audio/video (consented, one device, edge or upload)
- noise reduction + ASR (multilingual, child speech)
- diarisation (teacher vs. students vs. group talk; no speaker identity of children)
- utterance segmentation + discourse-move classification (question type open/closed, wait time, uptake/revoicing, talk share, student-to-student talk)
- metric aggregation per lesson and over time
- LLM coaching layer: pick 2-3 moments with transcript evidence, suggest strategies
- private teacher dashboard
- optional coach review
- goal setting
- raw audio deleted after processing (retention policy)
- Classification: fine-tuned encoder classifiers or prompted LLMs with a codebook; uptake models were trained on human-annotated classroom transcripts [3].
- Coaching text: an LLM must quote the transcript for every claim, to avoid invented praise.
- Human-in-the-loop: the teacher owns the data; coach or mentor reviews only with consent. Metrics are never fed automatically into appraisal.
Data preparation
The core data is consented classroom audio or video, plus lesson plans, subject/grade metadata and the observation rubric (CLASS, Danielson, national frameworks). Recordings usually sit on teachers' phones, school laptops or tablets; rubrics and lesson plans live in file shares, learning platforms or paper. Typical gaps: noisy rooms and overlapping child speech, inconsistent file formats, missing consent records per student and parent, no agreed codebook for discourse moves, and unclear access rights between teacher, mentor and school leadership.
Preparation steps: - Define one recording standard (device, microphone placement, file format, upload path) and a consent register, see PREP-025 for the legal-basis check. - Convert and normalise audio (ffmpeg), then test ASR and diarisation (Whisper, pyannote) on 10-20 real lessons. - Fix a written codebook for question types, wait time and uptake and build a small human-annotated test set, see PREP-018 and PREP-019. - Pseudonymise transcripts and set retention: raw audio deleted after processing, see PREP-022. - Restrict dashboards so only the teacher sees individual reports.
Rough share of effort: about 25-35 percent, mostly consent management, recording logistics and the annotated reference set. Transcription itself is covered in SPEECH-001.
Implementation paths
Which path for whom
- Enterprise (school operators, training providers): Path C, because lesson recordings of minors and teacher performance data are highly sensitive and Whisper plus open diarisation runs on HW2.
- Developer: Path C, because the open ASR/diarisation stack gives full control over retention and raw-audio deletion.
- Public (state schools, teacher-training institutes): Path C on sovereign infrastructure, because procurement rules and school-law data residency favour in-house or state-run hosting; Path A only for products with a documented EU-hosted DPIA basis.
Path A — Buy (off-the-shelf)
- Classroom talk analytics: TeachFX (US, widely used for talk time and coaching) [4]; Edthena (video coaching with AI feedback, used in teacher preparation) [5].
- Video observation platforms: Sibme, Swivl, and video-coaching tools from Teachstone (CLASS).
- Check: EU hosting or an EU-based processor, DPA, no emotion or engagement scoring, deletion of recordings, German/French/other-language ASR quality, child-speech accuracy, DPIA support.
- Few EU-native products exist; US vendors need transfer safeguards (SCCs, EU-US DPF) and a school-level legal review.
Path B — Build on cloud APIs
- ASR with diarisation: Azure AI Speech (EU regions), Google Cloud Speech-to-Text (EU), AWS Transcribe (Frankfurt), Speechmatics (UK/EU), Mistral Voxtral (EU provider).
- Classification and coaching: a frontier LLM (Claude, Gemini, GPT, Mistral) with structured output against a codebook; run on EU endpoints.
- Orchestration: queue-based pipeline plus a teacher-facing web app; store only transcripts and metrics.
- Choose B for custom rubrics, national curricula or languages that vendors do not serve.
Path C — Self-host (open source) (recommended)
- ASR: Whisper large-v3 / faster-whisper, NVIDIA Parakeet/Canary, or Voxtral open weights [6].
- Diarisation: pyannote.audio, NVIDIA NeMo (Sortformer) [7].
- Classifiers and LLM: fine-tuned BERT-class model for discourse moves; open-weight LLM (Qwen, Llama, Mistral families) via vLLM for coaching text.
- Hardware: HW2 handles a school district's daily lessons offline; on-premises processing keeps children's voices inside the institution.
- Noisy rooms, overlapping speech and child voices are the main engineering effort.
Path D — Custom training
Worth it for a national or university programme: fine-tune ASR on classroom and child speech in the local language, and train discourse-move classifiers on a few thousand utterances double-coded by trained educators. Reported inter-rater agreement of the human coding sets the ceiling.
State of the art (2026)
- Transcription: modern open and commercial ASR is good on teacher speech, but whole-class recordings with distant microphones, overlapping children and dialects still show much higher error rates; diarising individual children is unreliable and unnecessary.
- Discourse classification: models for uptake and question types approach but do not match expert coders [3]; LLM-based coding with a clear codebook is now the quickest baseline.
- Coaching: LLM-generated, transcript-grounded feedback works best when it is short, specific, non-evaluative and teacher-initiated [1].
- Limits: talk-time is not quality; high student talk can be off-task. Metrics are context-blind (subject, group work, language class) and can penalise teachers of younger or multilingual classes. Models may misattribute speakers and hallucinate coaching claims.
Shared resources & combinations
- Shared stack: asr + diarization + llm-general, identical to SPEECH-002 (call-centre QA) and SPEECH-001 (meeting transcription); one self-hosted server can serve all three.
- Combine with: EDU-009 (lesson planning, close the loop between plan and delivery), EDU-010 (role-play practice for trainees), EDU-012 (speech assessment of pupils), EDU-007 (student analytics, deliberately kept separate by purpose limitation).
Effort, cost & KPIs
- Path A: weeks to pilot; per-teacher or per-school licence. Main effort is consent, recording hardware and change management.
- Path B: 2-4 months incl. DPIA; cost drivers are ASR minutes and review effort.
- Path C/D: 4-9 months; GPU plus annotation of classroom data.
- KPIs: share of invited teachers who opt in and keep recording, metric stability against human observers (agreement), teacher-rated usefulness, change in targeted practice over a term, coach time saved. Student learning gains only in a controlled evaluation.
People & change
Involve management and the department lead (school head or programme director) first to fix the purpose: formative coaching, never appraisal. Bring the works council or staff council in at the start, before any pilot: the system logs and analyses teachers' behaviour, so §87(1) no. 6 BetrVG co-determination applies (objective suitability for monitoring is enough per the BAG), §90(1) no. 3 requires early information, and §80(3) allows an AI expert; in public schools the Personalrat under the state staff-representation law is responsible. A works agreement should ban use for appraisal or disciplinary purposes. The DPO must be involved early; a DPIA (Art. 35 GDPR) is needed because of systematic evaluation of people, children's voices and new technology, plus an AVV (Art. 28 GDPR) with any processor. Involve IT security for storage and deletion, HR for training links, and parents and students for consent.
Support AI literacy (AI Act Art. 4 asks employers to support it, no guaranteed level) with a short workshop on what the metrics do and do not mean. Recruit volunteer teachers as pilot group, share results only with them, and let them delete recordings at any time. Typical concern is surveillance: answer with transparency, private reports and opt-out. Humans stay in control; the coach interprets, the AI only suggests.
See GUIDE-004, GUIDE-005 and GUIDE-008.
Risks, regulation & compliance
- AI Act Art. 5(1)(f): prohibited. Inferring emotions of people in educational institutions (students and teachers) is banned except for medical or safety reasons; the Commission guidelines confirm education is covered [8]. Do not ship sentiment, enthusiasm, boredom, stress or "engagement" detection from voice or video. Text-based discourse labels (questions, uptake) are not emotion recognition, but vendors' feature sets must be checked.
- Risk class: formative, teacher-owned feedback is typically minimal or limited risk. If outputs are used for performance evaluation or promotion of staff, Annex III point 4 (employment, worker management) high-risk obligations are likely [9]. Using them to evaluate student learning outcomes would touch Annex III point 3. Keep use contractually private and formative.
- GDPR: recordings contain voices of minors (special protection), teachers (employee data) and possibly special-category content spoken aloud. Needs a legal basis (public task or consent; employee consent is weak), DPIA, transparency to parents and pupils, data minimisation, short retention, no secondary use, and works-council or staff-representative co-determination (in Germany BetrVG/PersVG, plus state school laws).
- Other: AI literacy (Art. 4); bias against accents and dialects; surveillance creep.
- Mitigations: opt-in, teacher-only visibility, delete audio after transcription, aggregate reporting with minimum group sizes, published purpose statement, regular audits.
Getting started
- Agree purpose and governance first: formative only, teacher-owned data, works-council sign-off.
- Run a DPIA and choose recording setup (one lapel or table microphone) and retention.
- Pilot with 5-10 volunteer teachers using one Path A tool or a Whisper + pyannote notebook.
- Compare AI metrics with a human observer on 10 lessons; adjust the codebook.
- Review feedback quality with teachers after a term; only then scale or customise.
Sources
- Demszky et al. — Can automated feedback improve teachers' uptake of student ideas? RCT — https://arxiv.org/abs/2105.07949
- Stanford NLP / Demszky — M-Powering Teachers and the NCTE transcripts — https://github.com/ddemszky/classroom-transcript-analysis
- Demszky et al. — Measuring conversational uptake (ACL 2021) — https://arxiv.org/abs/2106.03873
- TeachFX — https://teachfx.com
- Edthena — https://www.edthena.com
- OpenAI Whisper — https://github.com/openai/whisper
- pyannote.audio — https://github.com/pyannote/pyannote-audio
- European Commission — Guidelines on prohibited AI practices — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj