Courts, parliaments and municipal councils must keep an exact record of what was said, but stenographers and typists are scarce and slow. Speech recognition with diarisation produces a time-aligned draft with named speakers within minutes; trained staff correct it to official or evidentiary quality. The result is a searchable record linked to the audio, with the human remaining the author of the official text.
Problem & value
- Today: parliaments use stenographers and editing offices; courts in many EU states (e.g. Germany, criminal trials) keep only a summary protocol (Formalprotokoll) and judges' notes, so disputes about what a witness said are settled from memory. Councils often publish minutes weeks later.
- AI value: faster publication of records, full-text search across sessions, relief for scarce stenographers, and a reviewable evidence base for appeals. Correction effort replaces typing effort.
- Legal driver (DE): the Hauptverhandlungsdokumentationsgesetz (DokHVG, BT-Drs. 20/8096) provides for audio recording of first-instance trials at regional and higher regional courts with automated transcription; the Bundestag passed it on 17 November 2023, but the Bundesrat referred it to the mediation committee in December 2023, which adjourned in February 2024 without a result [1][9]. The law was not promulgated before the end of that legislative period, so there is currently no federal recording duty; check whether a new bill has been tabled before planning.
- No reliable public figures exist for time savings that generalise across institutions; measure your own correction time per audio hour in a pilot.
Variants
- Courts: criminal and civil hearings, tribunals, arbitration; evidentiary-grade transcript with chain of custody, optional image recording.
- Parliaments: plenary and committee verbatim reports, multilingual chambers with interpretation.
- Municipal councils & administrative hearings: minutes plus public streaming with captions (see SPEECH-004); often a small team, moderate requirements.
- Private/associations: general assemblies, shareholder meetings (lighter variant, see SPEECH-001).
How it works
- room audio (per-microphone channels) + session metadata (roster, agenda)
- capture with hash/timestamp (chain of custody)
- voice activity detection + diarisation (channel-aware) + speaker naming via roster
- ASR with domain vocabulary (names, statutes, dialect)
- word timestamps + confidence
- punctuation/normalisation to verbatim or "clean verbatim" rules
- human correction in editor synced to audio (low-confidence words highlighted)
- review/approval (clerk, editor, presiding judge)
- signed, archived record
- search index, subtitles, publication
- Speaker attribution: the strongest signal is the physical microphone channel plus the seating roster; acoustic diarisation is a fallback for shared microphones and remote participants. Naming must come from the roster or the clerk, not from voice identification (see risks).
- ASR: a large multilingual or language-specific model, adapted to domain vocabulary; dialect, overlapping speech and interjections (heckling, cross-talk) remain the hard cases.
- LLM role: optional and strictly secondary: fix punctuation, flag inconsistencies, extract agenda item boundaries or draft the formal minutes. The verbatim text itself must not be rewritten by an LLM.
- Human-in-the-loop: every official record is corrected and approved by an authorised person; the draft is never the record. Corrections and approvals are logged.
Data preparation
Data: multi-channel audio and video from courtrooms and council chambers, speaker rosters, seating plans, agendas, motions and earlier official records. These typically live in conference or recording systems, the document management or council information system (Ratsinformationssystem), file shares and, for older material, on paper.
Typical gaps: one mixed audio track instead of one channel per microphone, missing microphone-to-person mapping, scanned historical records without text, inconsistent spelling of names, place names and legal terms, and unclear access rights on recordings.
Preparation steps: 1. Record one channel per microphone where possible and document the mapping to seats and persons. 2. Build a custom vocabulary from agendas, rosters and legal terms, see PREP-015. 3. Digitise old protocols with OCR (PREP-006) and collect corrected official records as gold-standard examples and test sets (PREP-016, PREP-018). 4. Define retention, access rights and classification of recordings (PREP-024). 5. Capture clerk corrections as feedback to improve the vocabulary over time (PREP-021).
Tools: ffmpeg for audio conversion, Whisper-family models and pyannote for pilots, Paperless-ngx or a DMS for archive scans.
Share of effort: roughly 20 to 30 percent of the project, mostly audio setup and vocabulary work. See GUIDE-003 for a readiness check.
Implementation paths
Which path for whom
- public: C. Hearings and sessions are sensitive, courts and parliaments need sovereign control, and Whisper-class models with diarisation run on an own or German government cloud server.
- enterprise: C. Boards, arbitration and internal hearings involve confidential content and recurring volume, so a self-hosted pipeline on a shared internal AI platform pays off.
- developer: C. Open-weight ASR and diarisation give full control over custom vocabulary, audio retention and integration into court or council software.
Path A — Buy (off-the-shelf)
- Parliament/government record systems: specialised vendors such as Verbit (court reporting), Speechmatics (UK), Trint, and European specialists such as Phonexia (CZ) and Vocapia (FR); in German-speaking countries, court and parliament recording and stenography-software providers. Confirm current product scope with the vendor.
- Council/meeting tools: council-information systems (e.g. Somacos, Sternberg, ALLRIS-class systems) increasingly add minute-drafting modules.
- Choose A when: a small council or court wants a managed tool with EU hosting and an editor; check on-prem option, audit trail, role-based access, export formats and deletion functions.
Path B — Build on cloud APIs
- Speech APIs with EU regions: Azure AI Speech (batch transcription with diarisation, custom speech), Google Cloud Speech-to-Text (EU location), AWS Transcribe (Frankfurt/Ireland), Speechmatics, Deepgram. Use contractual EU residency and no-training terms.
- Constraint: courts and security-relevant bodies often may not use non-sovereign public cloud; a justice or government cloud (e.g. in Germany, the Bundes- and Länder-clouds) is usual.
- Choose B when: public sessions (open parliament, council streams) without sealed content.
Path C — Self-host (open source) -- usual choice for courts
- ASR: OpenAI Whisper large-v3 / large-v3-turbo via faster-whisper or whisper.cpp; NVIDIA Canary and Parakeet (NeMo) multilingual models; Kyutai and Mistral Voxtral speech models are open alternatives. Check licence and per-language error rates on your own recordings.
- Alignment & diarisation: WhisperX (forced alignment), pyannote.audio, NVIDIA Sortformer/NeMo diarisation, Silero VAD.
- Editor & stack: word-level correction UI synced to audio (e.g. open-source subtitle/transcript editors or a custom web app), PostgreSQL/OpenSearch for search, vLLM for an optional local LLM.
- Hardware: HW2 handles one or a few concurrent sessions; HW3 for many courtrooms plus live streaming.
Path D — Custom training
Worthwhile for strong dialects, legal jargon or low-resource EU languages. Fine-tune Whisper or a Canary/Parakeet model on hundreds of hours of audio with approved human transcripts (the archive of existing official records is the natural data source, subject to confidentiality and consent rules). Keep a held-out test set from different speakers and rooms.
State of the art (2026)
- Large open models transcribe clean, read speech in major EU languages at low word error rates, but spontaneous, overlapping, accented or dialectal courtroom speech is markedly harder. Benchmark on your own audio, not on vendor figures.
- Diarisation and overlap remain the weakest link; multi-channel capture and roster mapping beat purely acoustic methods. Far-field room microphones degrade accuracy; individual microphones help.
- Verbatim vs. clean: parliaments typically edit toward readable text; courts want content-faithful text including hesitations that matter. Decide the style rule and configure normalisation accordingly.
- Failure modes: hallucinated text in silence or noise (known for Whisper-type models), dropped words, wrong names, wrong speaker, numbers and legal citations. Highlight low confidence and never present the draft as an official record.
- Trend: end-to-end, streaming models and speech LLMs reduce latency enabling live drafts; still require correction before the record is official.
Shared resources & combinations
- Same ASR + diarisation stack as SPEECH-001 (meetings), SPEECH-005 (dictation), SPEECH-016/-020 (emergency and radio); one GPU server can serve several of these.
- Live subtitles for streams: SPEECH-004; interpreting chambers: SPEECH-003 (translate from the corrected record, not the draft).
- Case-file handling: DOC-020 (court-side case-file processing), DOC-005 (historical documents) for the same archive.
- Avoid coupling with SPEECH-010 (voice biometrics): speaker naming should use rosters.
Effort, cost & KPIs
- Path A: weeks to a few months incl. room audio installation and procurement.
- Path B/C: 6-18 months in courts (procurement, security accreditation, audio hardware, training, procedural-law rules); audio installation is often the largest cost driver.
- KPIs: word error rate and speaker-attribution error on a held-out sample; correction minutes per audio hour; time from session end to published record; share of segments accepted unchanged; search usage; incident count (wrong attribution found after approval).
People & change
Who and when: involve management and the department lead (court administration, clerk's office, council secretariat) at the start, because the process and the legal status of the record are theirs. Bring in the data protection officer and IT security before the first recording test, and IT for audio hardware and hosting. Clerks, stenographers and judges or chairpersons as end users join the pilot design early.
Works council / staff council: depends. Where clerks or stenographers are employees, the Personalrat (BPersVG or state law) or a Betriebsrat has co-determination (§87(1) no. 6 BetrVG, or the equivalent staff-council rules), because the system logs corrections and processing times and is objectively suited to monitor performance. Inform it under §90(1) no. 3 when work processes change, see GUIDE-004. Judicial independence needs separate clarification.
Data protection: hearings contain personal data and often special categories, so a DPIA is likely (Art. 35 GDPR), plus an AVV with any processor. See GUIDE-005.
AI literacy: support staff with training on error types, speaker mix-ups and verification duties (Art. 4 AI Act, GUIDE-006).
Pilot and communication: start with one non-sensitive committee or session type. Typical concern: replacing stenographers. Address it openly: the tool drafts, trained staff correct and sign off, and a human always owns the official record. See GUIDE-008.
Risks, regulation & compliance
- EU AI Act: AI systems intended to be used by or on behalf of judicial authorities to assist in researching and interpreting facts and the law are high-risk under Annex III point 8. A pure transcription tool is arguably an ancillary administrative function; classification depends on how far the output influences judicial decisions. Treat court deployments as high-risk (Art. 9-15: risk management, data governance, logging, human oversight, accuracy) until counsel confirms otherwise. Parliament and council record-keeping is generally minimal or limited risk. [3]
- Art. 5: do not add emotion recognition, credibility or "lie" scoring, or biometric categorisation of speakers; these are prohibited or high-risk and outside this use case.
- GDPR: hearings contain special-category and criminal-offence data (Art. 9, 10) of parties, witnesses, minors and victims; law-enforcement bodies fall under Directive (EU) 2016/680 and national law. Need legal basis, access control, retention and deletion rules, no vendor training on the audio, and DPIA. Public sessions may still need redaction (e.g. protected witnesses).
- Procedural law: evidentiary status of the transcript, who certifies it, rights of parties to review and object, and handling of the audio are set nationally (e.g. the StPO changes planned, but not enacted, by the DokHVG [1]).
- Judicial independence: the AI must not draft findings of fact; keep judges' control of the record.
- Mitigations: on-prem or sovereign cloud, hashing and immutable audio storage, edit audit log, mandatory human approval, bias checks across dialects, accents and genders.
Getting started
- Clarify the legal status of the record: official, evidentiary or merely auxiliary; define style rules.
- Inventory rooms and microphones; prefer one channel per seat plus a mixed feed.
- Collect 20-50 hours of representative audio with existing human transcripts; benchmark Whisper, Canary/Parakeet and two vendors with and without custom vocabulary.
- Build the correction workflow (editor synced to audio, roles, approval, audit log) before tuning accuracy.
- Pilot in one chamber or room, measure correction time per hour, then scale with a DPIA and AI Act check.
Sources
- Deutscher Bundestag — Gesetz zur Dokumentation der strafgerichtlichen Hauptverhandlung beschlossen — https://www.bundestag.de/dokumente/textarchiv/2023/kw46-de-hauptverhandlungsdokumentation-976608
- Bundestag Drucksache 20/8096 (Entwurf DokHVG) — https://dserver.bundestag.de/btd/20/080/2008096.pdf
- Regulation (EU) 2024/1689 (AI Act), Annex III — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- BMJV — Strafgerichtliche Hauptverhandlung wird digital dokumentiert — https://www.bmjv.de/SharedDocs/Pressemitteilungen/DE/2022/1122_Hauptverhandlungsdokumentationsgesetz.html
- WhisperX — https://github.com/m-bain/whisperX
- pyannote.audio — https://github.com/pyannote/pyannote-audio
- faster-whisper — https://github.com/SYSTRAN/faster-whisper
- Directive (EU) 2016/680 — https://eur-lex.europa.eu/eli/dir/2016/680/oj
- DATEV magazin — Vermittlungsausschuss vertagt Beratungen zu Justizreform (Hauptverhandlungsdokumentationsgesetz, 22 Feb 2024) — https://www.datev-magazin.de/nachrichten-steuern-recht/recht/vermittlungsausschuss-vertagt-beratungen-zu-justizreform-119379