Meetings produce decisions and tasks that are lost because nobody had time to take notes. An AI notetaker transcribes the conversation, separates speakers, and an LLM turns the transcript into a summary, decisions and action items with owners. Participants can concentrate on the discussion instead of typing, and absent colleagues can catch up in minutes. In the EU the decisive questions are not accuracy but consent, works-council agreement and data location.
Problem & value
- Today: one person takes notes (badly, and not in their own contribution), minutes are written hours or days later, action items are forgotten, people who could not attend re-ask the same questions.
- AI value: complete, searchable record; minutes drafted within minutes after the meeting; consistent task capture; multilingual teams can read the result in their own language.
- Benefit evidence: vendors claim time savings, but independent figures are scarce; measure your own baseline (minute-writing time, action-item completion) [1].
Variants
- Online meetings (bot or native): built-in recorder in Teams/Zoom/Meet, or a bot that joins as participant (Fireflies, Otter, Read.ai, tl;dv).
- Bot-free capture: desktop/mobile app records system audio (Granola, Jamie — German/EU) so no visible bot joins.
- In-room / hybrid: conference-room microphone arrays or a phone on the table; harder because of far-field audio and overlapping speech.
- Public sector & boards: committee minutes (Protokoll) drafted by AI, adopted by a human secretary; verbatim legal records are a separate use case (SPEECH-021).
- Private use: lectures, family or club meetings, voice memos on the phone (Apple/Google/Samsung built-ins).
How it works
- consent + notice
- capture (platform API / bot / local app / upload)
- voice-activity detection + chunking
- ASR (multilingual, word timestamps)
- speaker diarization (+ mapping to names via calendar roster / platform speaker events)
- transcript clean-up (punctuation, glossary, PII redaction optional)
- LLM: summary, decisions, open questions, action items (owner, due date) as JSON
- human review/edit
- distribute (mail, wiki, task tool, CRM)
- retention & deletion
- ASR: encoder-decoder or LLM-based speech models (Whisper family, Voxtral, Parakeet/Canary, commercial APIs). Word error rates are low for clean, major-language audio and rise with accents, code-switching, jargon, crosstalk and far-field microphones.
- Diarization: neural segmentation + speaker embeddings (pyannote); online platforms additionally provide active-speaker events and per-participant audio tracks, which are far more reliable than clustering a mixed track.
- Summarisation: long-context LLM with transcript, agenda and roster; JSON-schema output; long meetings chunked and merged.
- Human-in-the-loop: the organiser reviews the summary before it is shared; action items are proposals until confirmed. Never auto-send minutes with attributed statements without review.
Data preparation
The input is audio and video from Teams, Zoom or Meet, in-room microphones and uploaded recordings, plus the calendar entry. In an SMB these sit in the conferencing tool, mailboxes, phones and file shares; there is rarely any structured data to clean up. Typical gaps are poor audio (laptop microphones, room echo), missing speaker names, unknown product and customer names that the recogniser gets wrong, mixed languages, and unclear rules about who may record. Output archives also become scattered if every team uses a different tool.
Preparation steps: agree where recordings are stored and for how long (retention and deletion); set up a glossary of names, products and abbreviations (see PREP-015); buy a few conference microphones for the main meeting rooms; define a standard minutes template and task target (Planner, Jira) and collect two or three good example minutes (PREP-016); for the archive add metadata such as date, project and participants (PREP-010) and check access rights so that confidential meetings are not visible to everyone (PREP-023). Pseudonymisation is rarely useful here, so the focus is on consent, rights and retention (PREP-024, PREP-025).
Effort share: low, roughly 10 to 20 percent of the project; most effort goes into rules, rollout and acceptance. See also GUIDE-003.
Implementation paths
Which path for whom
- smb: Path A. Mature, EU-hostable notetaker products exist at SMB prices and meeting minutes are a standard process.
- enterprise: Path B. Own transcription and summary pipelines on cloud APIs integrate with the corporate calendar, task tools and archive, and keep seat costs predictable at scale.
- developer: Path B. Speech-to-text and LLM APIs are the fastest way to embed meeting summaries into a product.
- public: Path C. Confidential sessions and procurement rules favour Whisper plus a local model on sovereign EU infrastructure.
- private: Path C. Local apps with Whisper keep recordings on the own device at no running cost.
Path A — Buy (recommended for most SMBs)
- Platform-native: Microsoft Teams (Premium / Microsoft 365 Copilot recap), Zoom AI Companion, Google Meet "take notes for me" (Gemini), Webex AI Assistant. Easiest: data stays in the tenant you already govern [1].
- Independent notetakers: Fireflies.ai, Otter.ai, Read.ai, Fathom, tl;dv, Granola, Notta. EU/German vendors: Jamie (DE, bot-free), Amberscript (NL, transcription with EU hosting), Speechmatics (UK, enterprise APIs), Sonix; also meeting features of Notion AI and Fellow.
- Choose A when: standard online meetings, no special secrecy duties; verify EU data residency, DPA, no training on customer data, retention controls and admin-wide disabling of bots.
Path B — Build on cloud APIs
- ASR APIs: Mistral Voxtral (EU provider), OpenAI transcription models (
gpt-4o-transcribe, Whisper), Azure AI Speech (EU regions, batch transcription with diarization), Google Cloud Speech-to-Text (Chirp), AWS Transcribe (Frankfurt), Deepgram, AssemblyAI, ElevenLabs Scribe, Gladia (FR). - Summarisation: any frontier LLM with structured outputs (Claude, Gemini, GPT, Mistral) via an EU region.
- Capture: Recall.ai or Microsoft Graph/Teams media APIs for bots; Zoom RTMS; local capture in a desktop client.
- Choose B when: a product feature is missing (custom minute templates, CRM write-back, own UI).
Path C — Self-host (open source)
- ASR: OpenAI Whisper large-v3 / large-v3-turbo via faster-whisper or whisper.cpp, WhisperX (alignment + diarization), NVIDIA Parakeet/Canary, Mistral Voxtral open weights [2][3].
- Diarization: pyannote.audio, NVIDIA NeMo Sortformer [7].
- Summaries: open-weight LLM (Mistral, Qwen, Llama, gpt-oss) behind vLLM or Ollama.
- Hardware: HW1 for Whisper-small/medium on a laptop; HW2 runs large-v3 for many parallel meetings plus a ≤30B summariser.
- Choose C when: meetings are confidential (board, legal, R&D, works council, public authority) and must not leave your infrastructure.
Path D — Custom training
Rarely needed. Fine-tune ASR for a dialect, noisy domain or specialist vocabulary (e.g. Swiss German, medical Latin) with tens to hundreds of hours of transcribed audio; otherwise use hot-word lists and glossaries. Fine-tuning the summariser is usually replaced by prompt templates.
State of the art (2026)
- Large multilingual ASR is a commodity. Whisper (open, MIT licence) set the baseline; successors such as Voxtral, Parakeet/Canary and commercial streaming models compete on accuracy, latency and price [2][3]. Leaderboards (e.g. Hugging Face Open ASR Leaderboard) are dominated by clean read speech, so test on your own meetings [4].
- Diarization is the weakest link: overlapping speech, similar voices, many participants and a single mixed channel cause speaker confusion; per-participant tracks from the platform fix most of it.
- LLM summaries are good but not faithful by default: omissions, wrongly attributed statements, invented action items and "smoothed" disagreement are known failure modes. Mitigate by linking each summary point to transcript timestamps, asking for quotes, and keeping the transcript available.
- Trend: meeting follow-ups become agents (tickets, CRM updates, e-mails; AGENT-012, CONV-015) and speech-native LLMs take audio directly.
Shared resources & combinations
- Same stack (asr + diarization + llm-general): SPEECH-002 contact-centre analytics, SPEECH-005 dictation, SPEECH-004 live captioning, SPEECH-021 verbatim records — one GPU ASR server serves all.
- Combine with: SPEECH-003 (translated minutes / live interpreting), LANG-008 (summarisation of long documents), AGENT-012 (scheduling), CONV-015 (personal assistant), CONV-007 (agent assist).
Effort, cost & KPIs
- Path A: hours to days; per-user subscription, often bundled in collaboration suites.
- Path B: 2–6 weeks; cost driven by audio minutes plus LLM tokens.
- Path C: 1–3 months incl. capture, security and ops; GPU server plus maintenance.
- KPIs: word error rate and diarization error rate on a sample of own meetings; share of summaries edited by organisers; action-item recall/precision against a human reference; time to publish minutes; action-item completion rate; opt-out/complaint rate.
People & change
Involve management (decision, AI policy) and the department lead first, then IT and IT security (accounts, bot access to calendars, storage) and the data protection officer before the pilot. Recording and transcribing speakers means personal data of employees and external guests: legal basis, information of participants, AVV with the vendor and a DPIA check are needed (GUIDE-005). External participants must be told at the start of the meeting; no recording without notice.
Works council: yes, if one exists. Transcripts and speaker labels are objectively suited to monitor behaviour and performance, so §87(1) no. 6 BetrVG applies; §90(1) no. 3 requires early information about AI-supported work processes, and an expert can be engaged under §80(3). A works agreement should exclude performance evaluation from transcripts and fix retention (GUIDE-004). Public sector: Personalrat.
AI literacy: support staff with a short training on limits (errors in names and numbers, hallucinated action items) and on what may be recorded (GUIDE-006, GUIDE-007). Start with a volunteer pilot team, name AI champions, and communicate openly that the tool is no surveillance instrument (GUIDE-008). Typical concerns: "I am being monitored" (no evaluation, short retention, opt-out for sensitive meetings) and "people will talk less freely" (recording only with announcement). Humans stay in control: the organiser checks and releases minutes and tasks before distribution.
Risks, regulation & compliance
- Criminal law (DE, §201 StGB): recording the non-public spoken word without consent is an offence; no secret bots.
- EU AI Act: generally minimal risk; AI literacy duty (Art. 4) applies. Transparency under Art. 50 mainly concerns synthetic content and interaction with AI, not transcription itself. Art. 5 prohibition: inferring emotions of employees or students from voice (e.g. "sentiment", "engagement" or "stress" scores of participants) is prohibited in workplace and education settings except for medical or safety reasons; switch such features off [5][8]. Using transcripts for HR evaluation, promotion or dismissal decisions can move the system into Annex III high-risk (employment).
- GDPR: voices and statements are personal data; special categories can surface (health, union membership). Need a legal basis (consent for external participants, legitimate interest with balancing test or works agreement for internal use), transparency notice, data processing agreement, EU hosting or valid transfer mechanism, data minimisation, retention limits and DPIA for systematic recording [6]. Participants must know before recording starts.
- Labour law: in Germany the works council has co-determination rights (§87 BetrVG) on technical systems capable of monitoring behaviour; agree a works agreement first.
- Confidentiality: privileged talks (lawyer, doctor, tax adviser), trade secrets and export-controlled content — prefer Path C or exclude from recording. Third-party bots can leak meeting content to external services; restrict bot admission at tenant level.
- Mitigations: consent banner and visible recording indicator, default retention (e.g. delete audio quickly, keep minutes), role-based access to transcripts, no-training contract clauses, human review of summaries, opt-out for sensitive meetings.
Getting started
- Decide the policy first: which meeting types may be recorded, consent text, retention, who may access; involve the DPO and works council.
- Pilot the platform-native tool you already license (Teams/Zoom/Meet) with one team for four weeks.
- Test 2–3 alternatives on 10 recorded real meetings (with permission), including accents and crosstalk; score transcripts and summaries against human minutes.
- Define a minute template (decisions, actions with owner/due date, open points) and a review step before distribution.
- For confidential use, prototype Path C: faster-whisper + pyannote + local LLM on one GPU.
Sources
- Microsoft — Teams Premium (intelligent recap, AI notes) — https://www.microsoft.com/en-us/microsoft-teams/premium
- OpenAI Whisper — https://github.com/openai/whisper
- Mistral AI — Voxtral speech understanding models — https://mistral.ai/news/voxtral
- Hugging Face — Open ASR Leaderboard — https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
- European Commission — Guidelines on prohibited AI practices (AI Act, Art. 5) — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
- EDPB — Guidelines, recommendations and best practices — https://www.edpb.europa.eu/our-work-tools/general-guidance/guidelines-recommendations-best-practices_en
- pyannote.audio — https://github.com/pyannote/pyannote-audio
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj