Contact centres record thousands of calls but manually review only a tiny sample. Speech recognition plus LLM analysis turns every call into a transcript, summary, topic and scorecard, flags compliance problems and shows why customers call. Typical vendor-reported benefit: near-100 % coverage instead of a few percent, faster coaching and earlier detection of product or process problems. In the EU the hard part is not the technology but lawful design: employee monitoring, call-recording consent and the AI Act ban on emotion recognition in the workplace.
Problem & value
- Today: QA teams listen to a small random sample (often cited as only a few percent of calls; vendor-reported) and fill in scorecards by hand. Compliance breaches, churn signals and recurring defects are found late. Wrap-up notes written by agents are inconsistent and cost handling time.
- AI value: every call is transcribed, summarised and scored against the same criteria; topic and intent mining shows contact drivers; compliance phrases (identification, recording notice, cooling-off or risk disclosures) are checked automatically; supervisors get targeted calls to review and coach.
- Evidence level: vendor case studies report large QA-coverage gains and shorter after-call work, but independent, comparable figures are scarce. Measure your own baseline before relying on any number.
- Boundary: this file covers post-call (batch/near-real-time) analytics. Live guidance during the call is CONV-007; the voicebot itself is CONV-002; emergency lines are SPEECH-016.
Variants
- QA automation: AI pre-fills the scorecard, humans calibrate and handle disputes.
- Compliance monitoring: regulated sectors (banking, insurance, energy sales, debt collection) check mandatory statements, consent capture and mis-selling risk.
- Voice-of-customer / contact-driver analytics: topic clustering, root-cause and trend dashboards for product and operations teams (not individual agents).
- Sales / revenue intelligence (conversation intelligence): same ASR, diarisation and LLM stack, but outputs are deal signals, objection and competitor mentions, methodology scoring (e.g. MEDDICC) and CRM sync, bought by sales rather than service. See PRED-002 (pipeline prediction), CONV-005 (sales assistant) and EDU-010 (role-play coaching).
- Back-office and field calls: helpdesk, public-authority hotlines, collections, clinic or pharmacy phone lines.
- Agent self-coaching: agents see their own metrics and snippets (lowest works-council friction).
How it works
- call recording (CCaaS / SBC / SIPREC) + metadata
- ASR with diarisation (agent vs customer channel; telephony-tuned, multilingual)
- PII detection & redaction (PCI card numbers, IBAN, names, health data) in text and audio
- LLM / classifier analysis: summary, intent, topic, outcome, sentiment of the *conversation topic*, scorecard questions with evidence quotes, compliance rule checks
- aggregation (topic clusters, trend, outliers) in BI dashboard
- human review queue (calibration, disputed or low-confidence scores)
- coaching workflow
- ASR: stereo recordings give free speaker separation; mono needs diarisation. Telephone audio (8 kHz, accents, overlap) is the main accuracy driver, so test on your own calls.
- LLM scoring: one question per criterion, answer yes/no/NA plus quoted evidence; deterministic rules (regex/keywords) for mandatory phrases; LLM only for fuzzy criteria like empathy or clarity of explanation.
- Human-in-the-loop: supervisors calibrate the AI scorecard against human scores each month; no disciplinary or pay decision may rest on an AI score alone.
- Out of scope by design: inferring emotions of employees from voice is prohibited in the EU (see Risks). Analyse what was said, not the speaker's inner state.
Data preparation
The data are call recordings (often stereo, from the telephony or contact-centre platform), metadata from CRM and ticketing (customer, topic, outcome, agent) and the existing QA scorecards and policies. In an SMB recordings live on the PBX or cloud telephony, outcomes in the CRM or helpdesk, and the quality criteria in Word or Excel files and in the heads of team leads. Typical gaps: mono recordings that make speaker separation harder, missing call-to-ticket links, no consistent outcome labels, dialects and noise, product terms misrecognised, and unclear consent and retention for recordings.
Preparation steps: export recordings via API or telephony connector (PREP-005); join them with CRM and ticket metadata; clarify the legal basis and announcement, and delete or mask payment data and other sensitive passages (PREP-022, PREP-025); turn QA criteria into a precise, written scorecard (PREP-019); let experienced team leads rate 50 to 100 calls as a test set to compare AI scores against human ratings (PREP-018, PREP-020); build a product and term glossary (PREP-015). Restrict access to recordings by role (PREP-023).
Effort share: medium, about 25 to 35 percent, mainly for scorecard definition, test set and consent. See also GUIDE-003.
Implementation paths
Which path for whom
- enterprise: Path B. Call analytics is core to service quality; own pipelines on cloud APIs allow custom scorecards, integration with CRM and ticketing, and control over costs at high call volumes.
- smb: Path A. Small service teams are best served by mature analytics add-ons of cloud telephony or contact-centre platforms with EU hosting; custom builds do not pay off.
- developer: Path B. Speech and LLM APIs with batch processing are the quickest route to build analytics into a product.
Path A - Buy (recommended)
- Built into CCaaS: Genesys Cloud (Speech and Text Analytics), NICE CXone / Enlighten, Five9, Talkdesk, Amazon Connect Contact Lens, Google CCAI Insights, Microsoft Dynamics 365 Contact Center, Twilio Voice Intelligence.
- Dedicated QA / conversation analytics: Observe.AI, CallMiner, Verint, Calabrio (Teleopti), Level AI, Balto; sales: Gong, Chorus (ZoomInfo), Salesloft, Avoma. EU-based options to evaluate: Gladia and Mistral (speech models), Sipgate and Placetel (telephony); verify each vendor's actual EU hosting and subprocessor list.
- Choose A when: you already run a CCaaS platform, volume is below ~hundreds of agents, and QA process is standard. Check EU data residency, retention controls, redaction of audio, works-council configurability (anonymised agent views), and no use of emotion inference.
Path B - Build on cloud APIs
- ASR with diarisation: Azure AI Speech batch transcription (EU regions), Google Cloud Speech-to-Text v2 (EU), Amazon Transcribe Call Analytics (Frankfurt/Ireland, includes PII redaction), Deepgram, AssemblyAI, Speechmatics (UK/EU hosting options), Gladia (France), Mistral Voxtral (EU provider), OpenAI transcription models.
- Analysis: frontier LLMs with structured output (Claude, Gemini, GPT, Mistral) via EU-region endpoints; Azure OpenAI or Amazon Bedrock in EU regions for data-residency commitments.
- Orchestration: event-driven pipeline (object storage → queue → ASR → redaction → LLM → warehouse); BI in Power BI / Metabase / Superset.
- Choose B when: you need custom scorecards, multiple languages or a data warehouse integration the product does not give.
Path C - Self-host (open source)
- ASR: OpenAI Whisper large-v3 / turbo via faster-whisper or WhisperX; NVIDIA Parakeet / Canary (NeMo); Mistral Voxtral open weights; Kyutai and Moonshine for streaming. Diarisation: pyannote.audio or NVIDIA Sortformer.
- PII: Microsoft Presidio, GLiNER-style NER models; audio redaction by timestamp muting.
- LLM: open-weight models in the 30-120B class (Mistral, Qwen, Llama, gpt-oss) served with vLLM; embeddings (bge-m3, multilingual-e5) with BERTopic for topic discovery.
- Hardware: HW2 handles batch ASR for a mid-size centre; HW3 recommended for LLM scoring of all calls in-house. Batch processing overnight keeps cost low.
- Choose C when: recordings must not leave your infrastructure (banks, public sector, health).
Path D - Custom training
Fine-tune ASR on domain vocabulary (product names, drug names, dialects) with a few dozen hours of corrected transcripts, or train a small classifier for your own intent taxonomy with hundreds of labelled calls. Corrected transcripts from QA reviewers are the natural training data. Usually worth it only for low-resource languages or high volumes.
State of the art (2026)
- ASR: open and commercial models reach low word-error rates on clean read speech, but 8 kHz call audio, crosstalk, accents, code-switching and domain terms still produce clearly higher error rates; always benchmark on your own calls. Speech-aware LLMs (audio-in models such as Voxtral and Gemini) can summarise or answer questions from audio directly, though a transcript-based pipeline remains easier to audit.
- LLM scoring agrees well with humans on factual checklist items (was the greeting said, was the disclosure read) and less well on subjective criteria; calibration against human reviewers is mandatory. Keep evidence quotes so reviewers verify quickly.
- Shift from sampling to full coverage and from agent-ranking to contact-driver analytics and automatic summaries that replace agent wrap-up notes.
- Limits and failure modes: hallucinated evidence, misattributed speakers on mono audio, missed negation, bias against accents or dialects (lower ASR accuracy skews scores), over-trust in dashboards, and "metric gaming" by agents.
Shared resources & combinations
- Same stack (asr + diarization + llm-general): SPEECH-001 meeting transcription, SPEECH-016 emergency-call analysis, SPEECH-021 verbatim records; one ASR server serves all.
- Combines with: CONV-007 (live agent assist uses streaming ASR on the same transcripts), CONV-002 (QA of voicebot calls), PRED-002 and CONV-005 (sales intelligence), EDU-010 (turn best/worst calls into role-play training), DOC-018 (redaction patterns), SEC-018 (records of processing, retention and DSAR handling for recordings).
- Do not combine with SPEECH-019-style voice conversion without transparency review; do not feed outputs into automated HR decisions.
Effort, cost & KPIs
- Path A: weeks; priced per agent seat or per recorded hour; activation inside an existing CCaaS is fastest.
- Path B: 2-4 months incl. scorecard calibration and dashboards; ASR is billed per audio minute, LLM per token, usually small against QA labour cost.
- Path C: 3-6 months; GPU server plus operations; best cost at high volumes.
- KPIs: QA coverage %, AI-vs-human scorecard agreement (e.g. Cohen's kappa), ASR word-error rate on own calls, after-call-work time, first-contact resolution, compliance-breach detection rate, repeat-contact rate, CSAT, reviewer time per call.
People & change
Involve management and the service department lead from the start, together with agents or their representatives, IT (telephony integration), the data protection officer and HR. Call analytics evaluates employees systematically, so a DPIA is likely (Art. 35 GDPR), an AVV is needed with every processor, and callers must be informed (GUIDE-005).
Works council: yes. Call analytics logs and scores individual agents and is objectively suited to monitor performance, so §87(1) no. 6 BetrVG applies; inform early under §90(1) no. 3, engage an expert if needed (§80(3)). Agree a works agreement covering purposes, no sole automated personnel decisions, aggregation, access and deletion periods (GUIDE-004). Public sector: Personalrat.
AI literacy: train team leads in reading scores critically, and agents in how the tool works (GUIDE-006). Communicate that the goal is coaching and relief from random sampling, not control (GUIDE-008). Start with one team as pilot, share scores first only with the agent concerned, and involve agents in defining the scorecard (GUIDE-009). Typical concerns: "100 percent of my calls are judged" (use for coaching, transparent criteria) and wrong scoring (agents can contest). Humans stay in control: scores are input for coaching conversations, never an automatic basis for sanctions.
Risks, regulation & compliance
- EU AI Act: classification depends on use. Pure customer-experience and topic analytics: minimal or limited risk. Art. 5(1)(f) prohibits AI systems that infer emotions of natural persons in the workplace (exceptions only for medical or safety reasons); the Commission's February 2025 guidelines treat call-centre voice emotion monitoring of agents as prohibited, whereas detecting emotion of customers is not caught by that clause but needs care. Systems used to monitor and evaluate worker performance or behaviour, or allocate tasks, are high-risk under Annex III(4) (obligations from 2 December 2027 under the Digital Omnibus on AI, Reg. (EU) 2026/1744). Transparency duty (Art. 50) applies if AI interacts with callers. AI literacy (Art. 4) applies to staff.
- GDPR: call recordings are personal data, sometimes special-category (health, financial). Need a legal basis and purpose limitation (Art. 6), clear caller notice (recording announcement), data minimisation, short retention, DPIA (Art. 35) for systematic monitoring, processor agreements, and no third-country transfer without safeguards. Agent data is employee data: in Germany §26 BDSG and co-determination under §87(1) No. 6 BetrVG (works council) apply; CJEU case law constrains employee-data rules.
- Telecom secrecy and recording: in Germany recording without consent can be criminal (§201 StGB); consent or another valid basis and notification are needed.
- Sector rules: MiFID II call recording in investment firms, PCI DSS (never store card numbers or security codes; pause or redact), consumer-protection and debt-collection rules.
- Mitigations: redact before analysis, role-based access, aggregate or pseudonymised agent views, no emotion/stress inference on staff, human review of every consequential decision, appeal route for agents, bias testing across accents, logging of model and prompt versions.
Getting started
- Clarify legal basis, recording notice, retention and involve the data-protection officer and works council before any pilot; run the DPIA.
- Export 200-500 representative calls; benchmark 2-3 ASR options on word-error rate and speaker separation.
- Convert the QA form into 10-20 concrete, evidence-based questions; split rule-checkable from LLM-judged ones.
- Run AI scoring in shadow mode next to human QA for 4-6 weeks; measure agreement, adjust prompts.
- Roll out topic dashboards first (low employee risk), then coaching use with agreed rules.
Sources
- Regulation (EU) 2024/1689 (AI Act), Art. 5, Art. 50, Annex III - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- European Commission - Guidelines on prohibited AI practices (Feb 2025) - https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
- GDPR, Regulation (EU) 2016/679 - https://eur-lex.europa.eu/eli/reg/2016/679/oj
- Amazon Transcribe Call Analytics - https://docs.aws.amazon.com/transcribe/latest/dg/call-analytics.html
- Azure AI Speech - batch transcription and diarization - https://learn.microsoft.com/azure/ai-services/speech-service/batch-transcription
- OpenAI Whisper - https://github.com/openai/whisper
- pyannote.audio - https://github.com/pyannote/pyannote-audio
- Microsoft Presidio - https://github.com/microsoft/presidio
- Genesys Cloud speech and text analytics - https://help.mypurecloud.com/articles/about-speech-and-text-analytics/