Real-time agent assist for contact centres

Human agents handling calls and chats must search knowledge bases, type notes and remember compliance wording while a customer waits. A real-time assistant transcribes the conversation as it happens, retrieves the right answer, proposes the next step and drafts the summary. It keeps the human in charge, which makes it the lowest-risk way to bring generative AI into service operations. Field evidence shows the largest gains for new and less experienced agents.

Problem & value

• Today: agents juggle CRM, ticketing, knowledge base and telephony; new hires need months to become productive; after-call work adds minutes per contact; QA reviews only a small sample.
• AI value: shorter handling and search time, more consistent and compliant answers, faster onboarding, near-zero manual wrap-up, and coaching while the contact is still live.
• Evidence: a study of 5,179 customer-support agents using a generative-AI assistant that monitors chats and suggests responses found about 14% more issues resolved per hour on average and about 34% for novice or low-skilled agents, with minimal effect on highly experienced ones; it also reported better customer sentiment and higher retention [1]. This is one company, chat only; treat it as an indicator, not a guarantee.
• Difference to CONV-001/002: here the AI talks only to the agent; a human sends every message.

Variants

• Voice vs. chat/email: voice needs low-latency streaming speech recognition; chat is simpler.
• Knowledge, guided and compliance assist: answer cards, next-best-action/form fill, mandatory-disclosure reminders.
• Wrap-up only: automatic summary and disposition code, often the quickest-payback feature.

How it works

1. telephony / chat platform (CCaaS)
2. real-time media stream (consent + recording notice)
3. streaming ASR + speaker separation (agent / customer channel)
4. intent & entity detection (small LLM / classifier)
5. retrieval over knowledge base + CRM context (hybrid search, reranking)
6. LLM composes suggestion with citations, next step, compliance checks
7. agent desktop widget (accept / edit / ignore / thumbs feedback)
8. after call: summary, disposition code, CRM/ticket update (agent confirms)
9. analytics, QA and feedback loop

• ASR: streaming models with partial results; two-channel audio avoids most diarization issues.
• Retrieval-augmented generation: answers must come from approved content and show their source so the agent can verify in seconds (see KNOW-002).
• Latency: suggestions must arrive within seconds; use small fast LLMs for triggers.
• Human-in-the-loop: the agent decides; no suggestion is sent to the customer automatically. Summaries are confirmed by the agent.

Data preparation

Assistance quality depends on the knowledge base (product manuals, FAQs, process descriptions, tariff and policy documents), past tickets and call records, and CRM or ERP context. These usually live in a DMS, wikis, file shares, ticket systems and in experienced agents' heads. Typical gaps: outdated or conflicting articles, scanned manuals, duplicates, no ownership of articles, call recordings without consent documentation or transcripts, and unlabelled outcomes.

Preparation steps: (1) select the 50-100 most frequent contact reasons from ticket data; (2) write or clean one authoritative article per reason, with named owner and review date; (3) OCR and convert legacy manuals; (4) tag articles by product, process and audience; (5) define compliance phrases and wrap-up templates; (6) check the telephony set-up for transcription access and recording consent; (7) collect good example replies from top agents. Tools: ticket-system export, Paperless-ngx or OCR tools for scans, vendor knowledge connectors. Rough share of effort: 40 percent, plus a continuous content cycle fed by agent feedback.

Useful PREP use cases: PREP-013 (authoritative knowledge base), PREP-016 (gold-standard examples), PREP-017 (content lifecycle), PREP-021 (capturing agent corrections), PREP-006 (scanning). See also GUIDE-003.

Implementation paths

Which path for whom

• SMB — A: assist features in established cloud contact-centre or helpdesk suites cover standard needs without own AI operations.
• Enterprise — B: high call volume makes seat prices high, and the assistant must integrate with own CRM, telephony and knowledge systems; C when recordings must not leave the company.
• Developer — B: streaming speech-to-text plus LLM APIs and RAG allow tailored agent-assist at controllable cost.

Path A — Buy (recommended for most SMBs)

• Inside the contact-centre platform: Amazon Connect with its AI agents (the functionality formerly called Amazon Q in Connect; real-time recommendations, intent detection, agentic assistance) [2]; Google Customer Engagement Suite / Agent Assist with Gemini [3]; Genesys Cloud (announced availability on the AWS European Sovereign Cloud) [4]; NICE CXone and Cognigy Agent Copilot (Cognigy is Düsseldorf-founded, now part of NICE; EU-hosted speech options documented) [5]; Salesforce Service Cloud, Microsoft Dynamics 365 Contact Center, Five9, Talkdesk, Zendesk, Sprinklr.
• Add-on specialists: Observe.AI, Cresta, Uniphore, Balto; EU-based: Parloa (Berlin), Dixa and Trengo (chat-centric); verify each vendor's current agent-assist scope.
• Check: EU data residency, no vendor training on your audio, DPA/sub-processors, languages and dialects, and whether emotion features can be switched off per tenant (see Risks).
• Choose A when: you already run a major CCaaS platform and have a maintained knowledge base.

Path B — Build on cloud APIs

• Streaming ASR: Azure AI Speech, Google Cloud Speech-to-Text, Amazon Transcribe streaming (EU regions), Deepgram (EU endpoint), AssemblyAI streaming, OpenAI realtime transcription.
• LLMs: Mistral (EU provider), Claude, Gemini, GPT via EU regions; a small fast model for intent triggers and a stronger model for suggestions and summaries.
• Integration: CCaaS media-fork APIs (e.g. Genesys AudioHook, Twilio Media Streams, Amazon Connect Kinesis Video Streams), websocket to an agent-desktop widget (CTI / CRM side panel).
• Choose B when: you need custom logic, an unusual CRM, or several channels under one assistant.

Path C — Self-host (open source)

• ASR: Whisper-family (faster-whisper, whisper.cpp; low latency requires chunking tricks), NVIDIA Parakeet/Canary via NeMo or Riva, Kyutai streaming STT.
• LLM: open-weight models in the ~8–70B range (Mistral, Qwen, Llama, Gemma families) on vLLM; embeddings and reranker from open models (e.g. BGE, multilingual-e5); pgvector/Qdrant/OpenSearch.
• Pipelines: Pipecat or LiveKit Agents; Jambonz or FreeSWITCH for SIP.
• Hardware: HW3 for dozens of concurrent calls with ASR and a mid-size LLM; scaling is driven by concurrent streams, not total volume. Choose for strict sovereignty (public sector, banks).

Path D — Custom training

Fine-tune ASR for dialects or domain vocabulary, and a small classifier for intents and dispositions, using consented, transcribed audio. Rarely needed before prompt/RAG improvements are exhausted.

State of the art (2026)

• Agentic assist: vendors moved from "show a knowledge article" to assistants that look up orders, draft replies, fill forms and prepare actions that the agent approves [2].
• Streaming ASR is accurate for major EU languages on clean audio; accuracy drops for dialects, code-switching, telephone-band noise and spelled-out identifiers.
• Limits and failure modes: hallucinated policy or price statements; stale knowledge articles; suggestions arriving too late; over-reliance by novices (automation bias); alert fatigue; wrong summaries entering the CRM unchecked; prompt injection through customer chat text.

Shared resources & combinations

• Same stack: asr + diarization + llm-general serves SPEECH-001 meeting notes, SPEECH-002 post-call analytics and QA, and CONV-002 voicebots; one streaming-ASR cluster can feed all.
• Knowledge layer: KNOW-002/KNOW-003 provide the retrieval backend; CONV-001 and CONV-004 reuse the same content so bot and human give identical answers.
• Combinations: SPEECH-002 (QA on all calls), PRED-011 (staffing), LANG-012 (routing), LANG-013 (feedback), bot escalation (CONV-001/002) with handover summary.

Effort, cost & KPIs

• Path A: weeks to months (knowledge clean-up is the long pole); priced per minute, interaction or seat.
• Path B: 3–6 months for voice incl. CTI integration; ASR and LLM cost per minute is a variable cost to model against agent cost.
• Path C: 4–8 months; GPU servers plus an MLOps/telephony team.
• KPIs: average handling time, after-call work time, first-contact resolution, new-hire time to proficiency, QA/compliance scores, CSAT, suggestion acceptance rate, agent attrition, and hallucination/incorrect-suggestion rate from sampled reviews.

People & change

This use case touches employees directly, so start with the works council and agents, not at the end. Management and the contact-centre lead set the goal (quality and relief, not surveillance). Involve the works council before the pilot: live transcription, suggestion acceptance rates and quality scoring are suited to monitor behaviour and performance, so co-determination under §87(1) no. 6 BetrVG applies (the BAG requires only objective suitability); also §90(1) no. 3, and an expert under §80(3) is often considered necessary. A works agreement should rule out individual performance evaluation from assist data. In the public sector the staff council applies.

DPO and IT security are needed early: call content, recording consent, AVV with the provider and a DPIA are likely, since calls are processed systematically. HR handles training and role changes.

AI literacy: support agents with hands-on training on how suggestions arise, when to distrust them and how to report wrong ones (AI Act Art. 4). Name AI champions among the agents, and pilot with a volunteer team for 6-8 weeks. Typical concerns: surveillance, deskilling and job loss; address with transparent data use, no hidden scoring, agents decide and can ignore any suggestion, and a feedback button that visibly improves content.

Guides: GUIDE-004 (works council), GUIDE-006 (AI literacy), GUIDE-008 (change management).

Risks, regulation & compliance

• EU AI Act: the basic assistant (knowledge, drafts, summaries) is typically minimal risk or limited risk; the agent is the human interlocutor, so Art. 50 chatbot disclosure to the customer generally does not apply to pure agent-facing assist, but check if any text is sent unreviewed. Art. 4 AI literacy applies.
• Art. 5(1)(f) prohibition: inferring the emotions of employees in the workplace from biometric data (voice, face) is prohibited since 2 Feb 2025, with narrow medical/safety exceptions. The Commission's guidelines state that call-centre voice analysis of customers' emotions is not covered if agents' emotions are not tracked at the same time; text-based sentiment is outside the biometric definition [6][7]. Do not enable agent "stress/emotion" scores, wellbeing mood tracking or tone-based agent ranking. Sentiment dashboards must be configured on the customer channel only; legal review is advised because vendor features often mix both.
• High-risk boundary: if outputs feed decisions on agents' evaluation, task allocation or dismissal, Annex III (employment) may apply. Keep assist separate from automated performance scoring.
• GDPR: call recording and transcription are personal-data processing; need a legal basis, transparent notice at call start, retention limits, role-based access, DPA, and no vendor training on your data. Special-category data (health, finance) can appear in calls; redact PII/payment card data (PCI DSS) before storage or LLM calls. Under Germany's BDSG and works-council law (Betriebsverfassungsgesetz §87), employee monitoring needs a works agreement; other member states have comparable co-determination rules. Telephone-recording consent rules (e.g. §201 StGB) apply.
• Mitigations: show sources, agent confirmation, redaction, injection filters, kill switch, audit logs, sampled review, works agreement before launch.

Getting started

1. Measure the baseline (handling time, after-call work, FCR, onboarding time) and pick one queue.
2. Clean the top 100 knowledge articles; assign owners.
3. Start with wrap-up summaries and knowledge suggestions; no emotion or agent-scoring features.
4. Agree the rollout with works council and DPO; update recording notices.
5. Pilot with 20–50 agents for 8 weeks against a control group.

Sources

1. Brynjolfsson, Li, Raymond — Generative AI at Work, NBER Working Paper 31161 — https://www.nber.org/papers/w31161
2. AWS — Amazon Connect agentic assistance / Amazon Q in Connect docs — https://docs.aws.amazon.com/connect/latest/adminguide/agentic-assistance.html
3. Google Cloud — Customer Engagement Suite with Google AI — https://cloud.google.com/blog/products/ai-machine-learning/introducing-customer-engagement-suite-with-google-ai
4. Genesys — Experience orchestration on the AWS European Sovereign Cloud — https://www.genesys.com/company/newsroom/announcements/genesys-to-offer-experience-orchestration-services-on-aws-european-sovereign-cloud
5. Cognigy — Cognigy.AI 2025.17 release (EU-hosted speech) — https://www.cognigy.com/product-updates/2025.17
6. European Commission — Guidelines on prohibited AI practices (AI Act) — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
7. Bundesnetzagentur — Prohibited practices under the AI Act — https://www.bundesnetzagentur.de/EN/Areas/Digitalisation/AI/08_ProhibitedPractices/start.html
8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj