AI phone agent / voicebot for inbound calls

Phones remain the channel for urgent, complex or older-customer requests, yet many small businesses miss calls and contact centres queue them. A voice agent picks up every call at once, understands natural speech instead of "press 1", answers from approved knowledge, books, looks up or logs the request in the business systems and transfers to a person when it cannot help. Typical goals: no missed calls, shorter queues, 24/7 availability and humans freed for hard cases.

Problem & value

• Today: calls go to voicemail, a receptionist who is also doing other work, or a classic IVR menu that callers abandon. Contact centres staff for peaks, with high turnover and training cost.
• AI value: answer 100 % of calls without queueing, cover nights and peaks, collect structured data (name, reason, order number) instead of free-form voicemail, and resolve the repetitive share (opening hours, status, appointment, simple changes) end to end.
• Evidence: vendor case studies report high containment rates, but these are self-reported and depend heavily on the call mix; measure on your own calls before committing. Latency is the key quality factor: natural turn-taking needs a response start well under about a second [1][2].

Variants

• SMB reception/overflow: trades, medical and dental practices, law firms, hotels, restaurants: answer, take message, book (see CONV-003), forward urgent calls.
• Contact-centre front door: authenticate, identify intent, resolve or route with a summary; the human then continues with context (see CONV-007 for assistance during live calls).

How it works

1. call (PSTN/SIP trunk)
2. telephony gateway
3. AI disclosure + consent notice
4. streaming ASR (+ voice activity / turn detection, barge-in)
5. dialogue manager: LLM with tools (RAG on knowledge base, CRM/calendar/ticket APIs)
6. guardrails (scope, PII, prompt-injection, no unauthorised commitments)
7. streaming TTS
8. caller
9. escalation: warm transfer (SIP REFER) to human + transcript summary
10. post-call: transcript, summary, CRM entry, analytics (SPEECH-002)

• Two architectures: (1) cascaded ASR → LLM → TTS, which allows per-component choice, EU hosting, custom vocabulary and full logging; (2) speech-to-speech realtime models that take audio in and produce audio out with lower latency and more natural prosody but less control and transparency.
• Tool use: the LLM calls whitelisted functions (check booking, create ticket, send SMS link); authorisation and business rules live in code, not in the prompt.
• Human-in-the-loop: defined handover triggers (caller asks, repeated misunderstanding, complaint, sensitive or legal topic, low confidence, negative sentiment); a human reviews flagged transcripts and updates the knowledge base weekly.

Data preparation

A voicebot needs less bulk data than a chatbot but precise, current facts: opening hours, price lists, service areas, appointment rules, escalation numbers and the answers to the 20 most frequent call reasons. These typically live in the website, price lists in Excel or the ERP, calendars, and in the heads of the reception staff.

Typical gaps: undocumented call reasons, informal rules ("Mr Meier always gets a callback"), outdated hours, product names that speech recognition mishears, and no recordings or labels to test with.

Preparation steps: (1) log call reasons for two weeks (tally sheet or call-log export) to find the top intents, (2) write a short, authoritative fact sheet per intent, (3) collect product, street and person names plus abbreviations as a vocabulary list for speech recognition, (4) define clear escalation rules and callback data fields, (5) if recordings are used for testing, obtain consent and pseudonymise transcripts, (6) build 30 to 50 test calls with expected outcomes. Tools: Whisper for transcription of existing recordings, a spreadsheet or n8n for the fact sheet and call-flow.

Share of effort: roughly 25 to 40 percent. See PREP-015, PREP-016, PREP-018, PREP-022, PREP-025 and DATA-008.

Implementation paths

Which path for whom

• SMB: Path A. Ready-made voicebot and phone-assistant products with EU hosting exist, and the typical call flows (opening hours, appointments, callback) are standard.
• Enterprise: Path B. Telephony integration, volume pricing and control over latency and call flows favour building on cloud speech and LLM APIs.
• Developer: Path B. Cloud realtime speech APIs with a telephony provider are the quickest route for customer projects.
• Public sector: Path C. Call recordings and citizen data favour open speech models (Whisper-class) and open-weight LLMs on sovereign cloud.

Path A — Buy (recommended for most SMBs)

• Voice-agent platforms: ElevenLabs Agents (EU data residency on enterprise tier), Retell AI, Vapi, Bland, PolyAI (UK enterprise); EU/German-origin: Parloa (Berlin, enterprise contact centres), Famulor and other German AI-receptionist services, Deepgram/Speechmatics (UK) as speech components. Verify residency claims per vendor [4][5].
• CCaaS suites with built-in voice AI: Genesys Cloud, NICE CXone, Amazon Connect, Google CES, Microsoft Dynamics 365 Contact Center, Cognigy (Düsseldorf, now part of NICE), Talkdesk.
• Check: EU hosting of audio and sub-processors, DPA, EU languages, PBX handover.
• Choose A when: standard calls, few integrations, no in-house voice engineering.

Path B — Build on cloud APIs

• Realtime models: OpenAI Realtime API (gpt-realtime family) with native SIP support, also via Azure OpenAI/Microsoft Foundry for EU regions; Google Gemini Live API; Mistral (EU provider) for Voxtral speech-to-text and LLMs; Deepgram, AssemblyAI, Azure Speech, Google Chirp for ASR; ElevenLabs, Cartesia, Azure neural voices for TTS [1][6][7].
• Orchestration frameworks: LiveKit Agents, Pipecat, Twilio ConversationRelay, OpenAI Agents SDK; tool calls into CRM/calendar via MCP or REST; n8n for back-office steps.
• Choose B when: you need custom flows, deep integration or a specific EU-region setup, and have developers. Test accents, noise, phone codecs (8 kHz G.711) and barge-in early.

Path C — Self-host (open source)

• Stack: LiveKit (Apache 2.0) or Pipecat for media and agent loop; Asterisk/FreeSWITCH for telephony; ASR: Whisper-family (faster-whisper), NVIDIA Parakeet/Canary, Voxtral Realtime (Apache 2.0, streaming, 13 languages); LLM: Mistral, Qwen, Llama, gpt-oss class via vLLM; TTS: Piper, Kokoro, Coqui-successors, open models with licence checks (voice cloning licences vary) [6][8][9].
• Hardware: HW3 for an ASR + 20–70B LLM + TTS stack with several concurrent calls; HW2 works for few lines with a smaller LLM, at the price of quality in German and tool-calling reliability.
• Effort: needs voice engineering; choose for public bodies, health or strict sovereignty.

Path D — Custom training

Rarely required. Worth considering for domain vocabulary ASR fine-tuning (drug names, street names, dialects) using a few dozen hours of transcribed, consented calls, or a distilled small model for intent routing. Never train on recordings without a legal basis and purpose compatibility check.

State of the art (2026)

• Realtime speech-to-speech models with SIP, interruption handling and tool calling are generally available from the large labs; streaming open-weight ASR (Voxtral Realtime, sub-200 ms configurable delay) makes self-hosted cascaded pipelines viable [1][8].
• Agentic tool use lets voicebots complete transactions rather than only answer FAQs; reliability of multi-step tool calls over voice is lower than in text chat.
• Limits and failure modes: hallucinated policies or prices, mis-heard numbers/names/addresses (always read back and confirm), accents and background noise, code-switching, prompt injection by spoken instructions, latency spikes, silent failures on telephony edge cases (voicemail, DTMF, hold music), and voice-cloning abuse against voice-based authentication. Voice biometrics alone must not be used to authenticate callers.

Shared resources & combinations

• Same stack: CONV-008 outbound voice, CONV-025 voice ordering, CONV-014 in-car voice, CONV-019 voice interviews share ASR/TTS/realtime serving and telephony.
• Downstream: SPEECH-002 call QA, CONV-007 agent assist, CONV-003 booking, AGENT-002 follow-up automation.
• Text twin: CONV-001 web chatbot should share the same knowledge base and tool layer (KNOW family retrieval); one backend, two channels.

Effort, cost & KPIs

• Path A: days to 2–4 weeks; pricing typically per minute plus platform fee; compare total cost per resolved call rather than per minute.
• Path B: 1–3 months for a production agent with integrations and testing; costs: realtime model audio tokens or ASR+LLM+TTS per minute, telephony per minute.
• Path C: 3–6 months; GPU server plus operations.
• KPIs: containment/resolution rate (verified, not only "call ended"), transfer rate and reason, average handling time, abandoned calls, first-call resolution, CSAT, escalation quality, word error rate, response latency (p95), cost per resolved call, complaints about the bot.

People & change

Management decides scope and which calls the bot may take. The reception or service lead owns the call flows; the staff who answer the phone today are the key end users and should define escalations and test with real calls. IT integrates telephony, calendar and CRM; IT security reviews the voice channel. The DPO is essential: voice is personal data, recordings need a legal basis and notice, AVV with the provider, and a DPIA is likely at scale.

Works council: depends, but often yes in practice. Call analytics, transcripts and quality scoring log who handles which call and can monitor behaviour or performance (§87(1) no. 6 BetrVG; BAG: objective suitability is enough). Also inform it under §90(1) no. 3 and consider an expert under §80(3). Public sector: Personalrat.

Callers must be told at the start that they speak with an AI (Art. 50 AI Act) and be able to reach a human at any time. Support AI literacy of the phone team (Art. 4). Start with a pilot on after-hours calls, where the bot replaces voicemail, not people; this reduces fear and shows the benefit. Typical concerns are bad recognition of dialects and annoyed callers; address them with testing, a fast handover and weekly review of failed calls. See GUIDE-008, GUIDE-004, GUIDE-005.

Risks, regulation & compliance

• EU AI Act: limited risk. Art. 50(1) requires that people are informed they interact with an AI system unless obvious; the Commission's July 2026 guidelines treat customer-service and booking agents as in scope, including naming on whose behalf the agent acts. Art. 4 AI literacy applies to staff. Generated voices count as synthetic audio (Art. 50(2) marking, with grace periods per the omnibus). Becomes high-risk if the bot decides access to essential services or credit (Annex III), and prohibited if it infers emotions of employees (Art. 5(1)(f)); do not add emotion recognition for call-centre staff, and keep caller sentiment analysis out of decisions on individuals [3][10].
• GDPR: voice and transcripts are personal data; art. 6 basis, transparency notice at call start, DPA with processors, third-country transfers (US speech vendors need DPF or SCCs and a transfer assessment), retention limits, deletion on request. Voice-print biometrics are special-category data (Art. 9); avoid. Art. 22: no solely automated decisions with legal or similar effect.
• Telecom/recording: call recording needs consent or a clear legal basis (German §201 StGB, TKG/TTDSG rules; other member states differ); announce recording and offer to continue unrecorded.
• Mitigations: disclosure, narrow scope, read-back confirmation, human handover on request, logging, red-team calls.

Getting started

1. Export 2–4 weeks of call logs: top 10 reasons, durations, missed calls, peak hours; pick 3–5 intents that are frequent, simple and low-risk.
2. Write the call design: AI disclosure sentence, recording notice, handover rules, forbidden topics.
3. Pilot 2 platforms on overflow or after-hours lines with your real phone audio and dialects.
4. Connect one system of record (calendar or ticketing) and require read-back confirmation.
5. Review 50 transcripts weekly, update knowledge base, then extend intents and opening hours.

Sources

1. OpenAI — Realtime API with SIP (telephony guide) — https://developers.openai.com/api/docs/guides/realtime-sip
2. OpenAI — Introducing gpt-realtime and Realtime API updates for production voice agents — https://openai.com/index/introducing-gpt-realtime/
3. European Commission — Guidelines on transparency obligations for providers and deployers of certain AI systems (Art. 50) — https://digital-strategy.ec.europa.eu/en/policies/guidelines-ai-transparency-obligations
4. ElevenLabs — Agents platform — https://elevenlabs.io/api
5. Parloa — product release notes 2026 (EU voice processing) — https://www.parloa.com/blog/parloa_product_release_2026/
6. Microsoft — GPT Realtime API via SIP, Microsoft Foundry — https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/realtime-audio-sip
7. Twilio/LiveKit/Pipecat vendor comparison (secondary) — https://30elevate.com/en/blog/ai-voice-agent-platforms/
8. Mistral — Voxtral Transcribe 2 and Voxtral Realtime — https://mistral.ai/news/voxtral-transcribe-2/
9. Voxtral Realtime paper — https://arxiv.org/html/2602.11298v1
10. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj