Customers expect instant answers at any hour, while service teams drown in repetitive questions (where is my order, how do I return, what does my tariff include). Rule-based chatbots failed because they only matched fixed intents. LLM-based assistants retrieve answers from the company's own content, speak the customer's language, call back-office systems for simple actions and hand over cleanly to a human when they reach their limits. Done well, they absorb a large share of routine contacts; done badly, they hallucinate promises the company is legally bound to keep.
Problem & value
- Today: FAQ pages nobody reads, ticket queues, email with long response times, and legacy intent bots with dead-end menus. Agents spend much of their day on a few dozen recurring questions.
- AI value: 24/7 first response in many languages, deflection of routine contacts, shorter waiting times, consistent answers, and structured insight into what customers actually ask (knowledge gaps feed back into the help centre).
- Reported figures (vendor-reported, not independent): Klarna stated in 2024 that its assistant handled two-thirds of customer chats in its first month and cut resolution time from about 11 to 2 minutes [3]. In May 2025 Klarna's CEO said it was hiring human agents again so customers can always reach a person, after the AI push had cost service quality [9]. Treat such numbers as upper-end claims and plan for a hybrid model with humans.
- Liability is real: in Moffatt v. Air Canada (2024) a Canadian tribunal held the airline responsible for wrong refund information given by its website chatbot and rejected the argument that the bot was a separate entity [4]. Courts and regulators treat chatbot statements as the company's own.
Variants
- FAQ / deflection bot (SMB): answers only from a curated help centre; no system access.
- Transactional service agent: authenticated; checks orders, changes appointments, issues return labels, creates tickets via API (banking/insurance variant: CONV-013).
- Omnichannel messaging: WhatsApp Business, Instagram, Messenger, plus web widget.
- Multilingual EU service desk: one knowledge base, answers in many EU languages.
- Neighbours: voice front-end CONV-002; staff-facing CONV-007; public sector CONV-011; internal employees CONV-004; sales CONV-005.
How it works
- customer message (web / WhatsApp / app)
- AI-disclosure + privacy notice
- input guardrails (PII masking, prompt-injection and abuse filter)
- intent / language detection, authentication check
- retrieval (hybrid search + reranker over help centre, policies, product data)
- LLM answer grounded in retrieved passages, with citations
- optional tool calls (order lookup, ticket, return) with scoped credentials
- output guardrails (grounding check, policy and tone check)
- answer | escalation to live agent with summary and transcript
- logging, feedback (thumbs), analytics, knowledge-gap report
- Grounding (RAG): the model may only answer from retrieved content; if retrieval is weak, it says so and escalates. This is the most important control against hallucination.
- Actions: narrowly scoped tools (read-only first); every write action needs identity verification and, above a threshold, human confirmation.
- Human-in-the-loop: explicit "talk to a person" option at any time, automatic handover on low confidence, anger or vulnerability signals, complaints, legal or contract topics; agents see a summary.
Data preparation
The chatbot is only as good as its knowledge base. Typical sources are FAQ and help-centre pages, product manuals (often PDFs), return and warranty policies, past tickets in the helpdesk or CRM, and order data in the ERP or shop system. In an SMB this knowledge is usually scattered across the website, file shares, mailboxes and the heads of experienced service staff.
Typical gaps: outdated or contradictory articles, scanned manuals without text layer, duplicate FAQs, missing product or order identifiers, and no clear owner for content. Customer data in tickets must not leak into answers.
Preparation steps: (1) inventory sources and name an owner per topic, (2) pick an authoritative version and delete duplicates, (3) convert scans via OCR and rewrite long documents into short answer-sized articles, (4) mine the top 50 ticket topics and write missing answers, (5) pseudonymise ticket exports used for testing, (6) build a test set of real questions with approved answers. Tools: Paperless-ngx or the helpdesk export for OCR and collection, spreadsheet or n8n for cleanup.
Expect roughly 30 to 50 percent of the project effort in data and content work. See PREP-013, PREP-014, PREP-018, PREP-022, PREP-006 and DATA-003.
Implementation paths
Which path for whom
- SMB: Path A. Mature EU-hosted chatbot products exist at SMB prices and the process (FAQ, order status, handover) is standard.
- Enterprise: Path B. Service is a core process with high volumes and deep CRM/order integration; API usage costs and control favour a custom build on cloud APIs.
- Developer: Path B. Cloud APIs plus a RAG framework give the fastest, most flexible build for client projects.
- Public sector: Path C. Sovereignty, procurement rules and citizen data favour open-weight models on German or EU sovereign cloud.
Path A — Buy (recommended for most SMBs and mid-market)
- Helpdesk-native AI agents: Intercom Fin, Zendesk AI agents, Freshworks Freddy AI, HubSpot Customer Agent, Salesforce Agentforce Service, Ada.
- Conversational platforms: Cognigy (DE, part of NICE), Parloa (DE, strong in voice), Userlike (DE), Tidio (PL), Landbot (ES), Zowie (PL), Botpress, Kore.ai, Yellow.ai.
- Check: EU data residency, DPA and sub-processors (US LLM providers), AI-disclosure support, handover to your ticket system, citation of sources, per-resolution vs per-seat pricing.
- Choose A when: content lives in a help centre, the helpdesk is a mainstream product, and no deep custom back-end logic is required.
Path B — Build on cloud APIs
- Models: Mistral (EU provider), Claude, Gemini, GPT via EU regions (Azure OpenAI EU data zone, Vertex AI europe, AWS Bedrock Frankfurt/Paris); small fast models for routing, larger ones for answers.
- Stack: embeddings + hybrid search (Qdrant, Weaviate, pgvector, OpenSearch), reranker (Cohere Rerank, bge-reranker), orchestration with LangGraph / LlamaIndex / n8n, tool access via MCP or function calling, guardrails (Llama Guard, NeMo Guardrails, Microsoft Presidio for PII).
- Channels: WhatsApp Business Platform, Twilio, Bird (NL), or the helpdesk's API.
- Choose B when: you need custom actions, strict branding or want to own prompts and evaluation.
Path C — Self-host (open source)
- Models: open-weight Mistral Small, Llama, Qwen3, Gemma, gpt-oss; embeddings such as bge-m3 or Qwen3-Embedding; served with vLLM or Ollama.
- Frameworks: Rasa (DE-founded; CALM LLM-driven dialogue) [5], Botpress, Chatwoot (open-source inbox) [7], Haystack, Langflow, Dify; observability with Langfuse [8].
- Hardware: HW2 handles a ~8–14B model for modest traffic; HW3 for 30–120B models and high concurrency. Needed when data must not leave the premises (health, public bodies).
Path D — Custom training
Rarely necessary. Fine-tune a small model for tone, domain vocabulary or intent routing only after prompting + RAG plateau and you hold thousands of reviewed conversations; tune the reranker or router first, not the answer model.
State of the art (2026)
- From intent bots to agentic service: LLM agents combine retrieval with tool calls and follow written procedures ("playbooks"); several vendors price per resolved conversation.
- Evaluation is the bottleneck: teams build test sets from real tickets, use LLM-as-judge plus human review, and run regression tests on every prompt or knowledge-base change.
- Observability: tracing of retrievals and tool calls (Langfuse, LangSmith, Arize Phoenix) is standard.
- Limits and failure modes: hallucinated policies or prices, outdated knowledge-base content, prompt injection via user text or retrieved pages, over-eager actions, poor handling of emotional or vulnerable customers, and "doom loops" where users cannot reach a human. Quality depends more on knowledge-base hygiene than on model choice.
Shared resources & combinations
- Same RAG stack (embedding-text, reranker, vector-db) as KNOW-002, KNOW-004 and CONV-004; one retrieval service can serve customers and employees with different access scopes.
- Combine with: LANG-012 (ticket routing), LANG-013 (sentiment and feedback analysis), CONV-007 (agent assist after handover), CONV-002 (voice front-end), KNOW-011 (auto-maintained FAQ), AGENT-011 (CRM updates after conversations).
- A self-hosted LLM server (HW3) can be shared across several CONV and KNOW use cases.
Effort, cost & KPIs
- Path A: days to a few weeks; pricing per seat, per conversation or per resolution.
- Path B: 1–3 months incl. integrations, evaluation set and guardrails; token cost is usually small against agent time but rises with long histories and large retrieved contexts.
- Path C: 2–4 months plus GPU operations.
- KPIs: verified resolution rate (not just "no escalation"), CSAT of bot vs human conversations, escalation quality, first-response time, cost per resolved contact, grounded-answer rate in audits, unanswered-topic backlog, complaints about the bot.
People & change
Involve management early for budget and the decision on what the bot may do on its own. The service lead is the process owner and picks the topic scope; service agents are the end users and should co-write answers and test the bot, since they know the real questions. IT handles channel and CRM integration, the DPO checks legal basis, AVV with the vendor and whether a DPIA is needed (customer conversations at scale, possibly special categories). Data owners release the knowledge sources.
Works council: depends. A pure customer-facing bot does not automatically trigger co-determination, but handover logs, agent quality scoring or per-agent analytics make the system objectively suited to monitor performance (§87(1) no. 6 BetrVG; BAG: suitability is enough). In any case inform the works council in good time (§90(1) no. 3), because work processes change. Public sector: Personalrat.
Measures: support AI literacy of the service team (Art. 4 AI Act), disclose to customers that they talk to an AI (Art. 50), start with a pilot group on two or three topics, communicate that the bot takes repetitive questions so agents handle complex cases. Typical fear is job loss; address it with a clear statement of purpose and retraining. Keep humans in control through easy handover, review of unanswered topics and a kill switch. See GUIDE-008, GUIDE-004, GUIDE-009.
Risks, regulation & compliance
- EU AI Act: typically limited risk. Art. 50(1) requires that people are told they are interacting with an AI system unless this is obvious; per law-firm analyses the duty applies since 2 August 2026 and was not postponed by the AI Omnibus, which delayed high-risk rules and gave pre-existing generative systems until 2 December 2026 for content-marking only [1][2]. Art. 4 AI literacy applies to staff. Bots that decide access to essential services (credit, insurance) can become high-risk, and emotion recognition of employees is prohibited under Art. 5 [6].
- GDPR: chats contain personal data (names, orders, sometimes health or financial details). Needs a legal basis, DPA with providers, data minimisation, PII masking before model calls, retention limits and transfer safeguards for non-EU providers. Art. 22 applies if the bot makes decisions with significant effects (e.g. automatically refusing a refund).
- Consumer law: statements made by the bot bind the company (see Moffatt [4]); keep complaint and withdrawal rights reachable; the European Accessibility Act (since June 2025) covers e-commerce interfaces.
- Security: prompt injection, data leakage across customers, over-privileged tool credentials; use least privilege, session-bound identity and red-team tests.
- Mitigations: grounding with citations, read-only tools first, easy human escape hatch, logging for audits, regular transcript review.
Getting started
- Export 3 months of tickets and chats; rank the top 30 question types and measure how many are answerable from existing content.
- Clean and consolidate the help centre; assign an owner for knowledge-base freshness.
- Pilot one Path A product (or a small Path B prototype) on one channel with read-only access.
- Build a 100–200 question evaluation set from real tickets, including adversarial cases.
- Launch with AI disclosure and visible human handover; review transcripts weekly, then add actions.
Sources
- Goodwin — EU AI Act transparency obligations now in force (Aug 2026) — https://www.goodwinlaw.com/en/insights/publications/2026/08/alerts-technology-dpc-eu-ai-act-transparency-obligations-now-in-force
- Houthoff — AI Act: transparency obligations and the AI Omnibus Regulation — https://www.houthoff.com/insights/news/ai-act-transparency-obligations-and-the-ai-omnibus-regulation/
- CX Today — Klarna claims its new AI assistant does the work of 700 full-time agents — https://www.cxtoday.com/contact-center/klarna-claims-its-new-ai-assistant-does-the-work-of-700-full-time-agents/
- The Guardian — Air Canada ordered to pay customer who was misled by airline's chatbot — https://www.theguardian.com/world/2024/feb/16/air-canada-chatbot-lawsuit
- Rasa — open-source conversational AI framework — https://github.com/RasaHQ/rasa
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Chatwoot — open-source customer engagement platform — https://github.com/chatwoot/chatwoot
- Langfuse — open-source LLM observability — https://github.com/langfuse/langfuse
- Entrepreneur — Klarna is hiring customer service agents after AI couldn't cut it on calls (May 2025, CEO interview with Bloomberg) — https://www.entrepreneur.com/business-news/klarna-ceo-reverses-course-by-hiring-more-humans-not-ai/491396