Digital human / avatar for customer interaction

A text chatbot is efficient but impersonal; a human agent is personal but expensive and not available 24/7 in every language. A digital human puts a face and voice on the same conversational back end: speech recognition, an LLM grounded in company knowledge, speech synthesis and a real-time lip-synced video stream. It suits kiosks, reception desks, product advisers and onboarding, where a visible "someone" raises engagement. It must always be disclosed as AI.

Problem & value

• Today: websites rely on text chat or FAQs; stores, banks, airports and public counters rely on staff or static kiosks. Video "presenter" content is pre-recorded and cannot answer questions.
• AI value: an always-on, multilingual, on-brand presenter that answers questions, guides forms and onboarding, and hands over to people. Benefits are mostly engagement, accessibility (speech plus visual cues, lip-reading support) and brand experience, rather than hard cost savings.
• Evidence is thin: vendors publish engagement and conversion claims, but independent, peer-reviewed ROI data is scarce. Run your own A/B test against a plain voice or text bot (CONV-001, CONV-002) before investing; the avatar layer adds cost and latency over those.

Variants

• Brand/product adviser on websites and in retail (CONV-006, CONV-005).
• Kiosk or reception avatar in hotels, airports, hospitals, municipal offices (CONV-011, CONV-012).
• Onboarding/training coach (employees, customers, bank KYC explainers) - overlaps GEN-003 (pre-rendered avatar video).
• Sign-language or accessibility avatar - see LANG-030.
• Expert & creator AI clone (digital twin): a persona-styled RAG bot over a person's own books, videos and posts, plus voice and likeness cloning (SPEECH-007). Typical for coaches, authors, doctors-as-educators and executives. The stack is the same as above; what changes is the legal layer: written, specific, revocable consent of the person (GDPR Art. 6/7, and for voice/image personality rights, in Germany the general personality right and §22 KUG), contract terms on who owns the clone after the contract ends, and Art. 50(4) deepfake labelling because the clone imitates a real person. The clone should state what it is not (e.g. no medical or legal advice in the person's name). Posthumous clones are a separate case: CONV-028.

How it works

1. user speech/text
2. VAD + streaming ASR (language detection)
3. LLM agent (persona prompt + RAG over KB + tools: booking, order lookup) + guardrails
4. streaming TTS (consented voice)
5. avatar renderer (audio-driven lip-sync / face animation, idle behaviour, gestures)
6. WebRTC video+audio to browser/kiosk
7. transcript, analytics, human handoff

• Brain: identical to CONV-001/CONV-002: retrieval (KNOW-002), tool calls, guardrails, escalation rules.
• Voice loop: either cascaded ASR → LLM → TTS or a speech-to-speech model; turn-taking and interruption ("barge-in") matter more than looks. End-to-end latency under about a second feels natural; above that, users disengage.
• Avatar rendering: two approaches. (1) Hosted/neural video streaming: the vendor renders frames from audio and streams them. (2) Real-time 3D/engine avatars (Unreal MetaHuman, Unity, WebGL): lower compute per session, less photoreal, controllable. Photoreal neural rendering is GPU-heavy per concurrent session.
• Human-in-the-loop: disclosed AI status, "talk to a person" button always visible, escalation on low confidence, complaints, vulnerable-user signals; content owners approve persona and KB.

Data preparation

The avatar is only the face; the knowledge behind it decides quality. Needed are product and FAQ content (shop system, CMS, PIM), service knowledge (DMS, ticket system, e-mail answers), booking and order context (ERP, CRM), and persona material: script, tone of voice and, for a custom avatar, consented reference footage and voice recordings.

Typical gaps: marketing-style texts that are too long for spoken answers, FAQs scattered over wiki, PDFs and heads, outdated price or opening-hour information, missing consent documents for likeness and voice, and no list of topics the avatar must refuse.

Preparation steps: 1. Curate an authoritative knowledge base with a named owner (PREP-013, PREP-017). 2. Rewrite key answers short and speakable (PREP-014) and add a glossary for product names and abbreviations that speech recognition mishears (PREP-015). 3. Export CRM, booking and order data through APIs and restrict it to what the customer may see (PREP-005, PREP-023). 4. Document consent and licence for face and voice of any real person. 5. Build a test set of 50 typical conversations with domain experts (PREP-018).

Tools: DMS or CMS export, a spreadsheet for the glossary, a RAG index. Rough share of effort: 20-30 percent, mostly knowledge curation, low in data volume. See GUIDE-003.

Implementation paths

Which path for whom

• SMB: Path A. Avatar platforms with EU hosting deliver a ready persona, speech and video streaming; real-time lip-synced video is too demanding to build or run yourself.
• Enterprise: Path B. Own brand persona, CRM and booking integration and control over the dialogue brain justify composing avatar rendering with your own LLM and knowledge layer via APIs.
• Developer: Path B. Combine a streaming avatar API with your own agent logic; self-hosting needs HW3 GPUs and is only worthwhile for niche cases.
• Public sector: Path B on a sovereign EU or German cloud for the dialogue and knowledge part, with a certified avatar service where one is available.

Path A - Buy (recommended)

• Real-time avatar platforms: Tavus, HeyGen (LiveAvatar / interactive avatar), D-ID (agents/streaming), Synthesia (UK; mainly pre-rendered video, interactive agents emerging), Anam (UK), UneeQ and Soul Machines (digital-human specialists for enterprise brand deployments; both non-EU, verify vendor viability and ownership before contracting). Most are US/UK/NZ-based: check EU data residency, DPA, sub-processors, and retention of recordings and face/voice models. EU-based specialists and integrators exist for kiosk and 3D avatars; shortlist these first where sovereignty matters.
• Choose A when: you need a pilot in weeks, have no real-time video engineering, and volumes are modest. Check concurrency pricing (per-minute fees add up) and a clean exit path for your persona assets.

Path B - Build on cloud APIs

• Orchestrate with LiveKit Agents or Pipecat (open-source realtime agent frameworks with WebRTC and avatar plugins), LLM via EU-region endpoints (Azure OpenAI EU, Vertex AI EU, Mistral La Plateforme (EU), Claude via EU regions of cloud providers).
• ASR/TTS: Azure Speech (EU), Google Cloud, Deepgram, ElevenLabs (EU data residency option), Mistral Voxtral (EU).
• Plug a hosted avatar renderer (Tavus, HeyGen, D-ID, Anam) as the video layer, or Azure Speech's text-to-speech avatar.
• Choose B when: you want control of the brain (RAG, tools, guardrails) and swap-able avatar vendors.

Path C - Self-host (open source)

• LLM: Mistral/Qwen/Llama-class open weights via vLLM; ASR: Whisper variants or Voxtral; TTS: open models (e.g. Kokoro, Piper, Coqui-derived); speech-to-speech: Kyutai Moshi (EU, open).
• Face animation research code: MuseTalk, LivePortrait, Wav2Lip-class lip-sync, SadTalker-type models; check each licence (several restrict commercial use). 3D alternative: Unreal MetaHuman / Unity with audio-driven visemes (e.g. NVIDIA Audio2Face open components).
• Hardware: HW3 per few concurrent photoreal streams; HW2 for a small number of sessions or 3D engine avatars; scale linearly with concurrency. Substantial ops effort.

Path D - Custom training

Only for a bespoke likeness or brand voice with fine-tuned face/voice models: needs consented studio footage (typically minutes to hours of the person), a signed likeness licence and a re-consent process. Rarely worth it versus vendor "custom avatar" tiers.

State of the art (2026)

• Streaming neural avatars reached near-photoreal lip-sync with sub-second response in the better hosted products; speech-to-speech models cut turn latency. Independent comparisons of live-avatar providers remain mostly subjective [4].
• Limits and failure modes: uncanny valley and gaze/gesture mismatch; latency spikes on poor networks; hallucinated answers delivered with persuasive confidence (more trusted because of the human face); accents and noisy kiosks degrade ASR; per-session GPU cost; voice/face-clone misuse risk and prompt injection through speech. Over-anthropomorphising can mislead vulnerable users.
• Many deployments quietly fall back to a voice orb or text when users dislike the avatar: offer a choice.

Shared resources & combinations

• Shares the whole conversational stack with CONV-001/002/007 and RAG with KNOW-002; TTS/voice cloning with SPEECH-006/007; pre-rendered presenters with GEN-003; guardrails and logging with CONV-011 for public bodies.
• Deepfake and impersonation defence on your own persona: SEC-011.
• Hardware tier HW3 can be shared with other speech/vision workloads.

Effort, cost & KPIs

• Path A: 2-6 weeks for a pilot; costs are subscriptions plus per-minute streaming fees.
• Path B: 2-4 months incl. RAG, integration, consent flows. Path C: 4-8 months plus GPU operations.
• KPIs: containment/resolution rate vs. text bot, handoff rate, CSAT, session length, time-to-first-response, answer accuracy (sampled), opt-out rate from avatar mode, cost per conversation minute.

People & change

Involve management and the owner of the customer channel first, then IT and IT security (streaming, browser permissions, kiosk hardware, prompt injection), the DPO (conversation recordings, optional webcam, voice and likeness; DPIA if video of visitors is analysed) and legal for the Art. 50 AI Act disclosure that users are talking to AI. Frontline staff who today serve the counter or chat must join the design from day one.

Works council: usually no, because customers are the users. It becomes relevant when staff work alongside the avatar, when conversation analytics reveal employee performance (§87(1) no. 6 BetrVG) or when workplaces change (§90(1) no. 3 BetrVG). In case of doubt, inform it early. See GUIDE-004.

Support AI literacy of the staff who supervise the avatar (Art. 4 AI Act): limits, escalation, error reporting. Run a pilot at one location or one page for 6-8 weeks. Typical concerns: "uncanny valley" and customer rejection (test with real customers, offer text or human alternatives), job fear among service staff (position the avatar as relief for peaks and off-hours), brand risk from wrong answers (guardrails, refusal list). Keep humans in control through easy handoff and regular review of transcripts. See GUIDE-008 and GUIDE-009.

Risks, regulation & compliance

• EU AI Act (Reg. 2024/1689): typically limited risk. Art. 50(1) requires that people are informed they interact with an AI system unless obvious; for a lifelike avatar assume it is not obvious, so disclose visibly and verbally at start. Art. 50(2) obliges providers to mark synthetic audio/video in machine-readable form; Art. 50(4) requires deployers to disclose deepfakes (a clone imitating a real person). The Commission's Code of Practice on marking and labelling AI-generated content supports compliance [2]; Art. 50 transparency duties apply from 2 Aug 2026 under the original timetable (verify any Digital Omnibus grace periods).
• Art. 5 prohibitions: no manipulative or exploitative techniques that distort behaviour or target vulnerabilities (age, disability, social situation); emotion recognition from face or voice is banned in workplace and education and is regulated elsewhere. Do not add "mood detection" from webcam to an avatar used with employees or pupils.
• GDPR: voice, transcripts and any camera feed are personal data; face/voice templates used to identify a person are biometric data (Art. 9); avoid by not processing the user's face. Need DPIA, DPA, retention limits, no training on conversations without legal basis; Art. 22 if decisions are automated.
• Consumer/sector law: misleading advice (financial, medical), Digital Services Act dark-pattern limits, accessibility (EAA, in force June 2025).
• Mitigations: disclosure, human fallback, scoped persona, grounded answers with citations, red-team prompt injection, logging.

Getting started

1. Define one narrow, high-value dialogue (e.g. product adviser or reception) and a text/voice baseline.
2. Build the brain first with CONV-001-style RAG and escalation; test without the avatar.
3. Pilot 2 hosted avatar vendors on EU residency, latency and per-minute price.
4. Write the disclosure, consent (for any real-person clone) and DPIA before launch.
5. A/B test avatar vs. voice-only on resolution and satisfaction; keep the avatar only if it wins.

Sources

1. Regulation (EU) 2024/1689 (AI Act), Art. 5 and Art. 50 - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
2. European Commission - Code of Practice on marking and labelling AI-generated content - https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content
3. LiveKit Agents documentation - https://docs.livekit.io/agents/
4. The Live Avatar Landscape: APIs, Transport and Subjective Evaluation of 10 Leading Providers - https://medium.com/@ggarciabernardo/the-live-avatar-landscape-apis-transport-and-subjective-evaluation-of-10-leading-providers-5b5b6e8a54dc
5. Pipecat open-source realtime agent framework - https://github.com/pipecat-ai/pipecat
6. Tavus conversational video interface docs - https://docs.tavus.io
7. Kyutai Moshi - https://github.com/kyutai-labs/moshi
8. MuseTalk real-time lip-sync - https://github.com/TMElyralab/MuseTalk