Social companion & character chat (entertainment, role-play)

Apps such as Character.AI, Replika, Nomi or Talkie let people chat with persistent AI characters for friendship, fun, role-play and interactive fiction. Value comes from consistent personas and long-term memory; the risks come from the same features: emotional dependency, manipulation, sexual content and exposure of minors. In the EU this is a compliance-heavy product category (GDPR, AI Act Art. 50, DSA) and not a plain "chatbot".

Problem & value

• Who: consumers seeking company, entertainment, creative writing partners, language practice or low-stakes social rehearsal; studios and brands offering fictional characters or game NPC chat.
• Today: fan fiction forums, tabletop role-play groups, scripted game dialogue. These do not scale to personalised, always-available conversation.
• AI value: open-ended, personalised dialogue with a stable persona and memory, in text, voice and increasingly images/avatars. It is one of the largest consumer uses of conversational AI; figures for user numbers differ by vendor and are mostly self-reported, so treat them with care.
• Evidence of harm potential: the Italian Garante fined Luka Inc. (Replika) EUR 5M in 2025 for processing without a valid legal basis, inadequate transparency and weak age verification [1]; Character.AI ended open-ended chat for under-18s in late 2025 after lawsuits and regulator pressure [2]; California enacted SB 243, the first US companion-chatbot law, effective 1 Jan 2026 [3].

Variants

• Character-chat platforms: huge catalogue of user-created characters (Character.AI-style).
• Single-companion "friend/partner" apps: one persistent personalised companion (Replika, Nomi, Kindroid-style).
• Interactive fiction & game NPCs: scripted-world role-play, game masters, story engines (see GEN-016).
• Branded or celebrity characters: licensed personas; needs consent and likeness rights.
• Voice and avatar companions: real-time speech-to-speech with a rendered avatar (see CONV-018).
• Adjacent but separate: clinical wellbeing CONV-009, elderly assistive CONV-016, child toys CONV-024, memorial avatars CONV-028.

How it works

1. user message (text/voice)
2. age & account gate (age assurance, settings, jurisdiction)
3. input safety classifiers (self-harm, sexual content with minors, violence, PII)
4. context assembly (persona card + recent turns + retrieved long-term memories)
5. LLM generation in character (system prompt, style/LoRA, sampling)
6. output safety filters + AI-disclosure / break-reminder logic
7. optional TTS / avatar / image generation
8. memory write-back (summaries, facts, user-editable)
9. telemetry for safety review (privacy-minimised)

• Persona: structured character card (traits, voice, boundaries) in the system prompt; fine-tuned or preference-tuned models give stronger style consistency and fewer persona breaks.
• Memory: conversation summarisation plus vector retrieval of facts and events; users should be able to view, edit and delete memories (GDPR access/erasure).
• Safeguards: self-harm detection with escalation to crisis resources, refusal of content involving minors, rate/session limits, de-escalation of dependency patterns, no manipulative retention tactics (guilt-tripping when the user leaves).
• Human-in-the-loop: trust-and-safety review of flagged conversations, red-teaming before launch, incident response and appeals; no human reads chats routinely.

Data preparation

This use case needs little classic data preparation. The core data are the character definition (persona, tone, backstory, boundaries), safety and crisis-response rules, and optionally consented conversation history for memory. Within a company these live in marketing or brand documents, product specs and scattered style guides; for private users they are simply their own text files.

Typical gaps: persona descriptions that are inconsistent or contradict safety rules, no crisis-escalation scripts, no test set of risky conversations, and chat logs that contain personal and sensitive data without a retention policy.

Preparation steps: - Write one authoritative character sheet and a list of terms the character must not use (PREP-015, PREP-016). - Collect gold-standard dialogues, including red-team cases such as self-harm, minors and manipulation (PREP-018). - Define what conversation memory may be stored and pseudonymise logs before analysis (PREP-022, PREP-025). - Capture user feedback on bad replies for later tuning (PREP-021).

Effort is about 10-20% of the project; the larger work is safety design and testing. Only fine-tuning a persona (path D) requires curated dialogue data.

Implementation paths

Which path for whom

• private: C. Companion chats are intimate, so local apps such as LM Studio or Ollama with a character front end keep conversations on the user's own device; hosted apps (path A) are the easy alternative.
• developer: B. Cloud APIs with safety classifiers give the best quality and fastest iteration for a companion product.
• enterprise: B. Companies launching a branded character need EU-region APIs, age assurance and moderation more than their own models.

Path A — Buy (off-the-shelf)

• White-label / platform: character-chat SDKs and hosted companion engines exist (e.g. Inworld for game characters, Convai, Character.AI API partnerships); check minors policy, content rules and data location. Most consumer apps themselves are US-based (Character.AI, Replika, Nomi, Talkie); EU-based options are few (check vendor domicile and DPA individually).
• Choose A when: you only need characters inside a game or brand experience and accept the vendor's moderation and data terms. Not a fit for adult-oriented or high-volume consumer products.

Path B — Build on cloud APIs

• Models: frontier LLMs via EU regions or EU providers (Mistral AI, Aleph Alpha-style EU hosting, Anthropic Claude, OpenAI, Google Gemini through Vertex/Azure EU regions). Check each provider's usage policy: several restrict romantic/sexual role-play or require adult-only gating.
• Voice: ElevenLabs, Cartesia, OpenAI Realtime API or Gemini Live for speech-to-speech; EU residency options differ by vendor.
• Safety: moderation endpoints (OpenAI moderation, Azure AI Content Safety, Llama Guard-class classifiers), crisis-resource routing per country, age assurance vendors (e.g. Yoti, Persona, k-ID; Yoti is UK-based).
• Orchestration: custom service with a vector DB (pgvector, Qdrant — an EU vendor) for memory, prompt caching to cut cost on long persona prompts.

Path C — Self-host (open source)

• Models: open-weight chat/role-play models such as Llama, Mistral, Qwen and Gemma families and community role-play fine-tunes; license terms vary (check use restrictions).
• Stack: vLLM or llama.cpp/Ollama, SillyTavern-style front ends for personal use, Qdrant/pgvector, Llama Guard / ShieldGemma for safety, Piper/Kokoro/XTTS for voice, Whisper for ASR.
• Hardware: HW2 for one persona per user at small scale; HW3 for a multi-tenant service with 30–70B models.
• Why self-host: privacy of very intimate chat logs, control over moderation, no provider policy limits. Private hobbyists can run everything locally on a laptop/GPU.

Path D — Custom training

Preference-tune or LoRA-fine-tune a base model for persona consistency, tone and safe refusals. Needs curated dialogue data with documented rights and consent; never train on users' intimate chats without a clear legal basis (the Garante opened a separate inquiry into Replika's training data [1]).

State of the art (2026)

• Long context and memory: million-token contexts plus retrieval make multi-month continuity feasible, but memory still drifts, contradicts itself or invents shared history.
• Realtime voice and avatars: low-latency speech-to-speech is now standard in leading apps.
• Safety tooling: classifier guards, age assurance (behavioural signals plus selfie/ID checks) and crisis-escalation flows are becoming baseline. Character.AI combined in-house age models with third-party checks (Persona) when it restricted minors [2].
• Regulation catches up: California SB 243 (disclosure, suicide-prevention protocol, minor break reminders, annual reporting from 2027) [3]; the EU AI Act transparency duties apply from August 2026 [4].
• Limits and failure modes: sycophancy that reinforces delusions or self-harm, emotional dependency, persona jailbreaks into sexual content, hallucinated "memories", and weak age assurance. Evidence on long-term wellbeing effects is still thin and mixed.

Shared resources & combinations

• Same stack: llm-general + vector-db memory + guardrails + tts/asr are shared with CONV-015 (personal assistant), CONV-016 (elderly companion), CONV-009 (wellbeing), CONV-018 (avatar) and CONV-024 (toys).
• Hardware: one HW3 vLLM server can host persona models for several of these products.
• Combos: GEN-016 (illustrated stories) for visual storytelling; CONV-028 shares the memory/persona machinery but has far stricter ethical constraints (deceased persons).

Effort, cost & KPIs

• Path A: weeks for an embedded character; Path B: 3–6 months incl. safety, age assurance and legal review; Path C: 4–8 months plus GPU operations.
• Cost drivers: long persona + memory prompts per turn, voice synthesis minutes, moderation passes, trust-and-safety staffing, age-assurance fees per verification.
• KPIs: retention and session length (watch for dependency, not just maximise), safety-incident rate, self-harm escalation response time, share of verified-adult users, refusal precision/recall, user complaint and deletion-request turnaround.

People & change

For a company product, involve management (risk appetite, brand), the product lead, IT and information security (moderation, logging, abuse handling), the DPO and legal or compliance. Chats reveal very personal data and possibly health information, so a DPIA (Art. 35 GDPR) is likely, with an AVV (Art. 28) for each API provider. Check youth-protection rules (JMStV) and Art. 50 AI Act disclosure that users talk to an AI.

Works council: no, as long as the product serves customers or private users. It becomes relevant (depends) only if employees use the companion internally or their chats are logged; then §87(1) no. 6 BetrVG applies, and §90(1) no. 3 requires information about AI-supported work processes.

AI literacy: support staff who build, moderate and answer support requests with training on dependency risks, crisis signals and prompt injection (Art. 4 AI Act). Pilot with a small closed beta of adult testers and a clinical or psychological adviser; evaluate emotional dependency. Typical concerns: manipulation, addiction, harm to minors. Address them with clear AI labelling, session limits, a human-reachable escalation channel for crises, and no engagement-maximising dark patterns.

See GUIDE-005, GUIDE-007 and GUIDE-013.

Risks, regulation & compliance

• EU AI Act: typically limited risk: Art. 50 requires telling users they interact with an AI system (unless obvious) and labelling synthetic audio/image content. Art. 5 prohibitions apply to companion design: manipulative or deceptive techniques that materially distort behaviour and cause significant harm, and exploiting vulnerabilities due to age or disability, are prohibited. Emotion recognition is prohibited in workplaces and schools and is high-risk elsewhere; do not infer user emotions from voice or face for retention [4]. Mental-health-like claims can trigger medical-device rules.
• GDPR: intimate chats can reveal special-category data (health, sexuality, beliefs, Art. 9), so explicit consent or another Art. 9 basis is needed. Required: clear privacy notice, legal basis for training, age verification, data minimisation, erasure, DPIA. Replika's EUR 5M fine shows the risk [1].
• Minors: DSA Art. 28 and the Commission's guidelines on protecting minors online; national youth protection law (e.g. German JMStV). Sexual content with or involving minors is illegal; block it by design.
• US reference: California SB 243; litigation over teen harm [2][3]. Relevant for EU firms serving US users.
• Mitigations: age assurance, no sexualised minor personas, crisis protocol with local helplines, break reminders, transparent AI labelling, no dark-pattern retention, memory controls, regular red-teaming.

Getting started

1. Decide audience and content policy first (adults only vs. all ages); choose age assurance accordingly.
2. Run a DPIA and map Art. 5 / Art. 50 duties before building features such as romantic personas.
3. Prototype a single persona on an EU-hosted API with memory and a moderation layer.
4. Build crisis and dependency flows with local helplines; red-team with self-harm and minor scenarios.
5. Launch to a small adult beta; monitor safety KPIs and complaints before scaling.

Sources

1. Garante / Luka Inc. Replika fine summary — https://morrirossetti.it/en/insight/publications/replika-5-million-fine-for-the-u-s-based-company-the-italian-dpa-launches-new-investigation-into-ai-training-practices.html
2. TechCrunch — Character.AI is ending its chatbot experience for kids — https://techcrunch.com/2025/10/29/character-ai-is-killing-the-chatbot-experience-for-minors/
3. Skadden — New California companion chatbot law (SB 243) — https://www.skadden.com/insights/publications/2025/10/new-california-companion-chatbot-law
4. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. Orrick — 2026 State Chatbot Laws — https://www.orrick.com/en/Insights/2026/04/2026-State-Chatbot-Laws-Key-Provisions-and-Regulatory-Trends
6. Companyscope — Garante v Luka decision register entry — https://companyscope.io/register/air-2026-006