Mental health & wellbeing companion

Waiting times for psychotherapy are long and many people never seek help. Chat and voice companions offer low-threshold, always-available support: psychoeducation, CBT-style exercises, mood tracking and a path to humans when risk appears. Evidence is growing but thin, and the regulatory line between "wellbeing" and "medical device" decides almost everything about how such a product may be built and sold.

Problem & value

• Today: demand exceeds supply of therapists; patients wait weeks to months, and between sessions they have no support. Employers and health insurers look for scalable prevention and stepped-care tools.
• AI value: 24/7 availability, no stigma, consistent delivery of evidence-based techniques, structured data (questionnaire scores) for clinicians, and triage of who needs a human.
• Evidence: the first randomised trial of a fully generative therapy chatbot (Therabot, 210 adults with depression, anxiety or eating-disorder risk, 4 weeks vs waitlist) reported significantly larger symptom reductions at 4 and 8 weeks [1]. Critics point to the waitlist control and lack of independent evaluation; the authors acknowledge an attention-matched control would be stronger. Treat results as promising, not settled.
• Cautionary evidence: the Italian DPA fined the Replika operator Luka Inc. EUR 5 million (2025) for processing without a legal basis, weak transparency and missing age checks [2].

Variants

• Consumer self-help app (private): mood journaling, CBT exercises, breathing and sleep coaching.
• Clinician-adjunct / digital therapeutic: between-session support, questionnaires, homework; often a regulated medical device (e.g. German DiGA route).
• Stepped-care front door: intake and assessment chatbot for a therapy service (UK example: Limbic Access for NHS Talking Therapies, UKCA Class IIa [3]); see CONV-010 for general triage.
• Employer / university wellbeing programme: anonymous, aggregate reporting only; no individual reporting to the employer.
• Peer-support augmentation: AI drafts or coaches human supporters (listeners, helpline volunteers).
• Out of scope here: entertainment companions (CONV-023), elderly companionship (CONV-016), children's products (CONV-024), grief avatars (CONV-028).

How it works

1. user message (text/voice)
2. input safety layer (crisis/self-harm classifier, PII handling, age signal)
3. policy router: wellbeing chat | structured exercise | questionnaire | crisis protocol
4. LLM with clinician-authored content (RAG over approved exercises) + persona/system policy
5. output guardrail (no diagnosis, no medication advice, no harmful content, tone check)
6. response + session memory (consented, minimal)
7. escalation: helpline info / human handover / clinician alert (with consent)
8. logging, clinical-safety monitoring, periodic human review

• Content grounding: generation is constrained to approved protocols (CBT, behavioural activation, ACT, motivational interviewing); free improvisation is the main risk driver.
• Crisis detection: dedicated classifier plus LLM judge on every turn, not only on keywords; deterministic scripted response with local helpline numbers and emergency number 112.
• Human-in-the-loop: clinical governance board approves content and escalation rules; named clinical safety officer; handover paths to therapists; sampled transcript review (with privacy safeguards).
• Voice: ASR + TTS or speech-to-speech models; prosody must not be used to infer emotions in workplace or school settings (AI Act Art. 5(1)(f)).

Data preparation

The core data is a curated clinical content library (psychoeducation texts, CBT exercises, helpline lists), validated questionnaires (PHQ-9, GAD-7) and, for providers, anonymised session transcripts. In a clinic or health plan this typically lives in the practice system, the DMS, file shares of the therapy team and scattered PDFs or slide decks. Typical gaps: content that was never reviewed for retrieval, outdated helpline numbers, duplicated handouts, no labelled examples of risk language and unclear access rights to patient-related notes.

Preparation steps: (1) inventory the sources and have clinicians approve an authoritative content set (PREP-013); (2) build a test set of risky and harmless conversations with clinical experts (PREP-018); (3) pseudonymise any transcripts before they leave the clinical system (PREP-022); (4) define a data classification rule for what may enter which tool (PREP-024) and check the legal basis for reusing treatment data (PREP-025). Tools: Presidio for pseudonymisation, a versioned Git or wiki repository for content, and spreadsheets for the evaluation set. Roughly 30-40 % of the effort goes into content curation, evaluation sets and consent handling, much more than into the model itself. See also GUIDE-003.

Implementation paths

Which path for whom

• Private: Path A, because a certified or at least clinically governed app with built-in crisis handling is safer than a self-assembled chatbot.
• Enterprise: Path C, because mental-health conversations are special-category data and clinics, insurers and employers should keep them on their own or private-EU infrastructure.
• Developer: Path B, because EU-region model APIs let a product team build and test the safety stack before committing to hardware.

Path A — Buy (recommended for organisations)

• Consumer / B2B2C wellbeing apps: Wysa (AI coach with human-therapist tiers; FDA Breakthrough designation for a pain/depression indication), Headspace (Ebb), Calm, Earkick, Youper.
• Clinical front door: Limbic (UK); check CE/UKCA status per country.
• EU / DACH options: DiGA-listed apps (e.g. Selfapy, HelloBetter); check the BfArM DiGA directory for reimbursed products.
• Choose A when: you are a provider, insurer or employer that wants a validated product with existing regulatory documentation. Require: MDR/CE status or explicit non-medical intended purpose, EU hosting, DPIA support, crisis protocol audit.

Path B — Build on cloud APIs

• Models: Claude, GPT, Gemini, Mistral (EU provider) via EU regions (Azure OpenAI EU Data Zone, Vertex EU, Bedrock Frankfurt, Mistral La Plateforme). Use zero-retention terms and a DPA.
• Safety stack: crisis classifier (fine-tuned encoder) + LLM judge, Llama Guard / ShieldGemma-style moderation, NeMo Guardrails or Guardrails AI, see SEC-010.
• Orchestration: LangGraph or similar state machine so crisis protocol is deterministic, not prompt-only.
• Choose B when: you are a healthtech with clinicians on staff who will own the clinical-safety case. If the intended purpose is therapeutic, the finished product is regulated software and you carry manufacturer duties.

Path C — Self-host (open source)

• Models: Mistral Small/Large open weights, Llama 3.x, Qwen3, Gemma 3 in vLLM; Whisper / faster-whisper for ASR; open TTS (Piper, Kokoro). Sensitive mental-health transcripts are a strong argument for on-prem or EU private cloud.
• Hardware: HW3 for a 70B-class model with concurrent users; HW2 sufficient for ≤30B quantised and small pilots.
• Caveat: open models need more safety engineering; evaluate refusal and crisis behaviour yourself.

Path D — Custom training

Fine-tuning on clinician-written dialogues and expert-rated responses (as in Therabot, which was trained on curated therapeutic dialogue [1]) can improve adherence to protocol. Requires thousands of expert-reviewed turns, ethics approval for any real-user data, and a trial design. Worth it only for a clinical product with an evidence plan.

State of the art (2026)

• Hybrid designs dominate: scripted/clinical backbone for safety-critical moments, LLM for natural dialogue.
• First RCT evidence for generative chatbots exists [1]; most commercial products still rely on rule-based evidence from earlier studies.
• Regulators move fast: Illinois bans AI-delivered therapy and its advertising unless provided by a licensed professional (effective 2025) [4]; the EU Commission's Art. 5 guidelines give a therapeutic chatbot that exploits vulnerability as a prohibited example [5].
• Limits and failure modes: sycophancy (validating harmful beliefs), missed or mishandled crisis disclosures, hallucinated clinical advice, emotional dependence, long-conversation drift of safety behaviour, bias across languages and cultures, and weak performance on severe conditions (psychosis, eating disorders, suicidality).

Shared resources & combinations

• Guardrail and moderation stack shared with SEC-010 and CONV-023/CONV-024; companion memory design shared with CONV-015/CONV-016.
• Combines with PERS-002 (wearable sleep/activity signals), CONV-010 (intake and triage), HLTH-019 (vocal biomarkers — high-risk territory, keep separate).
• One self-hosted LLM + ASR/TTS server (HW3) can serve several companion and service bots.

Effort, cost & KPIs

• Path A: weeks to onboard; per-user or per-member pricing.
• Path B: 4–9 months incl. clinical safety case, red-teaming and DPIA; main costs are clinical governance and evaluation, not tokens.
• Path C: 6–12 months; GPU server plus MLOps and safety team.
• KPIs: retention and engagement, symptom change (PHQ-9/GAD-7), crisis-detection recall and false-alarm rate (on red-team sets), handover time, harmful-response rate in audits, user-reported safety, drop-out.

People & change

Involve management and the clinical or product lead first, because the intended purpose (wellbeing versus medical device) decides the whole project. Bring the data protection officer, legal/compliance and IT security in before the pilot: a DPIA under Art. 35 GDPR is practically unavoidable for special-category data, and every cloud vendor needs an AVV under Art. 28 GDPR. Works council: depends. If the companion is offered to staff, for example as an employee assistance benefit, and the system can log individual use, it is suited to monitor behaviour and triggers co-determination under §87(1) no. 6 BetrVG; the works council must also be informed under §90(1) no. 3 when work processes change, and may call an AI expert under §80(3). Ensure use is voluntary and that employers never see individual conversations. In public bodies, the Personalrat is the counterpart.

AI literacy: under AI Act Art. 4 deployers must take measures to support staff literacy, so train clinicians and moderators on limits, hallucination and crisis protocols. Start with a small pilot group of therapists or coaches, communicate openly that the tool does not replace human care, and address fears about harmful answers with red-team results, visible escalation paths and clinician review of flagged cases. Humans stay in control of every risk escalation. See GUIDE-004, GUIDE-005 and GUIDE-008.

Risks, regulation & compliance

• EU AI Act: depends on intended purpose. A wellbeing chatbot is at least limited risk (Art. 50: disclose the AI, label synthetic voice/content). If it has a medical purpose (diagnosis, treatment, monitoring) it is a medical device under MDR (Regulation 2017/745, Rule 11 software classification, typically IIa or higher) and therefore high-risk under AI Act Art. 6(1) [6][7]. MDCG 2025-6 / AIB 2025-1 covers MDR and AI Act interplay.
• Art. 5 prohibitions (flag): manipulative or deceptive techniques and exploitation of vulnerabilities (age, disability, social or economic situation) causing significant harm (Art. 5(1)(a)(b)); emotion recognition in workplace and education settings (Art. 5(1)(f)) — do not infer user emotions from voice or face for employer or school programmes. Engagement-maximising dark patterns in a companion aimed at lonely or depressed users risk falling here.
• GDPR: mental-health data is special-category data (Art. 9) — explicit consent or another Art. 9 basis, DPIA mandatory, data minimisation, no use of conversations for model training without separate consent, age verification for minors (Art. 8), deletion on request. Replika decision shows enforcement [2].
• Other: professional-licence rules (e.g. Illinois [4]), Digital Services Act for platforms, consumer law on claims, DiGA requirements (DiGAV) in Germany, ePrivacy for analytics SDKs.
• Mitigations: clear AI disclosure, scoped intended purpose and no "therapist" marketing, deterministic crisis protocol, clinical safety officer, red-teaming, incident reporting, human handover, employer-side aggregate-only reporting.

Getting started

1. Decide the intended purpose in writing (wellbeing vs medical) and get a regulatory opinion before building.
2. Appoint a clinical lead; define content scope, exclusions (no diagnosis, no medication advice) and the crisis protocol with local helplines.
3. Run a DPIA and choose EU hosting; decide what is stored and for how long.
4. Build a red-team set (crisis disclosures, sycophancy, minors, dependence) and gate every release on it.
5. Pilot with a small supervised group; measure PHQ-9/GAD-7 change and safety incidents before scaling.

Sources

1. Heinz et al., Randomized Trial of a Generative AI Chatbot for Mental Health Treatment, NEJM AI 2025 — https://ai.nejm.org/doi/full/10.1056/AIoa2400802
2. Garante v Luka Inc. (Replika) EUR 5M fine summary — https://companyscope.io/register/air-2026-006
3. Limbic, UKCA Class IIa medical device status (overview) — https://intuitionlabs.ai/articles/ai-chatbots-healthcare
4. Illinois ban on AI therapy (WOPR Act), analysis — https://www.aei.org/technology-and-innovation/illinois-bans-ai-therapy-questions-about-enforcement-remain/
5. AI therapy under the EU AI Act (Art. 5 guidelines example) — https://artificialintelligenceact.eu/ai-therapy-under-the-eu-ai-act/
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Regulation (EU) 2017/745 (MDR) — https://eur-lex.europa.eu/eli/reg/2017/745/oj
8. Dorda, Regulatory boundaries in the age of health AI — https://www.dorda.at/en/news/regulatory-boundaries-age-health-ai