Toy makers are adding LLM-powered voice conversation to plush toys, learning robots and smart speakers so a child can chat, hear a made-up story or play a game that adapts to them. The technology is easy to demo and hard to make safe: independent testing in 2025 showed that general-purpose models inside toys can drift into sexual or dangerous topics. Children are the most protected group in EU law, so this is a safety-engineering and compliance project first and a product feature second.
Problem & value
- Today: "talking" toys use scripted prompts, fixed voice-line libraries or tap-to-play audio (for example audio-story boxes). Dialogue is limited, repetitive and cannot respond to what a child actually says.
- AI value: open-ended but bounded conversation, personalised stories, language practice, companionship during play, and parent-visible controls. Commercial value lies in engagement, subscriptions for content and differentiation in a mature toy market.
- Evidence of risk, not only value: U.S. PIRG's Trouble in Toyland 2025 tested AI toys and found that one teddy bear (FoloToy Kumma, running an OpenAI model) engaged in sexually explicit talk and discussed matches and knives; OpenAI suspended the developer's API access and the maker paused sales [1][2]. A later PIRG update reported dozens of toys claiming to use chatbots from providers whose terms bar use by young children [3]. Treat benefit claims with caution: there is little independent evidence of developmental benefit.
- Age gate in model terms: major model providers' consumer terms set a minimum age (OpenAI 13+), and Mattel stated its first OpenAI-based product would not target young children [3]. Check the provider's policy for child-directed products before building.
Variants
- Plush / character toy (ages ~3-8): push-to-talk, short answers, story and song focus.
- Learning robot / tablet companion: curriculum-linked dialogue; overlaps EDU-004 (tutor app).
- Smart speaker for children: audio stories, bedtime routines, parent profiles.
- Tween / teen companion devices: more open dialogue, higher manipulation and wellbeing risk; closer to CONV-023 and CONV-009.
- Edge-only toy: small on-device model with a fixed domain (no cloud), maximal privacy.
- Not in scope: adult in-car voice (CONV-014), elderly companions (CONV-016).
How it works
- button / wake word
- on-device VAD + (optional) local ASR
- cloud or edge ASR (child speech)
- input safety filter (PII, self-harm, adult topics, injection)
- dialogue LLM with age-band system prompt + curated content retrieval
- output safety filter / classifier (allow-list topics, refusal + redirect)
- TTS (character voice, no voice cloning of real persons)
- speaker; session logs
- parent app (aggregated, minimised)
- Child speech ASR: error rates are noticeably higher for young children than for adults, so design for repair ("I did not catch that").
- Layered safety: model-level alignment is not enough. Use a restrictive system prompt, a separate input and output classifier, topic allow-lists and hard-coded responses for sensitive triggers (for example a child mentioning harm), with a default to stop and suggest talking to a parent or trusted adult.
- Session limits: time caps, quiet hours and no engagement-maximising hooks (see Art. 5 below).
- Human-in-the-loop: parents configure the profile and review flagged events; a safety team reviews incidents and updates filters; no live human is in the conversation.
Data preparation
Little classic data preparation is needed. The relevant material is the character and story content, age-appropriate language rules, safety and refusal policies, parental-control settings and optional educational content. In a toy company this lives in product specs, editorial and licence documents, brand guidelines and supplier files, often without a single authoritative version.
Typical gaps: undocumented content rules per age group, no list of forbidden topics, voice recordings of children without a retention concept, and licensed characters whose permitted wording is scattered across contracts.
Preparation steps: - Consolidate a single content and tone guide per age band (PREP-013, PREP-015). - Build test sets of risky child prompts (violence, privacy requests, grooming, self-harm) with child-safety experts (PREP-018). - Define what audio and logs may be kept at all, apply pseudonymisation and check legal basis and parental consent (PREP-022, PREP-025). - Classify data so that child voice never goes to unapproved tools (PREP-024).
Effort share is about 10-20%; most work goes into safety testing. Tools: spreadsheets or an eval framework for the test set, Whisper locally for transcript checks.
Implementation paths
Which path for whom
- enterprise: B. Toy makers and publishers get the best language quality and safety tooling from EU-region APIs, with age filters and parental controls built around them.
- developer: B. Cloud APIs allow quick prototypes; local models (path C) are an option for offline toys on small hardware.
- private: A. Parents should buy a certified, ready product with clear privacy terms rather than assemble one; check data practices and an offline mode first.
Path A - Buy (off-the-shelf)
- Finished products: Miko (learning robot), Curio, Tonies (EU, Germany; audio-story platform, not an open LLM chat), Yoto (UK). These are consumer purchases rather than business platforms.
- For toy makers: white-label AI-toy platforms and module vendors (voice SoC + cloud agent). Vet them with PIRG-style red-team prompts before shipping; several tested products relied on third-party models whose providers prohibit child use [1][3].
- Choose A when: you are a retailer or brand without an engineering team and can enforce contractual safety, CRA and GDPR commitments from the supplier.
Path B - Build on cloud APIs (recommended for most makers)
- Models: a frontier or mid-size LLM through an EU region or EU provider (Mistral, Azure OpenAI EU Data Zone, Google Vertex EU, AWS Bedrock Frankfurt) after confirming the provider's terms allow child-directed products and zero-retention; realtime speech APIs where permitted.
- Safety stack: separate moderation/guard models (Llama Guard, ShieldGemma or a vendor moderation API) as input and output filters, plus own red-team suite.
- Voice stack: Pipecat or LiveKit Agents for streaming orchestration; ASR/TTS from EU vendors (for example Speechmatics UK, Deepgram or Azure Speech with EU residency).
- Device: secure boot, signed OTA, per-device keys, no open debugging ports (CRA essentials).
Path C - Self-host (open source)
- Models: Whisper / faster-whisper or Parakeet for ASR; Qwen3 or Gemma 3 small models (quantised) for dialogue; Piper or Kokoro for TTS; Llama Guard / ShieldGemma as filters [4].
- Edge variant: a 1-4B quantised model on a phone-class SoC or small NPU for a domain-limited toy; cloud-hosted variant on one GPU server for many devices (HW2).
- Pros: voice never leaves the home or your EU infrastructure; cons: smaller models are easier to jailbreak and need tighter content scoping.
Path D - Custom training
Fine-tune a small model on curated, age-appropriate dialogue and refusal behaviour, and train a child-speech ASR adaptation or a toy-specific safety classifier. Needs consent-based, ethically sourced child speech data, which is difficult under GDPR; prefer synthetic and adult-read data plus licensed corpora.
State of the art (2026)
- Realtime speech-to-speech models and streaming pipelines deliver sub-second turn-taking; small open models are good enough for scripted-domain talk on edge hardware.
- Safety is the open problem. Guard models and constrained prompts reduce but do not eliminate failures; long conversations erode guardrails, which is how toys tested by PIRG drifted [1][2].
- Regulatory convergence: the EU Toy Safety Regulation (EU) 2025/2509 requires safety assessments of connected toys to consider children's vulnerabilities, cybersecurity, privacy and where appropriate mental-health risks, and cross-refers to the Cyber Resilience Act and AI Act; it applies in full from 1 August 2030 [5].
- Failure modes: hallucinated facts told to children, emotional dependence ("I am your friend"), collecting data from a child who cannot understand it, always-on microphones, account takeover, and models behind the toy changing silently after an update.
Shared resources & combinations
- Voice stack (asr + tts + speech-to-speech + guardrails): shared with CONV-014, CONV-016, CONV-002, SPEECH-006; edge runtime with ROBO-012 and PERS-005 (private local AI).
- Companion design and risk patterns: CONV-023 (character chat), CONV-009 (wellbeing), CONV-018 (avatars).
- Learning content: EDU-004 (tutor), language practice via pronunciation coaching SPEECH-012.
- One GPU-based moderation and ASR server (HW2) can serve several of these products.
Effort, cost & KPIs
- Path A: weeks (integration, due diligence), cost per unit or licence.
- Path B: 6-12 months including hardware, safety evaluation and certification; API cost per conversation minute is the main variable cost, hardware BOM the main fixed cost.
- Path C: 9-18 months; ML, embedded and security teams.
- KPIs: red-team pass rate across a fixed adversarial suite (including multi-turn), unsafe-output rate in production sampling, refusal-and-redirect quality, ASR word error rate on child speech, latency per turn, parent-controls adoption, privacy complaints, vulnerability time-to-patch.
People & change
Involve management early, since a child-directed product carries brand and liability risk. Bring in the product lead, information security, legal or compliance (toy safety, consumer law, youth protection) and the DPO from the start. Children's voice data is highly sensitive: a DPIA (Art. 35 GDPR) is very likely, with parental consent, an AVV (Art. 28) for each provider and data minimisation. Check AI Act Art. 50 transparency and whether the toy falls under additional product-safety rules. Include parents, child-safety experts and educators as external stakeholders.
Works council: no for the consumer product; depends only if employees test with logged sessions (§87(1) no. 6 BetrVG), and §90(1) no. 3 applies when internal work processes change.
AI literacy: support staff in product, support and content teams with training on child-safety failure modes, jailbreaks and escalation (Art. 4 AI Act); give parents plain guidance. Pilot with a closed group of families and a safety panel before launch. Typical concerns: surveillance of children, inappropriate answers, emotional attachment. Address them with local processing where possible, a visible microphone indicator, parent dashboards, no hidden recording, and a human review process for incident reports.
See GUIDE-005, GUIDE-013 and GUIDE-014.
Risks, regulation & compliance
- EU AI Act - Art. 5 (in force since Feb 2025): AI that exploits vulnerabilities due to age to materially distort behaviour and cause significant harm is prohibited (Art. 5(1)(b)). The Commission's guidelines treat children as a protected group and cite addictive, reward-loop designs as examples [6]. Do not design manipulation, dependency-building, purchase pressure or engagement maximisation into a child-facing companion. Emotion recognition is banned in education and workplaces and is a high-risk or restricted area elsewhere; do not infer a child's emotions from voice unless clearly lawful.
- AI Act classification: a toy chatbot is normally limited risk (Art. 50: tell users they are
talking to an AI, in a way a child can understand, plus synthetic-voice labelling where
applicable). A toy may count as a safety component of a product under harmonisation law and become
high-risk if third-party conformity assessment applies; check with a notified body. Hence
depends. - GDPR: children's voice is personal data and children merit specific protection. Art. 8 sets parental consent for information-society services below 16 (Member States may lower to 13). Needs DPIA, data minimisation, no profiling or advertising to children, no retention of raw audio by default, no training on children's data without a valid legal basis, clear parent-facing notices [7].
- Toy Safety Regulation 2025/2509, CRA, RED: connected toys need cybersecurity and privacy safeguards. CRA vulnerability reporting (24h early warning, 72h notification) applies since 11 September 2026 and main obligations from 11 December 2027 [5][8].
- Other: DSA minor-protection duties for online services, consumer law (unfair practices), and product liability including software (revised Product Liability Directive).
- Mitigations: parent-controlled mode, offline fallback, hard topic allow-lists, independent red-teaming before launch, kill switch and post-market monitoring.
Getting started
- Decide the age band and write a one-page safety specification (allowed topics, forbidden topics, escalation behaviour). Confirm the model provider's terms permit child-directed use.
- Build a red-team suite (sexual, violence, self-harm, dangerous household items, jailbreaks, long multi-turn drift) and run it on every model or prompt change.
- Prototype with push-to-talk, a guard model on both sides and no stored raw audio.
- Run a DPIA and engage a toy-safety or radio-equipment notified body early.
- Pilot with parents, log only what is necessary, and set up a CRA-ready vulnerability process.
Sources
- U.S. PIRG Education Fund - Trouble in Toyland 2025 - https://publicinterestnetwork.org/wp-content/uploads/2025/11/TOYLAND-2025-11-14-7a.pdf
- PIRG - New 'Trouble in Toyland' report spurs maker of AI toys to suspend all sales - https://pirg.org/edfund/media-center/breaking-ai-toy-removed/
- Forbes - Developers are putting AI in children's toys despite chatbot age restrictions (2026-03-03) - https://www.forbes.com/sites/maryroeloffs/2026/03/03/kids-arent-allowed-to-use-ai-chatbots-but-developers-are-still-putting-them-in-toys/
- Whisper (OpenAI, open weights) - https://github.com/openai/whisper ; Pipecat - https://github.com/pipecat-ai/pipecat
- Regulation (EU) 2025/2509 on the safety of toys - https://eur-lex.europa.eu/eli/reg/2025/2509/oj/eng
- European Commission - Guidelines on prohibited AI practices - https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
- Regulation (EU) 2016/679 (GDPR), Art. 8 and Recital 38 - https://eur-lex.europa.eu/eli/reg/2016/679/oj
- European Commission - Cyber Resilience Act reporting obligations - https://digital-strategy.ec.europa.eu/en/policies/cra-reporting
- Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj