Child online safety & parental assistance (on-device protection)

Children meet unwanted nude images, grooming attempts, bullying and self-harm content in chats, social media and games, usually where parents cannot see. On-device classifiers detect such content before it is viewed, warn the child with age-appropriate help and, depending on age and family settings, notify parents with a summary instead of a full transcript. The design question is less "can we detect it" than "how much should parents see" — child privacy and child rights are part of the product, not an afterthought.

Problem & value

• Today: parents rely on blanket filters, screen-time limits and conversations. Platforms moderate centrally (see LANG-014, VIS-027, ANOM-004), but private messages, end-to-end encrypted chats and cross-app contact are largely invisible to them. Parents of teenagers often cannot read everything and should not.
• AI value: context-aware detection (not keyword lists) of nudity, sexual solicitation, bullying and self-harm language in many languages; detection happens locally, so content never leaves the device; the child gets a chance to pause and ask for help.
• Benefit evidence: vendors publish usage figures, but independent, peer-reviewed effect sizes for AI monitoring on harm reduction are thin. Do not rely on marketing claims; evaluate against your own family's needs. Research on parental mediation generally finds that open communication and "enabling" mediation work better than covert restriction (general finding; not tied to a single source here).
• Regulatory driver: the Digital Services Act Art. 28 obliges online platforms accessible to minors to ensure a high level of privacy, safety and security; the Commission's guidelines on the protection of minors (2025) describe measures such as privacy-by-default settings and age assurance [1].

Variants

• OS-level nudity warning: blur-and-warn for images in messaging, AirDrop, video calls and shared albums, processed on the device (child accounts in a family group).
• Content and contact filtering: web/app categories, safe search, contact approval, time limits.
• Message-risk alerts: classifiers for grooming patterns, bullying, self-harm or suicidal language with alerts to the parent (Bark, Qustodio style).
• Keyboard-level protection: a keyboard that screens text and images as they are typed or sent (SafeToNet HarmBlock style) and nudges the child before sending.
• Conversational help for parents: an LLM assistant explaining apps, trends and how to talk to a child about an incident, without access to the child's data.
• School/club deployment: device management with filtering — school monitoring is a different, higher-regulation setting (see LANG-032 for workplace analogues).

How it works

1. content arrives / is composed on child device
2. on-device sensitive-content classifier (image/video: nudity; text: bullying, grooming, self-harm)
3. age band + family policy decide action (allow | blur+warn | block | warn before sending)
4. child-facing screen: explanation, "do not send / not your fault", help lines (e.g. 116 111)
5. optional escalation: category-level alert or weekly digest to parent (no full content by default)
6. parent dashboard: settings, digest, conversation guide; severe risk -> advice to seek professional help

• Image models: compact classifiers or small vision-language models running via Core ML / NNAPI / LiteRT; the content stays on the device.
• Text models: small encoder classifiers or small LLMs (≤ 8B or distilled) for intent, tone and context; multilingual and slang coverage is the main quality driver.
• Human-in-the-loop: the child decides on a blurred image (view or not); parents receive a summary, decide on follow-up; self-harm alerts should lead to human help, never to automated punishment. Crisis flows must be reviewed by child-protection professionals.

Data preparation

The data is produced on the child's device: messages, images, browsing, app usage and chat content. It lives in messenger apps, the photo library and the family account of the platform; nothing sits in an ERP or DMS. Typical gaps: end-to-end encrypted chats that cannot be read, slang and emoji not covered by classifiers, mixed languages, no age or consent information, and image classifiers biased by skin tone or context. Preparation steps: (1) fix the scope with the family, which content is checked and who sees what (PREP-024); (2) set age profiles and categories before any classification (PREP-019); (3) run classification on the device so images never leave it; (4) keep only labels, timestamps and a blurred thumbnail, not the content (PREP-022); (5) for developers, build a test set of synthetic and licensed examples with child-safety experts, never real abuse material (PREP-018); (6) collect parent and child feedback on false alarms (PREP-021). Rough share of effort: about 15 percent, since vendors ship trained models; the real work is policy and consent. See GUIDE-003 for general readiness.

Implementation paths

Which path for whom

• Private: Path A. Mature, EU-available parental-control products and platform features (operating system family controls) work out of the box, and families rarely have the skills or time to run their own stack.
• Developer: Path B. Teams building a child-safety product start with moderation and vision APIs on EU regions and move on-device for the child's images once the design is stable.

Path A — Buy (recommended for families)

• Platform built-ins (free): Apple Communication Safety (on-device nudity detection for child accounts in Family Sharing, extended to more apps via the Sensitive Content Analysis framework [2]), Google Family Link and Sensitive Content Warnings in Google Messages [3], Microsoft Family Safety.
• Commercial parental suites: Bark, Qustodio, Norton Family, Net Nanny, Kaspersky Safe Kids; European options include JusProg (DE, age-rating filter) and Salfeld Kindersicherung (DE); SafeToNet (UK) for keyboard-level protection. Check: data location, whether message content leaves the device, age-staged transparency, deletion on request.
• Choose A when: a family wants working protection within hours and no technical build.

Path B — Build on cloud APIs

• For app developers adding child protection to their service: moderation APIs (OpenAI moderation, Azure AI Content Safety, Google Cloud Vision SafeSearch, Amazon Rekognition content moderation), Mistral moderation (EU), Hive. Server-side scanning of private messages raises confidentiality and ePrivacy questions — prefer on-device inference with server fallback only for reports.
• Age assurance: EU age-verification blueprint / app, privacy-preserving age tokens, wallet-based proofs [1], instead of collecting ID copies.
• Choose B when: you build a messenger, game or kid-friendly app and need defence in depth.

Path C — Self-host (open source)

• Models: open NSFW image classifiers (e.g. Falconsai nsfw_image_detection on Hugging Face, OpenNSFW2), Llama Guard / ShieldGemma / Qwen3Guard for text and image safety classes, small multilingual toxicity classifiers (Detoxify, XLM-R based), small VLMs for context.
• Runtimes: ONNX Runtime Mobile, Core ML, TensorFlow Lite / LiteRT, llama.cpp on Android.
• Home gateway: DNS-level filtering (Pi-hole, AdGuard Home, OpenDNS-style category lists) on a Raspberry Pi (HW1) for network-wide filtering; no content analysis of encrypted traffic.
• Choose C when: privacy-focused families, schools or clubs with technical skills. Open classifiers have weaker slang, language and edge-case coverage than commercial ones.

Path D — Custom training

Fine-tune a classifier for a specific language, age group or platform culture (for example, German youth slang for bullying). Needs labelled data that is hard and ethically sensitive to collect; use synthetic and consented datasets, involve child-safety organisations, and never build or hold illegal imagery. For known illegal material, platforms use hash matching through authorised bodies, not family tools (see VIS-027, VIS-050).

State of the art (2026)

• On-device sensitive-content detection is standard in major OSes: images are analysed locally, and developers can reuse the OS capability through system frameworks [2].
• Small guard models (Llama Guard, ShieldGemma and similar) bring multilingual harm classification to the edge; context awareness (is this a joke, an adult or a peer?) is improving but not solved.
• Age assurance is converging on privacy-preserving proofs (EU age-verification app, wallets) and on-device age estimation; accuracy and bias limits remain, and the Commission's guidelines call for proportionality [1].
• Limits and failure modes: false positives (medical, art, family photos) embarrass and erode trust; false negatives give false security; end-to-end encrypted apps may be unreadable without on-device access; teenagers circumvent controls with VPNs, secondary accounts or new apps; coded language and emoji change quickly; AI-generated nudes of peers are a growing harm (classifiers must also treat synthetic content). Detection must never be the sole protection.

Shared resources & combinations

• Same stack (vlm, image-classification, guardrails, edge-runtime): VIS-027 image/video moderation and LANG-014 text moderation use the same model families server-side; PERS-005 private local assistant and PERS-009 family organisation share the home server and family account.
• Related: ANOM-004 platform abuse, PERS-007 assistive AI (accessibility settings for the same devices), VIS-050 forensic triage (law-enforcement, out of scope here; never a family tool).
• Hardware: child device NPU for inference (HW1); optional home gateway for DNS filtering.

Effort, cost & KPIs

• Path A: hours for setup; free built-ins to a modest annual subscription per family.
• Path B: 1–3 months for an app integration including age assurance and reporting flows.
• Path C: weeks for DNS filtering; months for a maintained on-device pipeline.
• KPIs: share of harmful exposures intercepted (red-team test sets), false-positive rate reported by children and parents, child help-seeking after a warning, parent-child conversation rate, time to remove content on reports, opt-out/circumvention rate.

People & change

The stakeholders are the family and, for developers or schools, the legal and data-protection side. Talk to the child first: explain what is checked, agree on rules age-appropriately and favour warnings to the child before any report to parents; secret surveillance damages trust and can be unlawful for older teenagers. Works council: no for private use; if a school, youth organisation or employer deploys it on staff or pupil devices, co-determination (section 87(1) no. 6 BetrVG) or the staff council (Personalrat) may apply (GUIDE-004). Data protection: children's data needs special care, so a DPO should assess a DPIA for any product offered to others (Art. 35 GDPR), with processor agreements and a clear legal basis (GUIDE-005). AI literacy: Art. 4 AI Act asks providers to support the AI literacy of staff, and parents benefit from a short guide on what the alerts mean and what they miss (GUIDE-006). Pilot with a few volunteer families. Typical concerns are privacy and false alarms: show how alerts are explained, allow appeal, and keep humans in control, since a parent decides and never the system alone. Point to counselling services for serious cases.

Risks, regulation & compliance

• EU AI Act: typically limited/minimal risk. Art. 5 prohibitions to avoid: emotion recognition in education settings, exploitative AI targeting children's vulnerabilities, and manipulative techniques; do not infer emotions from face or voice for child monitoring in schools. Biometric age estimation is not prohibited, but must avoid biometric categorisation of sensitive attributes. Chatbots for children need Art. 50 transparency [4].
• GDPR: child data is data of vulnerable persons; Art. 8 (consent age 13–16 by member state), data minimisation, purpose limitation, DPIA likely; the child's rights apply even when parents consent. UN CRC General Comment 25 requires balancing protection with privacy and participation.
• ePrivacy / secrecy of communications: a parent monitoring a child's messages is a family matter in most states, but vendors processing content in the cloud are not; Germany's JuSchG and JMStV set youth-protection duties, with the Kommission für Jugendmedienschutz (KJM) supervising.
• CSAM regulation: the EU "chat control" proposal was contested; family tools should not scan for illegal material or forward it; if a child receives such material, advise reporting to hotlines (INHOPE network, e.g. eco, jugendschutz.net) and police, preserving no copies.
• Risks: surveillance harming trust, outing LGBTQ+ children, abusive-household misuse of monitoring, vendor data breaches. Mitigations: age-staged transparency, child-visible monitoring, category-level alerts, on-device processing, deletion rights, no covert spyware.

Getting started

1. Talk with the child first: agree what is monitored and why, and adjust with age.
2. Turn on the platform built-ins (Communication Safety, Family Link, Family Safety) and review defaults.
3. Add a DNS-level or app-level filter only where gaps remain; read the vendor's data-processing terms.
4. Test with harmless, synthetic examples; check false positives and what parents actually see.
5. Review quarterly; relax controls as the child grows; bookmark help lines and reporting hotlines.

Sources

1. European Commission — Guidelines on protection of minors online under the DSA — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-protection-minors
2. Apple — Sensitive Content Analysis framework — https://developer.apple.com/documentation/sensitivecontentanalysis
3. Google — Sensitive Content Warnings in Google Messages — https://safety.google/intl/en_us/families/
4. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. UN Committee on the Rights of the Child — General comment No. 25 on children's rights in relation to the digital environment — https://www.ohchr.org/en/documents/general-comments-and-recommendations/general-comment-no-25-2021-childrens-rights-relation
6. Regulation (EU) 2022/2065 (Digital Services Act) — https://eur-lex.europa.eu/eli/reg/2022/2065/oj
7. Llama Guard (Meta) — https://github.com/meta-llama/PurpleLlama
8. Bark — https://www.bark.us