Image & video content moderation (nudity, violence, CSAM, terrorist content)

Any service that lets users upload pictures or video (social networks, marketplaces, dating, games, forums, cloud storage, ad platforms) must find and act on illegal and policy-violating visuals. Perceptual-hash matching catches known material, vision classifiers and VLMs catch new material, and trained human moderators decide the hard cases. In the EU the process is shaped by the DSA, the Terrorist Content Online Regulation and CSAM rules, so audit trails, appeals and reporting matter as much as detection accuracy.

Problem & value

• Today: volume is far beyond manual review. Platforms combine user reports, hash matching and ML classifiers; moderators (often outsourced) see disturbing material with documented wellbeing impact.
• AI value: automatic pre-filtering removes the bulk of clear violations and clearly benign items, prioritises the queue by severity, reduces moderator exposure (blur, grayscale, audio-off, cropping), and shortens time-to-removal, which matters under fixed deadlines such as the one-hour rule for terrorist content.
• Legal driver: the Digital Services Act (Reg. 2022/2065) requires notice-and-action, statements of reasons, internal complaint handling and, for large platforms, risk assessment and transparency reporting [1]. Regulation (EU) 2021/784 obliges hosting providers to remove terrorist content within one hour of a removal order from a competent authority [2].
• No sourced universal accuracy figure exists; precision/recall depend heavily on category, policy definition and content mix, so measure on your own data.

Variants

• Social / UGC platforms: images, short video, livestream, profile photos, with the full DSA process.
• Marketplaces and classifieds: listing photos (weapons, nudity, counterfeit, hate symbols); overlaps with product compliance.
• Messaging and cloud storage: hash-based detection of known CSAM; end-to-end encryption makes this legally and technically contested.
• Dating, gaming, children's services: stricter nudity and grooming-adjacent policies, combined with age assurance (VIS-029).
• Ad networks and publishers: creative review against ad policy.
• Public sector / hotlines / trusted flaggers: triage of citizen reports, hotline analyst tooling (e.g. INHOPE network, eco, FSM in Germany), TCO referral units; forensic seized-media work is separate (VIS-050).
• Livestream moderation: frame sampling plus audio. Live voice-chat moderation belongs to SPEECH-023; text to LANG-014.

How it works

1. upload / stream segment
2. normalise (decode, keyframes every n s, thumbnails, audio split)
3. known-material check: perceptual hashes (PhotoDNA, PDQ/TMK+PDQF, CSAI Match for video) vs. hash lists
4. classifiers/VLM: nudity, violence/gore, weapons, self-harm, hate symbols, extremist insignia, minors-risk signals
5. policy engine: category + confidence + context (account age, audience, geography)
6. action
7. auto-action only above high-confidence thresholds (block, blur, age-gate, limit reach)
8. human review queue (severity-ranked, wellbeing tooling, second reviewer on illegal-content categories)
9. statement of reasons, appeal, reporting to hotline/authority, transparency metrics
10. feedback: reviewer labels
11. threshold tuning, retraining, hash-list updates

• Hash matching finds re-uploads and near-duplicates of already verified material with very low false positives. It cannot see anything new. Hash lists come from NCMEC, IWF, other hotlines and industry sharing programmes; terrorist content hashes via GIFCT's shared database and Tech Against Terrorism's Terrorist Content Analytics Platform (TCAP) alerts.
• Classifiers score new material. For unknown CSAM, specialised classifiers (Thorn Safer Predict, Google Content Safety API) prioritise items for expert human review; they never make the final legal determination.
• VLMs add context ("is this a news report or glorification?") and zero-shot handling of new policy rules, but are slower and costlier, so they are usually used as a second stage on borderline items.
• Human-in-the-loop: mandatory for appeals, for context-dependent categories (news, art, education, satire) and for all reports of suspected CSAM. Reviewers need rotation, exposure limits and psychological support.

Data preparation

Moderation needs less classic data preparation than most use cases, because the main input is the content stream itself: user uploads, shop product images, chat attachments, comment sections. Content typically lives in the web platform, shop system, cloud storage or social media accounts. The real preparation is about rules, not files: a written content policy (what is forbidden, what is only age-restricted), severity levels and escalation paths are usually missing or only implied.

Preparation steps: define the policy and category scheme before configuring any tool (PREP-019). Assemble 100 to 200 borderline examples from past reports and decisions as a test set (PREP-018); handle these with strict access control because they may contain illegal material, and never store suspected CSAM, report it through the legally required channels instead. Harvest earlier moderator decisions as labels (PREP-020) and capture appeals and corrections (PREP-021). Decide which data may be sent to which provider (PREP-024) and check the legal basis (PREP-025). Set up logging for audit trails.

Rough share of effort: 10 to 20 percent. Most effort goes into policy, threshold tuning and building the review workflow rather than data cleaning.

Implementation paths

Which path for whom

• smb (A): Moderation APIs and platform features are mature, cheap and updated against new abuse patterns; building your own gives no advantage.
• enterprise (A): Moderation is a commodity function; a certified vendor with hash-matching and audit logs beats an in-house model.
• developer (B): Combine cloud moderation APIs with own policy rules and review tooling so the product's community guidelines are enforced precisely.
• public (B): Use EU-hosted moderation services with documented processing and add own escalation to reporting bodies; fully self-built models are hard to justify for illegal content.

Path A — Buy (recommended for most)

• Moderation APIs / platforms: Hive Moderation (visual classifiers, CSAM detection partnership with Thorn), Sightengine (France, EU), Checkstep (EU-founded policy and DSA workflow tooling), Unitary (UK), ActiveFence, ActiveFence (incl. the acquired Spectrum Labs), Cinder (trust-and-safety tooling), Besedo (EU, managed moderation), Amazon Rekognition content moderation (EU regions), Azure AI Content Safety (image and video, EU regions), Google Cloud Vision SafeSearch / Video Intelligence explicit-content detection.
• CSAM-specific: Thorn Safer (hashing, matching and Predict classifier, available to qualifying platforms), Microsoft PhotoDNA (cloud service for eligible organisations), Google CSAI Match and Content Safety API (approved partners), IWF hash-list membership and services. These are access-controlled; you apply and are vetted. Do not attempt to build or test with real CSAM.
• Choose A when: you lack a trust-and-safety team, need fast time to value, or must obtain legally sensitive hash lists through a vetted programme.

Path B — Build on cloud APIs

• Moderation endpoints (Azure AI Content Safety, AWS Rekognition, Google Vision) as first stage plus a frontier multimodal model (Claude, Gemini, GPT, Mistral Pixtral-class) as a policy-aware second stage returning JSON verdicts against your written policy.
• Orchestrate with queues (Kafka/SQS), keyframe extraction (FFmpeg), a rules/policy service and a reviewer UI (Checkstep, Cinder or custom). Use EU regions and check that the provider does not train on your data and permits processing of sensitive content (acceptable-use terms for CSAM handling differ; check before sending suspected illegal material to any general-purpose API).

Path C — Self-host (open source)

• Hashing: Meta's open-source PDQ (images) and TMK+PDQF (video) in the ThreatExchange repo; perceptual-hash libraries; vector search for near duplicates.
• Classifiers: open NSFW image classifiers, CLIP/SigLIP-based zero-shot and linear-probe heads, open VLMs such as Qwen-VL family or Llama Guard 4 (multimodal safety classifier), ShieldGemma 2 (image safety) from Google, served via vLLM.
• Review tooling: open-source options such as Osprey/Coop-style trust-and-safety frameworks and Label Studio for review queues; ROOST (open-source safety tooling initiative) publishes components. Check maturity before adopting.
• Hardware: HW2 handles image classification at high throughput and a ~7B VLM second stage on a fraction of traffic; HW3 for video-heavy loads.
• Self-hosting suits sovereignty needs (public sector, hotlines) and keeps sensitive images in-house, but legal access to CSAM hash lists still goes through vetted programmes.

Path D — Custom training

Worth it for platform-specific categories (local symbols, regional slang in memes, your own art-versus-nudity line) and for cost at large scale. Needs thousands of labelled examples per class from your own reviewed decisions; labelling of illegal material is restricted to authorised staff and hotlines, so synthetic or public benign proxies are used for development and real data only in controlled environments.

State of the art (2026)

• Layered stack: hash match first, fast classifier second, VLM third, human last. This pattern is standard at large platforms and in vendor products.
• Multimodal safety models (Llama Guard 4, ShieldGemma 2, Azure/Google safety services) allow policy-as-prompt: change a rule without retraining. Consistency and adversarial robustness are weaker than dedicated classifiers.
• Video and live: keyframe sampling plus audio/ASR signals; real-time latency and cost remain the limit for full-frame analysis.
• Generative AI content: AI-generated nudity and synthetic CSAM are a growing share of cases; hash lists do not cover new synthetic material, so classifiers and provenance signals (C2PA, watermarks, see SEC-011) matter. The EU is also moving to criminalise AI-generated child abuse material.
• Limits and failure modes: false positives on breastfeeding, medical, art, news and war-reporting content (a known source of over-removal and appeals); cultural and context dependence; adversarial evasion (cropping, overlays, mirroring beat weak hashes, so use robust hashes and re-scan); classifier bias across skin tones and body types; moderator trauma.

Shared resources & combinations

• Same stack (vlm + image-classification + guardrails): VIS-030 anonymisation, VIS-025 video tagging, VIS-022 visual search (near-duplicate index shared with hash lookup), VIS-029 age assurance.
• Sibling modalities: LANG-014 text moderation, SPEECH-023 voice chat, SEC-011 deepfake detection; one policy engine and one reviewer UI should serve all modalities.
• Forensic side: VIS-050 for law-enforcement triage; platforms hand reports over, they do not run investigations.
• Compliance workflow: shared with legal case-management and DSA transparency reporting.

Effort, cost & KPIs

• Path A: weeks for API integration; per-item pricing, plus reviewer costs which usually dominate. Access to CSAM tooling requires vetting time.
• Path B: 2-4 months including policy engine, reviewer UI and appeals flow.
• Path C/D: 4-9 months, a trust-and-safety engineering team and ongoing model upkeep.
• KPIs: precision and recall per category (on audited samples), prevalence of violating content viewed, median time-to-action (and time to removal for terrorist content orders against the 1-hour limit), share auto-actioned, appeal rate and overturn rate (a direct quality signal), moderator exposure hours and wellbeing indicators, number of reports to authorities, cost per 1,000 items.

People & change

Management and legal or compliance decide the content policy and liability stance, including duties as a platform operator. The department lead (community, customer service) owns thresholds and escalation; end users, the moderators, test the queue and give feedback. IT integrates the API and logging, and the DPO covers legal basis, retention and AVV with the provider. A DPIA is likely where user content is processed at scale.

Works council: depends. Moderation of customer content alone is not employee monitoring, but moderator dashboards that record decisions, speed or error rates per person are suited to monitor performance and trigger §87(1) no. 6 BetrVG. Moderators also face psychological strain; involving the works council and HR early on rotation, support and limits on exposure is advisable. §90(1) no. 3 requires timely information about AI-planned work processes.

AI literacy (Art. 4, support measures): train moderators on typical false positives and negatives, bias across skin tones and cultures, and how to override. Start with a pilot where the AI only pre-sorts the queue and humans decide every removal; widen automation to clear-cut categories only after measured precision is acceptable. Tell users transparently that automated checks exist and offer an appeal route. Typical concerns (wrongful removal, censorship, moderator wellbeing) are met by human review of borderline cases and visible appeal paths. See GUIDE-005, GUIDE-008 and GUIDE-014.

Risks, regulation & compliance

• EU AI Act: content moderation as such is not an Annex III high-risk use, so typically minimal risk (AI literacy under Art. 4). Check context: emotion recognition or biometric categorisation inside moderation tools may fall under Art. 5 prohibitions or high-risk rules; do not infer sensitive traits from faces. Synthetic-content labelling duties (Art. 50) concern generators, not moderators.
• DSA: notice-and-action (Art. 16), statements of reasons (Art. 17), internal complaints (Art. 20), out-of-court dispute settlement, reporting of suspected criminal offences involving threat to life or safety (Art. 18), transparency reports; VLOPs additionally carry systemic-risk duties [1]. Art. 14 requires clear disclosure of automated moderation in terms and conditions.
• TCO Regulation 2021/784: one-hour removal after order, specific measures if exposed to terrorist content, transparency reporting [2].
• CSAM: possession and handling of suspected CSAM is a crime in most jurisdictions; only authorised staff may view it, and reports go to NCMEC or national hotlines/police. Voluntary detection by interpersonal communication services in the EU has relied on the temporary ePrivacy derogation (Reg. 2021/1232, extended several times) while the proposed permanent CSA Regulation has been politically contested; verify the current legal status before deploying scanning of private messages [3].
• GDPR: images of people are personal data; special-category data can appear incidentally. Need legal basis (Art. 6(1)(c)/(f)), DPIA, Art. 22 care for solely automated decisions with significant effect (account bans), retention limits and processor agreements. Hash databases of illegal material have their own legal frameworks.
• Fundamental rights: over-blocking harms expression; document policy, sample audits, publish metrics, offer appeals.
• Mitigations: high-confidence thresholds for auto-removal, human review for context, reviewer wellbeing programme, red-team tests for evasion, region-aware policies.

Getting started

1. Write the policy taxonomy (illegal vs. policy-only categories, severity levels) and map each to DSA/TCO duties and actions.
2. Apply for hash-list and CSAM tooling access (NCMEC/IWF membership, Thorn Safer, PhotoDNA) and designate authorised staff and a reporting path.
3. Pilot a vendor moderation API on a labelled sample of your own benign and borderline content; measure per-category precision/recall.
4. Build the review queue with severity ranking, blur defaults, appeals and statement-of-reasons templates.
5. Run in shadow mode before auto-actions; review appeal overturn rates monthly and publish the transparency report.

Sources

1. Regulation (EU) 2022/2065 Digital Services Act — https://eur-lex.europa.eu/eli/reg/2022/2065/oj
2. Regulation (EU) 2021/784 on addressing the dissemination of terrorist content online — https://eur-lex.europa.eu/eli/reg/2021/784/oj
3. Regulation (EU) 2021/1232 (temporary derogation from ePrivacy for CSAM detection) — https://eur-lex.europa.eu/eli/reg/2021/1232/oj
4. Meta ThreatExchange: PDQ and TMK+PDQF hashing — https://github.com/facebook/ThreatExchange
5. Thorn Safer — https://safer.io
6. Microsoft PhotoDNA — https://www.microsoft.com/en-us/photodna
7. Azure AI Content Safety — https://learn.microsoft.com/azure/ai-services/content-safety/overview
8. Amazon Rekognition content moderation — https://docs.aws.amazon.com/rekognition/latest/dg/moderation.html
9. Google ShieldGemma 2 — https://ai.google.dev/gemma/docs/shieldgemma
10. Tech Against Terrorism — Terrorist Content Analytics Platform — https://terrorismanalytics.org
11. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj