Text content moderation & toxicity detection

Every service that lets people post text, from a regional newspaper's comment section to a marketplace, game chat or citizen-participation portal, has to keep illegal and abusive content out. Under the EU Digital Services Act it also has to handle notices, explain each decision and offer appeals. Modern moderation stacks combine fast classifiers with policy-following LLMs that read the platform's own written rules. They handle the clear cases automatically and send the ambiguous ones, with a rationale, to a human. The usual benefit is that moderators spend their time on hard cases instead of the volume. It also brings faster reaction times and a documented, auditable decision trail.

Problem & value

• Today: small and mid-sized operators rely on keyword blocklists, post-moderation by editors or community managers, and user reports. Blocklists miss paraphrases, misspellings and coded language. They also over-block reclaimed or quoted terms. Larger platforms run outsourced moderation teams at high cost and with documented psychological strain on staff.
• Scale of the obligation: the EU DSA Transparency Database showed about 4.7 billion statements of reasons submitted by 389 active platforms in the preceding 180 days, 42% of them flagged as fully automated decisions (statistics page, Nov 2026) [1]. Automated moderation is the norm, and it has to be explained.
• AI value: consistent first-pass triage of 100% of incoming text, near-real-time hiding of clearly violating posts, prioritised queues for human moderators, multilingual coverage without per-language staff, and machine-generated statements of reasons.
• Market shift 2026: Google/Jigsaw's free Perspective API, long the default toxicity scorer for news sites, stopped accepting new usage requests in February 2026 and shuts down on 31 December 2026, with no migration support [2]. Many publishers and NGOs are migrating now.

Variants

• News publishers & forums: pre- or post-moderation of reader comments; the main risks are hate speech, insults, illegal content (in DE e.g. §§130, 185 ff. StGB) and spam.
• Gaming & social apps: real-time chat filtering with latency budgets well under a second; usernames and clan names. Voice chat is a separate use case → SPEECH-023.
• Marketplaces & e-commerce: listings, seller messages and reviews (prohibited items, scams, off-platform payment solicitation). Fake reviews → ANOM-010; account abuse → ANOM-004.
• GenAI products: screening user prompts and model outputs. The classifier stack is the same, but LLM-application guardrails (jailbreaks, prompt injection) belong to SEC-010. Consumer co-creation campaigns (GEN-001) use this for input/output moderation.
• Public sector: citizen-participation portals, municipal social-media channels and consultation platforms; moderation must respect freedom of expression and equal treatment.
• Enterprise internal: intranet/Teams communities and employee-feedback tools (keep separate from conduct surveillance → LANG-032, which has different legal limits).
• Private / creators: comment filtering on own social-media channels (e.g. Bodyguard's social-media product). Family-side protection of minors → PERS-019.

How it works

1. ingest (post / message / listing / user notice)
2. normalisation (Unicode/leetspeak folding, language ID, dedup via embeddings / near-dup hashing)
3. stage 1: fast classifier (encoder or small guard model, < 50 ms) + blocklists/regex
4. clear allow
5. publish | clear severe violation
6. hide/hold
7. stage 2: policy-reasoning LLM on the grey zone (policy text in the prompt, thread context)
8. label + severity + rationale citing the policy clause
9. rules engine (thresholds per category, user history, rate limits, jurisdiction)
10. human review queue (priority by severity & reach) — moderator decides
11. action + DSA statement of reasons (Art. 17)
12. Transparency Database
13. appeals (Art. 20, human review)
14. labels feed back into evaluation & tuning

• Stage 1 handles the volume. Fine-tuned XLM-R/DeBERTa-class classifiers, small guard LLMs (Qwen3Guard 0.6B/4B) or a vendor API provide scores per category.
• Stage 2 is where the 2025–2026 change happened. Models such as OpenAI's open-weight gpt-oss-safeguard take the operator's own written policy at inference time and reason about whether content violates it. Policy changes therefore need no retraining [3].
• Human-in-the-loop: borderline content, account-level sanctions and all appeals go to people. The DSA requires that complaint decisions are not taken solely by automated means (Art. 20(6)) and that statements of reasons disclose the use of automated means [4]. Moderator wellbeing features (blurring, rotation, exposure limits) are part of the design.

Data preparation

Data needed: user-generated texts (comments, chats, reviews, listings, profiles), the written house rules or community guidelines, and past moderation decisions with reasons. In an SMB these live in the CMS, shop, forum or social media admin tools, the support mailbox and sometimes only in moderators' heads.

Typical gaps: house rules too vague to apply consistently; past decisions undocumented or without reason codes; few labelled examples of borderline cases; mixed languages, slang and emojis; personal data in the texts; reporter and moderator identities mixed with content.

Preparation steps: 1. Turn the house rules into a clear category scheme with examples (PREP-019, PREP-016). 2. Export past decisions and harvest them as labels (PREP-020); have two moderators label 200-300 borderline cases for a test set (PREP-018). 3. Remove user names and contact data from texts used for testing (PREP-022). 4. Log every later moderator correction as feedback (PREP-021).

Tools: CMS or platform exports, Excel or a labelling tool such as Label Studio, n8n for webhooks into the review queue.

Effort share: roughly 30 % of the project; the written policy and labelled borderline cases matter more than model choice.

Implementation paths

Which path for whom

• SMB — A: moderation tools and forum or shop platforms with built-in filters are mature, and the rules (hate, spam, insults) are standard.
• Enterprise — B: moderation rules, languages and appeal workflows are specific to the platform and must integrate with own case and DSA reporting systems.
• Developer — B: moderation and LLM APIs plus own policy prompts and review queues are the quickest route; self-host only for sensitive content.
• Public sector — C: citizen comments and procurement rules favour sovereign hosting with open classifiers and a human review queue.

Path A — Buy (recommended for most SMBs)

• Moderation SaaS with dashboards & queues: Bodyguard (France, 46 languages, social-media and API products) [5], Utopia Analytics (Helsinki, Finland; "language-agnostic" models trained on the customer's own past decisions) [6], Tremau (Paris; DSA trust-and-safety and compliance platform) (EU), plus Checkstep, Hive, ActiveFence and Besedo (Sweden, EU, human plus AI moderation services).
• Built-in options: comment systems and community platforms (Discourse, Disqus, Coral) and social-media native filters for owned channels.
• Check: EU hosting and DPA, German-language quality on your data, custom policy categories, appeal workflow, statement-of-reasons export to the DSA Transparency Database API, and an audit log.
• Choose A when moderation is not your core product and volumes are below several million items a month.

Path B — Build on cloud APIs

• Dedicated moderation endpoints: Mistral Moderation (EU provider; mistral-moderation-2603, 11 categories incl. hate, violence, self-harm, PII, jailbreaking; raw-text and conversational endpoints. The older -2411 model was deprecated on 31 March 2026 and retired on 30 June 2026) [7]. Azure AI Content Safety (EU regions; hate/sexual/violence/self-harm with severity levels, blocklists, custom categories; trained on 8 languages incl. German) [8]. OpenAI omni-moderation-latest (free, 13 categories, text + image) [9]. Amazon Comprehend DetectToxicContent (English only, max 10 × 1 KB segments per call) [10].
• Policy-as-prompt with a general LLM: send content + your guideline text to an EU-hosted frontier model (Mistral, Claude via Bedrock/Vertex EU, Gemini EU, GPT via Azure EU Data Boundary). Use structured outputs for label/severity/rationale, and use it only for the grey zone because of cost and latency.
• Orchestration: queue (Kafka/SQS) → classifier → rules → review UI. ROOST's open-source Osprey rules engine and Coop review console can be combined with cloud APIs [11].
• Choose B when you run your own product with custom policies but have no on-prem constraint.

Path C — Self-host (open source)

• Policy-reasoning guard models: gpt-oss-safeguard-20b (21B MoE, 3.6B active, fits 16 GB VRAM; -120b variant; Apache 2.0) [3]. Qwen3Guard (0.6B/4B/8B, 119 languages, safe/controversial/unsafe tiers, a Stream variant for token-level real-time checks; Apache 2.0) [12]. Llama Guard 4 12B (14 MLCommons hazard categories, text + image; Llama 4 Community License) [13]. IBM Granite Guardian 3.3 8B (Apache 2.0, custom criteria, English only) [14]. ShieldGemma 2B/9B/27B (4 harm types, Gemma licence) [15].
• Fast encoders: Detoxify (BERT/RoBERTa/XLM-R; multilingual model covers 7 languages, not German) [16], or your own fine-tuned XLM-R/mDeBERTa (Path D).
• Serving & tooling: vLLM or SGLang for guard LLMs, ONNX Runtime/Triton for encoders, Osprey + Coop for rules and review [11], Label Studio or Argilla for labelling, Langfuse for tracing.
• Hardware: HW2 (one 24–32 GB GPU) runs gpt-oss-safeguard-20b or Qwen3Guard-8B for grey-zone volumes plus an encoder for the bulk. Pure encoder setups run on HW1 (CPU). Use HW3 for gpt-oss-safeguard-120b or high throughput.

Path D — Custom training

Worth it when you have tens of thousands of past moderator decisions (newsrooms, large forums, games). It also makes sense when the community has its own slang or dialect, or when per-item cost must be minimal at very high volume. Typical recipe: fine-tune an XLM-R/mDeBERTa encoder or a small guard LLM on your own decision history, and use a policy-reasoning LLM to pre-label and to generate hard negatives (quotes, counter-speech, reclaimed terms). Audit the labels for annotator bias first. Retrain or re-calibrate whenever policy changes; the policy-as-prompt approach (stage 2) avoids this for fast-moving rules.

State of the art (2026)

• From fixed taxonomies to policy-following models. gpt-oss-safeguard (released with the ROOST model community) and the custom-criteria modes of Granite Guardian and Azure "custom categories" let operators enforce their rules, not a vendor's taxonomy [3][8][14].
• Multilingual small guards: Qwen3Guard covers 119 languages and adds a "controversial" tier that maps naturally to "send to human" [12]. Streaming classifiers allow moderation during generation and in live chat.
• Open trust-and-safety infrastructure: ROOST (backed by Google, OpenAI, Discord, Roblox and others) publishes production-grade open tools; it reports that Bluesky, Discord and Matrix use Osprey [11].
• Benchmarks: HateCheck/Multilingual HateCheck (functional tests incl. German), ToxicChat, OpenAI Moderation dataset, plus guard benchmarks from model cards (e.g. Granite Guardian 3.3 8B reports aggregate harm F1 0.81, vendor-reported) [14]. Results on public benchmarks transfer poorly to a specific community, so always evaluate on your own labelled sample.
• Limits & failure modes: over-flagging of identity terms, dialects and reclaimed slurs (bias against the groups moderation is meant to protect, a known Detoxify caveat [16]); missing sarcasm, coded language and context across a thread; adversarial obfuscation; German compound words and Swiss/Austrian variants; LLM rationales that sound plausible but cite the wrong clause. Legality (e.g. criminal insult) cannot be decided by a model alone.

Shared resources & combinations

• Same guard-model server (HW2/HW3, vLLM): SEC-010 LLM guardrails, CONV-001 chatbots (input/output filtering), GEN-001 consumer co-creation, CODE-018 evaluation/tracing. One Qwen3Guard or gpt-oss-safeguard deployment can serve all of them.
• Same text-classifier stack (fine-tuned encoders + labelling): LANG-012 routing, LANG-013 sentiment, LANG-018 social listening, LANG-024 promotional compliance review.
• Multimodal trust-and-safety pipeline: VIS-027 image/video moderation and SPEECH-023 voice moderation share the review console (Coop), rules engine (Osprey), statement-of-reasons and Transparency Database integration. ANOM-004 (fake accounts/bots) and ANOM-010 (fake reviews) add account-level signals. ANOM-015 adds AI-generated content provenance.
• Adjacent: LANG-021 misinformation detection, PERS-019 on-device child safety, SEC-014 red teaming of the moderation models themselves.

Effort, cost & KPIs

• Path A: days to a few weeks; SaaS priced per item/month or per seat.
• Path B: 4–8 weeks for API integration, rules, review UI and DSA reporting; dedicated moderation endpoints are cheap or free, while LLM grey-zone calls are the main cost driver.
• Path C: 2–3 months incl. evaluation set, serving and on-call operations; one GPU server.
• Path D: +1–3 months, mainly labelling and bias audit.
• KPIs: precision/recall per category on a labelled holdout (German and other key languages), share auto-decided vs. human-reviewed, time-to-action for severe content, appeal rate and appeal reversal rate, false-positive complaints, moderator hours per 1,000 items, moderator exposure to severe content.

People & change

Who and when: management decides how strict the platform is; the department lead (community or editorial) owns the policy; legal/compliance checks DSA duties (statements of reasons, appeals); IT integrates the queue; the DPO is involved before go-live. Moderators are the key end users and should help write the policy examples from day one.

Works council: yes in practice. Moderation tools log each moderator's decisions, speed and error rate, so they are objectively suitable for performance monitoring (§87(1) no. 6 BetrVG, BAG). Conclude a short works agreement excluding individual performance scoring; inform under §90(1) no. 3 and consider an expert under §80(3). In public bodies the Personalrat is responsible.

Data protection: AVV with the vendor (Art. 28 GDPR); a DPIA is likely when large-scale evaluation of user behaviour or sensitive categories occurs (GUIDE-005).

AI literacy and communication: support staff literacy under Art. 4 AI Act, especially on false positives and bias against dialects (GUIDE-006). Pilot on one comment section with AI only suggesting, moderators deciding.

Typical concerns: fear of being replaced or of psychological burden - AI pre-filters the worst content, humans decide borderline and every removal with appeal (GUIDE-008).

Risks, regulation & compliance

• EU AI Act: content moderation is not an Annex III high-risk category, so a typical deployment is minimal risk (AI literacy, Art. 4). Art. 50 transparency applies only if a chatbot talks to users. Moderation must not slide into prohibited practices: inferring emotions of employees (Art. 5(1)(f)) or biometric categorisation is excluded here. Guard LLMs are GPAI models whose providers carry the GPAI obligations [17].
• DSA (Reg. 2022/2065): notice-and-action (Art. 16), statement of reasons incl. use of automated means (Art. 17), internal complaint handling not solely automated (Art. 20), out-of-court settlement (Art. 21), trusted flaggers (Art. 22), transparency reports incl. description, accuracy and error rates of automated moderation (Art. 15, 24). VLOPs also face systemic-risk assessment and audits (Art. 34–37). In Germany the Bundesnetzagentur is the Digital Services Coordinator (DDG), and the DSA has largely replaced the NetzDG [4][18].
• Other law: TCO Regulation 2021/784 (terrorist content, one-hour removal order), national criminal law on hate speech and insults, the Audiovisual Media Services Directive for video-sharing platforms, and the JMStV/KJM (minor protection in DE).
• GDPR: posts and user histories are personal data, and moderation may reveal special categories (religion, sexual orientation, Art. 9). Legal basis is usually Art. 6(1)(c)/(f). Account bans based solely on automated processing can engage Art. 22, so keep humans in the loop. Minimise retention and use EU-hosted processors with a DPA.
• Fundamental rights & bias: over-removal chills lawful speech, especially from minorities and in dialects. Audit false-positive rates per group and language, and publish clear guidelines.
• Mitigations: a two-stage design with conservative auto-actions, human review of grey zones and all appeals, a versioned policy, a logged rationale per decision, a quarterly bias and accuracy evaluation, and moderator wellbeing measures.

Getting started

1. Write (or update) a concrete content policy with examples per category. It is both the moderator guideline and the LLM prompt.
2. Build a labelled evaluation set of 500–2,000 of your own items, incl. hard cases, German dialects, quotes and counter-speech.
3. Benchmark 2–3 options on it, e.g. a SaaS (Bodyguard/Utopia), Mistral Moderation and a self-hosted Qwen3Guard or gpt-oss-safeguard. If you still use Perspective API, migrate before 31 Dec 2026.
4. Set thresholds per category for auto-allow / auto-hold / human review; integrate a review queue and DSA statement-of-reasons export.
5. Monitor appeal reversals and false positives monthly; feed corrected labels back into the evaluation set and policy.

Sources

1. European Commission — DSA Transparency Database (statistics page, accessed 2026-11) — https://transparency.dsa.ec.europa.eu/
2. Jigsaw — Perspective API (sunset notice) — https://perspectiveapi.com/
3. OpenAI — gpt-oss-safeguard-20b model card — https://huggingface.co/openai/gpt-oss-safeguard-20b
4. Regulation (EU) 2022/2065 (Digital Services Act) — https://eur-lex.europa.eu/eli/reg/2022/2065/oj
5. Bodyguard — contextual moderation platform — https://www.bodyguard.ai/
6. Utopia Analytics — Utopia AI Moderator — https://utopiaanalytics.com/
7. Mistral AI — Moderation & guardrailing docs — https://docs.mistral.ai/capabilities/guardrailing/
8. Microsoft — Azure AI Content Safety overview — https://learn.microsoft.com/en-us/azure/ai-services/content-safety/overview
9. OpenAI — Moderation guide — https://developers.openai.com/api/docs/guides/moderation
10. AWS — Amazon Comprehend trust and safety (toxicity detection) — https://docs.aws.amazon.com/comprehend/latest/dg/trust-safety.html
11. ROOST — Robust Open Online Safety Tools (Osprey, Coop, model community) — https://roost.tools/
12. Qwen — Qwen3Guard-Gen-8B model card — https://huggingface.co/Qwen/Qwen3Guard-Gen-8B
13. Meta — Llama Guard 4 12B model card — https://huggingface.co/meta-llama/Llama-Guard-4-12B
14. IBM — Granite Guardian 3.3 8B model card — https://huggingface.co/ibm-granite/granite-guardian-3.3-8b
15. Google — ShieldGemma model card — https://huggingface.co/google/shieldgemma-2b
16. Unitary — Detoxify / toxic-bert model card — https://huggingface.co/unitary/toxic-bert
17. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
18. European Commission — The Digital Services Act — https://digital-strategy.ec.europa.eu/en/policies/digital-services-act