AI red teaming & adversarial robustness testing of models and LLM applications

LLM applications and agents fail in ways that classic penetration tests do not see: a polite sentence can leak a system prompt, an email can hijack an agent's tools, a crafted input can flip a classifier. AI red teaming systematically attacks the model or application before attackers do. Attacker LLMs generate thousands of probes, judge models score the responses, and humans explore the creative cases. The result is a measurable attack success rate, a fix list and audit evidence.

Problem & value

• Today: many teams ship LLM features after a handful of manual "try to break it" sessions, or rely on the provider's safety claims. Findings are anecdotal, not repeatable, and are lost when the model, prompt or guardrail changes.
• AI value: attack generation and scoring are automated, so the same suite runs on every model upgrade, prompt change or new tool. Coverage grows from dozens to thousands of probes; humans concentrate on novel multi-step attacks and business-specific harms.
• Obligation: providers of general-purpose AI models with systemic risk must perform and document adversarial testing (AI Act Art. 55(1)(a)); high-risk systems must reach appropriate accuracy, robustness and cybersecurity, including resilience to adversarial inputs (Art. 15) [1]. The GPAI Code of Practice (safety and security chapter) points to model evaluation and adversarial testing as expected practice [2].
• Maturity of guidance: OWASP GenAI Red Teaming Guide, NIST adversarial ML taxonomy (AI 100-2) and MITRE ATLAS give shared vocabulary and attack catalogues [3][4][5].

Variants

• Model-level red teaming: foundation-model providers and fine-tuners test safety refusals, CBRN/cyber uplift evaluations, bias and misuse (mostly GPAI providers, large enterprises).
• Application-level red teaming: chatbots, RAG assistants, copilots: prompt injection, data exfiltration, system-prompt leakage, excessive agency, cross-tenant leaks.
• Agent and tool red teaming: indirect injection via documents/web/MCP tool outputs, tool misuse, privilege escalation across tools (links to SEC-013).
• Classical adversarial robustness: evasion, poisoning and model extraction against vision, tabular and fraud models (e.g. fraud or malware classifiers).
• Continuous red teaming in CI/CD: a smaller probe set as a release gate.
• Public sector / SMB: buy a managed assessment before go-live of a citizen-facing or customer-facing chatbot rather than building a team.
• Private use: not applicable.

How it works

1. scope & threat model (SEC-012)
2. attack objectives per risk category
3. probe generation (static libraries + attacker LLM, multi-turn, encodings, languages)
4. execution against target (API / UI / agent with sandboxed tools)
5. scoring (rule checks + LLM judge + human sampling)
6. triage (severity, exploitability, business impact)
7. fix (prompt, guardrails SEC-010, tool permissions, model choice, data)
8. retest
9. report & evidence archive; regression suite into CI/CD

• Attacker models: an LLM rewrites seed goals into jailbreaks, role-plays and multi-turn escalations (PAIR/TAP/crescendo-style strategies) and mutates probes that partly succeed.
• Judges: LLM-as-judge classifies whether a response violated policy; judges are themselves imperfect, so a stratified human sample calibrates them.
• Human-in-the-loop: humans set scope and rules of engagement, review all high-severity findings, perform creative manual attacks, and decide on release. Testing against third-party hosted models requires the provider's terms and authorisation.
• Safety of the test itself: run against staging with test data and sandboxed tools; store harmful outputs under access control; do not test production with real customer data.

Data preparation

Red teaming needs little business data but good context: the system prompt and architecture of the AI application, its tool permissions, the knowledge base it can reach, and policy documents defining what counts as a violation. In an SMB these live in the developer's repository, the vendor's admin console, and in heads rather than documents.

Typical gaps: no written threat model or list of unacceptable outputs, undocumented tool permissions, no staging environment with realistic but synthetic data, and no regression test set from earlier incidents.

Preparation steps: 1. Document the application scope, data flows and tool rights (SEC-012 for the threat model). 2. Create a staging copy with synthetic or pseudonymised data (PREP-022); never test with real customer data. 3. Write down the policy as testable rules and collect first probes plus expected behaviour (PREP-018). 4. Define severity categories (PREP-019). 5. Keep findings as a growing regression suite (PREP-021 for feedback).

Tools: Markdown or a wiki for scope, a Git repository for probes, promptfoo or garak configurations. Rough share of effort: 10-20 percent; the main work is the testing itself. See GUIDE-003 for general readiness.

Implementation paths

Which path for whom

• SMB: Path C — open-source scanners such as garak, PyRIT or promptfoo run on modest hardware, use only test data, and the same eval tooling later serves other AI projects (Baukasten effect); commercial platforms are priced for larger programmes.
• Enterprise: Path B — a custom harness on cloud APIs (attacker and judge models, EU region) integrated into CI/CD and the governance process fits a core, repeated programme better than a single product; Path A for external pentest services on top.
• Developer: Path C — open-source frameworks are the standard tools and can be extended and run locally in CI.
• Public sector: Path C — self-hosted attacker and judge models on sovereign infrastructure avoid sending prompts, system prompts and findings to third parties; buy only certified services.

Path A — Buy (off-the-shelf)

• Commercial platforms and services: Giskard (French, EU; LLM scan and continuous red teaming), Lakera (Swiss; Lakera Red, now part of Check Point), Mindgard, HiddenLayer, Protect AI (Palo Alto Networks), Robust Intelligence (Cisco AI Defense), Promptfoo (enterprise edition; acquired by OpenAI in 2026, open-source core continues), and the AI-security offerings of large consultancies. Hyperscaler features also exist, e.g. Azure AI Foundry AI Red Teaming Agent (built on PyRIT).
• Choose A when: no in-house AI security skills, a go-live deadline, or a customer demands an independent report. Check EU hosting, whether test data leaves your tenant, and attack library update cadence.

Path B — Build on cloud APIs

• Use a frontier model (Claude, GPT, Gemini, Mistral on EU regions) as attacker and judge, with tools such as PyRIT or promptfoo orchestrating. Note that some providers restrict generating attack content; check usage policies and request red-team exemptions where offered.
• Orchestrate in CI (GitHub Actions/GitLab) and push results to a SIEM or ticketing tool.
• Choose B when: you want strong attacker/judge quality and have no GPU capacity.

Path C — Self-host (open source)

• Tools: NVIDIA garak (LLM vulnerability scanner), Microsoft PyRIT (Python Risk Identification Tool), promptfoo (MIT-licensed eval and red-team CLI), DeepTeam/DeepEval, Giskard open-source library, Meta Purple Llama / CyberSecEvals, UK AISI Inspect for evaluations, Adversarial Robustness Toolbox (ART) for classical ML, and datasets such as HarmBench, AdvBench and JailbreakBench [6][7][8].
• Models: open-weight attacker/judge models such as Llama, Qwen or Mistral families served via vLLM or Ollama; safeguard classifiers such as Llama Guard or ShieldGemma as additional judges.
• Hardware: HW2 is enough for scanning mid-size targets with a ~30B quantised judge; HW3 for higher judge quality. Ops effort: one engineer part-time to maintain probes and pipelines.
• Advantage: no sensitive prompts or system instructions leave your premises.

Path D — Custom training

Worth it for large providers: fine-tune a dedicated attacker model on successful past exploits, or train a domain-specific judge on labelled violations (hundreds to thousands of examples from your own policy). Adversarial training of classical models (ART) when evasion is a real threat.

State of the art (2026)

• Automated multi-turn attackers now find jailbreaks against most deployed models; published benchmarks (HarmBench, JailbreakBench) show that no model is robust to all attack classes [7][8].
• Agent attacks dominate: indirect prompt injection through tool outputs and documents is the main practical risk for agentic systems; OWASP lists prompt injection first in its LLM Top 10 [3][9].
• Tool convergence: probe formats are standardised around PyRIT, garak and promptfoo plugins mapped to OWASP and MITRE ATLAS categories; reports map findings to AI Act and ISO/IEC 42001.
• Limits and failure modes: attack success rate depends on the judge and attacker; results are non-deterministic and model-version specific; passing a suite proves little about unseen attacks; LLM judges miss subtle harms and over-flag benign text; coverage of non-English (e.g. German) attacks is often weaker; fixing one probe class can regress another. Treat the number as a trend indicator, not a safety guarantee.

Shared resources & combinations

• Shared tooling and GPU: the same HW2/HW3 server and judge models serve evaluation work in SEC-010 (guardrail tuning), and attacker LLM skills overlap with SEC-007 (infrastructure pentests).
• Upstream: SEC-012 delivers the threat model that scopes tests; SEC-013 supplies the inventory of AI systems and agents to test; SEC-016 covers malicious model artefacts.
• Downstream: SEC-010 receives new guardrail rules from findings; SEC-022 consumes the evidence for AI Act documentation.

Effort, cost & KPIs

• Path A: days to set up, subscription or per-engagement pricing; managed assessments take weeks.
• Path B: 2-6 weeks for a CI-integrated pipeline; cost is mainly API tokens for attacker and judge calls (thousands of multi-turn conversations per run).
• Path C: 1-3 months to mature; GPU server plus part-time engineer.
• KPIs: attack success rate per category and version, number of critical findings open, mean time to remediate, judge-human agreement, coverage of OWASP/ATLAS categories, share of releases gated by the suite, regression rate.

People & change

Involve management first for scope, budget and authorisation, since testing hosted models needs the provider's permission. The department lead who owns the AI application and IT security define rules of engagement together; IT provides the staging environment; the data protection officer confirms that no real personal data enters tests and decides about a DPIA for the application itself (GUIDE-005); legal checks provider terms and liability.

Works council: no, as a rule. The tests target systems, not employees, so §87(1) no. 6 BetrVG is normally not triggered. Exceptions: if human red-team exercises include social engineering of staff, or results are attributed to individual developers, involve the works council, and inform it early about AI projects under §90(1) no. 3 (GUIDE-004). In public bodies check the staff council.

Frame red teaming as quality assurance, not blame: findings are about the system. Choose one application with moderate risk as pilot (GUIDE-009) and keep the first report short with concrete fixes. Support AI literacy of developers and application owners with a workshop on prompt injection and jailbreaks (Art. 4 AI Act, GUIDE-006, GUIDE-013). Typical concern: fear of delaying launches; answer with automated regression checks in CI. Humans set scope, review all high-severity findings and decide on release.

Risks, regulation & compliance

• EU AI Act: the activity itself is minimal risk. It generates evidence for obligations elsewhere: Art. 15 (robustness, cybersecurity) for high-risk systems, Art. 55(1)(a) for GPAI with systemic risk, Art. 4 AI literacy for the team [1]. Red teaming should also check for Art. 5 prohibited behaviour of the target (e.g. manipulative techniques) and Art. 50 transparency.
• GDPR: attacks may extract personal data from training data or RAG corpora; use synthetic or test data, treat transcripts containing leaked data as personal data, and apply data minimisation and retention limits. Data processing agreements apply if a vendor runs the tests.
• Security and law: testing third-party services without permission may breach terms and computer-crime law; obtain written authorisation and rules of engagement. NIS2 and DORA (resilience testing) can make such testing part of a broader obligation.
• Risks and mitigations: harmful test outputs (restrict access, no wide sharing), false confidence (state coverage limits), leaking system prompts to external SaaS (prefer Path C for sensitive prompts), judge bias (human calibration).

Getting started

1. Inventory the AI systems in scope (SEC-013) and rank them by data access and tool power.
2. Write rules of engagement: scope, test environment, data, authorisation, escalation.
3. Run garak or promptfoo against one staging chatbot with OWASP LLM Top 10 plugins; record baseline.
4. Fix top findings (prompt, tool permissions, guardrails SEC-010), then rerun to confirm.
5. Put a reduced suite into CI as a release gate and schedule a quarterly human-led exercise.

Sources

1. Regulation (EU) 2024/1689 (AI Act), Art. 15 and Art. 55 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
2. European Commission — General-Purpose AI Code of Practice — https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
3. OWASP GenAI Security Project — https://genai.owasp.org/
4. NIST AI 100-2e2025 Adversarial Machine Learning taxonomy — https://csrc.nist.gov/pubs/ai/100/2/e2025/final
5. MITRE ATLAS — https://atlas.mitre.org/
6. NVIDIA garak — https://github.com/NVIDIA/garak
7. Microsoft PyRIT — https://github.com/Azure/PyRIT
8. promptfoo red teaming docs — https://www.promptfoo.dev/docs/red-team/
9. HarmBench — https://www.harmbench.org/
10. Giskard — https://www.giskard.ai/