AI threat modelling & secure design review

Threat modelling is the cheapest security activity and the one most often skipped, because it needs scarce experts and goes stale after the first workshop. LLMs can read architecture documents and diagrams, draft STRIDE or LINDDUN threat models, propose mitigations and re-run when the design changes. The expert moves from blank-page author to reviewer, so every significant design gets a first-pass model before code exists.

Problem & value

• Today: threat modelling is done in workshops by a few security architects, documented in slides or spreadsheets, and rarely updated. Most teams never model at all; development moves faster than the security team can review.
• AI value: a draft model in minutes instead of days, consistent coverage of a threat catalogue, re-assessment on every design change (pull request on an architecture file), and coverage of AI-specific threats (prompt injection, tool misuse, data poisoning) that classic checklists miss.
• Regulatory pull: the Cyber Resilience Act (Regulation (EU) 2024/2847) requires manufacturers of products with digital elements to base design and development on a cybersecurity risk assessment (Art. 13, Annex I); main obligations apply from 11 Dec 2027 [1]. NIS2 (Directive (EU) 2022/2555, Art. 21) requires risk-based measures including secure development for covered entities [2]. A living threat model is natural evidence for both.
• Evidence quality: academic evaluations of AI-assisted tools find they recover many threats of manual reference models but can hallucinate assets that are not in the system, so they are not yet suitable as standalone replacements for experts [3][4]. No credible published time-saving percentage is cited here.

Variants

• Product security (CRA): hardware/software vendors model devices and cloud back-ends.
• Internal application review: architecture-board intake; the AI pre-reads design documents.
• Agentic and LLM application modelling: agents, tools, memory and MCP servers as assets, using agent-specific frameworks (ASTRIDE, CSA MAESTRO, OWASP agentic guidance) [4][5].
• Privacy threat modelling: LINDDUN for GDPR data-protection-by-design and DPIA input.
• Cloud/IaC review: derive the data-flow model directly from Terraform or Kubernetes files.
• SMB: a lightweight checklist-driven assistant run by the lead developer instead of a security architect.

How it works

1. design artefacts (docs, diagrams, user stories, IaC)
2. extraction (VLM reads diagram
3. components, data flows, trust boundaries as structured JSON)
4. human confirms/corrects the system model [HITL 1]
5. threat generation (STRIDE per element / LINDDUN / agent-specific categories; RAG over catalogues)
6. attack trees & abuse cases; mapping to CAPEC, ATT&CK, ATLAS, CWE
7. risk rating and mitigation proposals (controls, ASVS / NIS2 / CRA Annex I requirements)
8. security expert review, accept/reject/adjust [HITL 2]
9. tickets in backlog + model stored as code (e.g. OWASP pytm, Threagile, Threat Dragon JSON)
10. trigger on design change (PR diff on architecture files)
11. delta assessment

• System model first: the most important step. Errors in extracted components propagate into every threat, so the model is confirmed by a human before threats are generated.
• Grounding: retrieval over curated catalogues (STRIDE, CAPEC, ATLAS, OWASP LLM Top 10) and the organisation's own control library reduces invented threats; every threat must cite a component.
• Threat-model-as-code gives diffs and review in version control.
• Human-in-the-loop: experts approve risk ratings and accepted risks; the AI never closes a threat.

Data preparation

The inputs are architecture descriptions, diagrams, user stories and infrastructure-as-code. They live in wikis (Confluence, SharePoint), drawing tools (draw.io, Visio, PowerPoint), the Git repository and sometimes only on whiteboard photos or in people's heads.

Typical gaps: diagrams exist only as images or outdated slides, no trust boundaries or data classifications drawn, user stories without security requirements, scattered knowledge about legacy systems, and documents on file shares with unclear currency.

Preparation steps: (1) store diagrams as diagram-as-code (Mermaid, Structurizr, draw.io XML) next to the code (PREP-014); (2) mark data classes and trust boundaries in the models (PREP-019); (3) collect the organisation's control library and previous threat models as a RAG source (PREP-013, PREP-016); (4) OCR and convert old Visio or PDF documentation (PREP-006, PREP-007); (5) define a glossary of component names (PREP-015); (6) build a few reviewed reference models as an evaluation set (PREP-018).

Tools: Git, draw.io, Mermaid, Paperless-ngx or any OCR for old PDFs. Rough share of effort: about 30 percent, mostly the one-off work of documenting the system model, which is useful beyond this use case. See GUIDE-003.

Implementation paths

Which path for whom

• smb: B, because few SMBs have a security architect and a thin script or workflow calling a cloud LLM over their own design documents beats the dedicated threat-modelling products, which assume a security team.
• enterprise: C, because architecture diagrams, infrastructure code and known weaknesses are trade secrets and a self-hosted model on shared AI platform hardware keeps them in-house.
• developer: B, because cloud APIs plus open tools such as OWASP pytm, Threagile or Threat Dragon integrate into pull-request workflows with little effort.

Path A — Buy

• IriusRisk (Spain/EU-rooted; threat modelling platform with AI-assisted model creation and CRA-oriented content), ThreatModeler, Microsoft Threat Modeling Tool (free, not AI), SD Elements (Security Compass), Apiiro (AI Threat Modeling from design documents), Cycode and similar ASPM tools (design-risk features vary and are vendor-reported). Verify current AI features with the vendor.
• Choose A when: many teams, need governance, workflow, reporting and audit trail, and integration with Jira and CI.

Path B — Build on cloud APIs (recommended)

• Models: a strong reasoning LLM plus a multimodal model for diagrams (Claude, Gemini, GPT, Mistral in EU regions; Mistral is an EU provider). Use structured outputs against a schema (components, flows, trust boundaries, threats).
• Orchestration: a small pipeline in CI; trigger on changes to docs/architecture/** or IaC; post the threat delta as a PR comment and open tickets.
• Starting point: STRIDE GPT, open source, supports several providers and agentic codebase analysis with OWASP agentic and MITRE ATT&CK/ATLAS mapping [3].
• Choose B when: you have a platform or security engineering team and design documents are confidential but cloud processing under a DPA with EU region is acceptable.

Path C — Self-host

• Models: open-weight reasoning/instruct models served with vLLM or Ollama, e.g. Qwen3, gpt-oss, Llama, Mistral families; a VLM such as Qwen-VL for diagrams. Architecture documents are among the most sensitive IP, so local inference is a real argument here.
• Stack: STRIDE GPT with a local model, OWASP pytm or Threagile for model-as-code, a vector store over catalogues, n8n or CI jobs.
• Hardware: HW3 for reasoning-grade models (~30-120B); HW2 works for smaller models with more human correction.

Path D — Custom training

Rarely justified. Possible: fine-tune on an organisation's past reviewed threat models to match its taxonomy and risk scales; needs hundreds of approved models, which few organisations have. RAG over past reviews usually achieves most of the benefit.

State of the art (2026)

• Agent-specific extensions of STRIDE: ASTRIDE adds a category for AI-agent attacks (prompt injection, unsafe tool invocation) and uses VLMs plus an LLM reasoning step on architecture diagrams [4]. STRIDE-AI (2026 preprint) combines STRIDE with NIST AI RMF and the OWASP LLM Top 10 [6]. CSA MAESTRO provides a layered model for agentic systems [5].
• Diagram-to-model with VLMs works for clean diagrams; messy whiteboard photos and diagrams without trust boundaries degrade results.
• Limits and failure modes: hallucinated components, generic threats ("attacker steals data"), missed business-logic and cross-system threats, inconsistent risk scoring between runs, and false assurance from a polished-looking output. Mitigate with human-confirmed system models, citation of components, repeated runs and expert sign-off. Model quality depends on the quality of the input documentation.

Shared resources & combinations

• Same stack: reasoning LLM + VLM + RAG as in SEC-010 (guardrails), SEC-014 (AI red teaming), SEC-004 (code scanning), CODE-006 (architecture documentation generation).
• Upstream/downstream: the threat model feeds SEC-007 (test scoping for pentests), SEC-014 (red-team scenarios for LLM apps), SEC-009 (compliance evidence for NIS2/CRA) and SEC-022 (AI system documentation and risk classification). SEC-013 supplies the inventory of AI agents to model. CODE-003 reviews code later against the mitigations defined here.

Effort, cost & KPIs

• Path A: weeks to onboard; per-seat or per-application licences.
• Path B: 2-6 weeks for a CI-integrated prototype; token cost is small (a design review is tens of thousands of tokens); the main cost is expert review time and prompt/catalogue curation.
• Path C: 1-3 months incl. GPU server and evaluation.
• KPIs: share of new designs with a threat model before implementation; expert edit rate (threats accepted vs rejected vs hallucinated); mitigations converted to tickets and closed; time from design change to updated model; design-stage findings vs findings later in pentest (SEC-007); audit evidence completeness for CRA/NIS2.

People & change

The project is led by the head of security or architecture; developers and product owners are the daily users and must confirm the extracted system model. IT supplies repository access, management sponsors the topic and data owners release documentation of critical systems. Involve the information security officer for classification of design documents.

Works council: no in the normal case, because the tool analyses systems, not people, and the output is not used to rate individuals. Check again if outputs are linked to individual developers' performance or commits are analysed per person; then §87(1) no. 6 BetrVG may apply. Still inform the council about the introduction, as §90(1) no. 3 BetrVG covers planned AI-supported work processes (GUIDE-004).

Data protection: usually no personal data; if designs include real personal data flows, involve the DPO, and sign an AVV with any cloud provider (GUIDE-005). Design documents are confidential, so the AI policy must state which tool is approved (GUIDE-007).

AI literacy: AI Act Art. 4 asks employers to support staff AI literacy; hold a short workshop on reviewing AI-generated threats critically (GUIDE-006). Run a pilot on one new design with an experienced security reviewer. Typical concern: "the AI will produce noise or replace security experts". Address it by measuring accepted versus rejected threats, and by making clear that experts approve risk ratings and the AI never closes a threat (GUIDE-009).

Risks, regulation & compliance

• EU AI Act: minimal risk. A design-review assistant is not an Annex III use; AI literacy (Art. 4) applies. If it models an AI system, the output can support Art. 9 risk management and Art. 15 cybersecurity duties of a high-risk system, but the tool itself stays minimal.
• GDPR: designs rarely contain personal data, but wikis and user stories may include names and customer data; minimise and redact before sending to external APIs. Where models describe data flows of personal data, LINDDUN output can feed the DPIA (Art. 35 GDPR).
• CRA and NIS2: the threat model does not prove compliance; manufacturers remain responsible for risk assessment, documentation and conformity. Keep human-approved versions as technical documentation.
• Confidentiality: architecture documents and vulnerabilities are crown-jewel information; use a DPA, no training on inputs, EU region or self-hosting, and restrict access to outputs.
• Prompt injection: documents ingested from wikis can contain instructions; treat inputs as untrusted, give the pipeline read-only access and no autonomous ticket closure.
• Over-reliance: automation bias; require named reviewers for high-risk systems.

Getting started

1. Pick one upcoming system design and write its architecture as diagram-as-code or structured text.
2. Run STRIDE GPT or a simple prompt with your own schema; compare against a manual workshop result.
3. Measure hallucinated and missed threats; tune prompts, add catalogue RAG, insist on component citations.
4. Add a CI trigger on architecture files and push accepted mitigations to the backlog.
5. Extend to LLM/agent applications with agent-specific categories and link to SEC-014 tests.

Sources

1. Regulation (EU) 2024/2847 (Cyber Resilience Act) - https://eur-lex.europa.eu/eli/reg/2024/2847/oj
2. Directive (EU) 2022/2555 (NIS2) - https://eur-lex.europa.eu/eli/dir/2022/2555/oj
3. STRIDE GPT (PyPI/GitHub, open source) - https://pypi.org/p/stride-gpt
4. ASTRIDE: A Security Threat Modeling Platform for Agentic-AI Applications - https://arxiv.org/html/2512.04785v1
5. Cloud Security Alliance - Agentic AI Threat Modeling Framework: MAESTRO - https://cloudsecurityalliance.org/articles/agentic-ai-threat-modeling-framework-maestro
6. STRIDE-AI: A Threat Modeling Framework for Generative AI Security Assessment - https://arxiv.org/abs/2605.17163
7. Aalborg University thesis comparing IriusRisk, ThreatCanvas and STRIDE GPT - https://projekter.aau.dk/projekter/files/782834690/thesis_threats_and_ai.pdf
8. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
9. Apiiro — Introducing AI Threat Modeling (vendor blog) — https://apiiro.com/blog/introducing-ai-threat-modeling/