LLM application security: prompt-injection defence & AI guardrails

Every company that connects an LLM to its documents, email or tools creates a new attack surface: text from an attacker can be read by the model as an instruction. Prompt injection is ranked first in the OWASP Top 10 for LLM Applications. No single filter solves it, so the practical answer is defence in depth: input/output guardrails, least-privilege tools, isolation of untrusted content, human approval for risky actions and continuous monitoring and testing.

Problem & value

• Today: teams ship copilots, RAG assistants and agents after a quick system-prompt tweak. Attacks are either direct (a user tells the bot to ignore its rules) or indirect (a hidden instruction in a web page, PDF, email or ticket that the model processes). Consequences: leaked system prompts and confidential data, manipulated answers, unauthorised tool actions (sending mail, writing to CRM, running code), data exfiltration through rendered links or images, and costly runaway usage.
• AI value: a guardrail layer is itself ML-based (small classifiers and LLM judges) and detects semantic attacks that regular expressions cannot. It also adds PII redaction, topic and toxicity control and groundedness checks, and produces the logs needed for audit.
• Honest limit: NIST's adversarial machine learning taxonomy states that mitigations for prompt injection are incomplete and that there is no foolproof defence [2]. Architecture (what the model is allowed to do) matters more than detection accuracy.

Variants

• Customer-facing chatbot: abuse, jailbreaks, brand-damaging output, PII leakage, off-topic use.
• Internal RAG assistant: indirect injection via documents, access-control leakage across users (document-level permissions must be enforced at retrieval time).
• Agents with tools / MCP servers: highest risk; tool poisoning, over-privileged tokens, confused-deputy actions. See OWASP's guidance on excessive agency [1].
• Coding assistants and browser agents: untrusted repository or web content steering the agent.
• Public sector / regulated: on-premises guardrails, logging and documented human oversight.
• Private use: limited to caution with AI browsers and mail assistants; no dedicated product category.

How it works

1. user input / retrieved content / tool result
2. input guardrails (injection & jailbreak classifier, PII detection, policy/topic check)
3. LLM with hardened prompt (instruction hierarchy, untrusted content delimited, no secrets in prompt)
4. tool-call gate (allow-list, argument validation, scoped short-lived credentials, human approval for sensitive actions)
5. output guardrails (PII/secret redaction, toxicity, groundedness, link/markdown/exfiltration filter)
6. response + structured logging (SIEM) + periodic red-team regression tests

• Detection models: small encoder classifiers (e.g. Prompt Guard, Lakera-style detectors) and safety LLMs (Llama Guard, Granite Guardian, gpt-oss-safeguard, Qwen3Guard) classify inputs and outputs against a policy in milliseconds to a few hundred milliseconds.
• Architectural controls (the strongest layer): separate privileged and quarantined models, never let one agent combine access to private data, exposure to untrusted content and an outbound channel (the "lethal trifecta" idea), restrict tools per task, and use design patterns such as plan-then-execute or the CaMeL approach that tracks data provenance [5].
• Human-in-the-loop: approval for irreversible actions (payments, external email, deletions), review of blocked-but-uncertain cases, and ownership of the guardrail policy by security plus the business owner.

Data preparation

This use case needs little business data, but it needs good test and policy data. The relevant material is the system prompt and tool definitions (usually in the code repository), the permissions of connected data sources (ERP, DMS, SharePoint, mail) and the application logs and traces.

Typical gaps: overshared SharePoint or file-share folders that a RAG bot can read on behalf of everyone, no data classification ("what may go into which AI tool"), no attack test set, logs that either do not exist or contain unredacted personal data, and API keys with too broad rights.

Preparation steps: (1) clean up permissions of every source the assistant can read (PREP-023); (2) define classification rules and a deny-list of secrets and PII patterns (PREP-024); (3) build a red-team and regression set of known injection prompts, together with domain experts (PREP-018); (4) collect real misuse cases and corrections as labels (PREP-021); (5) pseudonymise logs before storing or analysing them (PREP-022).

Tools: open prompt-injection test suites such as garak or promptfoo, the logging stack already in use, and a small spreadsheet for the test set. Rough share of effort: about 10 to 20 percent; permissions clean-up is the biggest item and benefits all other AI use cases.

Implementation paths

Which path for whom

• enterprise: C, because guardrail and detection models run inline on every prompt and a shared internal AI platform makes self-hosted safety models (e.g. Llama Guard, Granite Guardian) cheaper and keeps prompts inside the network.
• developer: B, because managed safety APIs and open guardrail libraries plug directly into the application code with little ops effort.
• smb: B, because the guardrails must be built into the company's own chatbot or agent and no product can replace the application-specific permission design.
• public: C, because prompts, retrieved documents and logs often contain citizen data and procurement rules favour sovereign, self-operated components.

Path A — Buy (off-the-shelf)

• AI security platforms / gateways: Lakera Guard (Check Point; Swiss-founded), Protect AI Guardian (Palo Alto Networks), HiddenLayer, Prompt Security (SentinelOne), Straiker, Zenity (agent security), Cisco AI Defense, Palo Alto Prisma AIRS, CrowdStrike (Pangea), F5 AI guardrails (CalypsoAI). EU/European options include Lakera (CH), Mindgard (UK, testing) and Giskard (FR, testing/evaluation).
• Built into model platforms: Azure AI Content Safety with Prompt Shields, Amazon Bedrock Guardrails, Google Model Armor, Mistral moderation API (EU provider), Cloudflare AI Gateway guardrails.
• Choose A when: many teams ship AI apps and you need central policy, logging and SIEM integration without building it. Check EU hosting, data retention of prompts, and DPA terms.

Path B — Build on cloud APIs (recommended default for SMBs)

• Turn on the guardrail service of the model platform you already use (Prompt Shields, Bedrock Guardrails, Model Armor) in an EU region, and route all calls through one gateway (LiteLLM, Kong, Azure API Management, Cloudflare AI Gateway) that does logging, rate and budget limits.
• Use structured outputs and function-calling schemas so the model cannot emit free-form commands.
• Add Microsoft Presidio for PII detection and anonymisation before data leaves your tenant.
• Enforce per-user permissions in the retrieval layer and give agents scoped, short-lived tokens.

Path C — Self-host (open source)

• Frameworks: NVIDIA NeMo Guardrails, Guardrails AI, LLM Guard (Protect AI), LlamaFirewall (Meta, with PromptGuard 2, Agent Alignment Checks, CodeShield), Rebuff, Microsoft Presidio.
• Models: Llama Guard 4, Llama Prompt Guard 2, IBM Granite Guardian, Qwen3Guard, gpt-oss-safeguard, ShieldGemma; served with vLLM or Ollama.
• Testing: garak, PyRIT and promptfoo for automated attack suites (see SEC-014).
• Hardware: HW1 for classifier-only guardrails (CPU is enough for encoder models); HW2 for a small safety LLM next to the main model. Ops effort: tuning thresholds and keeping policies current.

Path D — Custom training

Fine-tune a small classifier on your own attack logs and policy violations when generic detectors produce too many false positives for domain language (legal, medical, multilingual German/EU languages). Needs a few thousand labelled benign and malicious examples, mostly from red-team runs and production logs.

State of the art (2026)

• Consensus: prompt injection is not "solved"; vendors and researchers treat it as a residual risk to be limited through architecture. Adaptive attacks in the literature repeatedly bypass published detectors, so detector scores alone are not proof of safety [2][5].
• Agent focus: OWASP published a Top 10 for agentic applications alongside the LLM Top 10 (2025 edition), covering excessive agency, tool misuse and memory poisoning; MCP security (tool poisoning, token scope) is now a standard review item [1].
• Provenance-based designs (CaMeL, dual-LLM patterns) give provable properties for narrow tasks at the cost of flexibility.
• Benchmarks: AgentDojo and similar suites measure attack success against agents; use them for regression testing, not as certification.
• Failure modes: false positives blocking legitimate users, multilingual and encoded bypasses, multi-turn escalation, guardrail latency and cost, and guardrails that themselves are LLMs and can be injected.

Shared resources & combinations

• Shared stack: guardrails + llm-small + text-classifier; the same safety model and gateway serve all internal AI apps.
• Combines with: SEC-014 (red teaming feeds guardrail tuning), SEC-012 (threat modelling before build), SEC-013 (inventory and posture of agents), SEC-017 (PII/DLP), SEC-022 (AI governance and EU AI Act documentation), SEC-001 (SOC consumes guardrail alerts), AGENT-001 (agent design).

Effort, cost & KPIs

• Path A: days to weeks; typically licensed per request or per application.
• Path B: 2-6 weeks for gateway, policies, logging and tests; platform guardrail pricing is usage-based.
• Path C: 1-3 months plus ongoing tuning; GPU optional.
• KPIs: attack success rate on a fixed test suite, false-positive rate on real traffic, added latency, share of AI apps behind the gateway, share of agent tools with scoped credentials, mean time to update a policy, number of blocked PII leaks.

People & change

Management decides the risk appetite and should approve the AI policy (GUIDE-007). The information security officer and IT operate the guardrails; department leads and end users report blocked or suspicious cases, and the DPO checks that logged prompts are lawful and minimised (GUIDE-005). Involve the works council early.

Works council: yes in most cases. Guardrails log prompts, flag users and can reveal behaviour, so the system is objectively suited to monitor employees and §87(1) no. 6 BetrVG applies (Federal Labour Court: suitability is enough); §90(1) no. 3 adds the duty to inform on AI-supported work processes. Agree in a works agreement that logs serve security only, are pseudonymised and are not used for performance review (GUIDE-004). Public sector: staff council.

AI literacy: AI Act Art. 4 requires measures that support staff AI literacy; train users on prompt injection and not pasting secrets, and developers on secure agent design (GUIDE-006, GUIDE-013). Pilot with one internal assistant and a small team of volunteers. Typical concerns: "guardrails block legitimate work" (provide a feedback channel and measure false positives) and "I am being watched" (explain log purpose and retention). Humans approve irreversible tool actions and own the guardrail policy.

Risks, regulation & compliance

• EU AI Act: the guardrail layer itself is minimal risk. The protected application may be limited or high risk (e.g. HR or credit scoring); then Art. 15 requires accuracy, robustness and cybersecurity, explicitly including resilience against attempts to alter use or outputs via manipulated inputs (adversarial examples, model evasion, data poisoning) [3]. Chatbots need Art. 50 transparency. GPAI providers with systemic risk have additional cybersecurity duties. Guardrails must not implement Art. 5 prohibited practices (e.g. emotion inference of employees through chat monitoring).
• GDPR: prompts and logs contain personal data; apply data minimisation, short retention, DPA with guardrail vendors, access control on logs, and note that employee monitoring via logs needs a legal basis and works-council involvement in Germany. A leaked-data incident via injection can be a personal data breach (Art. 33 notification within 72 hours).
• NIS2 / DORA / Cyber Resilience Act: AI apps are part of the in-scope ICT risk management and incident reporting; BSI and ENISA publish guidance on LLM and generative AI risks [4].
• Mitigations: layered controls, least privilege, human approval, red-team regression tests, and incident playbooks for AI misbehaviour.

Getting started

1. Inventory your LLM applications and rate each on the three factors: private data access, untrusted content exposure, outbound actions. Fix the apps with all three first.
2. Route all model calls through one gateway with logging, rate limits and PII redaction.
3. Enable a managed guardrail service (or LLM Guard / NeMo Guardrails) and measure false positives on one week of real traffic before enforcing.
4. Reduce agent permissions: allow-listed tools, scoped tokens, human approval for sensitive actions.
5. Build a regression suite from the OWASP Top 10 for LLM Applications with garak or promptfoo and run it in CI on every prompt or model change.

Sources

1. OWASP — Top 10 for LLM Applications and GenAI Security Project — https://genai.owasp.org/llm-top-10/
2. NIST — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (AI 100-2 E2025) — https://csrc.nist.gov/pubs/ai/100/2/e2025/final
3. Regulation (EU) 2024/1689 (AI Act), Art. 15 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
4. ENISA — Multilayer Framework for Good Cybersecurity Practices for AI — https://www.enisa.europa.eu/publications/multilayer-framework-for-good-cybersecurity-practices-for-ai
5. Debenedetti et al. — Defeating Prompt Injections by Design (CaMeL) — https://arxiv.org/abs/2503.18813
6. NVIDIA NeMo Guardrails — https://github.com/NVIDIA/NeMo-Guardrails
7. Meta — LlamaFirewall — https://github.com/meta-llama/PurpleLlama/tree/main/LlamaFirewall
8. Microsoft — Azure AI Content Safety Prompt Shields — https://learn.microsoft.com/azure/ai-services/content-safety/concepts/jailbreak-detection
9. Microsoft Presidio — https://github.com/microsoft/presidio