Phishing & email security

Email remains the most common entry point for credential theft, ransomware and payment-diversion fraud. Classic gateways match known-bad signatures; attackers now use LLMs to write fluent, personalised lures in any language and to vary every message. Modern email security combines authentication standards with behavioural AI (who normally writes to whom, about what), language models that read intent, and automated triage of user reports, so that novel attacks are caught without a known signature.

Problem & value

• Today: a secure email gateway (SEG) or the mail platform's built-in filter handles spam and known malware; staff report suspicious mail to a mailbox that a small IT team reads by hand. Targeted BEC (fake CEO, supplier or IBAN change) has no malicious link or attachment and slips through signature-based filters.
• AI value: (1) detect socially engineered mail with no technical indicator by modelling normal communication behaviour and reading the intent of the text; (2) read QR codes, images and PDFs that hide links; (3) remove already-delivered messages in all mailboxes when a campaign is identified; (4) answer user reports within minutes instead of hours.
• Threat context: ENISA's Threat Landscape reports consistently list phishing among the leading initial-access vectors in the EU and describe the growing use of generative AI in social engineering [1]. The FBI IC3 annual reports rank BEC among the costliest cybercrime categories [2]. Vendor-reported detection rates should be tested on your own mail, not taken from marketing.

Variants

• SMB: mail-platform native protection (Microsoft 365 / Google Workspace) plus DMARC enforcement; optionally one add-on.
• Enterprise: API-based "integrated cloud email security" (ICES) next to or instead of a SEG, with SOAR integration and auto-remediation.
• Public sector / critical entities: NIS2-driven controls, sovereign or on-prem filtering, German/EU-hosted gateways.
• Beyond email: collaboration tools (Teams, Slack), SMS/QR phishing ("quishing"), voice phishing; deepfake-based CEO fraud is SEC-011.
• Outbound / data leakage: misdirected-recipient warnings and DLP (SEC-017).
• Private use: consumer providers' built-in filters plus passkeys; no dedicated product needed.

How it works

1. inbound mail
2. authentication checks (SPF, DKIM, DMARC, ARC; BIMI optional)
3. reputation + URL/attachment detonation (sandbox, SEC-003)
4. feature extraction (header anomalies, lookalike domains, rendered page screenshot, QR/OCR on images)
5. behavioural model (sender-recipient graph, tone, request type: payment, credentials, gift cards)
6. LLM / classifier intent verdict + explanation
7. action: deliver | warning banner | quarantine | post-delivery removal user report button
8. automated triage (cluster, verdict, reply to reporter)
9. analyst on ambiguous cases (SEC-001)

• Key AI components: fine-tuned encoder classifiers for text and URL features; graph/anomaly models on communication history; vision models to render and judge landing pages and read QR codes; LLMs for intent classification and for written explanations to users and analysts.
• Human-in-the-loop: analysts review borderline quarantine items and every auto-remediation rule at rollout; finance staff confirm payment-detail changes out-of-band by phone regardless of the verdict.

Data preparation

Email security products mostly work out of the box, so data preparation is light. What matters: the mail platform (Microsoft 365, Google Workspace, own mail server), the authentication setup (SPF, DKIM, DMARC), the user directory, and a collection of reported phishing mails and false positives from the past months. In an SMB these live in mailboxes, the helpdesk ticket tool and sometimes only in staff memory.

Typical gaps: no central place for reported mails, missing DMARC, outdated user and group lists, no list of high-risk roles (finance, HR, management), and mixed-up distribution lists.

Preparation steps: - Create a shared report mailbox or a report button and collect reported mails with their verdicts (see PREP-020). - Maintain a list of VIPs and payment-relevant roles for impersonation protection. - Fix SPF, DKIM and DMARC and clean the user directory. - Build a small test set of known phishing and legitimate mails, including internal newsletters, to check false positives (see PREP-018). - Define which mail content may be analysed by external AI services (see PREP-024).

Rough effort: about 10 to 15 percent of the project. Most time goes into policy decisions and tuning, not data cleaning. For a self-built pipeline (paths B and C), pseudonymise mail bodies before sending them to a model (see PREP-022).

Implementation paths

Which path for whom

• SMB: A - AI-based email security in Microsoft Defender, Google Workspace or an EU secure email gateway is mature, standard and affordable.
• Enterprise: A - email protection is a commodity function; the leading products already include AI detection, so building your own rarely pays off.
• Developer: C - open-source mail filters with a self-hosted classifier suit experimentation and own mail infrastructure.
• Public: B - authorities can add AI triage of reported phishing on EU-hosted or sovereign-cloud APIs, keeping the mail gateway itself as a product.

Path A — Buy (recommended)

• Platform-native: Microsoft Defender for Office 365 (Plan 2, with automated investigation and response), Google Workspace Gmail protections [3].
• ICES / API-based: Abnormal AI, Darktrace /EMAIL, Proofpoint (Core Email Protection, Nexus), Mimecast, Sublime Security, Check Point Harmony Email, Cisco Secure Email, Trend Micro. Characteristic: connect via Graph/Gmail API in minutes, no MX change.
• EU / sovereign options: Hornetsecurity (DE, 365 Total Protection), Retarus (DE), Vade (FR, now part of Hornetsecurity), Libraesva (IT), SEPPmail (CH, mail encryption plus filtering), Mail-in-Black style German gateway services; check EU data residency and sub-processors.
• Choose A when: you have no security-engineering team; run a proof-of-value with your own historic mail and a red-team sample before purchase.

Path B — Build on cloud APIs

• Use mail APIs (Microsoft Graph, Gmail API) to pull messages, call a frontier or EU model (Claude, Mistral, GPT, Gemini in EU regions) with structured output for "intent + risk + reasons", and feed verdicts to SOAR.
• Typical use: user-reported phishing triage and analyst assistance, not inline blocking (latency and cost). Strip or pseudonymise personal data before sending where possible.
• Tooling: n8n, Power Automate, Tines, Torq; URL scanning with urlscan.io or VirusTotal APIs.

Path C — Self-host (open source)

• Gateway layer: Rspamd, SpamAssassin, Postfix with OpenDMARC / opendkim; ClamAV; sandbox via CAPE or similar (SEC-003); Wazuh/Security Onion for correlation.
• AI layer: small open-weight LLMs (Qwen, Llama, Mistral, Gemma families) served with vLLM or Ollama for reported-mail triage; fine-tuned encoder models (DistilBERT/DeBERTa class) for inline text classification; PhishTank / OpenPhish / abuse.ch feeds for URLs.
• Hardware: HW1 for encoders and rules; HW2 for a ~8-30B triage LLM. Effort: continuous tuning, no vendor intelligence network.

Path D — Custom training

Worthwhile for large providers or very specific language/sector lures. Needs thousands of labelled phish and legitimate mails from your own tenant, which carry heavy GDPR and works-council constraints. Public corpora (e.g. Nazario, Enron-derived sets) are dated and not representative of LLM-written lures.

State of the art (2026)

• Behavioural + semantic detection has replaced indicators-only detection for BEC; the leading vendors build per-tenant models of communication patterns and add LLM-based intent analysis.
• Post-delivery remediation and API deployment are standard; Gartner's market guidance treats ICES as a distinct category complementing the platform filter and SEGs.
• LLM-generated attacks: research shows LLM-written spear phishing can match human experts in click-through rates [4]; defenders respond with intent models that do not depend on spelling or grammar cues, which are no longer reliable signals.
• Authentication enforcement: since 2024 Google and Yahoo require SPF/DKIM/DMARC for bulk senders, and Microsoft followed for consumer Outlook; DMARC at p=reject is the cheapest strong control against domain spoofing [5].
• Limits: false positives on legitimate urgent mail erode trust; adversaries abuse trusted services (shared documents, legitimate SaaS notifications, compromised supplier accounts); adversarial prompts hidden in mail can target LLM-based triage (see SEC-010); QR and multi-step redirection chains remain hard; no product catches everything, so MFA with phishing-resistant passkeys/FIDO2 and payment-change procedures remain essential.

Shared resources & combinations

• Detonation and file analysis: SEC-003 (malware analysis) shares sandboxes and YARA rules.
• SOC workflow: SEC-001 receives escalated incidents; SEC-008 supplies threat intelligence and indicator enrichment.
• Human layer: SEC-024 (AI-personalised simulations and training) uses the same lure-generation knowledge and measures report rates.
• Adjacent threats: SEC-011 (deepfake CEO fraud), SEC-021 (lookalike domains and brand impersonation), SEC-017 (outbound DLP), ANOM-017 (forged invoice documents).
• Hardware/stack: a single HW2 LLM server can serve reported-mail triage, SEC-001 summarisation and SEC-008.

Effort, cost & KPIs

• Path A: days to two weeks for API-based deployment; per-mailbox per-month licence; the platform-native tier often already exists in the Microsoft 365 / Workspace licence.
• Path B: 1-2 months for a triage pipeline; cost dominated by integration and tuning rather than API tokens.
• Path C: 2-4 months plus ongoing tuning and on-call.
• KPIs: phish delivered to inbox per 1,000 mailboxes, time to remediate after delivery, user report rate and report accuracy, false-positive rate (legitimate mails quarantined), analyst minutes per reported mail, DMARC alignment percentage, BEC incidents and losses.

People & change

Involve IT and IT-security for configuration and quarantine rules, management for the policy on what happens to suspicious mails, and the data protection officer because mail content and metadata are scanned (AVV, processing location, retention; see GUIDE-005). Tell end users early; they are both the target and the sensor, so a report button and quick feedback make them partners.

Works council: depends. If the product only scans mails for threats and does not evaluate individuals, co-determination is still likely, because systems that log mail and click behaviour are objectively suited to monitor staff (§87(1) no. 6 BetrVG). Phishing-simulation reports per person are particularly sensitive. Agree that results are used only in aggregated form for training (see GUIDE-004). In public bodies the staff council (Personalrat) takes this role.

AI literacy: combine awareness training with how AI-generated phishing looks and why quarantine decisions can be wrong; the AI Act asks employers to support staff AI literacy (see GUIDE-006). Pilot with IT and one department, monitor false positives and let users release mails under clear rules. Concern: legitimate mail gets blocked - answer with a fast release process. Humans decide on borderline cases and on blocking senders (see GUIDE-008).

Risks, regulation & compliance

• EU AI Act: minimal risk; security filtering is not an Annex III use. AI literacy duty (Art. 4) applies to staff. Emotion inference on employees from email text must be avoided (Art. 5(1)(f) prohibits emotion recognition in the workplace); keep models focused on message threat features, not on employee mood or performance [6].
• GDPR: mail content and metadata are personal data of employees and third parties. Legal basis usually legitimate interest (Art. 6(1)(f)) plus network and information security as a recognised interest (Recital 49); do a DPIA for deep content analysis, conclude a processing agreement, document sub-processors and third-country transfers, and keep human access to quarantined private content limited [7].
• Employee monitoring: in Germany the works council has co-determination rights (BetrVG s. 87(1) no. 6) over systems capable of monitoring behaviour; private mail use policy affects telecommunications secrecy (TDDDG).
• NIS2 / DORA: email security supports the required risk-management measures and incident reporting; NIS2 names basic cyber hygiene and training explicitly [8].
• Mitigations: pseudonymisation of reported mail sent to external LLMs, retention limits on quarantine, prompt-injection hardening for LLM triage, human approval for automated mass deletion, audit log.

Getting started

1. Check SPF, DKIM and DMARC for all sending domains; move DMARC from p=none to quarantine/reject.
2. Enable the strongest native protection already licensed (e.g. Defender for Office 365 safe links/attachments, impersonation protection) and a one-click report button.
3. Run a 30-day proof of value of one ICES in monitor-only mode on your own mail; compare misses and false positives.
4. Define a payment-change and credential-request procedure with out-of-band confirmation; roll out phishing-resistant MFA.
5. Involve the data-protection officer and works council before activating content analysis; then connect results to SOC tooling (SEC-001).

Sources

1. ENISA Threat Landscape — https://www.enisa.europa.eu/topics/cyber-threats/threats-and-trends
2. FBI Internet Crime Complaint Center annual reports — https://www.ic3.gov/AnnualReport/Reports
3. Microsoft — Microsoft Defender for Office 365 documentation — https://learn.microsoft.com/defender-office-365/mdo-about
4. Heiding et al. — Evaluating LLM-automated spear phishing (arXiv 2412.00586) — https://arxiv.org/abs/2412.00586
5. Google — Email sender guidelines (SPF, DKIM, DMARC requirements) — https://support.google.com/mail/answer/81126
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Regulation (EU) 2016/679 (GDPR), Recital 49 and Art. 6 — https://eur-lex.europa.eu/eli/reg/2016/679/oj
8. Directive (EU) 2022/2555 (NIS2) — https://eur-lex.europa.eu/eli/dir/2022/2555/oj
9. Rspamd documentation — https://rspamd.com/doc/