Network intrusion detection

Attackers who have passed the perimeter move through the network, talk to command servers and copy data out. Endpoint agents and logs miss unmanaged devices, IoT, and encrypted traffic. Network detection and response (NDR) combines signature-based IDS with machine-learning baselines of normal traffic to expose such behaviour, and LLMs now summarise and triage the resulting alerts.

Problem & value

• Today: many organisations run signature IDS/IPS on the firewall plus a SIEM. Signatures catch known tools; they miss novel malware, abuse of legitimate protocols, living-off-the-land activity and low-and-slow exfiltration. Rule tuning is manual and alert volume is high.
• AI value: unsupervised models learn per-device and per-segment behaviour and flag deviations. Supervised classifiers rate encrypted flows and DNS (e.g. DGA, tunnelling) from metadata without decrypting payload. LLMs explain alerts and cut analyst time (see SEC-001).
• Why now (EU): NIS2 requires essential and important entities to have measures for incident detection and handling, and reporting obligations (early warning within 24 h) make early detection a compliance matter [1]. DORA imposes comparable duties on financial entities [2].
• Caveat: published benchmark accuracies on datasets such as CIC-IDS2017 or UNSW-NB15 are often near-perfect but are known to overstate real-world performance because of dataset artefacts and distribution shift; always test on your own traffic [7].

Variants

• SMB: managed detection service or firewall with built-in IPS and cloud analytics; no own SOC.
• Enterprise / public sector: NDR sensors at data-centre, campus, cloud and remote-site chokepoints, feeding SIEM/XDR.
• Cloud-native: analysis of cloud flow logs and virtual traffic mirroring in AWS, Azure and GCP.
• OT/IoT segments: passive monitoring with protocol awareness — see SEC-015.
• Private use: home router IDS (e.g. OPNsense with Suricata) is possible but rarely needed.

How it works

1. traffic (SPAN/TAP, cloud mirroring, flow logs)
2. sensor: protocol parsing, metadata extraction (Zeek / Suricata EVE JSON)
3. signature engine (Suricata rules) + threat-intel matching
4. feature extraction (flows, DNS, TLS fingerprints such as JA3/JA4, timing)
5. ML scoring (baseline anomalies, supervised classifiers, graph analysis of host relations)
6. correlation & enrichment (asset, identity, vulnerability context)
7. LLM summary + ATT&CK mapping
8. SOC triage (human)
9. response: block / quarantine via firewall, NAC, EDR (approval for high-impact actions)

• Signature layer: deterministic, explainable, cheap; remains the foundation.
• Behavioural layer: autoencoders, isolation forests, sequence and graph models over flow features; per-entity baselines learned over days to weeks.
• Encrypted traffic: classification on handshake fingerprints, packet sizes and timing, not content.
• Human-in-the-loop: analysts confirm incidents; automated blocking is limited to high-confidence, low-blast-radius actions; feedback tunes thresholds.

Data preparation

Network detection needs traffic metadata or packet captures from the places where traffic crosses: firewall and switch mirror ports, DNS and proxy logs, NetFlow/IPFIX, and cloud flow logs. In an SMB this data sits in the firewall appliance, sometimes only in short rolling buffers, and is rarely stored centrally.

Typical gaps: no sensor at the right network position, encrypted traffic without visibility, flat networks where everything looks "internal", unknown devices (IoT, printers, machines), missing asset inventory, and no clean baseline because infections may already be present.

Preparation steps: 1. Build an asset and network inventory (see PREP-001, PREP-005) and draw the segments. 2. Place sensors or enable flow export at the perimeter and between key segments (Suricata, Zeek, ntopng). 3. Forward logs to one central store with synchronised time (NTP) and a common schema. 4. Collect 2 to 4 weeks of known-good traffic as baseline and label known scanners and admin tools (PREP-020). 5. Decide what payload may be stored; usually metadata only, with short retention for captures (PREP-025).

Rough share of effort: 30 to 40 percent, mostly sensor placement and inventory. Use GUIDE-003 for a quick readiness check.

Implementation paths

Which path for whom

• SMB: Path A. Next-generation firewalls and managed detection services with built-in ML (EU-hosted MDR) cover the standard case without in-house expertise.
• Enterprise: Path C. Network telemetry is sensitive and high in volume, so cloud or per-GB pricing is costly; Suricata/Zeek with own models integrate into the shared SIEM.
• Developer: Path C. Suricata, Zeek and open datasets on HW1 are the natural stack for research and tuning.
• Public: Path C. Sovereignty and procurement rules favour self-operated sensors with open, auditable rules.

Path A — Buy (recommended for most)

• NDR platforms: Darktrace, Vectra AI, ExtraHop Reveal(x), Corelight (Zeek-based), Cisco Secure Network Analytics, Arista NDR, Trend Micro, Fortinet FortiNDR, Palo Alto Networks, Check Point. EU vendors and operators: Stormshield (FR, network security), Gatewatcher (FR, NDR), Sekoia.io (FR, SIEM/XDR), Atos/Eviden and Orange Cyberdefense managed detection.
• Firewall-integrated IPS: FortiGate, Palo Alto, Sophos, Stormshield; adequate for SMBs.
• Check: EU data residency for telemetry, handling of encrypted traffic, sensor throughput, SIEM/SOAR integration, ATT&CK coverage evidence, explainability of detections.
• Choose A when: no in-house detection engineering team; most SMBs should prefer a managed service with a 24/7 SOC.

Path B — Build on cloud APIs

• Cloud-native detection: AWS GuardDuty and VPC Traffic Mirroring, Microsoft Defender for Cloud / Sentinel with Azure network analytics, Google Cloud IDS (Palo Alto-powered) [3].
• Pipeline: sensor logs → SIEM (Microsoft Sentinel, Elastic, Splunk, EU: Sekoia, Wazuh-hosted) → custom ML jobs and an LLM (EU-region endpoints) for alert summarisation, with PII-minimised prompts.
• Choose B when: you are cloud-first and need custom enrichment.

Path C — Self-host (open source)

• Sensors: Suricata (signature IDS/IPS, EVE JSON output) [4], Zeek (protocol metadata and scripting) [5], Arkime (full packet capture/search), Security Onion (distribution bundling these with Elastic) [6].
• Rules: Emerging Threats Open, community rule sets; Sigma-style detections on resulting logs.
• ML: scikit-learn, PyTorch, or Elastic ML on Zeek/flow data; JA3/JA4 fingerprints. Optional small local LLM (Qwen, Llama, Mistral class via vLLM/Ollama) for alert explanation.
• Hardware: HW1 for sensors up to a few Gbit/s with tuned capture (AF_PACKET); the SIEM/ML back end typically needs a separate server. Ops effort: one to two engineers part-time for rule tuning and baselines.

Path D — Custom training

Worth it for large networks with distinctive protocols or for research. Needs labelled benign traffic over weeks plus labelled attack or red-team data; public datasets are for method development only [7]. Plan continuous retraining due to drift.

State of the art (2026)

• Hybrid detection (signatures + behaviour + threat intel) is standard in commercial NDR; vendors add generative AI assistants for investigation and natural-language hunting.
• Research directions: graph neural networks over host communication graphs, transformer models on flow sequences, self-supervised pretraining on traffic, federated learning across sites.
• Encryption (TLS 1.3, QUIC, encrypted DNS) shifts detection to metadata and fingerprints; decryption at gateways adds privacy and legal burdens.
• Limits and failure modes: false positives from new but legitimate behaviour (migrations, backups); baseline poisoning by attackers present during training; adversarial evasion of ML classifiers; concept drift; benchmark datasets that do not represent production; blind spots where sensors are not placed; ML scores that analysts cannot explain.

Shared resources & combinations

• Same stack: SIEM/XDR and alert triage (SEC-001), UEBA (SEC-005), malware analysis (SEC-003), threat intelligence (SEC-008), OT/IoT monitoring (SEC-015) share log pipelines, anomaly-detection tooling and enrichment data.
• Compliance: SEC-009 maps detection coverage to NIS2/DORA/ISO 27001 controls.
• Hardware: sensors on HW1; optional LLM triage can share a HW2 server with other SEC use cases.

Effort, cost & KPIs

• Path A: weeks for SaaS/managed services; licence typically by throughput or sensor count plus sensor hardware.
• Path B: 2–4 months to integrate and tune.
• Path C: 3–6 months to reach stable detection quality; cost is mostly staff time.
• Cost drivers: sensor placement and throughput, storage for metadata/PCAP retention, tuning effort.
• KPIs: mean time to detect and respond, true-positive rate of alerts reviewed, false positives per analyst per day, ATT&CK technique coverage (validated by purple-team exercises), share of segments and cloud accounts monitored, sensor packet-drop rate.

People & change

Network analysis touches all employees' traffic, so announce it even though the goal is attack detection.

• Works council: yes. Network monitoring is objectively suited to monitor behaviour (§87(1) no. 6 BetrVG, BAG), even if only attacks are targeted. Agree in a works agreement on metadata-only analysis, retention, access rights and exclusion of individual performance evaluation; §90(1) no. 3 requires timely information. In public bodies the Personalrat is involved.
• DPO: IP addresses and DNS queries are personal data; clarify legal basis, retention and AVV (Art. 28 GDPR). A DPIA is usually required only if packet content or large-scale individual profiling is involved.
• IT and IT security plan sensor placement, alert triage and response playbooks; management decides on budget and responsibilities (internal vs. MDR).
• AI literacy: support the people handling alerts (Art. 4 AI Act) with short training on false positives and tuning.
• Pilot: start in one segment (e.g. server network) before rolling out company-wide.
• Typical concerns: "are you reading my traffic?" and alert fatigue. Answer with a clear data-scope statement and a tuning phase; blocking actions stay human-approved at first.

See GUIDE-004, GUIDE-005 and GUIDE-011.

Risks, regulation & compliance

• EU AI Act: typically minimal risk; a security anomaly detector is not an Annex III use. Exception: AI used as a safety component in the management and operation of critical digital infrastructure can be high-risk under Annex III point 2 — check with legal if the system controls infrastructure. Using network monitoring to infer employee behaviour or emotions would raise separate issues (Art. 5 prohibits emotion recognition in the workplace). AI literacy (Art. 4) applies [8].
• GDPR: IP addresses, DNS queries and URLs are personal data. Legal basis usually legitimate interest (Art. 6(1)(f)) supported by recital 49 on network security; document a balancing test, minimise payload capture, set retention limits, run a DPIA where monitoring is extensive [9]. In Germany, works council co-determination (§ 87 BetrVG) and TTDSG/TDDDG telecommunications secrecy apply to employee monitoring and private use.
• NIS2 / DORA / ISO 27001: detection and incident reporting duties [1][2].
• Risks and mitigations: alert fatigue (tune, prioritise), decryption privacy exposure (limit scope, access control), vendor telemetry leaving the EU (contractual and regional controls), model evasion (layered defence), over-automation (approval gates for blocking).

Getting started

1. Map network chokepoints and cloud accounts; decide where sensors or flow logs give the best visibility.
2. Start with Suricata plus Zeek (or a managed NDR trial) in passive mode on one segment; log to your SIEM.
3. Let baselines learn for 2–4 weeks; review and label alerts; tune rules and thresholds.
4. Run a purple-team or Atomic Red Team style exercise to measure ATT&CK coverage.
5. Complete the DPIA/works-council documentation and define retention, then add automated response for low-risk actions.

Sources

1. Directive (EU) 2022/2555 (NIS2) — https://eur-lex.europa.eu/eli/dir/2022/2555/oj
2. Regulation (EU) 2022/2554 (DORA) — https://eur-lex.europa.eu/eli/reg/2022/2554/oj
3. Google Cloud IDS documentation — https://cloud.google.com/intrusion-detection-system/docs
4. Suricata documentation — https://docs.suricata.io/
5. Zeek documentation — https://docs.zeek.org/
6. Security Onion documentation — https://docs.securityonion.net/
7. Engelen et al., "Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study" — https://intrusion-detection.distrinet-research.be/WTMC2021/Resources/wtmc2021_Engelen_Troubleshooting.pdf
8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
9. Regulation (EU) 2016/679 (GDPR), recital 49 — https://eur-lex.europa.eu/eli/reg/2016/679/oj
10. MITRE ATT&CK — https://attack.mitre.org/