Malware detection & analysis

Signature-based antivirus misses new, packed and obfuscated malware, and manual reverse engineering does not scale to thousands of samples. Modern endpoint protection uses ML classifiers on static and behavioural features, while LLMs now help analysts decompile, deobfuscate and summarise unknown samples in minutes. The result is faster triage, better coverage of unknown threats and fewer hours spent on routine samples. This file stays at defender, policy and architecture level.

Problem & value

• Today: AV/EDR signatures and reputation lookups catch known threats; unknown samples go to a small team of reverse engineers or an external incident-response provider. Polymorphic malware, living-off-the-land scripts (PowerShell, JavaScript, macros) and malicious installers or packages create a constant backlog.
• AI value: (1) ML classifiers generalise to unseen samples; (2) LLMs explain scripts and decompiled code in plain language, extract IOCs and draft detection rules; (3) agents automate the repetitive analysis steps (unpack, detonate, enrich, report).
• Evidence: VirusTotal reports that its AI-based analysis of Mach-O binaries surfaced real Mac and iOS malware that had zero antivirus detections at submission time, and Code Insight now covers scripts, documents, packages and extensions [1]. Elastic's EMBER2024 benchmark provides a public yardstick for ML detectors, including evasion-oriented test sets [2]. Do not expect vendor detection-rate claims to transfer to your environment without a trial.

Variants

• SMB / managed service: EDR/XDR with ML prevention run by a managed detection provider (MDR).
• Enterprise SOC / CERT: automated sandbox plus LLM summary for every suspicious attachment and file seen on endpoints; malware-analysis team uses AI-assisted reversing.
• Public sector / CERTs: shared analysis platforms (e.g. MWDB-style repositories) and national reporting duties.
• Developer / DevSecOps: scanning of build artefacts, containers and packages (see SEC-016).
• Mobile and macOS/Linux: platform-specific classifiers; browser extension and plugin scanning.
• Private use: consumer antivirus (Microsoft Defender, ESET, Bitdefender, G DATA, F-Secure) with built-in ML; no own pipeline needed.

How it works

1. file/script/attachment or endpoint event
2. pre-filter (hash reputation, YARA, allow-lists, file-type identification)
3. static analysis (PE/ELF/Mach-O features, strings, imports, entropy, packer detection)
4. ML classifier (gradient-boosted trees / neural net on features or raw bytes)
5. score
6. dynamic analysis in isolated sandbox (behaviour, network, persistence) if score uncertain
7. LLM assistant: deobfuscate script / decompiled code
8. behaviour summary, IOCs, ATT&CK mapping
9. verdict + evidence
10. SOC ticket / automated containment (isolate host, block hash)
11. analyst review
12. feedback into labels and rules

• Classifiers: feature-based models (EMBER-style, LightGBM) remain the workhorse for pre-execution detection; behavioural models score process and sandbox traces.
• LLM layer: code-capable LLMs read scripts, decompiler output (Ghidra, IDA) and capability reports (capa) and write a structured analysis. They are explainers, not the primary detector.
• Human-in-the-loop: analysts confirm verdicts for high-impact actions (mass quarantine, blocking business-critical binaries) and review every generated detection rule before deployment. Samples are handled only in isolated, access-controlled analysis environments.

Data preparation

Commercial detection needs hardly any preparation. Own analysis needs samples and context: suspicious files and URLs, EDR telemetry, sandbox reports, hashes and threat intelligence. In an SMB these sit in the EDR console, quarantine folders, mailboxes and the helpdesk tool.

Typical gaps: samples are deleted without being archived, no naming or case convention, sandbox reports in different formats, no list of approved software that triggers false positives, and legal questions about uploading files containing personal or confidential content to public services.

Preparation steps: - Define a sample intake process: isolated storage, hash, case ID, source (see PREP-004 for ownership). - Keep an allowlist of known internal tools and macros to cut false positives. - Export EDR and sandbox reports in a common format (see PREP-005). - Build a small labelled set of confirmed malicious and benign files for testing (see PREP-018, PREP-020). - Classify what may be uploaded to external analysis services (see PREP-024).

Rough effort: about 10 to 20 percent. Training a custom model (path D) would raise this considerably because it needs large labelled corpora; this is rarely justified outside security vendors. See GUIDE-003 for a general readiness check.

Implementation paths

Which path for whom

• SMB: A - EDR/XDR products with AI-based malware detection are mature and available from EU vendors, usually bundled with managed services.
• Enterprise: A - endpoint protection is a commodity function; AI-assisted analysis can be layered on via the vendor, and own analysis stays with the CERT.
• Developer: C - open-source tooling (YARA, sandboxes, local models for reverse-engineering help) keeps samples in-house and fits research.
• Public: B - authorities can add AI-assisted analysis through EU-hosted or sovereign cloud APIs while keeping sensitive samples out of public sandboxes.

Path A — Buy (recommended for most organisations)

• EDR/XDR with ML prevention: Microsoft Defender for Endpoint, CrowdStrike Falcon, SentinelOne Singularity, Palo Alto Cortex XDR, Trend Micro Vision One, Sophos, Bitdefender GravityZone (RO), ESET PROTECT (SK, EU), WithSecure Elements (FI, EU), G DATA (DE, EU), Trellix, Cisco Secure Endpoint.
• Analysis and intelligence platforms: Google Threat Intelligence / VirusTotal with Code Insight [1], Hybrid Analysis (CrowdStrike), ANY.RUN, Joe Sandbox (CH), Intezer, VMRay (DE/US).
• MDR providers: many EU MDRs run these tools for SMBs.
• Choose A when: no in-house reverse engineers; EU hosting and DPA with the vendor are available.

Path B — Build on cloud APIs

• Enrichment pipeline: hash/URL lookups via the VirusTotal API, sandbox APIs, then a frontier LLM (Claude, Gemini, GPT, Mistral — EU region or EU provider where required) to summarise decompiled functions, deobfuscate scripts and draft YARA/Sigma rules.
• Orchestration: SOAR (Shuffle, Tines, Cortex XSOAR) or n8n, with strict redaction: never send customer data or internal document contents embedded in samples to external APIs without a DPA.
• Choose B when: you have a SOC and want custom triage automation on top of an EDR.

Path C — Self-host (open source)

• Static/dynamic analysis: Ghidra (NSA, open source) and Rizin/Cutter for reversing, capa [3] for capability detection, YARA [4], CAPEv2 sandbox [5], MWDB-core (CERT Polska, EU) for sample repositories, ClamAV for baseline scanning, Zeek/Suricata for network traces.
• ML: EMBER2024 code and data [2] to train LightGBM baselines; open-weight code models (Qwen-Coder, DeepSeek-Coder, Mistral/Codestral class) via vLLM or Ollama for explanation.
• Hardware: HW2 is enough for a 14–32B quantised code model and a classifier service; sandboxes need separate isolated VM hosts (CPU/RAM, not GPU). Effort: a dedicated analyst or two.

Path D — Custom training

Worthwhile for vendors, large CERTs or organisations with unusual file types or proprietary formats. Needs a large, correctly labelled and time-split corpus (benign software is hard to collect), a feature pipeline and continuous retraining against concept drift. Pure academic accuracy numbers on random splits are misleading; evaluate on later-dated samples [2].

State of the art (2026)

• Public benchmark: EMBER2024 extends the original EMBER with more file types and challenge sets of hard-to-detect malware, so models can be tested against evasion and drift [2].
• LLM-assisted reversing: Code Insight and agentic analysis pipelines summarise scripts, packages and decompiled binaries; Google's threat-intelligence team also documents adversaries experimenting with LLMs, so defenders should expect AI-generated and AI-queried malware [1][6].
• Agentic analysis: agents call decompilers, sandboxes and lookups in loops; vendors ship these inside SOC platforms (see SEC-001).
• Limits and failure modes:
• Concept drift: models degrade as malware and benign software change.
• Adversarial evasion (padding, packing, feature manipulation) lowers classifier scores.
• LLMs hallucinate behaviour, struggle with large or heavily packed binaries and can be influenced by prompt-injection text planted in samples; treat output as a hypothesis and verify with sandbox evidence.
• False positives on legitimate admin tools and in-house software can disrupt operations.

Shared resources & combinations

• SEC-001 SOC alert triage: malware verdicts and summaries feed the AI analyst.
• SEC-002 phishing: attachment and URL detonation is the same pipeline.
• SEC-006 network intrusion detection and SEC-005 UEBA add behavioural signals.
• SEC-008 threat intelligence: IOCs and YARA rules are exchanged both ways.
• SEC-016 supply-chain security: malicious packages and model artefacts use the same static-plus-LLM analysis; SEC-010 covers prompt-injection defence for the LLM layer.
• Shared stack: llm-code and llm-reasoning endpoints (HW2/HW3) can serve SEC-004, SEC-007 and SEC-016.

Effort, cost & KPIs

• Path A: days to weeks to roll out an EDR; licence per endpoint per year.
• Path B: 1–3 months for SOAR integration; cost is API usage plus sandbox licences.
• Path C: 3–6 months and at least one skilled analyst; hardware is modest, expertise is the cost.
• KPIs: detection rate on a fixed, time-split test set; false-positive rate on in-house software; mean time to triage a sample; share of samples auto-closed correctly; analyst hours per sample; time from first sighting to deployed detection.

People & change

Involve IT-security and IT for rollout, exclusions and isolation rules; management for the decision on automatic containment; legal-compliance and the data protection officer because analysed files and endpoint telemetry can contain personal and confidential data and may be passed to the vendor (AVV, processing location; see GUIDE-005).

Works council: depends. Pure file and process analysis for threats is less intrusive, but EDR products usually log user and device activity and are objectively suited to monitor behaviour, so §87(1) no. 6 BetrVG often applies. Agree on purpose limitation, no performance evaluation and who may view telemetry (see GUIDE-004). Public bodies involve the staff council.

AI literacy: tell staff what the agent sees and blocks, how to report a false alarm, and that an AI verdict can be wrong; the AI Act asks employers to support staff AI literacy (see GUIDE-006). Roll out to a pilot group of IT-affine users first, run in detect-only mode, then switch to blocking. Typical concerns: work being interrupted by false positives and surveillance fears - handle them with a quick exception process and transparent logging rules. Humans decide on wiping or isolating devices and on submitting samples externally (see GUIDE-008).

Risks, regulation & compliance

• EU AI Act: minimal risk — security tooling is not an Annex III category and no Art. 5 practice is involved; AI literacy (Art. 4) applies. Cybersecurity of the AI systems themselves is covered by Art. 15 only if they are high-risk [7]. If a detector is also used to monitor employees' behaviour, reassess (labour-law and works-council issues).
• GDPR: samples and telemetry can contain personal data (documents, user names, paths, emails). Security processing can rest on legitimate interest (Art. 6(1)(f), recital 49), but needs a DPA, EU hosting or transfer safeguards, and minimisation before sending to cloud LLMs.
• NIS2 / DORA: essential and important entities must manage cyber risk and report incidents (NIS2 Directive (EU) 2022/2555 [8]); malware detection supports these duties but does not replace them. Financial entities also fall under DORA.
• Operational risks: accidental execution of live malware (isolated labs, no production network), leaking confidential files through uploads to public scanners (use private scanning), and over-blocking. Mitigations: air-gapped sandbox, upload policy, staged rule rollout, human approval for broad actions.
• Dual use: keep tooling defender-focused; do not build or share working malware generators.

Getting started

1. Inventory current coverage: EDR on all endpoints, mail attachment sandboxing, patch status.
2. Run a 30-day trial of two EDR/MDR options on pilot machines and compare alerts, false positives and explainability.
3. Define a private-upload policy for samples (no confidential files to public scanners).
4. Add automated sandbox detonation and an LLM-generated summary to the SOC ticket for suspicious files; measure analyst time saved.
5. Build a small time-split test set from your own incidents to compare tools objectively.

Sources

1. VirusTotal Blog — Reversing at Scale: AI-Powered Malware Detection for Apple's Binaries — https://blog.virustotal.com/2025/11/reversing-at-scale-ai-powered-malware.html
2. Elastic — EMBER2024 malware benchmark — https://github.com/FutureComputing4AI/EMBER2024
3. Mandiant FLARE — capa — https://github.com/mandiant/capa
4. VirusTotal — YARA — https://github.com/VirusTotal/yara
5. CAPEv2 sandbox — https://github.com/kevoreilly/CAPEv2
6. Google Threat Intelligence — Adversaries leverage AI for vulnerability exploitation, augmented operations and initial access — https://cloud.google.com/blog/topics/threat-intelligence/ai-vulnerability-exploitation-initial-access
7. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
8. Directive (EU) 2022/2555 (NIS2) — https://eur-lex.europa.eu/eli/dir/2022/2555/oj
9. VirusTotal Threat Intelligence docs, changelog Feb 2026 (Code Insight) — https://gtidocs.virustotal.com/changelog/february-9th-2026-code-insight-openclaw-skill-packages-private-scanning-livehunt-rules-and-more
10. CERT Polska — MWDB-core — https://github.com/CERT-Polska/mwdb-core