AIOps: incident detection & root-cause analysis

Modern systems emit millions of signals; on-call engineers drown in alerts and lose the first critical minutes searching dashboards. AIOps combines statistical anomaly detection, alert correlation and, since about 2024, LLM-based "AI SRE" agents that query telemetry, check recent changes and propose a root cause with evidence. The realistic benefit is faster triage and less alert noise; fully autonomous diagnosis is not yet reliable.

Problem & value

• Today: monitoring tools raise static-threshold alerts per component. A single failure (bad deploy, saturated database, expired certificate) triggers dozens of alerts. On-call staff correlate by hand across metrics, logs, traces, deployments and chat, often at night. Knowledge sits in a few senior heads and in old postmortems.
• AI value: (1) dynamic baselines instead of static thresholds; (2) grouping of related alerts into one incident; (3) topology- and change-aware ranking of probable causes; (4) an agent that runs the first investigation steps (queries, log search, diff of recent deploys) before a human is paged; (5) automatic timeline and postmortem drafts.
• Evidence is mixed: vendor material claims large MTTR reductions, but independent numbers are scarce. On public research benchmarks, LLM agents still recover the exact root-cause set in only a minority of cases (about 21% on OpenRCA 2.0, though they often name at least one correct service) [1][2]. Measure on your own incident history before trusting any vendor figure.

Variants

• Platform-native AIOps: causal AI inside an observability suite (Dynatrace, Datadog, Elastic, Grafana).
• AI SRE agent: a separate investigator over several tools, triggered by an alert or a Slack/Teams thread.
• Event-correlation layer: deduplication and noise reduction in front of paging (PagerDuty, BigPanda-style).
• Kubernetes/cloud-native troubleshooting: open-source agents explaining pod crashes and failed rollouts.
• Related but separate: security alerts belong to SEC-001/SEC-005; machine sensor anomalies to ANOM-005/PRED-006.

How it works

1. telemetry (OpenTelemetry metrics/logs/traces) + change events (CI/CD, feature flags, config)
2. anomaly detection (dynamic baselines, change-point detection, log-pattern anomalies)
3. alert correlation & dedup (time, topology, text similarity)
4. one incident
5. investigation: topology/causal graph + LLM agent (query metrics, search logs, diff deploys, retrieve runbooks and similar past incidents)
6. ranked hypotheses with evidence links and confidence
7. human-in-the-loop: on-call confirms; remediation only via approved runbook actions
8. timeline, status-page draft, postmortem draft
9. feedback to improve ranking

• Detection: seasonal time-series models, robust statistics, isolation-forest style detectors, log template mining; mostly classical ML, cheap and mature.
• Causal/topology reasoning: a dependency graph (service map, Kubernetes, CMDB) restricts candidate causes; deterministic causal engines are more predictable than pure LLM reasoning.
• LLM agent: reasoning model with tools (PromQL/SQL/log queries, deployment history, ticket search) producing hypotheses with cited evidence. Read-only by default.
• Human-in-the-loop: the agent proposes; engineers approve any state-changing action (rollback, restart, scaling). Ask/deny modes per tool are standard in leading products.

Data preparation

AIOps needs telemetry: metrics, logs and traces, change events (deployments, feature flags), the service topology and past incidents with runbooks. In an SMB these are spread across monitoring tools, log files on servers, a ticket system, wikis and chat threads. Typical gaps: inconsistent log formats, missing trace IDs, no service inventory or CMDB, unlabelled historical incidents, alert noise, and personal data or secrets inside logs.

Preparation steps: 1. Standardise collection with an OpenTelemetry Collector and consistent service, environment and version labels. 2. Record deployments and configuration changes as events, because most incidents correlate with a change. 3. Build a simple service map (even a maintained YAML file) with owners. 4. Clean up alerts first: remove duplicates and flapping rules. 5. Turn past post-mortems and runbooks into short, current documents and label incidents with confirmed root cause (PREP-013, PREP-019, PREP-020). 6. Redact personal data and tokens in the pipeline (PREP-022).

Rough share of total effort: 30-40 percent; the telemetry pipeline and labelled incidents are the bulk, while the AI layer is comparatively light. See PREP-005 for extracting data from legacy systems and GUIDE-003 for a readiness check.

Implementation paths

Which path for whom

• SMB: A. A mature AIOps or AI-SRE feature in an observability suite you already pay for (Dynatrace, Datadog, Elastic, Checkmk) is cheaper than building detection and correlation yourself.
• Enterprise: B. Core services, custom topology and CMDB integration, and an internal AI platform justify a tool-using agent on cloud APIs layered over your existing telemetry.
• Developer: C. HolmesGPT, K8sGPT and Prometheus with an Ollama or vLLM model run locally and cost nothing per seat.
• Public sector: C. Logs often contain citizen or employee data and procurement favours sovereign hosting, so self-hosted open-source detection with local models fits.

Path A — Buy (recommended for most organisations)

• Observability suites with AI: Dynatrace (Davis causal AI / Dynatrace Intelligence), Datadog (Watchdog anomaly detection, Bits AI SRE agent), Elastic Observability (AI Assistant, Attack Discovery-style correlation), Grafana Cloud (Assistant, adaptive alerting), New Relic, Splunk Observability, IBM Instana. Dynatrace and Elastic offer EU regions or self-managed deployment; Datadog has an EU site (datadoghq.eu).
• Incident and AI SRE tools: PagerDuty (event intelligence, SRE agent), incident.io (Slack-native incident workflow, AI-drafted timelines), Rootly, Komodor (Kubernetes), plus newer AI-SRE startups. Check: EU data residency, read-only default, audit log, which telemetry sources are supported.
• EU options: Checkmk (Germany, open core), Icinga/NETWAYS (Germany), Instana via IBM EU regions, Elastic (Netherlands-origin company, self-hostable).
• Choose A when: you already use one observability platform; the agent can then use data it already holds.

Path B — Build on cloud APIs

• Telemetry: OpenTelemetry collectors into an EU-hosted backend (Grafana Tempo/Loki/Mimir, ClickHouse-based stores, Elastic).
• Reasoning: frontier reasoning models (Claude, Gemini, GPT, Mistral) with tool use over an MCP server or function calls to your Prometheus/Loki/Kubernetes APIs; EU-region endpoints (Azure OpenAI EU, Vertex AI EU, Amazon Bedrock Frankfurt, Mistral).
• Choose B when: you need custom tools and data scope but accept a cloud model.

Path C — Self-host (open source)

• Detection and correlation: Prometheus + Alertmanager, Grafana, OpenSearch/Elastic, Netflix-style anomaly libraries, Merlion/Darts/Prophet for time series; Keep (open-source alert management).
• Agents: HolmesGPT (CNCF sandbox, Robusta) and K8sGPT for Kubernetes; custom agents on open-weight models (e.g. Qwen3, Llama, Mistral, gpt-oss families) served via vLLM or Ollama [4].
• Hardware: HW3 for a ~30-120B reasoning model; HW2 works for smaller models with weaker reasoning. Logs may contain personal data, which is a strong argument for self-hosting.

Path D — Custom training

Rarely worthwhile. Possible for very large operators: train detectors on own metric history, fine-tune a small model on labelled incident-to-cause pairs, or train graph models over topology. Needs hundreds of well-labelled incidents with confirmed root causes, which most teams do not have.

State of the art (2026)

• Anomaly detection and alert grouping are mature and commoditised in every major suite.
• Causal AI on topology (Dynatrace-style) is deterministic and explainable; LLM agents add breadth (logs, tickets, code) but are less repeatable.
• AI SRE agents went mainstream as products in 2025-2026: they investigate in parallel, cite queries and keep read-only access by default. Capabilities and pricing in vendor comparisons are largely vendor-reported [5].
• Benchmarks: RCAEval (735 failure cases, 3 microservice systems, 15 baselines) and OpenRCA (335 cases) test diagnosis; OpenRCA 2.0 adds step-wise causal annotations and shows that LLM agents often name a plausible service without a verified causal path (paper-reported) [1][2][3].
• Limits: hallucinated causes with convincing narratives; poor observability (missing traces, inconsistent labels) caps accuracy; novel failure modes absent from history; cascading failures where symptoms hide the cause; prompt injection via log content; cost of repeated queries.

Shared resources & combinations

• Telemetry and log stack shared with CODE-011 (log analysis and summarisation) and CODE-018 (LLM observability, same OpenTelemetry tracing).
• Agent and tool layer (reasoning LLM + MCP tools + workflow engine) shared with CODE-009 (infrastructure-as-code assistant), CODE-014 (bug triage and automated fixing; from root cause to fix PR) and AGENT-002 (workflow automation).
• Neighbours: SEC-001 and SEC-005 (security alert triage, UEBA) use similar correlation but have different threat and compliance profiles; ANOM-005 and PRED-006 apply anomaly detection to machine sensors.

Effort, cost & KPIs

• Path A: weeks to enable; cost is mostly platform licences and AI credits/consumption, which can grow with investigation volume. Check metering.
• Path B: 2-4 months for a useful read-only investigator; model cost per investigation is small versus engineer time, but telemetry query cost can dominate.
• Path C: 3-6 months, GPU server plus on-call/platform engineering time.
• KPIs: MTTD, MTTR, alerts per incident (noise ratio), share of incidents with correct top-3 hypothesis (measured against postmortems), false-page rate, on-call hours at night, postmortem lead time, engineer-rated usefulness of AI summaries.

People & change

Involve early: the operations/SRE lead (process owner) and the on-call engineers (end users) from the start, since they judge hypothesis quality. IT security checks read-only access and agent actions (GUIDE-013). The DPO is needed because logs and traces can contain user IDs and IP addresses. Management decides on budget and on which actions may ever be automated.

Works council: yes, in practice. Observability platforms record who did what, response times and on-call performance. Under §87(1) no. 6 BetrVG objective suitability for monitoring is enough (BAG), so co-determination applies even if monitoring is not the goal. §90(1) no. 3 requires timely information on AI-supported work processes; §80(3) allows an AI expert. Agree on a works agreement that excludes individual performance evaluation from incident data and limits access (GUIDE-004). Public sector: Personalrat.

Data protection: check personal data in logs, conclude an AVV with the vendor, and run a DPIA if logs identify individuals systematically (GUIDE-005).

AI literacy: train on-call staff to question ranked hypotheses and verify evidence (GUIDE-006).

Pilot and communication: run in suggestion-only mode on one team or service for several weeks and compare against human root-cause findings (GUIDE-009, GUIDE-008). Typical concerns are surveillance and de-skilling. Address them by publishing what is logged, banning use for individual appraisal, and keeping remediation behind human approval.

Risks, regulation & compliance

• EU AI Act: usually minimal risk. Exception: AI used as a safety component in management and operation of critical digital infrastructure (Annex III point 2) can be high-risk; assess for operators of energy, transport or similar infrastructure. Do not use AIOps for employee monitoring or performance scoring of on-call staff (HR/worker-monitoring rules, works council). AI literacy (Art. 4) applies to operators [6].
• GDPR: logs and traces often contain IP addresses, user IDs, e-mail addresses and sometimes payloads. Redact at the collector (OpenTelemetry processors), minimise retention, sign a DPA and keep telemetry and prompts in the EU.
• NIS2 / DORA: essential/important entities and financial firms have incident detection, reporting (early warning within 24 h under NIS2) and ICT third-party risk duties; AI vendors count as ICT third parties under DORA [7][8].
• Security: agents with production credentials are a high-value target. Use least privilege, read-only tokens, approval gates for write actions, and treat log text as untrusted input (prompt injection).

Getting started

1. Baseline MTTD/MTTR, alerts per incident and the top 10 noisiest alerts from the last 90 days.
2. Ensure telemetry quality first: OpenTelemetry instrumentation, consistent service names, deploy and feature-flag events in the same store.
3. Turn on alert grouping and anomaly detection in your existing platform; review for two weeks.
4. Pilot a read-only AI investigator on 20 past incidents (replay) and score top-3 accuracy against postmortems.
5. Roll out to one on-call team with ask-before-act permissions, then extend remediation via vetted runbooks.

Sources

1. RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data — https://arxiv.org/abs/2412.17015
2. OpenRCA 2.0: From Outcome Labels to Causal Process Supervision — https://arxiv.org/html/2606.27154
3. OpenRCA (Microsoft) benchmark repository — https://github.com/microsoft/OpenRCA
4. HolmesGPT (CNCF, Robusta) — https://github.com/robusta-dev/holmesgpt
5. Datadog Bits AI SRE documentation — https://docs.datadoghq.com/bits_ai/bits_ai_sre/
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Directive (EU) 2022/2555 (NIS2) — https://eur-lex.europa.eu/eli/dir/2022/2555/oj
8. Regulation (EU) 2022/2554 (DORA) — https://eur-lex.europa.eu/eli/reg/2022/2554/oj
9. OpenTelemetry documentation — https://opentelemetry.io/docs/