AI agent fleet governance, observability & evaluation (AgentOps)

Once dozens or hundreds of agents act on behalf of employees and departments, the question is no longer "does this prompt work?" but "which agents exist, who owns them, what may they touch, what did they do, are they still good, and how do we stop one?". AgentOps combines an agent registry, agent identity, tracing, continuous evaluation, approval policy and a kill switch into one operating layer, and produces the evidence needed for audits and the EU AI Act.

Problem & value

• Today: agents appear everywhere: copilots built by business users, vendor-embedded agents in CRM/ITSM/ERP, developer-built LangGraph or custom services, coding agents, and "shadow agents" created without IT knowledge. Typical gaps: no inventory, agents running under a human's shared credentials or long-lived API keys, no trace of what an agent did across tools, quality that silently degrades after a model or prompt change, and runaway token costs.
• AI/tooling value: a single inventory and risk view; per-agent identity with least privilege; reproducible traces for incident analysis; regression gates before release; cost and drift alerts; documented human oversight. Analyst and vendor claims about savings vary widely and are mostly unsourced, so measure your own baseline.
• Why now: Microsoft made Agent 365 generally available on 1 May 2026 (registry in the Microsoft 365 admin center, with Entra and Purview integration) [1]. Observability has converged on OpenTelemetry GenAI semantic conventions with invoke_agent / create_agent operations [2]. The AI Act's logging and oversight requirements for high-risk systems (Art. 12, 14, 26) make "evidence per action" a procurement criterion [3].

Variants

• Enterprise IT / platform team: central registry and policy across vendor and in-house agents (M365, Salesforce, ServiceNow, cloud platforms).
• Developer / product team: tracing plus offline and online evals for agents they build (overlaps CODE-018, which focuses on LLM application evaluation).
• Public sector: register of automated procedures, human-oversight documentation, sovereign hosting, transparency to citizens.
• Regulated industry (banking, insurance, health): model-risk-management integration, four-eyes approval for consequential actions, retention of logs for supervisory review.
• Agent-to-agent / multi-vendor: governance across agents that call each other via MCP or A2A (see AGENT-013).

How it works

1. discovery (platform APIs, IdP, code repos, network/gateway logs, MCP server inventory)
2. registry (owner, purpose, data classes, model, tools, risk class, lifecycle state)
3. identity & access (unique agent identity, short-lived scoped tokens, per-tool permissions)
4. gateway / policy enforcement (allow-lists, rate and spend limits, approval gates, PII filters)
5. runtime tracing (OTel spans: agent
6. model
7. tool; inputs/outputs with redaction)
8. evaluation (offline regression sets, online sampling, LLM-as-judge, human review, drift/cost alerts)
9. response (alert, throttle, quarantine, kill switch, rollback of prompt/model version)
10. evidence (immutable logs, oversight records, reports for DPO, auditors, regulators)

• Registry and discovery: inventories agents from platforms plus unmanaged ones found via identity and network telemetry. Each entry has a named human owner and a risk classification (including AI Act role: provider or deployer).
• Agent identity: each agent gets its own non-human identity (not a shared service account), with on-behalf-of delegation of user rights narrowed to the task; credentials are short-lived. Microsoft Entra Agent ID and comparable IdP features cover this [1].
• Tracing: every model call, retrieval and tool call is a span linked to a session and a responsible human or process. Content capture is configurable because prompts often hold personal data.
• Evaluation: task-success metrics against golden datasets, trajectory checks (right tools, right order, no forbidden actions), LLM-as-judge for open-ended output, calibrated against human labels, plus cost and latency budgets. Runs as a CI gate and continuously on sampled production traffic.
• Human-in-the-loop points: approval policies by action class (e.g. payments, external e-mail, deletion, access changes); periodic owner recertification; a named person authorised to trigger the kill switch; human review of flagged evaluation failures.

Data preparation

The "data" here is mostly telemetry and metadata, not business records, so preparation effort is low. Agent definitions live in the platforms and frameworks that host them (Copilot Studio, Agentforce, custom code), permissions in the identity provider and IAM, costs in cloud billing, and policies in PDFs or wikis. Typical gaps: unknown or shadow agents nobody registered, missing owners, inconsistent trace formats between frameworks, traces that contain personal data without a retention rule, and no evaluation dataset for the agent's task.

Concrete steps: first build an inventory of all agents and their tool scopes ([PREP-001]), assign a named owner per agent, standardise on OpenTelemetry-style traces, define data classes and what may appear in logs ([PREP-024]), and collect 30 to 100 real examples per agent as an evaluation set ([PREP-018]). Mask or pseudonymise personal data in stored traces ([PREP-022]). Tools: Langfuse, OpenTelemetry collectors, the IdP's app registry, spreadsheets for the first registry.

Rough share of effort: about 10 to 15 percent of the project; most effort goes into policy, ownership and integration.

Implementation paths

Which path for whom

• Enterprise: A. Platform-native governance and observability products are mature commodity functions and plug into the identity and cloud stack already in place.
• Developer: C. Open-source tracing and evaluation (e.g. Langfuse, OpenTelemetry) is cheap to run and keeps traces with prompts and personal data under your control.
• Public sector: C. Agent traces contain citizen data, so self-hosting on a sovereign EU or German cloud is easier to justify in procurement than a US SaaS console.

Path A — Buy (recommended for most enterprises)

• Platform-native governance: Microsoft Agent 365 (registry, Entra, Purview, Defender; standalone licence or Microsoft 365 E7 bundle, reported pricing should be verified with Microsoft) [1]; Salesforce Agentforce Command Center for Agentforce agents; ServiceNow AI Control Tower; Google and AWS agent platform consoles. Strong inside their own ecosystem; cross-cloud sync was still in preview per the Microsoft roadmap, so verify before relying on it.
• Observability and evaluation vendors: LangSmith, Arize, Datadog LLM Observability, Weights & Biases Weave, Braintrust; Langfuse (Berlin-founded, EU cloud region, EU/open-source) [4].
• AI governance platforms (registry, risk assessment, compliance workflows): e.g. Credo AI, IBM watsonx.governance, OneTrust, and EU players such as appliedAI-linked and German/French GRC vendors; check AI Act mapping and EU hosting.
• Choose A when: most agents sit on one or two platforms, and a fast audit story matters more than flexibility.

Path B — Build on cloud APIs

• Instrument agents with OpenTelemetry GenAI conventions [2] and send spans to an existing backend (Grafana, Elastic, Azure Monitor, Datadog) or an LLM-specific one.
• Put an LLM/agent gateway in front of model and tool access (LiteLLM, Portkey, Kong AI Gateway, Azure API Management) for keys, budgets, logging and policy; use MCP gateways for tool allow-lists.
• Identity via the existing IdP (Entra, Okta, Keycloak) with workload identity and token exchange.
• Judge models via EU-region APIs (Azure OpenAI EU, Vertex EU, Bedrock Frankfurt, Mistral) with a different model family than the agent's own to reduce self-preference bias.
• Choose B when: heterogeneous agent estate, strong platform team, need custom approval workflows.

Path C — Self-host (open source)

• Langfuse (MIT core, Docker/Kubernetes self-hosting, OTel ingest, datasets, LLM-as-judge) [4]; Arize Phoenix; OpenLIT; Opik; MLflow tracing; OpenLLMetry.
• Evaluation: promptfoo, DeepEval, Ragas, Inspect AI, lm-evaluation-harness for regression suites.
• Policy and guardrails: Open Policy Agent for authorisation decisions, NeMo Guardrails, Llama Guard / Granite Guardian classifiers (see SEC-010).
• Judge models: open-weight models such as Qwen3, Llama, Mistral or gpt-oss class served with vLLM.
• Hardware: telemetry stack runs on CPU servers (HW1); a self-hosted judge model for volume needs HW2-HW3.
• Ops effort: significant; storage for traces (ClickHouse-based in Langfuse) grows quickly.

Path D — Custom training

Rarely. Worth considering: a fine-tuned small judge or classifier for domain-specific quality criteria (trained on a few hundred to thousands of human-labelled traces), and anomaly detection on tool-call sequences. Reviewer labels from production are the natural training data.

State of the art (2026)

• Standardisation: OpenTelemetry GenAI semantic conventions define agent, model and tool spans; many frameworks and vendors now emit them, but attribute names have changed between versions (e.g. gen_ai.system became gen_ai.provider.name), so pin versions [2].
• Registry as a product category: hyperscaler and SaaS vendors ship agent registries; cross-vendor discovery of unmanaged agents is still the weakest part, relying on identity, network and repository signals.
• Agent identity: distinct non-human identities, token exchange and on-behalf-of flows are replacing shared API keys; standards work (OAuth profiles for MCP and agents) is ongoing.
• Evaluation: trajectory-level and outcome-level evals, simulation of users and tools, and LLM-as-judge with calibration are standard; agent benchmarks (e.g. tau-bench, SWE-bench Verified, WebArena/OSWorld for computer use) inform model choice but do not replace in-house evals.
• Limits and failure modes: LLM judges are biased (verbosity, self-preference, position) and drift with judge-model updates; traces with full prompts become a personal-data and secrets store; non-determinism makes regression thresholds statistical; kill switches only work if the agent's credentials are actually revocable and downstream side effects can be compensated; platform governance covers visibility, access and policy, not business correctness; vendors' "governance" claims often stop at their own ecosystem.

Shared resources & combinations

• SEC-010: runtime guardrails and prompt-injection defence are enforced by the policy layer here; AgentOps supplies registry, identity and evidence.
• CODE-018: the same tracing/eval stack (Langfuse, LangSmith, Phoenix) serves LLM application evaluation.
• AGENT-002 / AGENT-013 / AGENT-003: the agents being governed; AGENT-003 computer-use agents need the strictest approval and session recording.
• AGENT-018: continuous controls monitoring consumes the same logs; AGENT-016: ITSM tickets carry access requests and incidents for agents.
• Shared infrastructure: IdP, gateway, SIEM, log store (HW1), judge-model serving (HW2-HW3).

Effort, cost & KPIs

• Path A: weeks for a first registry and policies inside one platform; licence per user or per agent/message, plus integration with SIEM and GRC.
• Path B: 3-6 months for gateway, tracing, identity and approval flows; cost drivers are trace storage, judge-model tokens (evaluating can cost as much as running) and platform-team time.
• Path C: similar effort plus operations of the stack.
• KPIs: share of agents registered with owner and risk class; share of shadow agents discovered vs. remediated; share of agents with unique identity and least-privilege review in the last 12 months; task success rate and regression-gate pass rate; cost per successful task; mean time to detect and to stop a misbehaving agent; share of consequential actions with logged human approval; incidents per 1,000 runs.

People & change

Start with management and the IT security officer: they set the agent policy (who may deploy which agent with which permissions) before tooling is chosen (GUIDE-007, GUIDE-013). Involve IT for identity, logging and integration, the DPO for log content, retention and a DPIA where traces contain personal data (GUIDE-005), and legal/compliance for AI Act evidence (Art. 12, 14, 26; GUIDE-014).

Works council: yes in practice. Traces and audit logs record who triggered which agent action, so the system is objectively suited to monitor employee behaviour and performance (§87(1) no. 6 BetrVG, BAG case law). Inform the council early under §90(1) no. 3, consider an expert under §80(3), and agree on a works agreement that limits evaluation of individual staff (GUIDE-004). In public bodies the Personalrat applies.

Pilot with one or two agents and their owners. Typical concerns: surveillance of staff (address with purpose limitation and aggregated reporting), bureaucracy slowing teams (provide a lightweight registration path). AI literacy: the AI Act asks providers and deployers to take measures that support staff AI literacy; train agent owners and reviewers (GUIDE-006). Keep humans in control through approval thresholds and a working kill switch.

Risks, regulation & compliance

• EU AI Act: classification depends on what each agent does. Governance tooling itself is typically minimal risk, but governed agents may be high-risk (Annex III: e.g. employment, credit, essential services, public benefits). For high-risk systems: automatic event logging (Art. 12), human oversight designed in so people can understand, monitor, override and stop the system (Art. 14), deployer duties including log retention of at least six months, monitoring and informing workers (Art. 26) [3]. Art. 4 AI literacy applies to all deployers. Art. 50 transparency applies to agents that interact with people. Art. 5 prohibitions (e.g. emotion recognition at work, manipulative techniques, social scoring) must be screened for at registration; an agent in a prohibited category must be blocked, not just flagged. Annex III obligations apply from 2 Dec 2027 (Digital Omnibus on AI, Reg. (EU) 2026/1744, in force since 27 Jul 2026).
• GDPR: traces and eval datasets contain personal data; apply data minimisation, redaction before storage, retention limits, access control, DPIA where agents process personal data at scale, and Art. 22 safeguards if agents make solely automated decisions with legal or similarly significant effects. Monitoring employees via agent logs engages works-council co-determination in Germany (BetrVG section 87).
• Sector rules: DORA and EBA/BaFin outsourcing and ICT-risk rules, NIS2 (security of agent identities), ISO/IEC 42001 AI management systems, NIST AI RMF for structure.
• Risks and mitigations: over-trust in LLM judges (calibrate with human labels); log sprawl of sensitive data (redaction, tiered retention); governance theatre (registry without enforcement); tool lock-in (use OTel and open schemas); a kill switch that cannot undo actions (design compensating transactions, limit blast radius).

Getting started

1. Build an inventory in 30 days: platform APIs, IdP application and service-principal lists, repository scans, MCP/tool endpoints; assign a human owner to each agent.
2. Define a risk taxonomy (data classes, action types, AI Act category) and approval rules per class; screen for Art. 5 and Annex III.
3. Move agents off shared credentials to per-agent identities with scoped, short-lived tokens; put a gateway in front of model and tool access.
4. Instrument with OpenTelemetry GenAI spans; start with redacted traces for the top five agents by risk or spend.
5. Create 50-200 golden test cases per agent, add a regression gate in CI, sample production traces weekly for human-labelled judge calibration; rehearse the kill switch in a drill.

Sources

1. Microsoft Learn — Microsoft Agent 365 overview — https://learn.microsoft.com/en-ca/microsoft-agent-365/overview
2. OpenTelemetry — Semantic conventions for GenAI agent and framework spans — https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans
3. Regulation (EU) 2024/1689 (AI Act), Art. 12, 14, 26 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
4. Langfuse — open-source LLM engineering platform (self-hosting, EU cloud) — https://langfuse.com/docs
5. AI Act Service Desk / Art. 14 text — https://www.artificialintelligenceact.eu/article/14/
6. Open Policy Agent — https://www.openpolicyagent.org
7. Arize Phoenix — https://github.com/Arize-ai/phoenix
8. NIST AI Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework