LLM & agent application evaluation, tracing & observability (LLMOps)

GenAI features fail quietly: answers stay fluent while becoming wrong, costly or non-compliant after a prompt tweak, a model upgrade or a data change. LLMOps platforms record every model call, tool call and retrieval step as a trace, score outputs offline on golden datasets and online on live traffic, and block releases that regress. The result is measurable quality, controlled cost and the testing and logging evidence regulators increasingly expect.

Problem & value

• Today: teams ship a chatbot or agent after a handful of manual spot checks, then learn about failures from angry users. Prompts live in code or spreadsheets, nobody knows which version produced which answer, and a model swap is a leap of faith. Classic unit tests (CODE-004) do not work for non-deterministic text, and log tools (CODE-011) do not understand prompts, retrieval or agent trajectories.
• AI value: reproducible quality gates, faster debugging of multi-step agents (which tool call went wrong), safe model and prompt changes, cost and latency control per feature, and evidence for audits.
• Evidence base: LLM-as-a-judge agrees well with human raters on many tasks but has known biases (position, verbosity, self-preference), so judges must be calibrated against human labels [5]. No vendor ROI figure is quoted here; measure your own baseline.

Variants

• RAG evaluation: retrieval precision/recall, context relevance, faithfulness to sources (see KNOW-002).
• Agent trajectory evaluation: tool-selection correctness, step count, loop detection, task success.
• Chatbot and voice-agent testing with simulated users: before release, LLM-driven personas with goals, accents, interruptions and background noise call or chat with the agent in CI/CD; judges score task completion, policy adherence and tone. Tools: Coval, Cekura, Hamming (US-based SaaS; check hosting). Cross-reference CONV-001, CONV-002 and CONV-026.
• GDPR self-hosting: run tracing and evaluation entirely inside your own EU infrastructure so prompts, which often contain customer data, never reach a third-party SaaS (Path C).
• Red-team style evaluation: adversarial suites for jailbreaks and leakage, see SEC-014 and SEC-010.
• Private/hobby use: local Phoenix or Langfuse via Docker Compose to compare prompts and local models.

How it works

1. app instrumentation (OpenTelemetry / SDK)
2. trace store (LLM calls, tool calls, retrieval, sessions, cost, latency)
3. dataset curation (production failures
4. golden test cases)
5. offline evaluation (heuristics + LLM judge + human annotation queue)
6. regression gate in CI/CD (compare candidate prompt/model against baseline)
7. release
8. online evaluation on sampled live traces + user feedback
9. alerts and dashboards (drift, hallucination rate, cost, latency)
10. failures feed back into datasets

• Tracing: each request becomes a tree of spans. OpenTelemetry GenAI semantic conventions define attributes for model calls, agent steps and MCP tool calls, which keeps instrumentation vendor-neutral [2].
• Evaluators: deterministic checks (JSON schema, regex, exact match), embedding similarity, LLM judges with rubrics, and metric libraries such as Ragas or DeepEval for RAG and agents.
• Prompt management: versioned prompts with labels (staging, production) decoupled from code deploys.
• Human-in-the-loop: domain experts label samples in annotation queues; their labels calibrate the judges and promote hard cases into the golden set. Release approval and judge-rubric changes stay with humans.

Data preparation

The data are traces, test cases and labels, so preparation means building a quality baseline (roughly 25 to 35 percent of effort). Raw material lives in application logs, ticket systems, chat transcripts, support mailboxes, spreadsheets of prompts and in domain experts' heads. Typical gaps: no instrumentation, prompts scattered across code and wikis, no expected answers, personal data inside prompts and outputs, and no agreed definition of a "good" answer.

Preparation steps: - Instrument one application with OpenTelemetry or the vendor SDK and mask personal data at source (PREP-022). - Harvest 50 to 100 real cases, including known failures, from tickets and user feedback (PREP-020, PREP-021). - Let domain experts write expected answers or rubrics and label 30 or more samples to calibrate judges (PREP-018, PREP-016). - Define category schemes for failure types before classifying them (PREP-019). - Set retention and access rules for trace data (PREP-024, PREP-025). - Where source documents for RAG are messy, fix them first (PREP-013).

The expert time for labelling is the real bottleneck, not tooling. See GUIDE-003 for a readiness check.

Implementation paths

Which path for whom

• smb: Path C — Langfuse or Phoenix run on a small server (HW1), traces contain customer data and stay in your own EU infrastructure, and the same platform serves every AI feature you add later.
• enterprise: Path C — a shared internal observability platform for all GenAI teams avoids per-trace SaaS fees at volume, keeps prompts inside the company and can plug into the existing OpenTelemetry stack.
• developer: Path C — Docker Compose with Langfuse, Phoenix or Promptfoo gives full control, no vendor lock-in and works with local models.

Path A — Buy (off-the-shelf)

• Observability and evals SaaS: LangSmith (LangChain), Braintrust, Arize AX, Weights & Biases Weave, Datadog LLM Observability, Langfuse Cloud (EU data region; Langfuse GmbH, Berlin, now part of ClickHouse), Helicone, Maxim. Hyperscaler options: Azure AI Foundry evaluations, Amazon Bedrock evaluations.
• Voice/chat simulation: Coval, Cekura, Hamming.
• Choose A when: small team, no sensitive prompts, need results in days. Check data region, retention, PII masking, subprocessors and DPA.

Path B — Build on cloud APIs

• Judge models: a strong frontier model (Claude, GPT, Gemini, Mistral Large) via EU-region endpoints as judge, with structured outputs for scores plus reasoning. Use a different model family than the one under test to reduce self-preference.
• Glue: OpenTelemetry SDK or OpenLLMetry/OpenInference instrumentation into any OTLP backend, evaluation scripts in CI (GitHub Actions, GitLab CI).
• Choose B when: you already run an observability stack and only need the evaluation layer.

Path C — Self-host (open source; recommended for GDPR-sensitive teams)

• Platforms: Langfuse (Docker Compose for tests; Helm/Kubernetes, Terraform modules for AWS, Azure, GCP in production; components Postgres, ClickHouse, Redis/Valkey, S3) [1]; Arize Phoenix (OpenTelemetry-native, single container); Opik (Comet); Promptfoo and DeepEval for CI test suites; Ragas for RAG metrics.
• EU hosting: run on EU sovereign or EU-based clouds (e.g. Hetzner, IONOS, OVHcloud, STACKIT, Scaleway) or on-prem.
• Judges: open-weight models (e.g. Qwen3, Llama, Mistral, gpt-oss families) served via vLLM or Ollama; smaller judges need more calibration. The observability stack itself needs only CPU (HW1); add a GPU (HW2/HW3) only if judges run locally.
• Ops effort: ClickHouse and storage sizing, retention jobs, upgrades; about a few days of platform work.

Path D — Custom training

Fine-tune a small judge or reward model on your own human labels when judge cost at volume is high or domain rubrics (legal tone, medical safety) are not captured by general models. Needs hundreds to a few thousand consistently labelled examples and an inter-annotator agreement check.

State of the art (2026)

• Standardisation: OpenTelemetry GenAI conventions (model calls, agent spans, MCP) are converging, and most platforms ingest OTLP, reducing lock-in [2].
• Agent-level evaluation (trajectory, tool-use and multi-turn simulated conversations) has replaced single-prompt scoring as the focus; benchmark-style suites such as tau-bench test tool-using agents against simulated users [6].
• Online evals and annotation queues run on sampled production traces, with failures auto-promoted to datasets.
• Limits: judges are biased and can be gamed; scores on synthetic personas do not equal real-user behaviour; non-determinism requires repeated runs and confidence intervals; trace storage grows quickly and contains personal data; metrics like "faithfulness" are proxies, not ground truth.

Shared resources & combinations

• Quality gate for CONV-001/002/026 (chat and voice), KNOW-002 (RAG), AGENT-002 and AGENT-013 (agents), CODE-002 (coding agents).
• Adjacent: CODE-004 (deterministic tests), CODE-011 and CODE-010 (logs, AIOps; share the OTel collector and ClickHouse/Grafana), SEC-010 (guardrails run inline, evals measure them), SEC-014 (red teaming).
• Shared infrastructure: one self-hosted vLLM judge server can serve several teams; one trace store for all apps.

Effort, cost & KPIs

• Path A: days; priced per trace/event or seat. Path B: 2-6 weeks. Path C: 2-6 weeks incl. hardening; cost is infrastructure plus operations. Judge calls often dominate cost: sample online traffic.
• KPIs: eval pass rate versus baseline, hallucination/groundedness rate, judge-human agreement, regression escapes to production, mean time to diagnose failures, cost and p95 latency per request, share of production failures converted into test cases.

People & change

• Involve: management (quality and risk appetite), the owners of each GenAI feature, developers and domain experts who label samples, IT for hosting, IT security for access control and secrets in traces, and the DPO because traces hold prompts with personal data.
• Works council: yes. Traces of internal assistants record what employees ask and when; under §87(1) no. 6 BetrVG objective suitability for monitoring is enough (BAG), so a works agreement is advisable. Define purpose (quality, not performance review), pseudonymised user IDs, short retention and who may open traces. Engaging an AI expert is possible under §80(3) BetrVG.
• Data protection: legal basis and data minimisation for traces, AVV (Art. 28 GDPR) with any SaaS, DPIA if staff or customer conversations are evaluated systematically at scale (Art. 35 GDPR). Annotators see personal data, so restrict access.
• AI literacy: support annotators and developers with training on judge bias and how to read evaluation results (Art. 4 AI Act as amended).
• Pilot and communication: begin with one chatbot and a small expert panel; explain that the system evaluates the AI, not the users.
• Concerns: "this is surveillance" — address with the works agreement and pseudonymisation.
• Humans in control: release approval and judge rubrics stay with people.

See GUIDE-004, GUIDE-005 and GUIDE-011.

Risks, regulation & compliance

• EU AI Act: the tooling itself is minimal risk. It supports obligations for the systems it monitors: high-risk systems need automatic logging (Art. 12), accuracy and robustness testing (Art. 15), and post-market monitoring (Art. 72); chatbots need transparency (Art. 50) [3]. Monitoring simulated-user tests must not include emotion recognition of workers or students (Art. 5 prohibition) when evaluating workplace or education assistants.
• GDPR: traces hold prompts and outputs with personal data (customers, employees). Need a legal basis, data minimisation, PII masking before storage, retention limits, DPA with SaaS vendors, transfer safeguards for US vendors, and no use of trace data for vendor model training [4]. Self-hosting in the EU is the simplest answer. Human annotators see personal data: restrict access.
• Risks and mitigations: judge bias (calibrate against humans, rotate judge models); leaking secrets in traces (redaction at SDK level); vendor lock-in (OTLP); Goodhart effects when teams optimise to the eval set (hold out a private test set, refresh with production data).

Getting started

1. Instrument one application with OpenTelemetry or the Langfuse/Phoenix SDK; mask PII at source.
2. Collect 50-100 real cases (including known failures) into a golden dataset with expected behaviour.
3. Write 3-5 evaluators: schema/format checks, one groundedness judge, one policy judge; label 30 samples by hand and measure judge agreement.
4. Add the suite (e.g. Promptfoo) to CI and block merges that lower scores; move prompts into a registry.
5. Turn on sampled online evaluation, cost/latency alerts and a weekly review of failed traces.

Sources

1. Langfuse - self-hosting documentation - https://langfuse.com/self-hosting
2. OpenTelemetry - Semantic conventions for generative AI - https://opentelemetry.io/docs/specs/semconv/gen-ai/
3. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
4. Regulation (EU) 2016/679 (GDPR) - https://eur-lex.europa.eu/eli/reg/2016/679/oj
5. Zheng et al. - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - https://arxiv.org/abs/2306.05685
6. Yao et al. - tau-bench: tool-agent-user interaction benchmark - https://arxiv.org/abs/2406.12045
7. Arize Phoenix - open-source LLM tracing and evaluation - https://github.com/Arize-ai/phoenix
8. Promptfoo - LLM evaluation and red-teaming - https://github.com/promptfoo/promptfoo