Data quality monitoring & observability

Dashboards, reports and ML models silently break when upstream data is late, duplicated, truncated or drifts. Data quality monitoring combines explicit rules (tests, data contracts) with learned baselines that flag unusual freshness, volume, schema and distribution changes, and LLMs that draft rules, summarise incidents and trace likely root causes through lineage. The benefit is catching problems before business users do.

Problem & value

• Today: analysts discover wrong numbers in a report; engineers hand-write SQL checks for a fraction of tables; alerts are noisy and nobody knows which dashboards are affected.
• AI value: automatic baselines per table and column (seasonality-aware), rule suggestions from profiling, fewer false alarms through alert grouping, root-cause hints from lineage and recent changes, natural-language incident summaries.
• Why it matters in the EU: GDPR Art. 5(1)(d) requires personal data to be accurate; the AI Act Art. 10 requires data governance for training data of high-risk systems; BCBS 239 and Solvency II impose data quality duties in finance. Vendors publish ROI claims; treat them as marketing unless backed by your own baseline.

Variants

• SMB analytics stack: dbt tests plus freshness checks on a single warehouse, alerts to chat.
• Enterprise data platform: observability across hundreds of pipelines, lineage-based impact analysis, data contracts between domains (data mesh).
• ML/feature data: drift and training-serving skew monitoring, see CODE-018.
• Regulated reporting: quality gates before supervisory or ESG submissions (DATA-012, DATA-020).
• Public sector / open data: quality scoring of published datasets.
• Master data: match/merge quality tracked via DATA-003.

How it works

1. sources (warehouse, lakehouse, streams, orchestrator, BI)
2. metadata & metric collection (row counts, freshness, schema, profiles, query logs)
3. rules layer (declarative tests, data contracts, constraints)
4. learned monitors (time-series baselines, anomaly detection per metric)
5. lineage graph (upstream cause, downstream impact)
6. LLM layer (rule suggestion, incident summary, fix proposal as pull request)
7. alert routing (ticket/chat, owner from catalogue)
8. human triage
9. resolution & feedback

• Rules: deterministic checks (not null, unique, referential integrity, accepted values, freshness). LLMs can propose them from profiles and schema, but a human approves.
• Learned monitors: per-metric forecasts with prediction intervals; alerts when actuals leave the band. Classical statistics and gradient boosting are usually enough; deep models rarely pay off.
• Root cause: lineage plus change history (deploys, schema migrations, failed jobs) narrows candidates; an LLM summarises evidence but must cite it.
• Human-in-the-loop: data owners confirm or dismiss alerts; feedback tunes sensitivity. Blocking a pipeline run on a failed check is a policy decision, not an AI decision.

Data preparation

Monitoring needs access to tables and pipeline metadata in the ERP, warehouse, BI tool and the files that feed Excel reports. Typical gaps: no agreed definition of "correct", missing owners, undocumented transformations, manually maintained Excel inputs, no baselines or history, and alerts nobody is responsible for.

Preparation steps: 1. Inventory critical datasets and rank them by business impact (PREP-001, PREP-002). 2. Extract data reliably via APIs or connectors instead of manual exports (PREP-005). 3. Document KPI definitions and valid ranges with the department (PREP-012). 4. Name an owner per dataset (PREP-004) and agree what happens on an alert. 5. Start with simple rules (freshness, volume, nulls, uniqueness), let the AI propose further rules from profiling, and have owners approve them. 6. Collect confirmed incidents as labelled examples for anomaly detection (PREP-020).

Tools: Great Expectations, Soda Core, dbt tests. Rough share of effort: about 30 to 40 percent, mostly defining expectations and ownership rather than coding checks. See also DATA-007 and DATA-003.

Implementation paths

Which path for whom

• smb: Path C. Great Expectations, Soda Core or dbt tests run on HW1, keep data inside the company and serve the same checks across several pipelines.
• enterprise: Path B. Pipelines are business-critical and diverse; combine own tests with cloud APIs for rule suggestions and root-cause explanations on metadata, or buy an observability platform for commodity needs.
• developer: Path C. Open-source testing frameworks integrate directly into CI and orchestration with full control.
• public: Path C. Open-source tools on sovereign infrastructure keep operational data in the administration and avoid lock-in.

Path A — Buy (off-the-shelf)

• Data observability platforms: Monte Carlo, Bigeye, Anomalo, Acceldata, Sifflet (EU, France), Soda Cloud (EU, Belgium), Telmai, Metaplane (Datadog), Validio (EU, Sweden), Synq (EU, Denmark).
• Catalogue/governance suites with quality modules: Collibra, Informatica, Ataccama (EU, Czechia), Alation, Atlan, Talend/Qlik.
• Warehouse-native: Snowflake data quality monitoring and data metric functions, Databricks Lakehouse Monitoring and Unity Catalog, BigQuery/Dataplex data quality scans, Microsoft Purview.
• Choose A when: many pipelines, a small data team and a need for fast anomaly coverage without writing rules. Check EU hosting, whether only metadata (not row data) leaves your environment, and lineage coverage for your stack.

Path B — Build on cloud APIs

• Compute metrics in SQL in your warehouse, store them in a metrics table, and run forecasting with the warehouse's built-in anomaly functions or a managed time-series service.
• Use an LLM API (EU region or EU provider such as Mistral) to draft tests, summarise failed runs from logs and propose dbt/SQL fixes. Send metadata and aggregates, not raw rows, where possible.
• Orchestrate with Airflow, Dagster or Prefect; alert via chat and ticketing. Choose B for custom workflows on top of an existing warehouse.

Path C — Self-host (open source)

• Testing: Great Expectations (GX Core), Soda Core, dbt tests and dbt-expectations, Pandera, Deequ (Spark), Pydantic/Frictionless for files.
• Observability: Elementary (dbt-native anomaly tests and reports), OpenLineage with Marquez or DataHub/OpenMetadata for lineage and quality, Evidently for ML data drift.
• Data contracts: datacontract-cli with the Open Data Contract Standard.
• LLM assist: a small open-weight model (e.g. Qwen3 or Mistral Small class via Ollama/vLLM) is enough for rule drafting and summaries; metric forecasting runs on CPU.
• Hardware: HW1; a CPU server suffices, optionally HW2 for local LLM assistance. Ops effort is low to medium.

Path D — Custom training

Rarely needed. Worth considering for domain-specific anomaly models (e.g. sensor data quality, DATA-016) or a fine-tuned model to generate rules in your house style. Needs labelled past incidents; most organisations have too few, so start with Paths A-C.

State of the art (2026)

• Convergence: testing frameworks, catalogues, lineage (OpenLineage) and observability are merging into one "data reliability" layer; vendors add LLM assistants for rule generation, natural-language questions about incidents and agentic triage.
• Data contracts shift quality checks left to producers; contract tests run in CI before deployment.
• Unsupervised monitors work well for volume, freshness and schema, and moderately for distribution shifts; business-semantic errors (a valid but wrong currency) still need explicit rules or domain knowledge.
• Limits: alert fatigue, cold start for new tables, seasonality and holidays (use calendars), LLM root-cause hypotheses that sound plausible but are wrong, and the cost of scanning large tables. Prefer metadata-based and sampled checks.

Shared resources & combinations

• Catalogue and lineage: DATA-007 supplies owners and lineage; quality scores flow back into the catalogue.
• Cleaning and MDM: DATA-003 fixes what monitoring finds; DATA-022 applies checks to partner data onboarding.
• Consumers: DATA-001 and DATA-002 gain trust when answers show data freshness and quality status; ANOM-030 detects business-level KPI anomalies as opposed to data defects.
• Engineering: CODE-011 (log summarisation) and CODE-018 / AGENT-025 (LLM and agent observability) share tracing and alerting infrastructure.
• Shared stack: time-series baselines also serve PRED-family forecasting.

Effort, cost & KPIs

• Path A: days to weeks to connect; subscription pricing typically by number of tables or monitored assets.
• Path B: 1-3 months for a tailored setup.
• Path C: 2-6 weeks for dbt/GX/Elementary on one warehouse, then ongoing rule maintenance; compute cost from scans is the main driver.
• KPIs: share of critical tables covered, incidents detected by monitoring versus by users, mean time to detect and to resolve, false-positive rate, freshness SLA adherence, number of broken dashboards, data downtime.

People & change

Involve management for the priority, the department lead as process owner for the critical reports, and data owners who receive and resolve alerts. IT builds the checks and the alert channel; information security reviews access to production data. The DPO is consulted if tables with personal data are profiled, since profiling samples may expose them; an AVV is needed if a cloud vendor sees data.

Works council: no in general, because tables and pipelines are monitored, not people. If alerts or reports name the employee who caused an error, the setup could monitor behaviour and §87(1) no. 6 BetrVG applies; design alerts around datasets, not individuals, and inform the council under §90(1) no. 3 where processes change.

AI literacy: support the data team with training on reviewing AI-suggested rules (Art. 4 AI Act). Pilot with three business-critical tables for one month, and count incidents found before users noticed. Typical concern: alert fatigue. Address it by tuning thresholds with owners and keeping humans in control of every rule and every automatic action. Guides: GUIDE-011, GUIDE-009, GUIDE-003.

Risks, regulation & compliance

• EU AI Act: minimal risk for monitoring tools; AI literacy duty (Art. 4) applies. If the monitored data trains a high-risk AI system, Art. 10 data governance obligations (relevance, representativeness, error examination) apply, and quality evidence supports them.
• GDPR: profiling reads column values and may expose personal or special-category data in samples and alerts. Mitigate with metadata-only collection, masking, access control on example rows, EU hosting and a data processing agreement. Accuracy (Art. 5(1)(d)) is a legal argument for monitoring.
• Sector rules: BCBS 239 (banks), Solvency II (insurers), DORA ICT data governance, CSRD data assurance.
• Risks: silent false negatives creating false confidence; over-blocking pipelines; LLM-suggested fixes applied without review; vendor access to production metadata.
• Mitigations: critical-table rules defined by owners, review of AI-suggested tests, audit log, staged rollout in alert-only mode.

Getting started

1. Inventory the 20 tables and dashboards that drive key decisions and name an owner for each.
2. Add freshness, volume, uniqueness and not-null tests via dbt or Soda Core; run them in CI and on schedule.
3. Turn on anomaly monitoring (Elementary or a vendor trial) in alert-only mode for four weeks and tune.
4. Wire alerts to chat and ticketing with owner routing; record incident cause and fix.
5. Introduce data contracts for the most failure-prone producer interfaces; add an LLM assistant for rule drafting last.

Sources

1. Great Expectations documentation - https://docs.greatexpectations.io/
2. Soda documentation (Soda Core, data contracts) - https://docs.soda.io/
3. Elementary data observability for dbt - https://docs.elementary-data.com/
4. OpenLineage specification - https://openlineage.io/docs/
5. Open Data Contract Standard (Bitol) - https://bitol.io/
6. Snowflake data quality and data metric functions - https://docs.snowflake.com/en/user-guide/data-quality-intro
7. Databricks Lakehouse Monitoring - https://docs.databricks.com/aws/en/lakehouse-monitoring/
8. Regulation (EU) 2024/1689 (AI Act), Art. 10 and Art. 4 - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
9. Regulation (EU) 2016/679 (GDPR), Art. 5 - https://eur-lex.europa.eu/eli/reg/2016/679/oj
10. Dataplex automatic data quality (Google Cloud) - https://cloud.google.com/dataplex/docs/data-quality-overview