Data catalogue & metadata enrichment

Most companies hold far more data than anyone can find or understand: tables named kna1_x2, columns without descriptions, no idea who owns what or where personal data sits. AI drafts descriptions, tags, glossary links and sensitivity labels from schemas, profiles and query logs, so a catalogue becomes complete enough to be useful, and so natural-language BI and AI agents have the context they need to answer correctly.

Problem & value

• Today: data catalogues (and Excel lists) are filled by hand by overstretched data stewards. Coverage stays low, descriptions go stale, and analysts ask colleagues "which table has the real revenue?". Privacy teams need a record of where personal data lives (GDPR Art. 30) and compile it manually.
• AI value: (1) generate first-draft descriptions and glossary mappings for thousands of assets, reviewed by owners instead of written from scratch; (2) classify columns by sensitivity (names, e-mail, IBAN, health data) from names, values and context; (3) semantic search and Q&A over the catalogue; (4) quality metadata for text-to-SQL and agents, which is the main driver in 2026: LLM answers are only as good as the semantic layer and documentation behind them (see DATA-001).
• Benefit evidence: vendors report faster documentation and better discoverability, but independent, comparable figures are scarce. Measure your own baseline (share of documented assets, time-to-find a dataset) rather than relying on marketing numbers.

Variants

• Enterprise data platform: catalogue embedded in the lakehouse or warehouse (Databricks, Snowflake, Microsoft Fabric/Purview) with automated documentation and classification.
• Data mesh / data products: enrichment enforces product contracts (owner, SLA, glossary terms) at publishing time.
• Public sector / open data: harmonised metadata for portals (DCAT-AP, DCAT-AP.de, data.europa.eu), multilingual titles and keyword suggestions; relevant for the Data Governance Act and the Open Data Directive.
• Research and data spaces: FAIR metadata, Gaia-X / IDS / Eclipse Dataspace connectors; catalogue entries described semantically.
• SMB: a lightweight catalogue or a documented schema in the BI tool, plus AI-written column descriptions for the semantic layer.
• Unstructured content: enrichment of file shares, SharePoint and document stores with topic and sensitivity labels (links to DOC-014).

How it works

1. connectors (warehouse, lake, BI, ETL, SaaS, files)
2. technical metadata harvest (schema, lineage, usage, profiling stats, sample values)
3. context retrieval (glossary, existing descriptions, naming conventions, related assets, query logs)
4. LLM / classifier enrichment (descriptions, tags, glossary terms, domain, owner suggestion, PII class)
5. confidence scoring + policy rules
6. steward review queue (accept / edit / reject)
7. publish to catalogue + propagate (tags to access policies, masking, search index)
8. feedback loop (accepted edits become few-shot examples; re-run on schema change)

• Description generation: an LLM receives the table/column names, types, sample statistics, lineage neighbours and glossary, and drafts a short description. Sample values must be handled under privacy rules (profiles or masked values, not raw rows, when sensitive).
• Classification: rules and regex (IBAN, tax ID), small encoder/NER models and LLM judgement combined; the LLM resolves ambiguous names (dob, gnr) from context.
• Semantic search: text embeddings plus keyword search over assets; optional knowledge graph for lineage and relationships.
• Agent access: catalogues increasingly expose metadata through MCP servers so coding and analytics agents can look up definitions, owners and lineage.
• Human-in-the-loop: AI output is flagged as "suggested"; data owners approve before descriptions are marked certified; sensitivity labels that trigger access restrictions need steward sign-off or conservative defaults.

Data preparation

The raw material is metadata: table and column names, SQL, dbt models, BI reports, plus tacit knowledge in Excel files, ERP customising and people's heads. In an SMB this sits in ERP databases, file shares and a few BI workbooks. Typical gaps: cryptic column names, no descriptions, duplicate tables, unknown owners, undocumented KPIs and no sensitivity labels.

Preparation steps: 1. Inventory systems and sources (PREP-001, PREP-003) and connect them via connectors (PREP-005). 2. Name data owners and stewards per domain (PREP-004). 3. Write a first glossary of KPIs and business terms (PREP-012, PREP-015). 4. Define the tag and category scheme (sensitivity, domain, personal data) before the AI proposes tags (PREP-019, PREP-010). 5. Let the LLM draft descriptions and tags, then have stewards confirm them.

Tools: OpenMetadata, DataHub, Purview, or the catalogue of the warehouse. Rough share of effort: about 40 percent, mostly the human work of agreeing definitions and owners. This use case is itself a preparation step for DATA-008, DATA-003 and DOC-014.

Implementation paths

Which path for whom

• smb: Path A. Most SMBs have few data sources; use the AI documentation features of tools already in place (BI tool, cloud warehouse, Microsoft Purview) instead of running a catalogue.
• enterprise: Path B. A catalogue is core to data governance and must integrate with own warehouses and lineage; add LLM enrichment through cloud APIs on metadata only.
• developer: Path C. OpenMetadata or DataHub with a local model give full control and run on HW2.
• public: Path C. Open-source catalogues on sovereign infrastructure avoid lock-in and keep metadata inside the administration.

Path A: Buy (recommended for most)

• Platform-native: Databricks Unity Catalog (AI-generated comments and AI-assisted classification), Snowflake Horizon Catalog (AI-generated descriptions and sensitive-data classification), Microsoft Purview (classification and Unified Catalog, inside Fabric), Google Dataplex Universal Catalog (automated metadata and Gemini-generated insights), AWS SageMaker Catalog / DataZone (AI-generated business descriptions).
• Independent catalogues: Collibra, Alation, Atlan, Informatica CDGC, data.world, Secoda; EU: Collibra (Belgian origin, global), Actian Data Intelligence Platform (formerly Zeenea, France; Actian/HCLSoftware since 2024), Ataccama (Czech/Canadian, data quality + catalogue), Semarchy (France, MDM). Check EU data residency and which LLM sub-processor is used.
• Choose A when: you already run one of these data platforms or need a governed enterprise catalogue quickly.

Path B: Build on cloud APIs

• Harvest metadata with open connectors (DataHub or OpenMetadata ingestion, dbt artifacts, information_schema), then call an LLM API with structured output (JSON schema) for descriptions, tags and glossary matches.
• Models: Claude, GPT, Gemini via EU regions (Vertex AI EU, Azure OpenAI EU, AWS Bedrock Frankfurt) or Mistral (EU provider); a small model suffices for descriptions, a stronger one for ambiguous classification.
• Write results back through the catalogue API (DataHub GraphQL/REST, OpenMetadata REST, Unity Catalog and Purview APIs). Use dbt description YAML files via pull requests so every AI suggestion goes through code review.
• Choose B when: you need custom conventions, multi-catalogue consolidation or a PR-based documentation workflow.

Path C: Self-host (open source)

• Catalogues: DataHub (Apache-2.0), OpenMetadata (Apache-2.0, built-in AI features in recent releases), Amundsen (low activity), Apache Atlas, OpenLineage/Marquez for lineage, Unity Catalog OSS (Linux Foundation).
• Models: open-weight LLMs such as Qwen3, Mistral Small/Medium, Llama or gpt-oss via vLLM or Ollama; embeddings such as BGE-M3 or multilingual-e5; Microsoft Presidio for PII detection.
• Hardware: HW2 is enough for a 8-30B quantised model producing descriptions in batch; metadata volumes are small. This keeps sample values inside your network, which suits sensitive schemas.

Path D: Custom training

Rarely needed. Fine-tune a small classifier for company-specific sensitivity classes or an embedding model for domain jargon (ERP abbreviations, German field names) when you have a few thousand steward-approved labels. Accepted/rejected suggestions from the review queue are the natural training data.

State of the art (2026)

• Catalog as AI context layer: the major platforms position their catalogues as grounding for agents and natural-language analytics, with metadata served via MCP and semantic-layer standards.
• Auto-documentation is mainstream in platforms: AI-generated table and column comments exist in Unity Catalog, Snowflake, DataHub, OpenMetadata and Purview; quality is good for self-explanatory schemas, weak for cryptic legacy names without lineage or glossary context.
• Context beats model size: descriptions improve markedly when lineage, query logs, dbt models and glossary entries are supplied.
• Limits and failure modes: plausible but wrong descriptions that are then trusted; leaking sensitive sample values to external models; classification misses on free-text columns; description drift after schema changes; reviewers rubber-stamping suggestions. Counter with "suggested vs certified" status, sampling audits and re-validation triggers.

Shared resources & combinations

• Same stack (llm-general + embedding-text + vector-db): DATA-001 (text-to-SQL uses the catalogue as grounding), DOC-014 (RAG over documents), DATA-003 (MDM and deduplication).
• Combines with: DATA-008 (quality monitoring attaches quality scores to assets), DATA-016 (industrial tag and asset-model mapping), DATA-022 (schema mapping for partner onboarding), SEC-022 (AI inventory and governance uses the same catalogue entries), SEC-020 (access reviews rely on sensitivity labels).
• One self-hosted LLM server (HW2/HW3) can serve all enrichment batch jobs.

Effort, cost & KPIs

• Path A: days to weeks to switch on native features; licences for enterprise catalogues are a significant cost driver and usually negotiated per platform.
• Path B: 1-3 months incl. connectors and review workflow; LLM cost is negligible compared with steward time because metadata volumes are small.
• Path C: 2-4 months; plus operating the catalogue itself.
• KPIs: share of assets with certified description/owner/domain, steward acceptance rate of AI suggestions, time-to-find a dataset, PII classification precision/recall on a labelled sample, share of sensitive columns with policy applied, accuracy of text-to-SQL before/after enrichment.

People & change

Involve management for the mandate, the department leads and data owners early, because they must confirm AI-drafted descriptions; IT connects the sources. The DPO contributes the classification of personal data and checks that column samples sent to an LLM contain no real personal data; a DPIA is normally not needed for metadata alone.

Works council: no in the typical case, as the catalogue documents data, not people. It becomes relevant when usage logs show which employee queries what; then §87(1) no. 6 BetrVG applies and the council should see the configuration. Inform it under §90(1) no. 3 if work processes change.

AI literacy: brief stewards on checking generated descriptions (Art. 4 AI Act, support measures). Pilot with one domain such as sales data, show analysts the time saved finding the right table, and publish the glossary. Typical concerns: extra documentation work and fear of exposing poor data quality. Address both by letting the AI draft and people only approve, and by treating gaps as findings, not blame. Guides: GUIDE-003, GUIDE-008, GUIDE-005.

Risks, regulation & compliance

• EU AI Act: minimal risk; AI-literacy duty (Art. 4) applies. Classification could change if outputs feed a high-risk system (e.g. data governance evidence for Art. 10 of a high-risk AI system), so keep the audit trail.
• GDPR: the catalogue is a tool for Art. 30 records, data-minimisation and DPIAs. Metadata can itself hold personal data (owner names, sample values), so mask samples, use DPAs with LLM providers and prefer EU processing. Do not rely on AI classification alone to conclude that a column contains no personal data.
• Data Act and Data Governance Act: machine-readable metadata supports data-sharing obligations and data-space participation.
• Security: catalogues expose the map of your data; apply access control to metadata, and defend against prompt injection from description text that agents read.
• Mitigations: human approval, conservative sensitivity defaults, source and model recorded on each suggestion, periodic audits.

Getting started

1. Inventory the sources and measure baseline coverage (documented assets, owners, sensitivity labels).
2. Pick 1-2 high-value domains (e.g. sales and customer) and load glossary plus lineage first.
3. Switch on native AI documentation or run a script that drafts descriptions into a staging area.
4. Have stewards review 100 suggestions, record the acceptance rate, and tune prompts and context.
5. Connect the certified metadata to text-to-SQL (DATA-001) and to access policies; repeat per domain.

Sources

1. Databricks docs, AI-generated comments in Unity Catalog: https://docs.databricks.com/aws/en/comments/ai-comments
2. Snowflake docs, Sensitive data classification: https://docs.snowflake.com/en/user-guide/classify-intro
3. DataHub project: https://github.com/datahub-project/datahub
4. OpenMetadata project: https://github.com/open-metadata/OpenMetadata
5. Microsoft Purview documentation: https://learn.microsoft.com/purview/
6. OpenLineage: https://openlineage.io/
7. Microsoft Presidio: https://github.com/microsoft/presidio
8. DCAT-AP, data.europa.eu: https://interoperable-europe.ec.europa.eu/collection/semic-support-centre/dcat-ap
9. Regulation (EU) 2016/679 (GDPR) Art. 30: https://eur-lex.europa.eu/eli/reg/2016/679/oj
10. Regulation (EU) 2024/1689 (AI Act): https://eur-lex.europa.eu/eli/reg/2024/1689/oj