Catalog / Data Readiness for AI / PREP-001

AI data-readiness assessment & data inventory

Build a lean inventory of where business data and documents live, rate quality, format and access state per source, and decide which planned AI use cases are ready to start.

Parts needed to run it yourselfAgents & automationDocument readingLanguage model

Most SMB AI projects stall not on the model but on the data: nobody knows which files exist, which are current, who may see them and whether they are readable by a machine. A short, structured inventory plus a readiness rating per source turns "we should do something with AI" into a ranked list of what can start now, what needs two days of cleanup and what must wait. Small scripts and a local model can do the boring counting and sampling; people decide.

Problem & value

  • Who: IT managers and managing directors of 20-500 person companies who have bought or been offered a Copilot, a chatbot or a document-AI tool and now have to say "which data do we feed it?".
  • Today: knowledge sits in file-share folders, mailboxes, Excel sheets, DATEV, an ERP and paper. Nobody owns the overview; the answer comes from the one colleague who "knows where things are".
  • Why AI helps: crawling a file share and counting types, ages, duplicates and sizes is a script job; classifying a sample of documents ("contract / offer / personal data / obsolete") is a small-LLM job. This cuts the inventory from weeks of interviews to days.
  • Value: avoids failed pilots (wrong data, oversharing, unreadable scans) and gives a defensible priority list. Measure your own baseline: sources, share rated "ready", cleanup hours per use case.

Variants

  • Light (SMB): spreadsheet inventory of 10-30 sources, interviews, file-share statistics via script.
  • Per use case: start from a candidate use case (see PREP-002) and assess only the sources it needs.
  • Enterprise: continuous scanning with catalogue or DSPM tools (DATA-007, SEC-017).
  • Public sector: same method, plus records-management plans (Aktenplan).

How it works

  1. interviews + system list
  2. file-share / mailbox / export crawl
  3. sampling
  4. local LLM classifies sample (type, topic, personal data, language, obsolete?)
  5. scoring sheet (quality, format, access, legal)
  6. traffic light per source & use case
  7. gap list + effort
  8. backlog (PREP-005, -006, -009, -023 …)
  • Crawl: a Python or PowerShell script (or n8n workflow) lists path, type, size, owner, last modified and permissions; Microsoft Graph for SharePoint/OneDrive, Google Drive API, Nextcloud WebDAV for file stores.
  • Sampling and classification: 50-200 documents per source go to a small local LLM (via Ollama) or an EU-hosted API that returns JSON labels. Results are estimates, not facts.
  • Scoring: five criteria, 0-2 points each: findable (known location and owner), readable (text layer, standard format), current (last review), clean (duplicates, versions), allowed (legal basis, permissions).
  • Human in the loop: department heads confirm owner and sensitivity per source; the DPO confirms the personal-data flags. The model never decides access rights.

Data preparation

The assessment is itself data preparation, so the inputs are the system list and the raw listings. - Where it lives: file shares and SharePoint (usually the largest part), mailboxes, ERP/DATEV exports, Excel sheets on desktops (PREP-003), paper binders (PREP-006), CRM and ticketing systems. - Typical gaps: scans without text layer, five versions of the same file, folders nobody owns, "Everyone" permissions, data older than retention periods, exports that only the one administrator can run. - Steps and effort (realistic, SMB with ~10-30 sources): system list and owners 0.5 day; file-share crawl 0.5-1 day (script exists after first run); sampling and classification 1 day; scoring workshop with department heads 0.5 day; report 0.5 day. Total 3-4 working days plus calendar time for interviews. - Follow-up use cases: PREP-002 mapping to use cases, PREP-003 Excel shadow systems, PREP-005 exports, PREP-009 restructuring, PREP-023 oversharing clean-up, PREP-024 classification rules, PREP-025 legal-basis check. See also GUIDE-003.

Implementation paths

Which path for whom

  • smb: C - the sample contains contracts, HR and customer data, so classification should stay in-house; a script, n8n and a small Ollama model run on an existing server or a laptop (HW1) and the same stack serves later use cases.
  • enterprise: A - catalogue and discovery products (Microsoft Purview, Collibra, Atlan) are commodity and already integrate with identity and storage.
  • developer: C - scripts plus open-source parsers and local models, reusable across clients.
  • public: C - sovereignty, procurement rules and sensitive records favour on-premise tooling.

Path A - Buy (off-the-shelf)

  • Microsoft Purview (data map, sensitivity labels, content explorer; licensing tied to Microsoft 365 E5/Compliance tiers, check your contract); Collibra, Atlan or Alation for catalogues at enterprise scale; Varonis or similar for permission analytics.
  • EU-hosted or EU-based options: Purview supports EU data residency in the Microsoft EU Data Boundary; European catalogue vendors exist (verify hosting in the AVV). Prices are quote-based and usually exceed SMB budgets.
  • Choose when you already hold the licence tier or have thousands of users.

Path B - Build on cloud APIs

  • Crawl with Microsoft Graph or Google Drive API, classify samples with an EU-region model API (for example Mistral La Plateforme, hosted in the EU; Azure OpenAI in an EU region; IONOS AI Model Hub or STACKIT AI Model Serving as German/EU clouds).
  • Needs an AVV (Art. 28 GDPR) and a decision on whether document content may leave the premises; for pure metadata (names, sizes, dates) no model is needed.

Path C - Self-host (open source)

  • Crawl: Python (pathlib, msal/Graph SDK), rclone for cloud stores, czkawka or fdupes for duplicates; orchestration in n8n (self-hosted, EU-installable).
  • Parse: Docling or Apache Tika for text extraction; OCRmyPDF to test whether scans have a text layer.
  • Classify: Ollama with a small instruction model (Qwen, Llama, Mistral or Gemma class, 7-14B) for labels as JSON; runs on CPU or a modest GPU for a few hundred documents.
  • Record: inventory in an Excel/Nextcloud table, Baserow or NocoDB; later promoted to a catalogue (DATA-007).
  • Ops effort: a few hours initial setup, almost none afterwards.

Path D - Custom training

Almost never. A fine-tuned classifier (text-classifier) only pays off if you repeatedly scan large, uniform archives; start with prompts.

State of the art (2026)

  • Small local models classify document type and topic well on clean text, enough for sampling; weak on scans, mixed languages, jargon and personal-data detection.
  • Vendor tools (Purview, DSPM) bundle AI-assisted discovery, but report what exists, not fitness for a specific use case. That judgement stays human.
  • Failure modes: counting a folder as "ready" because files exist; trusting LLM labels without sampling check; ignoring permissions so that an assistant later surfaces HR files (see PREP-023).

Shared resources & combinations

  • Shared stack: the n8n + Ollama + parser setup is the same "Baukasten" used for PREP-006 (OCR), PREP-010 (tagging), PREP-022 (pseudonymisation) and later RAG assistants.
  • Combines with: PREP-002 (what each use case needs), PREP-004 (owners), DATA-007 (catalogue), SEC-017 (sensitive-data scanning).

Effort, cost & KPIs

  • Path A: licences dominate; setup weeks.
  • Path B: 3-5 days; API cost for a few hundred samples is small.
  • Path C: 3-5 days including learning curve; no licence cost; existing hardware.
  • KPIs: percentage of sources inventoried with named owner; share of sources rated green/yellow/red; hours of preparation per use case; share of documents with text layer; duplicate rate; number of folders with overly broad permissions.

People & change

Involve early: management (sponsor, scope), IT (accounts, read access), department heads as owners of their sources, the data owners for ERP/DMS, IT security for access, and the DPO because the inventory itself touches personal data. End users are asked, not scanned: tell them why you look at folders (GUIDE-008).

Works council: depends. A crawl listing last editor and modification time per file can make behaviour visible, and the BAG holds objective suitability is enough under §87(1) no. 6 BetrVG. Agree scope up front: no per-person evaluation, results aggregated per source. §90(1) no. 3 requires timely information if work processes are planned with AI; an expert under §80(3) can help (GUIDE-004). Public bodies involve the Personalrat.

Data protection: the crawl processes personal data (names in paths, content in samples). Document purpose and legal basis, sign an AVV with any cloud API, run the model locally where possible. A DPIA (Art. 35 GDPR) is typically not needed for the inventory itself, but may be for the use cases it enables (GUIDE-005).

AI literacy: Art. 4 AI Act, since 27 Jul 2026, requires providers and deployers to take measures that support staff's AI literacy; give the project team a short briefing on what the classifier can and cannot judge (GUIDE-006).

Pilot and concerns: start with one department and its two most important sources; typical worry is "you are checking my folders" - state that nothing is deleted or moved (GUIDE-009). Humans keep control: owners sign off the rating.

Risks, regulation & compliance

  • EU AI Act: minimal risk; the tooling is an internal analysis aid. Art. 4 AI literacy applies. The inventory helps spot whether a later use case (for example HR) falls under Annex III, whose duties apply from 2 Dec 2027 (GUIDE-014).
  • GDPR: crawling and sampling is processing; limit to what is needed, store results without document content, apply retention, AVV for processors.
  • Security: the inventory maps your sensitive data; protect it accordingly, use read-only service accounts (GUIDE-013).
  • Mitigations: read-only access, aggregate reporting, sample review by owners, no content stored in the inventory, logged runs.

Getting started

  1. Write the system list: every place where business data lives, with a named owner (0.5 day).
  2. Run a read-only listing of the biggest file share or SharePoint site and look at types, ages, duplicates and permissions.
  3. Classify 100 sampled documents with a local model, review 20 by hand to measure error.
  4. Score sources in a half-day workshop (findable, readable, current, clean, allowed) and mark them green/yellow/red.
  5. Hand the red/yellow items to the matching PREP use cases and match results to your first AI use case (GUIDE-001, GUIDE-003).

Sources

  1. Microsoft Learn - Microsoft Purview data governance and Data Map documentation - https://learn.microsoft.com/purview/
  2. Microsoft - EU Data Boundary for the Microsoft Cloud - https://learn.microsoft.com/privacy/eudb/eu-data-boundary-learn
  3. Ollama - project documentation - https://github.com/ollama/ollama
  4. Docling project - https://github.com/docling-project/docling
  5. OCRmyPDF documentation - https://ocrmypdf.readthedocs.io/
  6. n8n - documentation - https://docs.n8n.io/
  7. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  8. Regulation (EU) 2016/679 (GDPR) - https://eur-lex.europa.eu/eli/reg/2016/679/oj
  9. Betriebsverfassungsgesetz (BetrVG), §§ 80, 87, 90 - https://www.gesetze-im-internet.de/betrvg/