Catalog / Data Readiness for AI / PREP-003

Discovery of Excel shadow systems & personal data stores

Find the business-critical spreadsheets, Access databases, personal drives and mailbox folders that hold data outside the core systems, and decide which to consolidate before AI projects start.

Parts needed to run it yourselfAgents & automationLanguage model

In most SMBs the real process knowledge lives outside the ERP: a price-calculation workbook one colleague maintains, an Access database for service tickets, a personal OneDrive folder, a mailbox folder called "Offers 2019". Before any AI project can use this data, someone must know it exists. Scripts, a small local language model and a few structured conversations produce a ranked inventory in two to four weeks, with a decision per file: keep, consolidate, migrate or retire.

Problem & value

  • Who: every company that grew organically. Departments build their own tools in Excel and Access because the core system is too rigid. The result: no owner on paper, no backup concept, no access control, often personal data inside.
  • Today: nobody has a list. IT sees file servers and licences, not which of 40,000 workbooks drive quotes or production planning. Discovery happens when the author leaves or the file breaks.
  • Why AI helps: search finds file names; a small model reads sheet names, headers and a few sample rows and answers "what is this file for, and is it critical?". It flags personal data and groups near-duplicate versions. People make the decisions.
  • Value: you avoid building an assistant on the wrong copy of a price list, you find sensitive data before Microsoft 365 Copilot can surface it (PREP-023), and you get a migration list. EuSpRIG documents that errors in business-critical spreadsheets are common; we quote no percentage.

Variants

  • Spreadsheet-only sweep (.xlsx, .xlsm, .csv): fastest start.
  • Wider sweep: Access files, personal OneDrive/Google Drive folders, shared mailbox folders, Teams file tabs.
  • Interview-led: under about 30 staff, a half-day workshop per department finds most of what matters; scripts only confirm it.

How it works

  1. file listing
  2. profile (sheets, headers, formulas, macros, editors)
  3. local small LLM labels purpose and data types
  4. score
  5. owner interviews
  6. decision list
  1. Crawl: PowerShell, Python (openpyxl, pandas) or Microsoft Graph lists path, size, dates, last modifier and sharing status. Content stays in your environment.
  2. Profile: sheet names, header rows, formula count, external links, macros, first 5-10 rows.
  3. Classify: a 7-8B local model answers a fixed question list as JSON: purpose, department, data types, master list or copy, personal data yes/no.
  4. Score: rules add usage (edited in last 12 months, number of editors), uniqueness (no ERP/CRM equivalent), risk (personal data, macros, single owner) and AI value (consistent headers).
  5. Verify: owners confirm or correct the top 50-100 files in 20-minute talks. Human-in-the-loop: nothing is moved or deleted on the model's word alone.
  6. Decide: keep, consolidate, migrate to a system, retire.

Data preparation

  • Where: file servers, SharePoint/OneDrive, Google Drive, Nextcloud, laptops, USB drives, mail attachments, Access files.
  • Gaps: versions ("final_v3_neu.xlsx"), password-protected files, merged cells, files of departed staff.
  • Effort: exclusions 0.5 day; crawler 0.5-1 day with an existing script, 2-3 days from scratch; model profiling of 5,000-20,000 files runs overnight on a CPU server; interviews 2-4 hours per department.
  • Feeds: PREP-001, PREP-002, PREP-011, PREP-009. Out of scope: ROT clean-up (KNOW-025) and analysing spreadsheet content (DATA-006).

Implementation paths

Which path for whom

  • smb: C. Files are often confidential (prices, salaries, customer lists), the task is simple, and a local model runs on a normal PC or small server. The same Ollama and n8n setup later serves other everyday automations (Baukasten effect).
  • enterprise: A. Commodity capability; posture and catalogue products integrate with access rights.
  • developer: C. Full control, easy to extend.

Path A — Buy (off-the-shelf)

  • Microsoft Purview data security posture management and sensitivity labels for Microsoft 365; the EU Data Boundary covers most services; licences vary, check with your reseller.
  • Google Workspace: Drive labels and security investigation tool in higher editions.
  • Catalogues: Collibra, Atlan; OpenMetadata is open source and self-hostable (EU-hostable). Posture tools find sensitive content, but are weak on business purpose.

Path B — Build on cloud APIs

Not recommended for file content. Possible if you send only sheet names and headers to an EU-region endpoint (Azure OpenAI EU region, Mistral from France, both EU-hosted options) under an AVV, orchestrated with Power Automate or n8n.

Path C — Self-host (open source)

  • Crawl: Python, PowerShell, rmlint for duplicates, Microsoft Graph for SharePoint.
  • Model: Ollama with a 7-8B instruct model and structured JSON output. HW1 suffices; seconds per file on CPU, so 10,000 files take overnight to a couple of days.
  • Orchestration: self-hosted n8n; results into Excel, Nextcloud tables or PostgreSQL.
  • Effort: 3-5 days setup; re-run quarterly at near zero cost.

Path D — Custom training

Not needed. A fine-tuned classifier only pays off in very large estates with thousands of verified labels.

State of the art (2026)

  • Small local models summarise a sheet's purpose from headers and samples and produce reliable JSON.
  • Vendor tools focus on sensitive content and oversharing; "what does this file do" is still best answered with people.
  • Limits: the model sees a sample, so hidden sheets, formulas and macros need rule-based checks; cryptic headers get mislabelled; password-protected files are skipped. "No personal data found" is never proof.

Shared resources & combinations

  • Same stack (llm-small + workflow-automation): Ollama and n8n also serve e-mail triage and document filing.
  • Neighbours: PREP-004 assigns owners, PREP-005 extracts data from replacement systems, PREP-023 cleans up permissions, PREP-006 digitises paper.

Effort, cost & KPIs

  • Path A: days to weeks of configuration; licences you may already own.
  • Path C: 3-5 days setup, 1-2 weeks including interviews for 20-100 staff, existing hardware. The main cost is interview time, not compute.
  • KPIs: share of files profiled, critical shadow systems with a named owner, files with personal data found, migrations decided, time to answer "where is data X?".

People & change

Involve early: management (mandate), department leads (they know what matters), IT and IT security (access, hosting), data owners, and the DPO before files with personal data are scanned. Tell staff what is scanned and why (GUIDE-008).

Works council: depends. The scan reads files, not behaviour, but "last modified by" lists can reveal who works when; objective suitability to monitor suffices for §87(1) no. 6 BetrVG (BAG). Keep user-level activity out of the report or agree it in a works agreement. §90(1) no. 3 requires timely information about AI-supported work processes; §80(3) lets the works council engage an AI expert. Public bodies involve the Personalrat (GUIDE-004).

Data protection: files hold employee and customer data; check legal basis and a DPIA (Art. 35 GDPR) if employee files are systematically evaluated; local processing reduces the AVV question (GUIDE-005). AI literacy: Art. 4 AI Act requires measures that support staff's AI literacy; a short briefing for the project team suffices (GUIDE-006).

Pilot and tone: start with one department, frame it as "rescuing your tools", not an audit, and name a champion. Concerns: "you will take my Excel away" (nothing is removed without agreement); fear of exposure for informal practices (offer amnesty for the first run).

Risks, regulation & compliance

  • EU AI Act: minimal risk; internal classification aid.
  • GDPR: scan metadata where possible, do not copy content elsewhere, record it in the processing register, and protect the inventory itself because it shows where sensitive data sits.
  • Risks: false confidence, files moved by mistake, lost owner trust, salary or health files exposed during the scan. Mitigations: read-only access, human confirmation, restricted report, deletion only after backup and sign-off.

Getting started

  1. Agree scope and send a one-page note to staff and the works council.
  2. Run a metadata-only crawl; look at the top 100 files by editors and recent changes.
  3. Interview department leads with ten questions (which files do you rely on, who maintains them, what if they break).
  4. Profile the top files with a local model and compare with the interviews.
  5. Decide per file, hand the list to PREP-001/PREP-004, repeat every six months.

Sources

  1. Microsoft — Microsoft Purview Data Security Posture Management — https://learn.microsoft.com/purview/data-security-posture-management-learn-about
  2. Microsoft — EU Data Boundary for the Microsoft Cloud — https://learn.microsoft.com/privacy/eudb/eu-data-boundary-learn
  3. Microsoft — Microsoft Graph overview (files in SharePoint and OneDrive) — https://learn.microsoft.com/graph/overview
  4. n8n — Documentation (self-hosting) — https://docs.n8n.io/hosting/
  5. Ollama — Structured outputs — https://docs.ollama.com/capabilities/structured-outputs
  6. OpenMetadata — project documentation — https://docs.open-metadata.org/
  7. European Spreadsheet Risks Interest Group (EuSpRIG) — https://eusprig.org/
  8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  9. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj