Catalog / Data Readiness for AI / PREP-019

Defining category schemes before AI classification

Agree with the people who do the work on categories, definitions, boundary rules and example items for documents, tickets, e-mails or defects, and test the scheme with a small LLM before any classifier is set up.

Quick win: proven, manageable effort, first results within weeks, low regulatory risk.

Parts needed to run it yourselfAgents & automationLanguage model

Many AI classification projects fail on the whiteboard, not in the model: two people label the same e-mail differently, "Complaint" and "Warranty" overlap, and nobody knows what "Other" means. Spending one to two weeks on a clear scheme, with definitions and examples, and testing it with a small LLM on real items, is cheaper than any model tuning. The result also helps without AI: it makes routing and reporting consistent.

Problem & value

  • Who: every team that sorts incoming items: service desks, mailrooms, quality departments, sales support, accounting inboxes.
  • Today: categories grew over years. Ticket queues are named after people, folders after projects, and the Excel list of defect codes has 140 entries of which 30 are used. Each employee applies them a bit differently.
  • Why AI helps: an LLM applied to 200 real items shows quickly where categories are ambiguous: where it disagrees with the humans, the scheme is usually unclear, not the model wrong. It can also cluster unlabelled items to suggest missing categories and draft first definitions.
  • Value: a scheme everybody understands, a labelled sample that doubles as test set (PREP-018), and a realistic view of what a classifier can achieve. We quote no percentages; measure agreement between your own reviewers.

Variants

  • E-mail and ticket routing: categories map to teams (LANG-012, AGENT-001).
  • Document types in mailroom or DMS: invoice, offer, contract, complaint (DOC-008, PREP-010).
  • Defect and quality codes: shop-floor catalogue, often with a standards or customer-specified base.
  • Private use: a personal folder and tag system for paperwork; same method, one afternoon.
  • Out of scope: running the classification itself (LANG-012, DOC-008) and labelling tools (DATA-023).

How it works

sample 200 real items (mixed, incl. odd ones)
  -> cluster or let an LLM propose draft categories
  -> workshop: merge, rename, define "use when / not when", add examples
  -> two people label the same 100 items independently
  -> compare (agreement, confusion list) + run a small LLM with the draft scheme
  -> fix definitions where humans or model disagree
  -> freeze v1, name an owner, set change rules
  • Human-led, AI-assisted: the LLM suggests and tests; the department decides. Definitions are written in the vocabulary of the work, not of IT.
  • Rules that matter: one primary category per item (or a clear multi-label rule), a defined "unclear/other" bucket with a maximum share, and a rule for what happens to it.
  • Hierarchy: keep it to two levels at most; 8-15 top-level categories are easier to apply than 60.
  • Human-in-the-loop later: low-confidence items go to a person; their corrections are captured (PREP-021).

Data preparation

Which data: a representative sample of items, current categories and counts per category, routing rules, and customer-facing wording.

Where it lives: shared mailboxes (Outlook/Exchange, Gmail), ticket systems (Zammad, Freshdesk, Jira Service Management, Zendesk), DMS or file shares, Excel lists, quality systems, and paper forms.

Typical gaps: categories used inconsistently, rare but important types missing from the sample, personal data in the texts, mixed languages, items that belong to two categories.

Preparation steps: 1. Export 200-300 items across the last 6-12 months, including rare types (see PREP-005); remove or pseudonymise personal data where not needed (PREP-022). 2. Count usage of existing categories; drop or merge the ones used rarely. 3. Hold a 2-3 hour workshop with the people who do the work and one consumer of the data. 4. Two people label 100 items independently; list disagreements (PREP-018, PREP-020). 5. Collect gold-standard examples per category (PREP-016) and align terms (PREP-015).

Tools: Excel or Google Sheets, SharePoint lists, Paperless-ngx tags, an Ollama model for testing, n8n or Power Automate for the batch run.

Rough share of effort: medium. About 3-8 person-days in total, most of it workshop time and labelling. See GUIDE-003.

Implementation paths

Which path for whom

  • SMB - Path C: tickets and mails contain customer and employee data, and a simple stack (Excel, n8n and a small Ollama model on HW1/HW2) is enough to test a scheme. The same server later runs classification, tagging (PREP-010) and summaries, so the Baukasten effect applies.
  • Enterprise - Path B: several departments and languages, existing tools; cloud APIs in an EU region plus workflow tooling fit, with Path A features (below) where already licensed.
  • Developer - Path C: open-weight models with structured outputs and an evaluation script give a repeatable scheme test.

Path A - Buy (off-the-shelf)

  • Ticket systems with built-in classification or auto-tagging (Zendesk, Freshdesk, Jira Service Management, Zammad with its AI ticket categorizer agent), Microsoft Purview trainable classifiers or SharePoint Premium for documents, DMS vendors such as DocuWare, d.velop and ELO (EU vendors). Check EU hosting and AVV.
  • Choose A when: the categories are standard (invoice, offer, contract) and the vendor's own scheme is accepted. A vendor scheme does not replace your own definitions and examples.

Path B - Build on cloud APIs

  • Models: Mistral (EU provider) or Azure OpenAI, Google Gemini or Claude in EU regions, with structured outputs.
  • Orchestration: n8n or Power Automate reads a Sheet or list, calls the model with the scheme, writes the label and a short reason back.
  • Choose B when: large item volumes, several languages, no highly sensitive texts or an AVV with EU processing.

Path C - Self-host (open source)

  • Stack: Ollama with a small open-weight model (7-8B class, e.g. Qwen or Mistral class) and JSON-schema output, n8n self-hosted, Excel/Nextcloud tables or Paperless-ngx tags; clustering with an open embedding model.
  • Hardware: HW1 is enough for a few hundred items; HW2 for faster runs.
  • Effort: 1-2 days setup, then iterations alongside the workshop.

Path D - Custom training

Not part of this use case. A fine-tuned classifier comes only after the scheme is frozen and some thousand labelled items exist (DATA-023).

State of the art (2026)

  • Small instruction-tuned models classify clean text into well-defined categories with definitions and few-shot examples; schema-constrained decoding prevents invalid labels.
  • LLM-assisted category discovery (clustering embeddings, then naming clusters) is common practice for finding missing categories.
  • Limits: models follow the wording of definitions literally, drift on near-synonyms, and are weak on categories that depend on unwritten context ("VIP customer"). If two trained humans agree only moderately, no model will do better; fix the scheme first.

Shared resources & combinations

Effort, cost & KPIs

  • Path C: 1-2 days setup plus workshop and labelling, 3-8 person-days total (planning estimate, not a benchmark). Licence cost near zero.
  • Path A: configuration 1-3 days plus licence.
  • Path B: low API cost for a few hundred items; 1-3 days integration.
  • KPIs: agreement between two human labellers, agreement between model and humans, share of "other/unclear", number of categories merged or dropped, share of items needing review.

People & change

Involve early: department leads (scheme owner), end-users who label daily, the consumers of the data, IT, the DPO and, where one exists, the works council. Customers or suppliers are involved if categories are visible to them (for example ticket forms).

Works council: depends. A scheme workshop is no monitoring. But if categories later feed statistics per employee or team, or a system logs who labelled what, objective suitability to monitor suffices for §87(1) no. 6 BetrVG (BAG); agree purpose limits in a works agreement. §90(1) no. 3 requires timely information when work processes are planned with AI; §80(3) lets the works council engage an expert. Public bodies involve the Personalrat (GUIDE-004).

Data protection: mails and tickets contain personal data; check legal basis, AVV (Art. 28 GDPR) for cloud models and DPIA (Art. 35) only if staff or customers are systematically evaluated (GUIDE-005). AI literacy: Art. 4 AI Act, as amended by Regulation (EU) 2026/1744, obliges providers and deployers to take measures that support staff's AI literacy; brief the labellers (GUIDE-006).

Pilot and tone: one team, one item type. Frame it as "ending the arguments about where this belongs". Anyone may challenge a category; the owner decides (GUIDE-008, GUIDE-009).

Risks, regulation & compliance

  • EU AI Act: minimal risk for routing and document types. If categories drive HR decisions or access to essential services, high-risk rules (Annex III, from 2 Dec 2027) may apply; do not reuse such schemes without review (GUIDE-014).
  • GDPR: labelled samples are personal data; keep them in your systems and delete after the test.
  • Risks: biased or political categories ("difficult customer"), too many categories, scheme frozen too early or never maintained. Mitigations: neutral wording, owner and review date, quarterly check against new items (GUIDE-011).

Getting started

  1. Pick one stream (for example the service mailbox) and export 200 items.
  2. Count current categories; propose 8-15 candidates with an LLM and cut them down in a workshop.
  3. Have two people label 100 items independently and list the disagreements.
  4. Run a small model with the scheme; rewrite definitions where it and the humans disagree.
  5. Freeze v1, name an owner and hand the labelled sample over as test set (PREP-018).

Sources

  1. Microsoft - Trainable classifiers in Microsoft Purview - https://learn.microsoft.com/purview/trainable-classifiers-learn-about
  2. Ollama - Structured outputs - https://docs.ollama.com/capabilities/structured-outputs
  3. Paperless-ngx - Documentation - https://docs.paperless-ngx.com/
  4. n8n - Documentation - https://docs.n8n.io/
  5. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  6. Regulation (EU) 2016/679 (GDPR) - https://eur-lex.europa.eu/eli/reg/2016/679/oj