Every SMB already owns thousands of labelled examples: the category a service agent picked, the G/L account the bookkeeper chose, the defect cause the quality team recorded, whether a quote was won. Instead of paying people to annotate data, you export these historical decisions, check how trustworthy they are and use them to train, test or configure AI. Typical benefit: a usable evaluation set in days instead of weeks of manual annotation.
Problem & value
- Today: AI projects stall at the question "how do we know it works?" Teams assume they must annotate from scratch. Meanwhile the answer is sitting in the ERP, CRM, ticket system or DATEV booking history.
- AI value: past decisions become (a) an evaluation set to compare tools and prompts, (b) few-shot examples for an LLM, (c) training data for a small classifier or a tabular model (e.g. win probability).
- Caveat: a recorded outcome is not automatically the truth. Agents choose the first matching category, ticket categories change over time, and "lost" quotes often carry no real reason. Reliability checking is the core of this use case. No sourced benchmark figure on typical noise rates in SMB business records is cited here; measure it on your own data.
Variants
- Service desk: closed-ticket category, priority, team and resolution as labels for triage (see PREP-019 for the category scheme).
- Finance: final booking account and cost centre per supplier/invoice line as labels for booking proposals (DOC-001).
- Sales: won/lost per quote plus product mix, discount and customer segment for win-probability models.
- Quality/production: defect cause codes from complaint or 8D records.
- Enterprise: data warehouse extracts at scale; developer: reusable harvesting pipelines per source system.
How it works
source system (ERP/CRM/ticketing/DATEV export)
-> extract (API, CSV, n8n/Power Automate)
-> join record + outcome, keep timestamps and who-decided
-> reliability checks (class balance, drift over time, rule conflicts, model-based label-issue detection)
-> human review of a sample / flagged conflicts [human-in-the-loop]
-> frozen labelled set: train / validation / evaluation split by time
Key components: - Time-based split: evaluate on the most recent months, never on a random mix, to avoid leakage and to see drift. - Weak supervision: several imperfect sources (category field, resolution text keywords, later re-assignment) can be combined as "labeling functions" whose noise is modelled; open-source Snorkel popularised this idea [1]. Treat the result as probabilistic, not as truth. - Label-error detection: train a simple model with cross-validation and let the open-source library cleanlab rank examples whose recorded label looks likely wrong, for human review [2]. - LLM as second opinion: a small local or EU-hosted LLM re-labels a sample; disagreements with the recorded label go to a human.
Data preparation
- Where it lives: ticket tools (Jira Service Management, Zendesk, Freshdesk), ERP (Business Central, SAP Business One, Odoo), CRM (HubSpot, Salesforce, Dynamics), DATEV booking exports, Excel quality logs.
- Typical gaps: free-text "other" categories, categories renamed or merged over the years, defaults never changed by agents, outcomes entered weeks later, duplicates, personal data inside free text.
- Steps and effort (one use case, one source): define the target label and exclude unusable periods (0.5 day); export and join (1-2 days, more if no API: see PREP-005); profile class balance and drift in Excel or a notebook (1 day); review 100-300 flagged or sampled records with the process owner (1-2 days); freeze and document the dataset (0.5 day). Realistic total: 4-6 working days.
- Related: PREP-011 for tidy tables, PREP-018 for expert-built test sets, PREP-022 for pseudonymisation before export. See also GUIDE-003.
Implementation paths
Which path for whom
- SMB: C. Data is customer and employee related; the work is small enough to run on a laptop or CPU server with Python, cleanlab and n8n, and the same pipeline serves many use cases (Baukasten effect).
- Enterprise: B. Larger volumes and central data platforms; cloud ML services and EU-region LLM APIs fit existing governance.
- Developer: C. Open-source stack, versioned datasets, reproducible pipelines.
Path A - Buy
Labels are mostly a by-product of tools: ticket systems with built-in AI categorisation learn from your history (Zendesk, Freshdesk, Jira Service Management); HubSpot and Dynamics offer lead/deal scoring from won/lost history. Data-labelling platforms (Label Studio Enterprise, Snorkel Flow) exist but are rarely needed at SMB scale. Good when the vendor already trains on your history and offers an EU data region and AVV.
Path B - Build on cloud APIs
Use an EU-region LLM API (Mistral, Azure OpenAI EU Data Boundary, Aleph Alpha-style sovereign offers) as second-opinion labeller on pseudonymised samples; orchestrate with n8n (EU-hostable) or Power Automate; write results back to a spreadsheet or table for review.
Path C - Self-host
Python (pandas, scikit-learn, cleanlab), optionally Snorkel, Label Studio Community for the review step, Ollama with a small model for second-opinion labels. Runs on HW1; HW2 only if you re-label large text volumes with a 14-30B model. Ops effort: a few hours per month once pipelines exist. n8n self-hosted on Nextcloud/own server keeps data in the EU.
Path D - Custom training
Only after the harvested set is reliable: fine-tune an encoder classifier when several thousand consistent labels exist per class. Usually unnecessary; start with few-shot LLM or a gradient-boosting model.
State of the art (2026)
- Small tabular/text classifiers trained on clean historical labels still match or beat zero-shot LLMs on narrow in-house categories, at far lower cost per item; LLMs excel when labels are few.
- Confident-learning tools such as cleanlab are established for flagging label issues [2]; weak supervision combines noisy sources [1].
- Limits and failure modes: historical labels encode past mistakes and past bias (e.g. who got discounts); concept drift after process changes; survivorship bias (only closed/won records are recorded in detail); a model trained on programmatic labels can learn their assumptions as ground truth [1].
Shared resources & combinations
- Shares extraction (PREP-005), tidy tables (PREP-011), evaluation sets (PREP-018), category schemes (PREP-019) and feedback capture (PREP-021).
- Feeds invoice coding (DOC-001), ticket triage and sales scoring. The same n8n instance and Python environment serve all of them.
- Out of scope: manual annotation tooling (DATA-023) and synthetic data (DATA-005).
Effort, cost & KPIs
- Path A: hours of configuration; licence cost of the ticket/CRM tier. Path B: 4-8 days plus small API fees. Path C: 4-8 days, no licence cost, existing hardware. Cost drivers: data access (no API), reviewer time of process owners.
- KPIs: share of records with usable label; estimated label noise in the reviewed sample; agreement between recorded label and LLM second opinion; time to build an evaluation set; later: accuracy of the downstream tool on the frozen evaluation set.
People & change
- Involve: department lead and end-users early; they know why categories were chosen and must review the flagged records. Data owners grant exports; IT builds the pipeline; DPO checks purpose and legal basis (GUIDE-005).
- Works council: harvested labels often contain who decided what. If the dataset or resulting models could assess individual agents' performance (e.g. "agent X mislabels"), co-determination under §87(1) no. 6 BetrVG is likely, and §90(1) no. 3 requires timely information when work processes are planned with AI. Use aggregated, pseudonymous analysis and agree this upfront (GUIDE-004). Public sector: staff council under BPersVG/state law.
- Communication: say clearly that the goal is to improve the process, not to grade people; let experts correct labels and show them the result (GUIDE-008).
- AI literacy (Art. 4): deployers must take measures to support staff AI literacy; short sessions on "labels are noisy" for reviewers (GUIDE-006).
Risks, regulation & compliance
- AI Act: preparing datasets is minimal risk. Risk changes with the downstream use: models for hiring or worker evaluation are Annex III high-risk (duties from 2 Dec 2027 under Regulation (EU) 2026/1744). Ticket, booking and quality labels are normally unproblematic (GUIDE-014).
- GDPR: reuse of records for AI is a new purpose; check compatibility and legal basis (Art. 5(1)(b), Art. 6; see PREP-025). Strip or pseudonymise names and free-text identifiers (PREP-022). AVV (Art. 28) with every tool processing the data; DPIA (Art. 35) where employee performance or special categories are involved.
- Risks: hidden bias, leakage, labels containing secrets. Mitigate with sampling review, time-based evaluation, access limited to the project, documented dataset version.
Getting started
- Pick one decision with a recorded outcome and a clear owner (e.g. ticket category).
- Export 12-24 months to CSV and profile classes and drift per quarter in Excel.
- Have the process owner review 100 random plus 50 flagged records; note the error rate.
- Freeze the cleaned set as the evaluation set before choosing any tool.
- Define which fields to capture better from now on and feed corrections back (PREP-021).
Sources
- Snorkel - rapid training data creation with weak supervision (VLDB Journal): https://link.springer.com/article/10.1007/s00778-019-00552-1 (accessed 2026-10)
- cleanlab documentation - finding label issues: https://docs.cleanlab.ai/v2.2.0/index.html (accessed 2026-10)
- GDPR text (Art. 5, 6, 28, 35): https://eur-lex.europa.eu/eli/reg/2016/679/oj (accessed 2026-10)
- Snorkel research - Interactive Programmatic Labeling for Weak Supervision: https://snorkel.ai/research-paper/interactive-programmatic-labeling-for-weak-supervision/ (accessed 2026-10)