Records management: auto-classification, retention labelling & ROT clean-up

Organisations hold years of unstructured content - shares, SharePoint, mailboxes, Teams - that nobody has classified. Statutory retention periods, GDPR storage limitation and legal holds all require knowing what a document is. AI classifies content against the records schedule or Aktenplan, proposes retention labels, finds redundant, obsolete and trivial (ROT) files and routes deletion decisions to a records manager. It is also the unglamorous first step before rolling out enterprise RAG or copilots, because assistants otherwise index outdated and over-shared content.

Problem & value

• Today: records are filed by users (or not at all). Retention is applied per site or folder, if at all. Deletion is rarely done because nobody can prove it is safe, so storage, risk and e-discovery cost grow. Over-sharing and stale drafts are surfaced by AI assistants.
• Legal drivers (Germany): accounting records 10 years, accounting vouchers 8 years, business letters and other tax-relevant documents 6 years - starting at the end of the calendar year of the event; the period does not end while an assessment period is still open (§ 147 AO; § 257 HGB, which sets 10 years for vouchers of banks, insurers and securities institutions) [1][2]. GDPR Art. 5(1)(e) requires that personal data is kept identifiable no longer than necessary [3]. Both pull in opposite directions - hence the need for per-class, documented schedules.
• AI value: classifying millions of items by hand is infeasible; rules and keywords alone miss most content. Classifier + LLM + clustering reduce the manual part to reviewing samples, exceptions and disposition lists. Vendors report large ROT shares in typical unmanaged corpora, but such figures are vendor-specific and should be measured in your own scan (run an inventory first).

Variants

• Records management of unstructured corpora (M365/file share): label and retain by content, e.g. contract, HR file, invoice, project record.
• E-file / Aktenplan filing (public sector): propose the Aktenplan position, Aktenzeichen and Aufbewahrung/Aussonderung (offer to archive) for incoming and legacy documents in the e-Akte; archive-law transfer rules apply. Keep AI proposals separate from the formal filing decision.
• GoBD retention and legal hold: tag tax-relevant documents (Belege, Handelsbriefe) with the correct period and block deletion during holds or tax audits (Betriebspruefung); see DOC-001.
• ROT clean-up / migration: remove duplicates, abandoned drafts, empty and personal files before a DMS or cloud migration; reduces storage and e-discovery volume.
• Copilot/RAG readiness: find sensitive or outdated content that assistants could surface (see KNOW-002) and fix permissions before rollout.
• Email and chat retention: classify mailbox and Teams content into business record vs. transient.
• Private use: household document archive with suggested keep-periods - see PERS-013 / DOC-002.

How it works

1. inventory (connectors to shares, SharePoint, DMS, mail archive; metadata + ACLs + hashes)
2. text extraction (OCR + layout parsing for scans/PDF, language detection)
3. dedup & near-dup clustering (hashes, embeddings)
4. ROT candidates
5. classification against filing plan / schedule (rules + trainable classifier + LLM with the schedule as context)
6. sensitivity & PII detection (names, health, HR, contract terms)
7. proposal: class, retention label, trigger event (creation, contract end, employee exit), hold check
8. human review: sample audit per class, exception queue, disposition approval (four-eyes)
9. write-back (label/metadata), retention engine executes, audit log kept

• Cluster first: embed documents, cluster, let the records manager label clusters rather than single files; propagate to members and review outliers.
• Retention trigger events (contract end, employee leaving, end of project) usually are not in the document - they come from ERP/HR/CRM data and need integration.
• Human-in-the-loop: the retention schedule itself (legal basis per class) is a human decision; AI proposes classes and surfaces conflicts. Deletion of records above agreed thresholds always needs approval; legal hold overrides everything. Default is "keep", never "auto-delete on low confidence".

Data preparation

Data needed: the retention schedule or filing plan, the unstructured content itself (file shares, SharePoint, mail archives, DMS, paper archives) and the metadata on owner, date and use. It typically lives on historically grown file shares, in personal mailboxes, in the ERP attachments and in paper boxes.

Typical gaps: no up-to-date filing plan, many duplicates and copies, scans without a text layer, mail archives in legacy formats, missing owners, unclear legal-hold status, and over-broad permissions that expose personal data.

Preparation steps: inventory repositories and owners ([PREP-001], [PREP-004]); define the category scheme and map each class to a retention rule before any AI is used ([PREP-019]); find and measure duplicates and ROT; convert formats and OCR scans ([PREP-006], [PREP-007]); clean permissions ([PREP-023]); check purpose and legal basis for processing personal data ([PREP-025]); label a sample of 200 to 300 documents by records managers as the gold set ([PREP-018], [PREP-020]). Tools: DMS or M365 Purview, Paperless-ngx, deduplication scripts.

Expect 60 percent or more of the effort in this preparation, since the quality of the filing plan sets the ceiling for any automatic classification.

Implementation paths

Which path for whom

• smb: A - retention labels and auto-classification already exist in the DMS or the M365 licence, and the process is standard; no custom build is needed for a modest archive.
• enterprise: B - a large, heterogeneous archive and a company-specific filing plan call for classification on cloud APIs integrated with the existing records system.
• public: C - personal data and procurement rules favour sovereign or on-premises operation of open-source classification, for example with Paperless-ngx or an e-file system on HW2 hardware.
• developer: C - open models and an open-source DMS give a reusable classification service for several clients.

Path A - Buy

• Platform-native: Microsoft Purview (retention labels, records management, auto-apply labels by sensitive-information types, keyword queries or trainable classifiers; simulation mode to test a policy before activation) [4][5][6]. Natural choice if content lives in M365.
• ECM / records platforms: OpenText, Hyland, Laserfiche, M-Files; EU/German/Austrian vendors: ELO Digital Office, d.velop, OPTIMAL SYSTEMS (enaio), Fabasoft (Austria; common in public administration), Materna (DOMEA). Check: retention schedule/Aktenplan modelling, GoBD/IDW PS 880 certification where tax-relevant, EU hosting, audit trail, archive export.
• Data-governance/discovery tools: Varonis, BigID, Congruity360, Concentrix-type cleanup services and e-discovery vendors for ROT reduction (verify current product names and EU hosting).
• Choose A when: you already run one of these platforms - AI classification is then a licence or module. Limits to check: Purview's custom trainable classifiers are English-only, cannot be retrained after publishing, and auto-apply to existing SharePoint/OneDrive/Exchange items covers only the last six months; pre-trained classifiers have their own language list [5][6].

Path B - Build on cloud APIs

• LLM classification with the schedule in the prompt: frontier models with structured outputs (Claude, Gemini, GPT, Mistral Large via EU regions or EU provider) classify text chunks plus metadata into filing-plan classes and return JSON with rationale and confidence.
• Cost control: pre-filter with hashes, embeddings and rules; call the LLM only on cluster representatives and ambiguous items. Use batch endpoints for the backlog.
• Extraction: Azure AI Document Intelligence, Google Document AI or Mistral OCR for scans (EU regions). Write labels back via Microsoft Graph, SharePoint APIs or DMS REST APIs; orchestrate with n8n or Power Automate.
• Choose B when: your filing plan is bespoke, content is multilingual or the vendor tool does not fit; no strict on-prem constraint. Personal data goes to the provider - conclude a DPA and use EU regions.

Path C - Self-host (open source)

• Parsing: Docling, PaddleOCR, MinerU for scans and layout [7]. Embeddings/rerank: bge-m3 and similar multilingual models; vector search: pgvector, Qdrant, OpenSearch.
• Classification: fine-tuned multilingual encoder (e.g. XLM-R/German BERT variants) for the stable, high-volume classes; open-weight LLM (Qwen3, Mistral, Gemma family; 7-32B, via vLLM or Ollama) for rationale and the long tail; dedup with MinHash/simhash.
• Repository: Mayan EDMS, Paperless-ngx (small scale), Alfresco Community, OpenKM or Nuxeo with their records/retention modules; the retention engine must be tamper-evident and auditable.
• Hardware: HW2 is enough for scan-and-classify of millions of documents as a batch job over weeks; HW1/CPU for pure dedup and rules. Sensitive HR, health or defence archives are the typical reason for C. Ops effort is high: connectors, ACL handling, re-scan jobs.

Path D - Custom training

Worth it when the filing plan has dozens or hundreds of classes and an LLM per document is too expensive or inconsistent. Train a classifier on a few hundred reviewed examples per class (the records manager's accepted or corrected labels are the natural training data) and keep a hold-out set. Retrain when the schedule changes. Document the training data for audit purposes.

State of the art (2026)

• Layered approach is standard: exact/near duplicate detection and rules first, then trainable classifiers, then LLM-based classification for ambiguous or novel documents. LLMs follow a filing plan given in context well, but long hierarchical plans (hundreds of classes) need retrieval over the plan and a two-step "narrow, then decide" classification.
• Platform-native AI labelling (Purview-class) is mature for sensitive-information types, keywords and pre-trained classifiers; custom classification is more constrained (language, retraining, coverage of old items; labels apply only to unlabelled content and an existing label is not replaced) [4][5][6].
• AI readiness drives demand: copilots and RAG make stale, duplicated and over-shared content immediately visible, so clean-up projects are often funded as part of AI rollout.
• Retention triggers and legal context remain the hard part: whether a contract is still active, an employee has left, or a tax audit is open lies in business systems, not in the document.
• Limits: misclassification of a long-retention record as trivial (premature deletion) is far worse than the reverse; scanned, handwritten and mixed-language content lowers accuracy; one document can belong to several classes (the longest period usually wins); renaming/moving content breaks assumptions. No public benchmark exists for filing-plan classification - evaluate on your own labelled sample.

Shared resources & combinations

• Same building blocks (ocr-docparse, embedding-text, vector-db, text-classifier, llm-general): DOC-008 (classification at intake - the same classes feed retention), DOC-014 (parsing for RAG), KNOW-001/KNOW-002 (the cleaned and labelled corpus is the index), KNOW-011 (retire outdated knowledge articles), KNOW-013 (entity extraction for retention triggers).
• Governance: SEC-018 (record of processing, DSAR - retention classes are part of the GDPR deletion concept; the same PII detection supports DSAR search), DOC-001 (GoBD vouchers and their archive periods).
• Hardware: one HW2 GPU server with a vLLM endpoint and a vector DB can serve KNOW-025, DOC-008, DOC-014 and KNOW-002; batch jobs at night, interactive RAG by day.
• Workflow engine (n8n/Camunda) for the review and disposition queue is shared with AGENT back-office use cases.

Effort, cost & KPIs

• Path A: weeks to configure labels and policies on an existing platform, but the records schedule and stakeholder alignment take months.
• Path B: 2-4 months for connectors, classification pipeline and review UI; cost driven by number of documents processed by an LLM - pre-filtering is the main lever.
• Path C: 4-6 months with connectors and operations; one GPU server plus storage.
• Path D: adds labelling effort per class; only after A-C show where the classifier fails.
• KPIs: share of content classified (by volume and by risk class), classification precision/recall on an audited sample (especially for long-retention classes), ROT share removed, storage and e-discovery volume reduced, share of disposition proposals accepted by reviewers, over-shared sensitive items fixed, time to answer a legal-hold or DSAR request.

People & change

Involve management for the deletion policy, the records manager or process owner, IT and IT security for repositories and access, legal and compliance for retention duties and holds, and the DPO because deletion and classification touch personal data ([GUIDE-005]). Data owners of each repository must approve deletion rules.

Works council: depends. Content such as mail and files contains employee data, and classification logs can show who stored what, so §87(1) no. 6 BetrVG likely applies under the BAG's objective-suitability test. Inform the council early under §90(1) no. 3, and agree scope in a works agreement ([GUIDE-004]). In public bodies the Personalrat applies instead. A DPIA is likely where mailboxes or HR files are scanned at scale (Art. 35 GDPR).

AI literacy: employers must take measures that support staff AI literacy under Art. 4 AI Act; train records staff on reviewing suggestions and exceptions ([GUIDE-006]).

Pilot and concerns: start with one department's file share in suggestion mode only. Typical worries are accidental deletion and surveillance. Address them with human approval before any deletion, a dry run with reports, holds that override all rules, and clear communication on what is analysed and what is not ([GUIDE-008]).

Risks, regulation & compliance

• EU AI Act: typically minimal risk - an internal document-governance tool, no Annex III use. Art. 4 AI literacy applies. It becomes more sensitive if the same pipeline is used to profile employees from their documents or mail; then HR rules (works council, Art. 5 and Annex III employment) must be assessed [8].
• GDPR: the scan itself processes personal data (HR files, health, customer data); legal basis, data-protection impact assessment for large-scale scans and works-council involvement for mail and file shares are common requirements. Retention classes must be consistent with the deletion concept (Loeschkonzept, DIN 66398) and the record of processing; keep classification logs free of content copies.
• Retention law: GoBD, § 147 AO, § 257 HGB, sector rules (BaFin/DORA, MDR, GMP) and, for public bodies, archive laws and transfer to the archive (Anbietungspflicht) [1][2]. AI proposals are not a legal determination - the responsible person signs off the schedule.
• Risks: premature deletion, deletion during legal hold or tax audit, wrong labels being written back at scale (labels may not be reversible), exposure of content to the LLM provider, model drift.
• Mitigations: keep-by-default, simulation or dry-run before write-back, staged rollout by site, sample audits per class and confidence band, two-person approval for deletion, immutable audit log, backups before first disposition run.

Getting started

1. Obtain or build the retention schedule: classes, legal basis, retention period and trigger event (involve legal, tax, HR, IT, works council); this is the precondition for any AI.
2. Run a metadata and hash inventory of one repository; quantify duplicates, age and last access.
3. Pilot classification on one department (e.g. finance or HR) with 500-1000 documents labelled by the records manager; measure precision/recall per class and compare platform-native vs. LLM.
4. Run label policies in simulation or dry-run mode, then roll out by site; set up the disposition review.
5. Define KPIs and a quarterly re-audit; feed corrected labels into the classifier.

Sources

1. Abgabenordnung § 147 (Ordnungsvorschriften fuer die Aufbewahrung von Unterlagen) - https://www.gesetze-im-internet.de/ao_1977/__147.html
2. Handelsgesetzbuch § 257 (Aufbewahrung von Unterlagen, Aufbewahrungsfristen) - https://www.gesetze-im-internet.de/hgb/__257.html
3. GDPR Art. 5 - Principles relating to processing of personal data - https://gdpr-info.eu/art-5-gdpr/
4. Microsoft Learn - Learn about retention policies & labels - https://learn.microsoft.com/en-us/purview/retention
5. Microsoft Learn - Automatically apply a retention label to Microsoft 365 items (simulation mode, trainable classifiers, limits) - https://learn.microsoft.com/en-us/purview/apply-retention-labels-automatically
6. Microsoft Learn - Learn about trainable classifiers - https://learn.microsoft.com/en-us/purview/trainable-classifiers-learn-about
7. Docling project - https://github.com/docling-project/docling
8. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj