Catalog / Data Readiness for AI / PREP-017

Knowledge content lifecycle: review cycles & feedback from assistant errors

Give every source document of an AI assistant an owner, a review date and an expiry, and route wrong or missing assistant answers back to the owner for correction.

Parts needed to run it yourselfAgents & automationLanguage modelSearch & RAG

An AI assistant is only as current as its sources. After the launch, price lists change, a procedure is replaced and the assistant keeps quoting the old version with full confidence. This use case is the unglamorous maintenance loop: every source gets an owner, a review date and an expiry; expired content leaves the index; and every wrong or unanswered question lands in the owner's task list. It costs hours per month and decides whether staff still trust the assistant after six months.

Problem & value

  • Who: every company that runs an internal assistant over its own documents (see PREP-013). Typically 5-50 people maintain content "on the side".
  • Today: documents are uploaded once. Nobody knows who owns them, nothing expires, and complaints ("the assistant said the wrong thing") arrive by hallway talk or not at all.
  • Why AI helps: a small model can cluster the questions that failed, point to the source document that probably caused the error, and draft a correction proposal. The core of the loop is still process and ownership, not AI.
  • Value: fewer outdated answers, visible content gaps, and a clear owner per topic. We quote no percentages; measure the share of thumbs-down answers and expired documents before and after.

Variants

  • Review-date only (no AI, cheapest), feedback loop (thumbs into tickets) or AI-assisted triage (LLM groups failed questions and drafts fixes).
  • Out of scope: automatic FAQ generation (KNOW-011) and LLM tracing and observability (CODE-018).

How it works

source documents (+ owner, review_by, valid_until)
  -> nightly job: due soon? -> reminder to owner -> escalate to lead
  -> expired and not renewed -> flag "expired" -> removed from index
assistant answers (+ sources, thumbs, comment)
  -> failed / unanswered questions -> weekly triage (LLM groups, proposes source)
  -> task for owner -> correct document -> re-index -> re-test question (PREP-018)
  • Metadata: four fields per document: owner, last_reviewed, review_by, valid_until (see PREP-010). The retrieval filter ignores documents past valid_until.
  • Human in the loop: the AI only proposes. The owner decides whether the document, the question or the assistant is wrong; nothing is rewritten automatically.

Data preparation

Which data: the source inventory with owners (PREP-004, PREP-013), a review interval per document type (for example price lists quarterly, safety procedures yearly), and the assistant's question log.

Where it lives: SharePoint or OneDrive, Google Drive, Nextcloud, Confluence or a wiki, Paperless-ngx; the log in the assistant's database (Open WebUI, Copilot Studio analytics, a custom bot).

Steps: 1. Export the document list with last-modified dates and folder owners; assign a named owner or a team mailbox per area (1-2 days with department leads). 2. Define 4-6 review intervals and add the metadata columns. 3. Switch on feedback and source logging in the assistant; define how long logs are kept. 4. Delete or archive documents nobody claims after a deadline.

Rough share of effort: medium, mostly clarifying ownership. See GUIDE-003.

Implementation paths

Which path for whom

  • SMB - Path C: the logs contain questions from employees and may include customer data, and the same n8n and Ollama stack that tags documents (PREP-010) and serves the assistant can run the loop on HW1 (Baukasten effect). Microsoft 365 shops with only SharePoint and Copilot can stay on Path A for the reminder part.

  • Enterprise - Path B: existing Microsoft 365 or ServiceNow processes plus EU-region cloud APIs for triage. Developer - Path C: open-source stack with metadata filters.

Path A - Buy (off-the-shelf)

  • Microsoft 365: SharePoint columns with Power Automate reminders; Microsoft Purview retention labels for expiry; Copilot Studio analytics shows unanswered questions (EU Data Boundary).
  • Wiki and knowledge tools: Guru verification cycles and Notion page verification (vendor-reported; check plan, EU hosting and AVV); other wiki tools may offer similar features. Elements in the EU: Nextcloud and Paperless-ngx (self-hosted), DocuWare, d.velop, ELO.
  • Choose A when: the assistant is Copilot or a vendor product and its built-in review features suffice.

Path B - Build on cloud APIs

  • Models: Mistral (EU provider) or other models in EU regions for grouping failed questions and drafting fixes.
  • Orchestration: Power Automate or n8n reads the log, calls the model, creates Planner, Jira or Teams tasks.
  • Choose B when: you have many systems and an AVV with EU processing exists.

Path C - Self-host (open source)

  • Stack: n8n (self-hosted) for reminders and weekly reports, Paperless-ngx or Nextcloud for documents, Open WebUI or a RAG framework with feedback buttons, Ollama with a small model for triage.
  • Effort: 2-4 days setup, then 1-2 hours per week of owner time.

Path D - Custom training

Not needed; fine-tuning does not fix outdated sources.

State of the art (2026)

  • Enterprise knowledge tools ship verification cycles (owner, review interval, "verified" badge) as a standard feature.
  • Assistant platforms log retrieved sources and user ratings; LLM-based clustering of failed questions is practical.
  • Limits: thumbs-down is sparse and biased; models misattribute errors to the wrong source; fixes can break other answers; the loop fails when owners ignore reminders.

Shared resources & combinations

Effort, cost & KPIs

  • Path A: licence cost; 1-3 days configuration.
  • Path C: 2-4 days setup plus about half a day per month for admin (planning estimate, not a benchmark).
  • Path B: small per-token cost plus 3-5 days integration.
  • KPIs: share of sources with owner and valid review date, number of expired documents in the index, thumbs-down rate, average days from reported error to correction, share of unanswered questions per topic, owner reaction rate to reminders.

People & change

Involve early: management (ownership as a duty), department leads (name owners), owners as end-users of the loop, IT, the DPO and the works council.

Works council: yes. Question logs and feedback show who asked what, and objective suitability to monitor behaviour suffices for §87(1) no. 6 BetrVG (BAG). Agree what is logged, who sees it (aggregated by topic, not by person) and how long it is kept in a works agreement. §90(1) no. 3 requires timely information on AI-supported work processes; §80(3) allows an expert. Public bodies involve the Personalrat (GUIDE-004).

Data protection: questions can contain personal data; define a legal basis, retention, an AVV (Art. 28 GDPR) for any cloud model, and involve the DPO; a DPIA (Art. 35) is likely only if logs are evaluated per person (GUIDE-005). AI literacy: Art. 4 AI Act, as amended by Regulation (EU) 2026/1744, requires measures that support staff's AI literacy; teach owners what the assistant can and cannot do (GUIDE-006).

Pilot and tone: start with one area, such as HR policies or product FAQs, and one champion. Frame it as "the assistant is only as good as our documents", not as control of staff; owners answer for content, not model behaviour (GUIDE-008).

Risks, regulation & compliance

  • EU AI Act: minimal risk. Art. 50 transparency applies to the assistant itself (users must know they talk to AI). If answers feed HR selection decisions, Annex III high-risk duties apply from 2 Dec 2027.
  • GDPR: log minimisation, short retention, no personal data in feedback comments where avoidable.
  • Risks: owners ignore reminders, so documents expire silently; over-aggressive expiry empties the assistant; feedback is gamed or used to monitor staff. Mitigations: escalation to the lead, grace period, aggregated reporting, regular sampling (GUIDE-011, GUIDE-013).

Getting started

  1. List the assistant's sources and name one owner per area.
  2. Add review_by and valid_until columns; set intervals per document type.
  3. Build the reminder flow in Power Automate or n8n and test it with one department.
  4. Enable thumbs, comment and source logging; agree on log retention with DPO and works council.
  5. Hold a 30-minute weekly triage for four weeks, then judge with KPIs (GUIDE-009).

Sources

  1. Microsoft - Power Automate documentation - https://learn.microsoft.com/power-automate/
  2. Microsoft - Retention policies and labels in Microsoft Purview - https://learn.microsoft.com/purview/retention
  3. Microsoft - EU Data Boundary for the Microsoft Cloud - https://learn.microsoft.com/privacy/eudb/eu-data-boundary-learn
  4. n8n - Documentation - https://docs.n8n.io/
  5. Paperless-ngx - Documentation - https://docs.paperless-ngx.com/
  6. Open WebUI - Documentation (feedback and evaluation) - https://docs.openwebui.com/
  7. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  8. Regulation (EU) 2016/679 (GDPR) - https://eur-lex.europa.eu/eli/reg/2016/679/oj