Catalog / Data Readiness for AI / PREP-021

Capturing user feedback & corrections on AI output

Collect thumbs-up/down, corrections and edits from employees who use AI tools and turn them into a reviewed improvement log for prompts, knowledge bases and test sets.

Parts needed to run it yourselfAgents & automationLanguage model

Most companies roll out an AI assistant and then never learn whether it is right. Employees quietly fix wrong answers, give up on the tool, or paste results into Word without telling anyone. A small, routine feedback loop (a thumbs button, a reason, a place where corrections land and a person who reads them every week) turns everyday use into the cheapest improvement data an SMB can get. It is mostly organisation, not technology.

Problem & value

  • Today: feedback is anecdotal ("the chatbot is useless for contracts"). Vendor thumbs buttons often send data to the vendor, not to your team. Corrections made in Word or the ERP are never compared with the AI draft.
  • AI value: a structured log shows which use cases fail, why, and for whom. It feeds knowledge-base fixes (PREP-017), test sets (PREP-018) and gold-standard examples (PREP-016), and gives management evidence for go/no-go decisions.
  • Caveat on figures: no reliable cross-industry benchmark exists for how much feedback loops improve accuracy; measure your own baseline. Note that Microsoft documents that Copilot feedback is used to improve Copilot, not to train foundation models, and is reviewable by admins; it is not a full improvement log of your own [1].

Variants

  • Chat assistants: ratings plus reason inside the tool (Open WebUI, LibreChat, Copilot).
  • Drafting use cases: compare AI draft with the final text sent (mails, offers, minutes) to see what staff change.
  • Extraction/classification (invoices, mail routing): every manual correction on a review screen is a label (PREP-020).
  • Public sector: same routine, usually self-hosted, with staff council involved from the start.

How it works

  1. AI answer shown to employee
  2. one-click rating (up/down) + optional reason category + optional comment
  3. correction captured (edited text, corrected field, "right answer is …")
  4. log entry with context (use case, prompt, sources, model version, date) in one table
  5. weekly triage by a feedback owner (cause: knowledge gap, prompt, wrong source, model limit, user error)
  6. fix (update document, adjust prompt/template, add test case) and tell users what changed
  • AI components are optional and small: an LLM or classifier can pre-sort comments into categories and cluster duplicates; a human confirms. Everything else is forms, lists and a workflow engine.
  • Human-in-the-loop: a named person triages; nothing is fed back into a model automatically. Pseudonymise before any comment text is sent to another AI service (PREP-022).

Data preparation

Which data: ratings, reasons, corrected outputs, and the minimal context to reproduce the case (use case name, prompt, retrieved sources, model/version, timestamp).

Where it lives: inside the AI tool (Open WebUI evaluations, Copilot admin feedback), in Microsoft Forms/Lists or Google Forms/Sheets, in a Nextcloud Forms table, or in the review screen of an automation (Paperless-ngx corrections, ERP booking changes).

Typical gaps: no common category scheme, feedback without context, personal data in comments (names, customer details), no owner, and feedback that stays in the vendor's console.

Preparation steps: 1. Define 5-8 reason categories (wrong fact, outdated source, wrong tone, missing source, too long, privacy concern) - PREP-019. 2. Decide the log table: one row per case with the context fields above; a Microsoft List, Excel on SharePoint, Nextcloud table or a database. 3. Add a free-text hint: "no names or customer data in comments". 4. Define retention (for example 12 months) and who may read raw comments. 5. Name the feedback owner and a weekly 30-minute slot.

Tools: Microsoft Forms/Lists, Google Forms, Nextcloud Forms, n8n or Power Automate, Open WebUI, Excel.

Rough share of effort: about 15 percent; the work is agreeing on categories and ownership. See GUIDE-003.

Implementation paths

Which path for whom

  • SMB - Path C: the same self-hosted chat front end (Open WebUI with Ollama on HW1/HW2) serves several use cases, keeps comments on your own server and has ratings built in; n8n links the log to your tables. If you only use Copilot or ChatGPT Team, start with Path A-style forms (see below) instead.
  • Enterprise - Path B: an internal AI platform already calls cloud APIs, so feedback is logged through an observability layer plus your BI tool.
  • Developer - Path C: open-source tracing with user-feedback scores, self-hosted.
  • Public - Path C: comments may contain citizen or staff data; self-operated or sovereign hosting is the safe default.

Path A - Buy (off-the-shelf)

  • Built-in thumbs in Microsoft 365 Copilot (admin policy, admin centre review) and Gemini for Workspace; a short Microsoft Forms/Google Form linked from the tool. Microsoft announced in late October 2026 that thumbs-down will show issue categories and an optional comment [1][2]. Good for a first signal, but the data mostly flows to the vendor and cannot be joined with your context.
  • Many EU AI assistants (for example Mistral Le Chat Enterprise, EU-hosted) offer ratings; check what is exported.

Path B - Build on cloud APIs

  • Wrap your own calls (n8n, Power Automate, small Python service) so each answer gets an ID, then store ratings against that ID in a database. Langfuse Cloud (EU region) or Azure Monitor can hold scores. Choose B when an internal platform exists.

Path C - Self-host (open source)

  • Open WebUI: thumbs up/down with reason; ratings feed an admin Elo leaderboard (only for responses that have sibling responses, e.g. regenerated or side-by-side), and rated chats can be exported [3].
  • Langfuse (MIT, self-hostable): user feedback stored as scores on traces, via Browser SDK or backend; filter for low-rated answers [4].
  • n8n + Nextcloud/Paperless-ngx: workflow that copies corrections into the log table and sends the weekly digest.
  • Hardware: HW1 for the logging stack; HW2 only if a local LLM pre-sorts comments.

Path D - Custom training

Only when thousands of validated corrections exist and a fine-tune is justified (DATA-023 for annotation tooling). Most SMBs never get here; use the log to fix knowledge and prompts first.

State of the art (2026)

  • Explicit feedback is sparse: only a small minority of answers get a rating, and it skews to extremes, so implicit signals (edits, regenerations, abandoned answers) are valuable. These are also the more monitoring-sensitive signals.
  • LLM-assisted triage of comments into causes works well enough for pre-sorting; humans decide.
  • Feedback to evaluation: the best practice is converting real failures into regression test cases (PREP-018) so fixes are verified.
  • Limits: biased and tiny samples, ratings that reflect tone rather than correctness, privacy leakage in comments, and feedback fatigue if nothing visibly changes.

Shared resources & combinations

  • Same stack as PREP-017 (content lifecycle), PREP-018 (test sets), PREP-016 (gold examples), PREP-020 (labels from records) and PREP-013 (knowledge base).
  • Technical tracing is out of scope here (CODE-018); this use case is the organisational routine.
  • One chat platform plus n8n serves all AI use cases (Baukasten effect).

Effort, cost & KPIs

  • Path A: 1-2 days to switch on thumbs and a form.
  • Path B/C: 3-8 days for log, workflow and weekly report; ongoing about 30-60 minutes per week for the feedback owner. Software cost is low (open source or existing licences).
  • KPIs: share of answers with feedback, share rated helpful, top 5 failure causes, time from report to fix, number of feedback cases turned into test cases.

People & change

Involve early: the department lead (owner of the use case), end-users (they know the errors), IT (log, access) and the DPO. Management names the feedback owner.

Works council: yes in practice. Feedback logs link a person, a prompt and a judgement of output, and logging of user activity is objectively suited to monitor behaviour or performance, which is enough for co-determination under §87(1) no. 6 BetrVG (per the Federal Labour Court). §90(1) no. 3 requires timely information on AI-supported work processes. Public bodies involve the Personalrat. Agree on a works agreement: feedback is pseudonymous or aggregated, not used for performance evaluation (GUIDE-004).

Data protection: ratings tied to named users are employee data; comments may contain customer data. Use a legal basis check, restrict access and retention, and sign an AVV with any hosted tool (GUIDE-005). A DPIA is rarely needed for a small internal log, but ask the DPO.

AI literacy: Art. 4 AI Act requires measures that support staff's AI literacy; a 30-minute briefing on what to report and how to word it (GUIDE-006).

Communication and pilot: say clearly that the aim is to fix the tool, not rate people. Start with one team and one use case for four weeks, publish a monthly "you said, we fixed" note, name AI champions (GUIDE-008, GUIDE-009).

Typical concerns: surveillance ("who rated what"), extra work (one click, not a form), nothing happens (visible fixes).

Humans in control: the feedback owner decides what changes; no automatic retraining.

Risks, regulation & compliance

  • EU AI Act: minimal risk as an internal quality process; Art. 4 literacy support applies. If the log were used to evaluate employees, the context could change, so exclude that explicitly (GUIDE-014).
  • GDPR: purpose limitation (quality improvement only), data minimisation, retention, access control. Check what the vendor does with feedback and screenshots (Microsoft lists optional screenshots in the new feedback flow) [2].
  • Risks: personal data in comments, shadow use of feedback for performance review, biased samples. Mitigation: pseudonymous logging, aggregated reports, clear rules (GUIDE-007).

Getting started

  1. Pick one live AI use case and name a feedback owner.
  2. Switch on thumbs and reasons in the tool, or add a Forms link; define the categories.
  3. Create the log table and a weekly 30-minute triage slot.
  4. Agree the rules with the DPO and works council (if one exists) before go-live.
  5. After four weeks, turn the top 10 failures into test cases (PREP-018) and report fixes to users.

Sources

  1. Microsoft Learn - Welcome users, create organizational messages, and enable feedback for Microsoft Copilot - https://learn.microsoft.com/en-us/microsoft-365/copilot/microsoft-365-copilot-enable-users
  2. Microsoft 365 Message Center MC1530595 - Microsoft Copilot Feedback UX update (via Merill's mirror) - https://mc.merill.net/message/MC1530595
  3. Open WebUI docs - Evaluation - https://docs.openwebui.com/features/administration/evaluation/
  4. Langfuse docs - User Feedback - https://langfuse.com/docs/observability/features/user-feedback
  5. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj