Building test & evaluation sets with domain experts

Before an AI assistant goes live, someone has to answer one question: "Is it right often enough, on our questions?" Vendors and IT cannot know that, the people who do the work can. A test set of 50-100 realistic questions with expected answers, written by domain experts in a spreadsheet, turns opinions ("seems fine") into a measurable pass rate and keeps quality visible after every model update, prompt change or new document.

Problem & value

• Today: pilots are judged by a few impressive demos or by the loudest sceptic. After go-live nobody notices that answers got worse when the vendor changed the model or the knowledge base grew.
• Why experts: only the accounting clerk, the service technician or the HR officer knows the real edge cases, the correct answer and which wrong answer is dangerous. IT can run the test, but cannot write it.
• Value: an objective go/no-go criterion (GUIDE-009), early detection of regressions, a shared language between business and IT, and acceptance among staff because their expertise defines "good". Test sets are also the cheapest data-preparation step: a few hours of expert time instead of weeks of rework later.

Variants

• RAG assistant / chatbot: question, expected answer, source document.
• Extraction / classification: document plus known correct fields or category (for example invoices, see DOC-001).

How it works

1. collect real questions & documents
2. experts write expected answers + sources
3. tag (topic, difficulty, risk)
4. run against the AI (manual or automated)
5. score (pass / partly / fail, optional LLM as judge)
6. fix causes
7. re-run on every change

• Mix: typical questions (60 percent as a rule of thumb, not a measured value), hard edge cases, questions the AI should refuse or answer "I do not know", and trick cases (outdated policy, similar product names).
• Scoring: humans rate first; an LLM as judge can pre-score large sets but must be spot-checked by an expert.
• Human-in-the-loop: experts own the expected answers; they review disagreements between AI judge and human.

Data preparation

Which data: real questions, example documents, the authoritative source for each answer.

Where it lives: helpdesk or ticket tool, shared mailboxes, Teams or WhatsApp groups, policy PDFs in SharePoint, Nextcloud or file shares, and in the heads of experts.

Typical gaps: no single authoritative document per topic, contradicting versions, no one owns the answer, personal data inside real tickets and mails.

Preparation steps: 1. Pull 100-200 candidate questions from tickets and mails; remove names, customer numbers and other personal data (PREP-022). 2. Define categories first (PREP-019). 3. Each expert writes expected answer plus the source passage; mark the authoritative document (PREP-013, PREP-016). 4. Keep a held-back part (about 20 percent) that is never used for tuning prompts, so results stay honest. 5. Store the set in one spreadsheet with columns: ID, question, expected answer, source, category, difficulty, risk, author, date, version.

Tools: Excel or Google Sheets, SharePoint list, Microsoft Forms for collecting questions, Nextcloud with a shared spreadsheet (EU-hosted, self-hostable). See GUIDE-003.

Effort: about 2-4 hours per expert for 25-30 cases; a first set of 50-100 cases takes 2-4 working days of calendar time.

Implementation paths

Which path for whom

• SMB - Path C: no product is needed; a spreadsheet plus a small open-source runner on a laptop or small server (HW1) does the job, and the same set serves every future assistant (Baukasten effect). Test data often contains customer or business secrets, which favours keeping it in-house.
• Enterprise - Path B: a shared evaluation platform or cloud services with API access fit when many teams run many assistants and need central reporting.
• Developer - Path C: open-source tooling in the CI pipeline gives full control and reproducible runs.

Path A - Buy (off-the-shelf)

Few buyable products exist for the expert-authored part; it is mostly process. Built-in features help: Microsoft Copilot Studio agent evaluation imports a CSV of questions and expected responses, up to 100 cases per set [1], and evaluation modules inside commercial LLM-ops platforms exist. Check EU hosting and AVV. Use A when you already run Copilot Studio agents and only need a quick check.

Path B - Build on cloud APIs

Run the test set through the model API you use in production (Azure OpenAI in EU regions, Mistral (EU provider), or other EU-region endpoints) with an n8n or Power Automate flow that reads the spreadsheet, calls the assistant, and writes answer and score back. An LLM judge can be a second, stronger model from an EU region. Choose B when the assistant itself runs on cloud APIs and results should land in SharePoint or Excel.

Path C - Self-host (open source)

• Promptfoo (MIT) reads test cases from CSV or Excel files and supports expected-value checks per row [2]; runs on any laptop with Node.
• Langfuse (MIT, self-hostable; all features open-sourced in June 2025) offers datasets, experiments and annotation queues for expert review [3]; needs Docker and a database, so plan 1-2 days for setup and an admin.
• n8n self-hosted (EU) as the glue to schedule nightly runs.
• Local judge model via Ollama (for example a 7-14B model) on HW1/HW2 when test content must not leave the company; judge quality of small models is limited, keep human spot checks.

Path D - Custom training

Not needed for the test set itself. Later, a well-curated set (and its corrections) can seed fine-tuning data (PREP-020), but a few hundred high-quality examples are needed and the held-back set must stay separate.

State of the art (2026)

• LLM-as-judge is common and cheap but biased towards verbose answers and its own model family; calibrate against expert ratings.
• Limits: small test sets give noisy percentages (one case at n=50 equals 2 points); non-deterministic models need repeated runs; test sets age when policies change, so they need an owner and a review date (PREP-017).
• A test set measures what you wrote into it; it is not a safety or red-team assessment (see SEC-014, CODE-018).

Shared resources & combinations

• Same set, many uses: PREP-013 knowledge base, DOC-001 extraction, any chatbot; one spreadsheet and one runner serve all.
• Feeds: PREP-021 (corrections from users become new test cases) and PREP-020 (labels from business records).

Effort, cost & KPIs

• Path B: 1-3 days to build the n8n flow; API cost for 100 cases is small.
• Path C: 1-2 days for Promptfoo; 1-2 days plus admin for Langfuse.
• Expert time (rule of thumb): 2-4 hours per expert for the first 25-30 cases, then 1-2 hours per month for upkeep.
• KPIs: pass rate per category, share of critical cases passed (target 100 percent), regressions per release, share of real user complaints already covered by a test case, number of experts contributing.

People & change

Involve early: the department lead names 2-4 domain experts and frees their time; end-users supply real questions; IT runs the tests; data owners confirm the authoritative source; management accepts the pass-rate threshold for go-live. The DPO checks that real tickets are anonymised before they enter the set and that any external judge model has an AVV (Art. 28 GDPR); a DPIA is normally not needed for the test set alone (GUIDE-005).

Works council: depends. The test set evaluates the AI, not people, so it is not itself a monitoring system. Where results are broken down by person, or where logs show who asked what, §87(1) no. 6 BetrVG applies; §90(1) no. 3 requires timely information on AI-supported work processes. Public bodies involve the Personalrat. Agree that test results are never used for individual performance assessment (GUIDE-004).

AI literacy: Art. 4 AI Act requires measures that support staff's AI literacy; writing test cases is a very effective hands-on training because experts see the strengths and failures themselves (GUIDE-006).

Communication: invite experts as co-designers: "you define what good looks like". Pilot with one team, publish the pass rate openly, and let staff add cases when they find errors (GUIDE-008, GUIDE-009).

Risks, regulation & compliance

• EU AI Act: the test set is minimal risk. For high-risk uses (for example HR selection, Annex III, duties from 2 Dec 2027) the test and data-governance evidence becomes part of the documentation; see GUIDE-014.
• GDPR: real questions and documents often contain personal data; anonymise, restrict access to the set, delete when no longer needed.
• Risks: overfitting prompts to the test set (use held-back cases), outdated expected answers, a false sense of safety from a small set, LLM judge errors, trade secrets in the set stored in an uncontrolled spreadsheet.
• Mitigations: version control, owner and review date, access limited to the project group, periodic refresh with real failures (PREP-021).

Getting started

1. Appoint 2-4 domain experts and a set owner; block two hours each.
2. Collect 100 real questions from tickets and mails and anonymise them.
3. Write expected answers and sources in one spreadsheet using the column template above; mark about 10 critical cases.
4. Run the set manually against the pilot assistant, record the pass rate, and agree the go-live threshold with management.
5. Automate with Promptfoo or an n8n flow and re-run on every model, prompt or document change.

Sources

1. Microsoft Learn - Create a test set for agent evaluation in Copilot Studio - https://learn.microsoft.com/en-gb/microsoft-copilot-studio/analytics-agent-evaluation-create
2. Promptfoo documentation - Test cases (CSV / Excel) - https://www.promptfoo.dev/docs/configuration/test-cases/
3. Langfuse - Open source LLM engineering platform (datasets, experiments, self-hosting) - https://langfuse.com/self-hosting
4. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj