Running an AI pilot: from proof of concept to production (scope, KPIs, go/no-go)

Building an AI demo that impresses people in a meeting is easy. Getting the same system to work every day, on real data, for real users, at a known cost is much harder. This guide shows how to run a pilot that ends in a clear decision: go to production, change the scope, or stop. You will have a one-page pilot charter, a KPI sheet with a baseline, and a go/no-go template.

Why it matters

• Many pilots never reach production. In July 2024 Gartner predicted that at least 30% of generative AI projects would be abandoned after the proof of concept by the end of 2025. The reasons it gave were poor data quality, inadequate risk controls, rising costs and unclear business value [1]. A good pilot tests all four.
• German companies are now past the "trying it out" stage. In Bitkom's survey of 603 companies with 20 or more employees (published 14 September 2026), 57% use AI, up from 36% a year earlier. Yet none of the AI users said they already use its potential fully [2]. The bottleneck is now getting pilots into daily operations.
• A failed pilot can still be a good result. The Mittelstand-Digital AI handbook treats a proof of concept as a feasibility check, and a negative result as a useful finding rather than a setback [6]. The real failure is a pilot that ends without a decision.
• A pilot is not exempt from the rules. With real personal data or real employees, GDPR, the BetrVG and the AI Act apply from day one.

When you need it

• After discovery and prioritisation (GUIDE-001), once one or two candidate use cases have been chosen.
• After a first decision on the implementation path (GUIDE-002) and a quick data readiness check (GUIDE-003).
• Before any licence commitment longer than the pilot period, and before the tool reaches more than a small test group.
• Also for "buy" cases: switching on an AI feature in your ERP, DMS or M365 deserves a pilot too.

Typical duration: 6–12 weeks (1–2 preparation, 4–8 live operation, 1–2 evaluation).

Who is involved

In a 50-person company one person often holds several roles; just make sure every role is covered.

Step by step

1. Write a one-page pilot charter (0.5–1 day). Name one process and one user group, the problem in numbers ("a clerk spends about 6 minutes per incoming invoice"), what is in and out of scope, the duration, the budget cap and the decision date. If the problem does not fit on one page, the scope is too wide.

2. Measure the baseline before the AI arrives (1–2 weeks, partly in parallel). Without a "before" figure, you cannot show an "after" figure. Measure throughput time, effort per case, error or rework rate and backlog over at least two normal weeks. For low volumes, 30–50 timed cases are enough.

3. Define 3–5 KPIs with thresholds in advance. Use one from each group: - Quality: correct extraction or answer rate measured on a fixed test set, share of outputs that need correction. - Efficiency: minutes saved per case, throughput time, share of cases handled without manual work. - Adoption: weekly active pilot users, share of cases actually processed with the tool. - Cost: cost per case (licence, API usage, internal effort), projected to full volume. - Risk: number of critical errors (wrong amount booked, wrong answer to a customer), data protection or security incidents. For each KPI, write down the "go" threshold and the "stop" threshold before the pilot starts.

4. Build a test set from your own data (2–5 days). Collect 50–200 real, representative cases with the correct answer, including difficult ones: bad scans, unusual suppliers, dialect in transcripts, edge cases. Use the same set for every vendor or model version. Data and test-case preparation usually takes longer than configuring the AI (GUIDE-003).

5. Clear the legal and co-determination points before real data flows (1–3 weeks lead time). - DPO: legal basis, AVV under Art. 28 GDPR with the provider, and whether a DPIA under Art. 35 GDPR is likely (systematic evaluation of people, special categories of data, new technology at scale). The DSK guidance on AI and data protection lists the points to check [5]. - Works council: if the tool logs user activity, it is objectively suitable for monitoring behaviour or performance. Under Federal Labour Court case law that is enough to trigger co-determination under §87(1) no. 6 BetrVG, and a pilot is no exception. §90(1) no. 3 BetrVG requires timely information when work processes involving AI are planned [9]. A short pilot agreement is often the fastest route: time limit, named users, no performance evaluation of individuals, and how the data is deleted. The public sector deals with the staff council (Personalrat) instead. - AI Act: check early whether the use case is high-risk (for example, AI in hiring or performance evaluation). The Annex III duties apply from 2 December 2027. The Art. 50 transparency duties, such as telling users when they are talking to a chatbot, have applied since 2 August 2026 (GUIDE-014). - Under Art. 4 of the AI Act, you must take measures to support the AI literacy of the staff who use the system. A short briefing for pilot users on capabilities, failure modes and permitted inputs is one such measure (GUIDE-006).

6. Set up the pilot with production in mind. Use real accounts with single sign-on rather than shared logins, set access rights the way they will be in production, and enable logging of inputs, outputs and corrections. Keep a human in the loop for every output that has external effects. BSI recommends a risk analysis for the specific use case before integrating large language models into workflows [3]. For cloud AI services, the BSI AIC4 criteria catalogue is a useful set of questions for the vendor [4].

7. Run the pilot and collect feedback every week (4–8 weeks). Hold a 30-minute check-in each week with the pilot users: what worked, what failed, and examples to add to the test set. Track the KPIs weekly, not just at the end. Document every change to prompts, configuration or process.

8. Calculate the cost of production honestly. The pilot costs are not the production costs. Add the integration into ERP or DMS, user and rights management, monitoring, support, training for all users, licence or usage fees at full volume, and the internal owner's time. Compare this with the measured savings from step 3, not the vendor's figures.

9. Hold the go/no-go meeting on the date set in the charter. Decide on one of three outcomes and record it in writing: - Go: the KPIs met their thresholds and no open legal or security issues remain. Fix the rollout plan, budget, owner and operations handover (GUIDE-011). - Pivot: promising results with a fixable problem, such as a narrower scope, a different vendor or better data. Run one more short, time-boxed iteration with the same KPIs. - Stop: thresholds missed, or production would cost more than it saves. Document what you learned, end the licences and delete the pilot data as agreed.

10. Plan the rollout in stages. Roll out department by department, keep the KPIs running, and update the works agreement, DPIA and staff training from pilot scope to production scope. Communication with all staff: GUIDE-008.

Checklist

• [ ] One-page charter with scope, duration, budget cap and decision date
• [ ] Baseline measured over at least two weeks or 30–50 cases
• [ ] 3–5 KPIs with written go and stop thresholds
• [ ] Test set of 50–200 real cases, including hard ones
• [ ] AVV signed; DPIA question answered by the DPO
• [ ] Works council informed; pilot agreement in place if §87(1) no. 6 applies
• [ ] Pilot users briefed (AI literacy measure under Art. 4 AI Act)
• [ ] Logging, access rights and human review set up
• [ ] Weekly KPI tracking and feedback round
• [ ] Production cost calculated at full volume
• [ ] Written go / pivot / stop decision with an owner

Templates & tools

• KPI sheet: one row per KPI; columns for baseline, go threshold, stop threshold, weeks 1–8, final value.
• Mittelstand-Digital AI handbook (Mittelstand-Digital Zentrum Kaiserslautern): an approach for introducing AI in SMBs, including the role of the proof of concept [6].
• Mittelstand-Digital Zentren: free, vendor-neutral support including hands-on implementation projects with companies [7]. The current network is being restructured, and funding for the existing centres runs out by the end of 2026 at the latest, with a new network planned from 2027 [8]. See GUIDE-012 for funding.
• BSI: "Generative KI-Modelle – Chancen und Risiken" for the risk analysis [3]; the AIC4 criteria catalogue for questions to cloud AI vendors [4].
• DSK: Orientierungshilfe "Künstliche Intelligenz und Datenschutz" as a checklist for the DPO [5].

Common pitfalls

• Demo instead of pilot. Vendors show their best examples. Always test on your own test set.
• No baseline. Without "before" figures, the result comes down to gut feeling, and the loudest voice in the go/no-go meeting decides.
• Scope creep. Park extra wishes for a later phase; do not add them to the running pilot.
• Pilot that never ends. Without a decision date it becomes unofficial production with no budget, owner or works agreement.
• Underestimating data preparation. Cleaning master data and labelling test cases often take longer than the AI part.
• Hand-picked enthusiasts only. Include at least one sceptic who knows the process well (GUIDE-008).
• Skipping the works council because "it's only a test". Co-determination covers introducing a monitoring-capable system, including a pilot. A late agreement can stop a successful pilot just before rollout.
• Comparing pilot costs with production savings. Free trial licences flatter the business case.

Sources

1. Gartner: "Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025", press release, 29 July 2024. https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025
2. Bitkom: "Erstmals nutzt die Mehrheit der Unternehmen KI", press release, 14 September 2026 (n=603 companies with 20+ employees). https://www.bitkom.org/Presse/Presseinformation/Erstmals-nutzt-Mehrheit-Unternehmen-KI
3. BSI: Generative KI-Modelle – Chancen und Risiken für Industrie und Behörden (English edition: Generative AI Models). https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/KI/Generative_KI-Modelle.pdf
4. BSI: AI Cloud Service Compliance Criteria Catalogue (AIC4). https://www.bsi.bund.de/EN/Themen/Unternehmen-und-Organisationen/Informationen-und-Empfehlungen/Kuenstliche-Intelligenz/AIC4/aic4.html
5. Datenschutzkonferenz (DSK): Orientierungshilfe "Künstliche Intelligenz und Datenschutz", version 1.0, 6 May 2024. https://www.datenschutz-bayern.de/dsbk-ent/DSK_OH_KI_und_Datenschutz.pdf
6. Mittelstand-Digital Zentrum Kaiserslautern: Handbuch Künstliche Intelligenz. https://www.ptw.tu-darmstadt.de/media/fachgebietptw/dokumente_3/wissenssammlung_ptw/leitfaeden_2/Handbuch-Kuenstliche-Intelligenz_Mittelstand-Digital-Zentrum-Kaiserslautern.pdf
7. IHK Cottbus: "Mittelstand-Digital Zentrum Spreeland gut aufgestellt" (free, vendor-neutral support and implementation projects). https://www.ihk.de/cottbus/presse/pressemitteilungen/mittelstand-digital-zentrum-spreeland-gut-aufgestellt-7100164
8. NRW.BANK: Unterstützungsangebote Mittelstand-Digital (restructuring of the network). https://www.nrwbank.de/de/unternehmen/nrw-bank-innovationspartner/aktuelles/unterstuetzungsangebote-mittelstand-digital.html
9. Betriebsverfassungsgesetz (BetrVG), §§ 87, 90. https://www.gesetze-im-internet.de/betrvg/

Accessed 2026-10.