Guides / GUIDE-011

Operating AI in production: monitoring quality, costs, model updates, incident handling

How to run an AI system after go-live: who owns it, what to measure, how to control token costs, how to survive model updates and how to handle AI incidents.

After go-live the questions change: is the AI still as good as in the pilot, what does it cost, what happens when the vendor swaps the model, and who reacts when it produces something harmful? This guide gives you a lean operating model with one owner, a handful of metrics, a cost guardrail, a change process for model updates and an incident playbook.

Why it matters

  • AI systems change under your feet. Microsoft's lifecycle policy for its Foundry/Azure OpenAI models gives generally available models an 18-month lifecycle (12 months for some partner models), promises at least 60 days' notice before a GA model retires, and auto-upgrades standard deployments to a replacement model at retirement. Preview models can be force-upgraded with 30 days' notice. Retirement dates "aren't extendable" [1]. Without planning, the replacement arrives untested against your prompts and extraction rules.
  • Costs are variable and can run away. Usage-based AI is billed per token or per call. OWASP lists "Unbounded Consumption" as one of the ten top risks for LLM applications: uncontrolled inference leads to service outages and financial loss ("denial of wallet"). OWASP recommends rate limits, quotas and usage monitoring [2]. The FinOps Foundation recommends tying AI spend to unit costs such as cost per query, per user or per workflow, because token prices alone leave out fixed costs [3].
  • The law expects monitoring, at least for high-risk AI. Under Art. 26 AI Act, deployers of high-risk systems must assign competent human oversight, monitor operation, suspend use and inform provider and market surveillance authority if they see a risk, and keep logs they control for at least six months [4]. After the Digital Omnibus, Regulation (EU) 2026/1744, these duties apply from 2 Dec 2027 for Annex III systems (see GUIDE-014). For other systems it is simply good practice.
  • AI incidents are often also security and data protection incidents. A chatbot that reveals customer data is a personal data breach. Under Art. 33 GDPR you must notify the supervisory authority within 72 hours [5]. If your company falls under NIS2, the German implementing act has been in force since 6 Dec 2025: significant security incidents need an early warning to the BSI within 24 hours, a notification within 72 hours and a final report after one month [6].

When you need it

  • Before go-live (last phase of GUIDE-009): name an owner, set up the metrics, budget alerts and incident path. Do not hand over a system that nobody operates.
  • From day one in production: weekly checks in the first month, then monthly.
  • Whenever the vendor announces a model change, a new version or a new feature (for SaaS tools also "silent" changes visible in release notes).
  • Whenever your own data or process changes (new document layouts, ERP release, product range).
  • Once a year: full review including contract, cost, risk class and whether the system is still worth running.

Who is involved

Role Part in operations
System owner (usually the department lead) Accountable for business value and quality; decides on changes and shutdown
IT operations Accounts, integration, monitoring, cost dashboards, vendor tickets, model switch
IT security Abuse detection, prompt-injection and data leakage incidents, NIS2 reporting (see GUIDE-013)
End users / key users Report errors, give feedback, run monthly spot checks. They notice drift first
Data owners Keep the knowledge base and master data current (stale data is the most common quality problem)
DPO Log retention, personal data in prompts, breach assessment under Art. 33 GDPR (see GUIDE-005)
Works council Co-determination on usage logs and dashboards (see below and GUIDE-004)
Management Budget, risk acceptance, decision to switch vendor or stop

Works council point: monitoring an AI system nearly always means logging who did what (prompts, answers, corrections, handling times). Under §87(1) no. 6 BetrVG the Federal Labour Court holds that objective suitability to monitor behaviour or performance is enough, so the logging and reporting set-up is subject to co-determination. Agree in the works agreement which logs exist, who may see them, retention periods and that quality metrics are evaluated per system or team, not per person. Without this, your monitoring dashboard can stop the rollout.

Step by step

1. Write a one-page operating sheet (half a day). Per system: purpose, owner and deputy, vendor and model version, data sources, risk class (GUIDE-014), log location, budget, vendor contact and the "off switch" (disable the feature without stopping the process).

2. Fix the baseline and a test set (1-3 days, reuse the pilot). Take 50-200 real pilot cases (GUIDE-009) with known correct answers, e.g. correctly booked invoices or approved answers to customer questions. This "golden set" is your regression test for every model change. Store it with the prompts and configuration used.

3. Choose five to eight metrics, not fifty. A workable set: - Quality: accuracy on the golden set; share of outputs corrected by users; "thumbs down" rate; for RAG assistants, the share of answers with a correct source. - Usage: active users, cases processed, share of cases that still go fully manual. - Cost: total spend and cost per case (per invoice, per ticket, per conversation). - Operations: availability, latency, error rate of API calls. - Risk: flagged outputs, blocked prompts, security events. Report them monthly on one page.

4. Collect feedback where the work happens. A thumbs-up/down button, a "report wrong answer" link or a correction field in the ERP mask costs little and gives the best early warning. Feedback is evaluated anonymously (works agreement). Key users run a monthly spot check of 20-30 random cases.

5. Set cost guardrails (1 day). Set budget alerts at 50/80/100% in the cloud console or vendor portal, hard quotas per application and per user, rate limits, maximum input and output length, and a separate API key per application so you can see who causes which cost [2][3]. For seat licences, reassign unused ones monthly. Review the model choice every six months; a smaller model often suffices.

6. Run a change process for model updates. Subscribe to the vendor's lifecycle notifications (e.g. service health alerts and emails for Azure [1]). Where possible, pin model versions and switch off automatic upgrades, so that you decide when to move. When a change is announced: 1. run the golden set on the new model in a test environment; 2. compare quality, cost and latency with the baseline; 3. adapt prompts and extraction rules; 4. switch with a fallback (keep the old deployment until the new one has run cleanly for one to two weeks, if the vendor allows it); 5. record the change in the operating sheet. Plan two to five person-days per model change and system. For SaaS tools where you cannot pin versions, at least rerun the golden set after each announced change.

7. Watch data and knowledge drift. For RAG assistants, a data owner removes outdated documents and adds new ones. For forecasting and inspection models, compare predictions with outcomes monthly and retrain when the error exceeds the agreed threshold (GUIDE-003).

8. Prepare the incident playbook (1 day). AI incidents include harmful output, leaked confidential or personal data, successful prompt injection, wrong automated decisions reaching customers, runaway costs and outages. For each, define who is notified (IT security, DPO, owner), how to stop the damage (disable the feature, revoke the API key, go manual), how to preserve evidence (logs, prompts, outputs) and the legal checks: Art. 33 GDPR (72 hours) [5], NIS2 reporting if applicable [6], informing the vendor, and for high-risk systems informing the provider and market surveillance authority (Art. 26 AI Act) [4]. Add it to your existing IT emergency process, not a parallel one. The BSI paper on generative AI lists threats and countermeasures to reuse here [7].

9. Keep logs, but deliberately. Keep what you need for quality and security, and for high-risk systems at least six months [4]. Agree retention, access and pseudonymisation with the DPO and works council; prompts often contain personal data. The DSK guidance on AI and data protection addresses deployers of AI systems as controllers, so you are responsible for this data even when the vendor stores it [8].

10. Review yearly, and dare to switch off. Once a year: is the use case still delivering the planned benefit, has the risk class changed, is the contract still fit (GUIDE-010), would a different implementation path be cheaper now (GUIDE-002)? NIST's generative AI profile explicitly includes deactivating systems that operate outside defined limits as a management action [9]. A documented "stop" is good governance.

Checklist

  • [ ] System owner and deputy named for each production AI system
  • [ ] One-page operating sheet with model version, data sources, off switch
  • [ ] Golden test set and baseline stored with prompts/config
  • [ ] Five to eight metrics defined and reported monthly
  • [ ] In-tool feedback button and monthly key-user spot check
  • [ ] Budget alerts, quotas, rate limits, one API key per application
  • [ ] Vendor lifecycle notifications subscribed; versions pinned where possible
  • [ ] Model change process with regression test and fallback
  • [ ] Data owner for knowledge base / training data
  • [ ] AI incident playbook integrated into IT emergency process (GDPR 72 h, NIS2 if applicable)
  • [ ] Log retention and access agreed with DPO and works council
  • [ ] Annual review with an explicit continue / change / stop decision

Templates & tools

  • AI Act Service Desk (Commission): consolidated Art. 26 text incl. Digital Omnibus [4].
  • BSI "Generative KI-Modelle": threats and countermeasures for risk analysis and playbook [7].
  • OWASP Top 10 for LLM Applications 2025: especially LLM10 Unbounded Consumption for cost guardrails [2].
  • NIST AI 600-1 Generative AI Profile: voluntary catalogue of monitoring and incident actions [9].
  • FinOps Foundation materials on AI cost and token economics: unit cost metrics [3].
  • Vendor tools: cost budgets and alerts, usage dashboards, model lifecycle APIs [1]; for custom LLM apps, LLM observability tools (use case CODE-018).
  • Your IHK for NIS2 basics [6].

Common pitfalls

  • "Fire and forget" after the pilot. Nobody owns the system, quality drops quietly. Name the owner before go-live.
  • Auto-upgrade surprises. Extraction fields shift after a model swap and nobody notices until month-end. Pin versions, keep the golden set.
  • Monitoring individuals. Per-person reports trigger co-determination conflicts and destroy trust. Measure system and team.
  • One shared API key. Cost spikes become untraceable. Separate keys and budget alerts take a day.
  • Stale knowledge base. The assistant quotes last year's price list. Data ownership is an operations task.
  • Keeping all logs forever. Data protection risk and works council resistance. Define purpose and retention.

Sources

  1. Microsoft Learn: Foundry Models lifecycle and support policy (model deprecations and retirements), updated 2026-07-24. https://learn.microsoft.com/azure/ai-services/openai/concepts/model-retirements
  2. OWASP GenAI Security Project: LLM10:2025 Unbounded Consumption. https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/
  3. FinOps Foundation: Token economics, the atomic unit of AI value. https://finops.org/insights/token-economics-the-atomic-unit-of-ai-value/
  4. European Commission, AI Act Service Desk: Article 26, Obligations of deployers of high-risk AI systems (consolidated text as of 27 Jul 2026 incl. Digital Omnibus). https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-26
  5. Regulation (EU) 2016/679 (GDPR), Art. 33 Notification of a personal data breach. https://eur-lex.europa.eu/eli/reg/2016/679/oj
  6. IHK Magdeburg: NIS2, implementation in Germany (registration and reporting duties). https://www.ihk.de/magdeburg/digitalisierung/it-sicherheit/nis2-6096160
  7. BSI: Generative KI-Modelle – Chancen und Risiken für Industrie und Behörden. https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/KI/Generative_KI-Modelle.pdf
  8. Datenschutzkonferenz (DSK): press release on the guidance "Künstliche Intelligenz und Datenschutz", 6 May 2024. https://datenschutzkonferenz-online.de/media/pm/2024_05_06_DSK_PM_OH_KI_und_Datenschutz.pdf
  9. NIST AI 600-1: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile, July 2024. https://doi.org/10.6028/NIST.AI.600-1

Accessed 2026-10.