Process quality prediction & virtual sensors (soft sensors)

Many of the values that matter most in production, such as viscosity, melt index, residual moisture, concentration, layer thickness or tensile strength, are measured only a few times per shift in the lab, or by expensive and failure-prone analysers. Between samples the plant runs blind. A soft sensor (virtual sensor) is a model that estimates such a value continuously from the signals the plant already records. Operators see quality drifting hours before the lab result comes back. Labs can sample less often. Advanced process control can then regulate on predicted quality. Semiconductor fabs use the same idea under the name virtual metrology.

Problem & value

• Today: quality is checked by periodic lab samples (often with a delay of 1–8 h), offline metrology on sampled wafers or parts, or online analysers that drift and need constant upkeep. Off-spec material is found late and becomes rework, downgrade or scrap. Engineers' rule-of-thumb correlations sit in spreadsheets.
• AI value: continuous quality visibility, earlier interventions, less sampling and analyser effort, a backup when a physical analyser fails, and a controllable variable for APC/MPC.
• Established practice: the process industry has used data-driven soft sensors for over two decades. The classic review by Kadlec, Gabrys and Strandt (with Evonik) covers chemical, bioprocess and steel applications. It names the typical data problems: measurement noise, missing values, outliers, co-linear features, varying sampling rates and the need for model adaptation [1].
• Regulatory acceptance in pharma: ICH Q13 (continuous manufacturing) explicitly names soft sensors, "a model that is used in lieu of physical measurement to estimate a variable or attribute", as a process-monitoring tool. It also requires ongoing model maintenance and performance monitoring during commercial manufacture [2]. This builds on the FDA's PAT framework [3].
• Vendor-reported benefits exist (e.g. on the Seeq and Imubit sites [8][11]), but they are unaudited marketing figures. Measure your own baseline: off-spec rate, lab-sample frequency, time-to-detect.

Variants

• Continuous process industry: inferential quality for distillation columns (impurities, cut points), polymer melt index and density, refinery product properties, cement fineness and free lime.
• Batch / bioprocess & pharma: golden-batch models and end-of-batch prediction of titer, yield and CQAs. Raman/NIR spectra are turned into concentrations via chemometric models (PLS) as a PAT soft sensor [5].
• Semiconductor virtual metrology: predict film thickness, critical dimension or etch depth per wafer from tool sensor traces (FDC) instead of measuring every wafer. Published examples include CVD nitride thickness and lithography CD [6]. Wafer-map defect classification is VIS-056.
• Discrete manufacturing (predictive quality): predict part quality from machine data: injection moulding (dimensions from cavity pressure/temperature curves), welding (seam quality from current and voltage), machining (surface roughness), extrusion, battery-cell production (electrode quality).
• Food & beverage: moisture, Brix, fat content or taste-related attributes from line data. Freshness/shelf-life virtual sensors for perishables [1].
• Water & environment (public sector): effluent COD/NH₄ or emission estimates (predictive emission monitoring) as a backup to analysers. Treatment optimisation itself is OPT-024.
• Product-side virtual sensors (automotive/machinery): estimate quantities without a physical sensor, e.g. temperatures or loads in a component, to save hardware cost. Deployed on ECUs or edge runtimes.
• Private use: not relevant.

How it works

1. historian / PLC / FDC traces + LIMS lab values (sparse, delayed)
2. contextualise tags (asset, unit, recipe, batch) [DATA-016]
3. align lab timestamps with process data (dead time, residence time, sample delay)
4. cleaning (outliers, frozen sensors, missing values, steady-state / phase detection)
5. feature engineering (lags, moving windows, batch-phase statistics, spectra pre-processing)
6. model: PLS/PCR · gradient boosting · tabular FM · sequence model · hybrid first-principles + ML
7. validation on later time periods (never random splits)
8. deployment: stream scoring every few seconds/minutes
9. historian tag / DCS / dashboard
10. monitoring: applicability domain, residual vs. each new lab value, drift
11. recalibrate

• Model families: linear latent-variable models (PLS, PCA-based) are still the workhorse because they are robust with co-linear sensors and easy to explain and validate [1][5]. Gradient-boosted trees and neural nets handle non-linear regimes. Hybrid (grey-box) models combine mass/energy balances or a process simulator with an ML correction term and extrapolate better.
• Bias update: each new lab value corrects the model's offset (recursive/moving-window updates). Online-learning libraries such as River support this pattern [10].
• Human-in-the-loop: process engineers choose inputs and approve models. Operators see the estimate with its uncertainty and validity flag. Quality release decisions stay with QA unless a validated real-time-release strategy has been approved.

Data preparation

Soft sensors live or die with aligned process data. Sensor time series sit in the historian (PI, Wonderware), SCADA or PLC logs; lab and quality results sit in LIMS, QM software, Excel or even paper inspection sheets; recipes, lots and orders sit in MES or ERP. In many SMB plants the historian is missing and data is only on local PLC or machine PCs.

Typical gaps: lab values with manual timestamps that do not match the sensor clock, dropouts and drift, changed tags, missing batch or lot IDs, too few lab samples, and undocumented maintenance or calibration events that explain jumps.

Steps: inventory tags and sources (PREP-001, PREP-002), get data out of historian, MES and ERP (PREP-005), digitise paper lab records (PREP-006, PREP-008), align timestamps and resample, join lab values to the process window, label downtime and maintenance periods (PREP-020), and agree a tidy table design (PREP-011). Tools: OPC UA, Node-RED, Telegraf with InfluxDB or TimescaleDB, pandas, Power Query. Define the evaluation set with the process engineers (PREP-018). See DATA-003 and GUIDE-003.

Plan about half to two thirds of the effort for this work, including OT access and time-sync fixes. Do not expect a good model before the lab data is traceable.

Implementation paths

Which path for whom

• smb: C. Process and recipe data are trade secrets, simple tabular models run on HW1, and one open-source stack (Python, MLflow, a time-series database) can serve soft sensors, anomaly detection and reporting together.
• enterprise: B. Multiple plants need a managed ML platform in an EU region connected to the historian and MES; industrial suites (Path A) fit where the plant already runs them.
• developer: C. Open libraries give full control of feature engineering, validation and drift monitoring.
• public (labs, utilities, research plants): C. Sovereign hosting and open tooling fit procurement and critical-infrastructure rules.

Path A — Buy (fits plants that already run a historian and industrial suites)

• Self-service process analytics on top of the historian: TrendMiner (Belgian origin, ML and agentic analytics layer, EU-friendly) [9], Seeq (Agent Q / Seeq Intelligence) [8], AVEVA PI System with its analytics ecosystem.
• Inferential/APC suites from automation vendors: AspenTech (now part of Emerson) Aspen IQ inferential models with Aspen DMC; Yokogawa Platform for Advanced Control and Estimation; Honeywell Profit Suite; Siemens APC in SIMATIC PCS 7/neo.
• Batch & pharma: Sartorius SIMCA / SIMCA-online (German owner, Umetrics MVDA, golden-batch and real-time predictions, Raman models) [5].
• Semiconductor: virtual metrology and FDC modules from fab-software vendors (Applied Materials SmartFactory, PDF Solutions Exensio, which also targets battery-cell manufacturing [12]).
• Choose A when: a historian already exists, and you want engineers (not data scientists) to build, deploy and maintain models with an audit trail.

Path B — Build on cloud APIs

• Platforms: Azure Machine Learning / Fabric Real-Time Intelligence (EU regions), AWS SageMaker (Frankfurt) with IoT SiteWise, Google Vertex AI (EU locations), Databricks. Sovereign EU options: STACKIT, IONOS, OVHcloud running the open stack from Path C.
• Industrial data platforms for contextualised data and model deployment: Cognite Data Fusion (Norway), Siemens Insights Hub, Microsoft Fabric / Azure IoT Operations.
• Integration: OPC UA / MQTT (Sparkplug B) into a unified namespace, scoring service returns the estimate as a tag. LLMs are optional, as a copilot that explains models or drafts feature hypotheses. They are not the predictor.
• Choose B when: many sites share one model factory and the data is already in the cloud. Latency and OT/IT segmentation must allow it. Many plants keep scoring on-prem even when training is in the cloud.

Path C — Self-host (open source)

• Modelling: scikit-learn (PLS, ridge, GPR), XGBoost/LightGBM/CatBoost, sktime/Darts for lagged time-series features, River for online/bias-updated models [10]. TabPFN (Prior Labs, Freiburg/Germany) is a tabular foundation model for small lab-value datasets. Note that the v2.5+ weights carry non-commercial licences, so check licensing before production use [7]. Chronos-2 (Apache-2.0) supports covariate-informed forecasting when you need to predict quality ahead in time [4].
• Data & ops: TimescaleDB/InfluxDB or the existing historian, Node-RED/Telegraf for OPC UA, MLflow for model registry, Grafana for operator views. Export models to ONNX for edge PCs next to the line.
• Hardware: HW1 is enough. Training and scoring of PLS/boosting models run on a CPU server or industrial edge PC. HW2 only for deep sequence models on high-frequency traces or spectra.
• Ops effort: model monitoring and recalibration are the main ongoing cost, not compute.

Path D — Custom training

Every soft sensor is technically "custom-trained" on plant data. Path D here means going further: deep sequence models (1D-CNN/transformer) on high-frequency machine traces (virtual metrology, welding, moulding curves), hybrid physics-ML models built on a process simulator, or transfer learning across similar lines/tools. Worth it when linear/boosting models plateau and there are thousands of labelled samples, or when one model must be transferred to many identical machines. Data needed: several months to years of historian data with consistent lab results that cover the operating window (grades, loads, seasons).

State of the art (2026)

• Robust baselines still win often: PLS and gradient boosting with careful time alignment and domain features stay the default in production. Data quality, not model class, decides accuracy [1].
• Tabular foundation models (TabPFN-3.x, up to 1M rows per its documentation) give strong out-of-the-box regression on small datasets. Licensing is the main limit [7].
• Time-series foundation models with covariates (Chronos-2) make "predict next hours of a quality variable from process inputs" feasible zero-shot as a baseline [4].
• Hybrid modelling and agentic analytics: vendors add physics-informed models and LLM agents that search the historian, propose features and explain deviations (TrendMiner, Seeq Agent Q) [8][9].
• From open-loop to closed-loop: soft sensors increasingly feed APC and AI controllers. Autonomous setpoint writing is OPT-038.
• Failure modes: extrapolation outside the trained operating range (new grade, raw material, catalyst age), sensor drift and recalibration, changed lab methods, leakage from mis-aligned timestamps, spurious correlations with operator actions, and silent degradation if nobody compares predictions with new lab values. Mitigate with applicability-domain checks, residual monitoring and scheduled re-validation (as ICH Q13 requires for pharma [2]).

Shared resources & combinations

• Same data pipeline (historian + OT contextualisation, streaming): ANOM-005 industrial anomaly detection, PRED-006 predictive maintenance, PRED-013 production yield prediction, DATA-016 tag mapping and unified namespace. One OPC UA/MQTT ingestion layer, one historian and one model registry serve all.
• Downstream optimisation: OPT-009 parameter optimisation and OPT-038 closed-loop control use the soft-sensor estimate as the target variable. OPT-024 does the same in water treatment.
• Quality stack: VIS-001 camera inspection (end-of-line verification of the predicted quality), VIS-056 wafer yield analytics (pairs with virtual metrology), SCI-018 spectral interpretation (PAT spectra as inputs), PRED-025 warranty early warning (field feedback on quality).
• Simulation: SCI-005 surrogates and process simulators supply the physics half of hybrid models.
• Hardware: HW1 CPU/edge PCs shared with predictive maintenance and anomaly detection. An HW2 GPU only if deep models on traces or spectra or an on-prem LLM copilot (KNOW-004) are added.

Effort, cost & KPIs

• Path A: first soft sensor in 4–12 weeks on an existing historian. Cost driver is the platform licence (per site/user/tag) plus engineer time.
• Path B/C: 2–4 months for the first production model including data alignment, OT integration and monitoring. Every further sensor on the same pipeline is much cheaper.
• Hidden costs: lab data quality (wrong sample timestamps are the most common problem), OT network approvals, and ongoing recalibration ownership.
• KPIs: prediction error vs. lab (RMSE/MAE relative to lab method repeatability), share of time inside the validity domain, time-to-detect off-spec, off-spec/scrap rate, lab samples per day, analyser downtime covered, APC uptime when the soft sensor is the controlled variable.

People & change

Involve the department lead (production or quality manager) and the process engineers and operators who will use the estimate; they know which signals matter and when the model is plausibly wrong. IT and IT security must approve OT access (read-only data tap first, IEC 62443). Data owners for historian, LIMS and MES clarify exports. Management sets scope and accepts that the first model needs weeks of data work.

Works council: depends. A soft sensor on pure process data is not aimed at employees. However, when operator IDs or shift logs enter the dataset, or alarms and corrections are logged per person, the system is objectively suited to monitor behaviour, and §87(1) no. 6 BetrVG applies (BAG: suitability is enough). Always inform under §90(1) no. 3 when work processes change, and consider an expert under §80(3). Agree in writing that the model is not used for performance evaluation.

Data protection: pseudonymise operator IDs; usually no DPIA, but ask the DPO (GUIDE-005). AI literacy: train operators on what the estimate means, how to read confidence and when to trust the lab (GUIDE-006, Art. 4 requires support measures).

Pilot on one line in shadow mode: show the estimate next to the lab value without acting on it. Typical concerns are job loss and "the model overrides my judgement"; address them by keeping operators in the loop and the safety systems independent of the model. See GUIDE-009 and GUIDE-008.

Risks, regulation & compliance

• EU AI Act (depends): a soft sensor that only informs operators or quality engineers is usually minimal risk (Art. 4 AI literacy applies). It becomes high-risk if it is used as a safety component in the management and operation of critical infrastructure (supply of water, gas, heating or electricity; Annex III point 2) [13]. It can also become high-risk as a safety component of machinery or other Annex I products (the Machinery Regulation (EU) 2023/1230 applies from Jan 2027). Quality-only use is not a safety function. Keep safety instrumented systems (IEC 61511) independent of ML estimates. Art. 5 and Art. 50 are not relevant.
• Pharma (GMP): soft sensors used for control or release are part of the control strategy. They need validation, change control and model maintenance (ICH Q13, EU GMP Annex 11 for computerised systems, data integrity / ALCOA+) [2][3].
• Other sector rules: emission estimates used for permit reporting need regulator acceptance (EN 14181 for analysers). Food: HACCP, so critical control points must not rely only on an unvalidated model. Cybersecurity: IEC 62443 and NIS2 for OT integration. Write-back into a DCS needs a reviewed interface.
• GDPR: process data is normally not personal data. Operator IDs or shift logs in the dataset can make it personal, so pseudonymise them and do not use the system for staff performance monitoring without works-council agreement (BetrVG §87(1) no. 6 in Germany).
• Mitigations: validity flags, fallback to the last lab value, alarms on residual drift, versioned models with approval workflow, and periodic comparison against reference measurements.

Getting started

1. Pick one variable with high business impact and slow measurement (e.g. a lab value that drives off-spec decisions). Quantify today's lab delay, sampling frequency and off-spec cost.
2. Export 6–24 months of historian data plus LIMS results. Fix sample timestamps and agree dead times with process engineers.
3. Build a PLS and a gradient-boosting baseline with time-based validation. Check that the error is close to the lab method's repeatability.
4. Run the model in shadow mode for 4–8 weeks, displaying it next to lab values, before operators or APC act on it.
5. Define ownership for monitoring and recalibration, then scale to further variables on the same pipeline (and to ANOM-005/PRED-006 on the same data).

Sources

Note: the web-search budget ran out during research. Vendor facts were verified by fetching the primary pages listed below where possible. 1. Kadlec, Gabrys, Strandt — Data-driven Soft Sensors in the Process Industry, Computers & Chemical Engineering 33(4), 2009 — http://eprints.bournemouth.ac.uk/8498/1/CACE_KadlecGabrysStrandt_2008.pdf 2. ICH Q13 — Continuous Manufacturing of Drug Substances and Drug Products (Step 4, 2022), incl. soft-sensor and model-maintenance definitions — https://database.ich.org/sites/default/files/ICH_Q13_Step4_Guideline_2022_1116.pdf 3. U.S. FDA — PAT: A Framework for Innovative Pharmaceutical Development, Manufacturing, and Quality Assurance (Oct 2004), FDA guidance for industry — https://www.fda.gov/regulatory-information/search-fda-guidance-documents/pat-framework-innovative-pharmaceutical-development-manufacturing-and-quality-assurance 4. Amazon Science — Chronos forecasting models incl. Chronos-2 (covariates, Apache-2.0) — https://github.com/amazon-science/chronos-forecasting 5. Sartorius — SIMCA / SIMCA-online MVDA software — https://www.sartorius.com/en/products/process-analytical-technology/data-analytics-software/mvda-software/simca 6. Virtual metrology overview with references (Purwins et al. 2014 CVD thickness; Susto et al. 2015 lithography CD) — https://en.wikipedia.org/wiki/Virtual_metrology 7. Prior Labs — TabPFN repository (versions, size limits, weight licences) — https://github.com/PriorLabs/TabPFN 8. Seeq — industrial analytics platform, Agent Q — https://www.seeq.com/ 9. TrendMiner — industrial analytics, ML and analytics agent — https://www.trendminer.com/ 10. River — online machine learning in Python — https://riverml.xyz/latest/ 11. Imubit — closed-loop AI process models (vendor-reported results) — https://www.imubit.com/ 12. PDF Solutions — Exensio platform incl. battery manufacturing modules — https://www.pdf.com/ 13. EU AI Act Annex III (high-risk areas, point 2 critical infrastructure) — https://artificialintelligenceact.eu/annex/3/ 14. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj