Visual quality inspection & defect detection in manufacturing

Manual end-of-line inspection is slow, tiring and inconsistent between inspectors. Machine-vision systems with deep-learning models inspect every part at line speed, detect scratches, cracks, missing features, contamination or surface flaws, trigger rejects and log each decision. Modern anomaly-detection models need only good samples to start, which makes the approach accessible to small and mid-sized manufacturers, not just automotive and electronics giants.

Problem & value

• Today: human visual inspection (sampling or 100 %) plus rule-based machine vision (thresholds, template matching, blob analysis). Humans fatigue and disagree; rule-based systems break on natural variation (textures, castings, wood, textiles, reflective metal) and need re-tuning per product.
• AI value: deep-learning models learn what a defect looks like (supervised) or what "normal" looks like (unsupervised anomaly detection), handle variable appearance, and scale to 100 % inspection with consistent criteria. Every decision is stored with its image, which supports traceability, complaint handling and process improvement.
• Benefits: fewer escapes, less scrap and rework, freed inspector capacity, traceable defect data. Vendor figures are self-reported; measure your own baseline [1][3].

Variants

• Surface inspection (sheet metal, glass, film, paint, wood, textile): line-scan cameras, segmentation of scratches/dents.
• Part and feature verification (dimensions, presence, orientation): see VIS-002, VIS-026.
• Packaging and label inspection (print quality, seals, codes): see VIS-018.
• Food and agricultural grading: see VIS-006.
• Electronics: PCB/AOI solder-joint checks; wafer-level analytics see VIS-056.
• Small workshop / SME: one smart camera at one station, no data scientist.

How it works

1. part triggers camera (PLC/encoder) + controlled lighting
2. image acquisition (GigE Vision / USB3 Vision / GenICam, line-scan or area-scan)
3. preprocessing (ROI crop, normalisation, alignment)
4. model inference on edge device (a) anomaly detection: good-only training, anomaly map + score (b) detection/segmentation: defect classes, boxes or masks (c) classification: OK / NOK / defect type
5. decision logic (thresholds per defect class and size, recipe-specific rules)
6. PLC signal: accept / reject / sort / stop line
7. storage: image + result + lot ID
8. dashboard, SPC, MES/QMS
9. human review of uncertain or rejected parts
10. labels fed back to retraining

• Models: unsupervised/few-shot anomaly detection (PatchCore, EfficientAD, FastFlow-style methods), supervised detectors and segmenters (YOLO family, RT-DETR, U-Net/Mask-style models), classifiers; vision-language models help with labelling, defect explanation and zero-shot triage but are generally too slow and not deterministic enough for the cycle-time-critical decision.
• Human-in-the-loop: inspectors review borderline scores, confirm new defect types and release or scrap held lots; a retraining and model-release process with change control is part of the system.

Data preparation

The data are images of good and defective parts, plus lot, recipe and PLC context from MES or ERP. Images typically sit on the inspection PC, on file shares or on USB sticks; defect knowledge sits in inspectors' heads and in paper reject logs. Typical gaps are inconsistent lighting and camera setup, too few defect examples, no agreed defect catalogue, wrongly labelled or unlabelled images, and no link between image and lot number.

Preparation steps: (1) fix optics, lighting and triggering first, since software cannot compensate for poor images; (2) agree a defect catalogue and acceptance limits with quality staff (PREP-019); (3) collect several hundred golden good samples and all available defects, labelled by experienced inspectors (PREP-018, DATA-023); (4) store images with lot, time and station metadata, deduplicated and cleaned (DATA-003); (5) keep a held-out test set; (6) feed operator corrections back as new labels (PREP-021).

Tools: vendor labelling tools, CVAT or Label Studio, simple scripts for metadata. Rough share of total effort: 40-60 percent, mostly optics and labelling. Anomaly-detection models need only good samples, which lowers the labelling burden.

Implementation paths

Which path for whom

• SMB — Path A: mature, EU-hostable machine-vision products with defect libraries exist at SMB prices and the inspection task is standard.
• Enterprise — Path C: inspection is core to quality, volumes over many lines make licences expensive, and one internal edge platform can serve all plants.
• Developer — Path C: open-source anomaly-detection and detection frameworks on edge hardware give full control over models, data and latency.

Path A — Buy (recommended for most SMBs)

• Deep-learning machine vision platforms: Cognex (In-Sight / VisionPro Deep Learning), Keyence (vision systems with AI-based inspection), Zebra (Aurora Vision), Landing AI (LandingLens), MVTec (HALCON, MERLIC; EU/Germany), Basler (cameras and software; EU/Germany), SICK (Inspector, AppSpace; EU/Germany), ISRA Vision / Atlas Copco machine vision (EU/Germany), Neurocle, Instrumental (electronics), plus EU start-ups such as Inspekto (now Siemens), Mikrolo and Eyedea offerings. Siemens and Rockwell offer vision integrated with their PLC ecosystems.
• Choose A when: a standard station (one product family, stable optics) must go live in weeks, you lack a vision engineer, or the system must integrate with a specific PLC/MES. Ask for a proof of concept with your own good and bad parts and defined acceptance criteria.

Path B — Build on cloud APIs

• Cloud services (Azure AI Custom Vision successors in Azure AI Foundry, Google Vertex AI Vision, AWS SageMaker; Amazon Lookout for Vision was discontinued, so check service lifecycle before committing) suit training and labelling in EU regions rather than line-speed inference, where latency and connectivity are issues.
• Typical pattern: train in the cloud, export (ONNX/OpenVINO/TensorRT) and run inference on the edge; use multimodal frontier or EU models (Mistral, Gemini, Claude) for defect description, labelling assistance, shift-report summaries and handling of rare cases offline.

Path C — Self-host (open source)

• Anomaly detection: Anomalib (Intel/OpenVINO, Apache-2.0 library with PatchCore, EfficientAD, PaDiM, etc.); detection/segmentation: Ultralytics YOLO (AGPL-3.0, commercial licence available), RT-DETR, MMDetection, Detectron2, SAM 2 / SAM 3 for semi-automatic mask labelling; open-vocabulary detectors (Grounding DINO, OWL-ViT) and open VLMs (Qwen-VL, PaliGemma, SmolVLM) for bootstrapping labels.
• Hardware: HW1 (industrial PC with a small GPU, Jetson Orin, Intel iGPU/NPU, Hailo) is enough for inference of one to several cameras; training fits HW2.

Path D — Custom training

Worth it when the product is highly specific, defects are tiny or subtle, throughput is high, or you build the inspection system as a product. Start with anomaly detection on good parts only, then add supervised defect classes as examples accumulate (often tens to a few hundred per class).

ground truth that is inconsistent between labellers.

State of the art (2026)

• Unsupervised anomaly detection reached near-saturation on the public MVTec AD benchmark; the field moved to harder, more realistic benchmarks (e.g. MVTec AD 2, Real-IAD, VisA) and to few-shot and zero-shot approaches using foundation models (CLIP/DINOv2-based, VLM-based) [2].
• Foundation models (DINOv2/v3, SAM family) reduce labelling effort and give robust features for new products with few images.
• VLM-assisted inspection is emerging but not yet a replacement for specialised models in cycle-time-critical decisions.
• Limits and failure modes: domain shift (new lighting, lens dirt, new material batch, tool wear) degrades accuracy silently; high false-reject rates when anomaly thresholds are too tight; tiny defects below pixel resolution; reflective parts; rare defects with few examples; benchmark scores that do not transfer to your parts. Plan drift monitoring and periodic revalidation.

Shared resources & combinations

• Same stack (object-detection + segmentation + edge-runtime): VIS-002 assembly verification, VIS-006 food grading, VIS-018 industrial OCR, VIS-026 measurement, VIS-007 PPE monitoring, VIS-020 waste sorting; one annotation platform and one MLOps pipeline serve all.
• Specialised siblings: VIS-053 (NDT/X-ray), VIS-056 (wafer maps), VIS-057 (additive process monitoring).
• Downstream: PRED-014 and PRED-006 use defect trends.

Effort, cost & KPIs

• Path A: weeks to a few months per station incl. optics and PLC integration; costs are hardware (camera, lens, lighting, compute) plus software licence per station or camera.
• Path B/C: 3–9 months for a first robust line; ongoing MLOps for drift and new products.
• KPIs: defect detection rate (recall) and false-reject rate (precision) on a frozen validation set, customer escape rate (ppm), scrap/rework cost, inspection cost per part, cycle time impact, share of uncertain parts sent to human review, model drift alerts.

People & change

Involve management for budget and scrap-rate targets, the quality department lead as process owner, and IT and IT security early for camera networks, PLC integration and remote access. Data owners (MES, ERP) release metadata. Inspectors are the key end users: they define what a defect is and later verify uncertain cases, so bring them in during the pilot rather than at go-live.

Works council: depends. Cameras aimed at parts are normally not a monitoring system, but if images capture workers or results are evaluated per station, shift or person, co-determination under §87(1) no. 6 BetrVG applies. The works council must also be informed in good time about AI-supported work processes (§90(1) no. 3); where in doubt, involve it early. In public bodies the staff council applies. A DPIA is usually not needed when no personal data are captured; check with the DPO.

AI literacy: the AI Act (Art. 4) requires measures that support staff's AI literacy; short hands-on training for inspectors and maintenance covers it. Communicate that the system assists, not replaces, inspectors; address job fears openly. Start with one line and a pilot group of experienced inspectors, keep a human review for uncertain or new defect types, and keep the manual fallback. See GUIDE-008, GUIDE-009 and GUIDE-004.

Risks, regulation & compliance

• EU AI Act: typically minimal risk. An inspection system that only rejects production parts is not an Annex III use. It can become high-risk (Annex I, via product-safety legislation such as the Machinery Regulation (EU) 2023/1230) if the AI acts as a safety component, for example when vision output is relied on to stop a hazardous machine. Check with your conformity assessment route. Art. 4 AI literacy applies to staff operating it [4][5].
• No Art. 5 issue as long as cameras do not infer emotions or evaluate workers. Do not reuse the line cameras to assess worker performance without works council involvement and a separate legal basis.
• GDPR / employee data: images of parts are not personal data, but cameras near workstations may capture employees; mask or crop to the part area, avoid recording operators, involve the data protection officer and works council (in Germany co-determination under §87 BetrVG).
• Quality and sector rules: automotive (IATF 16949, VDA), medical devices (ISO 13485, MDR), food (HACCP) require validated inspection processes, documented acceptance criteria and change control for model updates. Keep an audit trail of model version per decision.
• Mitigations: golden-sample checks at shift start, periodic re-validation, challenge parts with known defects, human review of rejects, versioned models, fallback to manual inspection.

Getting started

1. Define defect catalogue, acceptance criteria (size, location) and the cost of an escape versus a false reject; collect 200+ images of good parts and every defect type available.
2. Get optics right: test lighting and camera angles on the real part before choosing a model.
3. Run a proof of concept with 2 vendors (Path A) or Anomalib baseline (Path C) on the same frozen test set including borderline cases; compare recall, false rejects and speed.
4. Integrate with the PLC in shadow mode (decide but do not reject) for 2–4 weeks and compare with human inspection.
5. Go live with human review of rejects, drift monitoring and a model change-control procedure.

Sources

1. Cognex — Deep learning for machine vision, VisionPro Deep Learning — https://www.cognex.com/products/deep-learning
2. Bergmann et al. — The MVTec Anomaly Detection Dataset (IJCV 2021) and MVTec AD 2 — https://www.mvtec.com/research-teaching/datasets/mvtec-ad
3. Landing AI — LandingLens visual inspection documentation — https://landing.ai/
4. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. Regulation (EU) 2023/1230 (Machinery Regulation) — https://eur-lex.europa.eu/eli/reg/2023/1230/oj
6. Directive (EU) 2024/2853 (Product Liability) — https://eur-lex.europa.eu/eli/dir/2024/2853/oj
7. Anomalib (Open Edge Platform, Intel) — https://github.com/open-edge-platform/anomalib
8. Ultralytics YOLO documentation — https://docs.ultralytics.com/
9. MVTec HALCON / MERLIC — https://www.mvtec.com/products