Synthetic image data generation for vision model training (rare defects, edge cases)

Visual inspection models need many labelled examples of defects, but good production lines produce few defects, and the rare ones matter most. Generative models (diffusion, GAN, controllable inpainting) and procedural 3D rendering can create realistic, automatically labelled defect images from a handful of real samples. Used carefully and validated on real held-out data, this lowers the number of real defect images needed and speeds up commissioning of inspection systems.

Problem & value

• Today: quality teams wait months to collect enough defect images, or label them by hand. Rare defect classes (cracks, contamination, a new supplier's scratch pattern) and new product variants have almost no data. Models then miss exactly those defects, or produce false rejects.
• AI value: inpaint or render defects onto good parts with masks produced for free, cover lighting/pose/material variations, and balance class frequencies. Industrial vendors report that inspection models can be trained from small real sample sets using generated defects [1][2]; academic work on defect-image generation shows gains mainly when real data is scarce [3].
• Honest framing: synthetic data supplements real data; it does not replace a real validation set. Gains depend on domain gap, and no general figure should be assumed. Measure on your own parts.

Variants

• Defect inpainting on real good parts: a diffusion model with mask control adds scratches, dents, stains onto real images (most common in manufacturing).
• Procedural 3D rendering: CAD + physically based rendering with domain randomisation (lighting, pose, texture); labels are exact. Used for assembly checks (VIS-002), logistics, robotics.
• Few-shot defect synthesis: fine-tune a small LoRA/adapter on 10-50 real defect crops.
• Edge-case scene generation: rare weather, lighting, occlusions for automotive and drone perception; video world models create driving or site scenarios.
• Medical, agriculture/food, private use: rare pathologies, foreign bodies or hobby datasets; synthetic rare pathologies or cell images (research-heavy; strict validation).

How it works

1. real samples (good + few defects) + defect taxonomy + CAD (optional)
2. generator choice (inpainting | LoRA-tuned diffusion | procedural render | GAN)
3. controlled generation (mask/position/size/severity prompts, ControlNet-style conditioning)
4. automatic labels (mask
5. box/class) + quality filters (realism score, duplicate, plausibility)
6. human expert review of a sample (is this defect physically possible and correctly named?)
7. train detector/segmenter/anomaly model on real + synthetic mix
8. validate ONLY on held-out real images (domain-gap check)
9. iterate

• Generators: latent diffusion with inpainting and structural conditioning (ControlNet, depth, edges), GAN variants for narrow textures, rendering engines with randomisation.
• Label propagation: the generated mask defines the label, so annotation cost drops; check boundary accuracy because soft diffusion edges can shift the true extent.
• Human-in-the-loop: quality engineers approve defect definitions and review samples; real validation images are never mixed into synthetic training pools.

Data preparation

The data is the hard part. Real defect and good-part images sit on inspection PCs, camera systems, MES/QMS databases, USB sticks and quality reports; CAD models come from PLM or file shares. Defect information is often only in free-text complaint records or on paper.

Typical gaps: very few defect examples, inconsistent camera settings, no or contradictory labels, duplicates, no defect catalogue with agreed class definitions, and no clean held-out real validation set. Access rights are scattered between production, quality and suppliers.

Preparation steps: (1) inventory image sources and export them with timestamps, line and part IDs; (2) define defect classes with quality staff and fix the labelling guideline; (3) label a small gold set with domain experts; (4) split off a real validation set before any synthetic generation and never mix it in; (5) log provenance of every synthetic image; (6) harvest labels from existing quality records where possible. Tools: CVAT or Label Studio, DVC for versioning, Blender or Omniverse for rendering.

Rough share of effort: 50 to 60 percent of the project. See PREP-018 for evaluation sets, PREP-019 for category schemes, PREP-020 for harvesting labels and PREP-005 for getting data out of legacy systems.

Implementation paths

Which path for whom

• smb: Path B, because the generator must fit the own parts, cameras and defect classes and no off-the-shelf product covers this for small series.
• enterprise: Path C, because production images are trade secrets, volume is high and a shared internal ML platform can run diffusion and rendering on HW2 and above.
• developer: Path C, because open diffusion models, Blender and similar tools give full control over labels and domain-gap experiments.

Path A - Buy (off-the-shelf)

• Inspection platforms with built-in synthesis: UnitX (GenX / defect generation) [1], Overview.ai (Defect Creator) [2], plus general platforms such as Landing AI LandingLens, Cognex (e.g. ViDi/In-Sight), Zebra Aurora, MVTec HALCON (EU, Germany), Neurocle, Sualab.
• Synthetic data services: NVIDIA Omniverse Replicator partners, Unity Computer Vision; EU: Anyverse (Spain, automotive/sensor simulation). Tabular vendors belong to DATA-005.
• Choose A when: you want an inspection system and have no ML team; verify EU hosting and on-prem option for images of your products (trade secrets).

Path B - Build on cloud APIs

• Image models with inpainting/editing APIs: Black Forest Labs FLUX (EU company, FLUX Fill / Kontext for editing) [4], Stability AI Stable Diffusion family, Google Imagen/Gemini image editing, OpenAI image editing, Adobe Firefly (commercially safe training data) for non-technical content.
• Pipeline: cloud storage in EU region, orchestration with Python/Airflow, a VLM (Claude, Gemini, Mistral, GPT) as a "judge" that flags unrealistic or mislabelled outputs, then training on Azure ML, Vertex AI or SageMaker (EU regions).
• Check: licence terms for generated output, no training of the vendor on your images, cost per image at 10^4-10^5 images.
• Choose B when: quick pilots, varied products, no GPU team.

Path C - Self-host (open source)

• Models: open-weight FLUX.1 [dev] / Fill (non-commercial licence, check commercial terms [4]), Stable Diffusion 3.5 / SDXL (community licence), diffusers library with ControlNet and inpainting pipelines [5]; segmentation by SAM 2 for mask refinement; detectors via Ultralytics YOLO (AGPL), RT-DETR, anomaly libraries such as Anomalib (Intel, Apache-2.0) [6].
• Procedural: Blender (BlenderProc [7]), NVIDIA Omniverse Replicator [8], NVIDIA Isaac Sim, Unity Perception.
• Hardware: HW2 (24-32 GB GPU) for inpainting and LoRA fine-tuning of SDXL-class and FLUX quantised; HW3 for FLUX full-precision at volume; rendering benefits from RTX GPUs.

Path D - Custom training

Worth it when you have a recurring product family and a stable defect catalogue. Fine-tune a diffusion model or LoRA on 20-500 real defect crops per class (labelled by experts), or train a conditional GAN for narrow textures. Keep a clean real test set; track every synthetic batch with generator version, seed and prompts.

State of the art (2026)

• Mask-conditioned diffusion inpainting is the practical default; it outperforms older GANs on realism and variety. Recent defect-generation papers focus on few-shot, mask-aligned generation and on verifying that the defect looks like the target class [3].
• Procedural rendering with domain randomisation remains strongest where geometry and exact labels matter (assembly, robotics, logistics); generative models add texture realism.
• World/video models (e.g. NVIDIA Cosmos [8]) are used to expand driving and robot scenes with varied conditions; outputs still need real-data validation.
• Alternatives: anomaly detection (PatchCore-family, Anomalib [6]) needs only good images and can reduce the need for defect data at all; synthetic anomalies also feed these methods.
• Limits and failure modes: physically impossible defects; label noise at mask boundaries; mode collapse and low diversity; the model learns generator artefacts ("shortcut learning"), scoring well on synthetic and poorly on real images; bias from the few seeds copied across thousands of images; benchmark gains on MVTec AD-like datasets do not guarantee line performance.

Shared resources & combinations

• Same stack (image-gen + segmentation + object-detection): GEN-011 image editing and inpainting, GEN-008 3D asset generation (CAD/asset sources for rendering), GEN-001 product imagery.
• Consumers: VIS-001 (inspection), VIS-002 (assembly), VIS-056 (wafer maps), VIS-057 (additive manufacturing), VIS-043 (sewer pipes); ROBO-010 (simulation for robot training).
• Scope boundary: DATA-005 covers tabular/text synthetic data; this entry covers image-level synthesis.

Effort, cost & KPIs

• Path A: weeks; licence per line/camera, often bundled with the inspection system.
• Path B: 1-3 months for a pilot; image-generation cost is usually small versus GPU training and expert review time.
• Path C: 2-4 months; one GPU workstation, ML engineer, quality engineer time for review.
• KPIs: recall of rare defects on the real test set, false-reject rate, number of real defect images needed to reach a target recall, time to commission a new product variant, share of generated images rejected by reviewers, domain-gap (synthetic vs. real score).

People & change

Involve management for budget and risk appetite, the quality or production lead as process owner, the inspectors who know the defects and will validate the results, IT for GPU and storage, and the data owners of the image and CAD sources. Legal or compliance should check supplier contracts and customer confidentiality, because production images often belong to the customer.

Works council: usually no co-determination, since generating training images does not monitor employees (§87(1) no. 6 BetrVG); it becomes relevant if images show workers or later feed performance evaluation. The council must be informed in good time about planned AI-supported work processes (§90(1) no. 3), and may call in an AI expert (§80(3)). A DPIA is rarely needed; check when images may show people and ask the DPO.

AI literacy: train quality staff on domain gap, bias of synthetic data and why real validation stays mandatory (Art. 4 AI Act: measures to support literacy). Start with one defect class on one line as pilot. Typical concern: "synthetic data makes the inspection unreliable". Address it with a documented real-test comparison and a go/no-go gate. Humans own the defect definitions and release decisions.

See GUIDE-009 for pilots, GUIDE-003 for data readiness and GUIDE-008 for acceptance.

Risks, regulation & compliance

• EU AI Act: typically minimal risk for quality inspection. It becomes high-risk if the model is a safety component of a product under Annex I legislation (e.g. machinery, medical devices, vehicles) or in critical infrastructure; then Art. 10 data governance applies, including documentation of data origin and bias examination, which fits synthetic data provenance [9]. Art. 50 marking duties concern content shown to people (deepfakes); internal training images mostly fall outside, but label them synthetic anyway. Art. 4 AI literacy applies to staff.
• GDPR: mostly no personal data in parts images; personal data arises with images of workers, faces, number plates or patients. Do not use generators fine-tuned on real faces without legal basis; for medical data, apply anonymisation and DPIA. Generated images that memorise training samples can leak personal data.
• IP and licences: model licences (FLUX dev non-commercial, SD community), copyright of training data in generators, customers' CAD and trade secrets - avoid sending to external APIs.
• Safety risk: over-trusting synthetic gains can lead to missed real defects in safety-critical parts.
• Mitigations: real held-out test set from production, staged rollout in shadow mode, drift monitoring, dataset cards with generator version and share of synthetic data, expert sign-off.

Getting started

1. Define defect classes and the cost of misses; collect whatever real samples exist and fix a real validation set that will never be used for generation.
2. Train a baseline on real data only (or an anomaly model such as PatchCore via Anomalib).
3. Generate 500-2000 defect images per class by mask-guided inpainting; have a quality engineer review 5-10%.
4. Retrain with several real-to-synthetic ratios; compare on the real test set and keep the ratio that helps.
5. Document provenance and run in shadow mode before releasing to the line.

Sources

1. UnitX - synthetic defect generation (GenX) - https://www.unitxlabs.com/
2. Overview.ai - Defect Creator - https://overview.ai/
3. Zhang et al. - "DefectGAN: Synthesizing Data for Industrial Defect Inspection" and later diffusion-based defect generation (arXiv) - https://arxiv.org/abs/2103.01593
4. Black Forest Labs - FLUX models and licensing - https://bfl.ai/ and https://huggingface.co/black-forest-labs
5. Hugging Face Diffusers - inpainting and ControlNet - https://huggingface.co/docs/diffusers
6. Anomalib (open-edge-platform) - https://github.com/open-edge-platform/anomalib
7. BlenderProc - https://github.com/DLR-RM/BlenderProc
8. NVIDIA - Omniverse Replicator and Cosmos - https://developer.nvidia.com/omniverse/replicator and https://github.com/nvidia-cosmos
9. Regulation (EU) 2024/1689 (AI Act), Art. 10 and Art. 50 - https://eur-lex.europa.eu/eli/reg/2024/1689/oj