Food quality grading & sorting

Natural products vary in size, colour, ripeness and defects, and packhouses still rely on manual graders and simple colour sorters. Modern sorters combine high-speed cameras, NIR or hyperspectral sensing and deep-learning classifiers to grade every item, eject defects and foreign material, and report grade distribution per lot. Typical benefits are consistent grading, less good product lost to rejects, and fewer foreign-body complaints.

Problem & value

• Today: grading is done by trained people on inspection tables plus rule-based optical sorters (colour/size thresholds). Human graders tire, differ between shifts and cannot see internal defects or non-visible contaminants. Over-rejecting wastes food; under-rejecting causes complaints and recalls.
• AI value: learned classifiers handle natural variation (variety, season, lighting) where threshold rules fail; hyperspectral/NIR and X-ray add information beyond human vision (sugar, bruising under skin, shell fragments, bone); data per lot supports pricing and grower feedback.
• Benefit figures: vendors report higher yield and lower give-away; treat these as vendor claims and measure on your own product with a baseline trial [1][2].
• Regulatory driver: EU marketing standards define quality classes for many fruits and vegetables, and food law makes the operator responsible for safe food and traceability [3][4].

Variants

• Fresh produce packhouse: apples, citrus, potatoes, tomatoes, berries; size/colour/blemish grading, often with weight.
• Processing lines: frozen vegetables, fries, French-style potato products, defect and foreign-material removal.
• Nuts, seeds, grains, dried fruit, coffee: high-throughput free-fall belt/chute sorters; aflatoxin-risk screening with NIR/UV (indicative only).
• Meat, poultry, fish: trim, fat/bone detection, fillet grading, portioning (X-ray and 3D).
• Bakery and ready meals: shape, topping and colour checks (overlaps VIS-001).
• Farm and small-grower level: low-cost camera graders, smartphone grading apps.

How it works

1. product singulation & presentation (belt / chute / free fall, controlled lighting)
2. multi-sensor capture (RGB, NIR, hyperspectral, laser/3D, X-ray, weight)
3. detection & segmentation of each item
4. classification: size, colour, shape, defect type, ripeness, foreign material
5. grade decision against specification (rules + learned thresholds)
6. ejector/diverter control (air jets, flaps, robotic pick) in milliseconds
7. lot reporting, traceability, drift monitoring, retraining loop

• Models: compact CNN/transformer detectors and segmenters running on edge GPU/FPGA/industrial PCs for latency of a few milliseconds per item; anomaly detection for rare contaminants where defect examples are scarce.
• Spectral sensing: hyperspectral and NIR give chemistry-related signals (moisture, sugar, fat, plastic vs organic) that RGB cannot.
• Human-in-the-loop: QA staff define grades and tolerance, review rejects and borderline samples, approve model updates, and run regular reference samples and seeded-contaminant tests.

Data preparation

Data: product images from the line under production lighting, defect examples, grade standards (customer specifications, marketing standards), NIR or hyperspectral spectra and reference lab measurements. In an SMB these live in the line camera, QA spreadsheets, paper inspection sheets, the ERP (batches, customers) and the heads of senior sorters.

Typical gaps: too few defect examples, grading differing between shifts and customers, images without batch link, unlabelled data, and standards that exist only as PDF or tacit knowledge.

Preparation steps: 1. Write down grade definitions and borderline examples with QA (PREP-016, PREP-019). 2. Record images at the real line with fixed lighting and link them to batch IDs from the ERP (PREP-005, PREP-020). 3. Label with two or more sorters and measure agreement, using CVAT or Label Studio (PREP-018). 4. Keep a frozen test set per product and season. 5. Digitise paper inspection logs (PREP-008).

Effort: Path A mostly means commissioning and calibration (10-20%); Path B/C/D 40-60%, mostly labelling and defect collection. Check readiness with GUIDE-003.

Implementation paths

Which path for whom

• SMB (packhouses, small processors): A, because established sorter vendors deliver camera, optics and ejector as a tested package and the grading task is standard for common produce.
• Enterprise (large processors): B, because grading must feed own quality, ERP and traceability systems and high volumes justify tailored models on cloud or edge inference.
• Developer: C, because open-source detection and segmentation models on HW1 edge devices allow custom products and defect classes.

Path A — Buy (recommended for most processors)

• Optical sorters and graders: TOMRA Food, Key Technology (Duravant), Bühler (CH), Satake, Raytec Vision (IT, EU), Sortex (Bühler), Greefa (NL, EU), Aweta (NL, EU), Compac (NZ), Unitec (IT, EU), Sesotec (DE, EU, contaminant sorting), Marel and Baader (fish/meat), plus X-ray inspection from Mettler-Toledo, Eagle (Eagle PI), Minebea Intec (DE, EU) and Ishida.
• Vendors increasingly offer deep-learning classification trained on your product, remote support and cloud dashboards [1][2].
• Choose A when: a standard commodity with an established machine class exists, throughput is high, and you need food-safety documentation and service.

Path B — Build on cloud APIs

• Cloud vision services (Azure AI Vision custom models, Google Vertex AI Vision or AWS SageMaker, EU regions) suit offline grading, photo-based intake checks and analytics, not millisecond ejection.
• VLMs (Claude, Gemini, GPT, Mistral Pixtral) help with labelling, defect-taxonomy bootstrapping, report generation and low-volume manual grading by photo; latency and cost rule them out of the real-time line.
• Choose B when: you prototype graders, grade at intake by smartphone, or analyse data from existing sorters.

Path C — Self-host (open source)

• Detection/segmentation: Ultralytics YOLO (AGPL, check licence) [6], RT-DETR, RF-DETR, SAM 2 for labelling; torchvision/MMDetection.
• Anomaly detection: Anomalib (Intel, OpenVINO export), PatchCore-style methods for unlabelled good-product training [5].
• Runtime: ONNX Runtime, TensorRT, OpenVINO on industrial PCs or Jetson Orin; GenICam/Aravis for industrial cameras; Label Studio or CVAT for annotation.
• Hardware: HW1 (edge GPU or industrial PC) per line; training on HW2. Ejector timing, lighting and mechanics are the hard part, not the model.

Path D — Custom training

Often needed even with Path A/C: train per product, variety and season. Hundreds to thousands of labelled images per defect class under production lighting; add synthetic or augmented images for rare contaminants (see GEN-020). Plan regular retraining as harvest quality changes.

State of the art (2026)

• Deep learning on RGB plus NIR/hyperspectral is mainstream in commercial sorters; academic reviews describe hyperspectral imaging with CNNs as the leading approach for internal and chemical quality attributes, with real-time deployment limited by cost and data volume.
• X-ray and 3D extend detection to bone, glass, stones and shell fragments that colour cameras miss.
• Foundation models (SAM, DINOv2, open-vocabulary detectors) speed labelling and few-shot setup; they are not yet used for the hard-real-time ejection loop.
• Limits and failure modes: distribution shift across varieties, seasons and lighting; wet, touching or overlapping product; subtle defects (early rot, internal browning) invisible to cameras; class imbalance; and false rejects that erode yield. Sorters reduce but never eliminate foreign bodies, so they are one control inside HACCP, not a replacement.

Shared resources & combinations

• Same stack (object-detection, segmentation, anomaly-detection, edge-runtime): VIS-001 industrial defect detection, VIS-002 assembly checks, VIS-026 counting and volume estimation, VIS-053 NDT/X-ray evaluation, VIS-020 waste sorting. One edge platform and MLOps pipeline can serve several lines.
• Upstream/downstream: VIS-005 crop health in the field, PRED-013 yield prediction, ROBO-007 harvesting robots, SCI-018 spectral data interpretation, VIS-033 food waste tracking.
• Data: annotation tools, synthetic data (GEN-020) and drift monitoring are shared.

Effort, cost & KPIs

• Path A: machine purchase or lease plus installation, typically weeks to a few months with recipe setup; cost scales with throughput and sensor set.
• Path B/C: 3-9 months for a production-grade line integration including lighting, mechanics, PLC and validation.
• Cost drivers: sensors (hyperspectral/X-ray), mechanical integration, labelling per season, validation and maintenance.
• KPIs: false-reject rate and false-accept rate per defect class, detection rate of seeded test contaminants, yield and give-away, throughput, grade-distribution consistency across shifts, customer complaints and recall events, uptime.

People & change

Involve early: management (investment, ROI), the production and QA lead as process owner, sorters and line staff who know the defects, IT and maintenance for the line integration, IT security for connected machines, and customers or suppliers whose specifications define the grades. Legal or compliance reviews food-safety and liability aspects.

Works council: depends. Where a works council exists, a camera line is mostly product-focused, but §87(1) no. 6 BetrVG is triggered if images, reject statistics or line speed can be attributed to individual workers, since objective suitability suffices. Inform it in good time under §90(1) no. 3 and consider a short works agreement (GUIDE-004).

Data protection: product images are generally not personal data; avoid capturing staff in frame. If staff can be identified, involve the DPO (GUIDE-005).

AI literacy: support sorters and QA with training on interpreting rejects, drift and recalibration (GUIDE-006).

Communication and pilot: present it as relief from monotonous inspection and move people to higher-value QA tasks (GUIDE-008). Run a shadow-mode pilot on one lane, where the system suggests grades while humans keep sorting, then compare (GUIDE-009).

Typical concerns: job loss (reskilling, honest plan), fear of wrongly rejected goods (tune thresholds, manual re-check bin), and trust (QA keeps the right to override and to change grades).

Risks, regulation & compliance

• EU AI Act: minimal risk in most cases. A sorter is typically not a safety component under Annex I and not an Annex III system; AI literacy (Art. 4) applies. Reassess if the vision system becomes a safety component of regulated machinery (Machinery Regulation (EU) 2023/1230, applicable from January 2027) [7].
• No Art. 5 issue unless cameras also observe workers (then see VIS-007 and employment rules).
• Food law: Regulation (EC) 178/2002 (operator responsibility, traceability, recall) and 852/2004 (HACCP) apply; automated sorting must be validated, documented and verified as a control step, with seeded-contaminant tests [3][4].
• Marketing standards: classes (Extra, I, II) for fruit and vegetables follow EU marketing rules; the model must reflect the current legal specification [8].
• GDPR: generally no personal data; check grower/lot data and any worker-visible cameras.
• Risks and mitigations: drift (periodic reference samples), over-reliance (retain metal/X-ray detection and manual checks), audit trail of model versions per lot, supplier lock-in for recipes.

Getting started

1. Define grades and defect classes with QA, and collect 500-1,000 labelled images per main class under real line lighting.
2. Benchmark 2-3 vendors (Path A) with your own product, including seeded foreign-material tests.
3. If building, start with a camera and anomaly-detection prototype on one product before touching the ejector.
4. Integrate with PLC, lot traceability and HACCP records; define reject handling and review.
5. Monitor false-reject/accept rates weekly and retrain each season.

Sources

1. TOMRA Food - sorting and grading solutions - https://www.tomra.com/food
2. Bühler - optical sorting - https://www.buhlergroup.com/global/en/products/optical-sorting.html
3. Regulation (EC) No 178/2002 (General Food Law) - https://eur-lex.europa.eu/eli/reg/2002/178/oj
4. Regulation (EC) No 852/2004 on food hygiene - https://eur-lex.europa.eu/eli/reg/2004/852/oj
5. Anomalib (Intel / OpenVINO) - https://github.com/open-edge-platform/anomalib
6. Ultralytics YOLO documentation - https://docs.ultralytics.com
7. Regulation (EU) 2023/1230 Machinery Regulation - https://eur-lex.europa.eu/eli/reg/2023/1230/oj
8. European Commission - marketing standards for fruit and vegetables - https://agriculture.ec.europa.eu/farming/crop-productions-and-plant-based-products/fruit-and-vegetables_en
9. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj