Barcodes need one human motion per item. Computer vision removes it in two ways: point recognition (a camera at a scale, kiosk, bakery counter or smart cart identifies the item placed in front of it) and checkout-free stores (ceiling cameras and shelf sensors track which items each shopper takes, then charge automatically on exit). Point recognition is mainstream and low-risk; full checkout-free stores are operationally demanding, capital-intensive and carry the most privacy obligations.
Problem & value
- Today: queues at tills, staff time on scanning, produce keyed by PLU code (often wrongly), bakery and canteen items without barcodes, and shrinkage from missed scans.
- AI value: faster throughput, fewer queues, correct pricing of unbarcoded goods (bakery, produce, canteen trays), and a continuous shelf-level view of stock as by-product.
- Evidence is mixed. Amazon announced in 2024 that it was moving its Just Walk Out technology out of its Fresh grocery stores while continuing it in third-party venues such as stadiums, airports and campuses [1]. In January 2026 Amazon announced it would close almost all of its own Amazon Go and Amazon Fresh stores to focus on Whole Foods [9]. Treat checkout-free as strongest in small, high-traffic, controlled-range formats (venues, travel retail, convenience, micro-markets) and not as a default for full-range supermarkets. No universal ROI figure exists; build a business case per site.
Variants
- Unbarcoded-item recognition: scale or counter camera identifies produce, bakery or canteen trays (e.g. cafeteria tray recognition for staff restaurants).
- Smart carts and handheld/mobile scan: camera in the cart recognises items added; supermarkets use it for scan-as-you-shop.
- Checkout-free store: whole-store overhead cameras plus shelf weight sensors and an app or card gate; "grab and go".
- Micro-markets and smart fridges: a few cameras in a fridge or unattended kiosk; the most SMB-friendly variant.
How it works
- cameras (ceiling / shelf / cart / scale) + optional weight sensors
- person detection & multi-camera tracking (anonymous track IDs)
- interaction detection (hand reaches shelf, picks / returns item)
- product recognition (detector + embedding match against product catalogue)
- sensor fusion (weight change, shelf position / planogram prior)
- virtual cart per track, reconciled at exit gate
- payment (app account, card tap) + receipt
- low-confidence events
- human review queue / post-visit audit
- Product recognition is a fine-grained, open-set problem: tens of thousands of SKUs, new packaging monthly, near-identical variants (flavours, sizes). Practical systems use a detector for localisation plus an embedding (retrieval) model matched against reference images per SKU, so new products are added by enrolling images, not retraining.
- Fusion with planogram and weight resolves ambiguity: a pick from the shelf facing for a known SKU with a matching weight delta is far more certain than vision alone.
- Humans in the loop: uncertain baskets are resolved by remote reviewers or by a post-visit receipt correction; staff handle age-restricted goods; disputes go to customer service.
- Edge inference is the norm: video streams stay in-store, only events and cropped item images leave the site.
Data preparation
Recognition needs a clean product master (EAN/GTIN, names, variants, dimensions, weight) and reference images for every item. In an SMB this master usually lives in the ERP or POS system and in supplier price lists; images sit on phones, in the web shop or in supplier portals. Typical gaps are duplicate or inconsistent article numbers, missing variant data, no images of loose goods (bakery, produce), outdated packaging and no ground truth for how items look on your own counter lighting.
Preparation steps: export the article master from POS/ERP (see PREP-005), de-duplicate and add missing GTINs, photograph each item from several angles under real store lighting (20-50 images per item is a common starting point), label them with the article ID, and collect a test set of real baskets with a human-verified result (PREP-018). Tools: a spreadsheet or the ERP export, a simple labelling tool such as Label Studio, and the vendor's enrolment app. Plan roughly 30-40 percent of total effort for data; it drops sharply after the first range because new items only need new images. For camera grids, also document where cameras may point and where shoppers must stay unrecorded.
Implementation paths
Which path for whom
- SMB: Path A, because shelf-ready scale, kiosk and smart-cart products with EU hosting exist and the checkout process is standard.
- Enterprise: Path B, because chains need integration with their own ERP, POS and product master data and the volume makes per-store licences costly.
- Developer: Path C, because open detection and embedding models on edge hardware let you build a product-recognition module without per-camera fees.
Path A — Buy (recommended)
- Checkout-free platforms: Amazon Just Walk Out (technology licensed to third parties) [1], Zippin, Standard AI, AiFi, Trigo (Israel-based, European grocery deployments), Grabango (US; reported to have shut down after failing to raise funding). Fully managed hardware + software + support; check EU data hosting and who is the controller.
- Smart carts and scales: Caper (Instacart), Veeve, Mettler Toledo, Bizerba (DE), Toshiba, NCR Voyix, Diebold Nixdorf (produce and item recognition at the scale or self-checkout).
- Canteen/bakery tray recognition: specialised vendors (e.g. Wasteless, Mashgin-type kiosks for self-service lanes). Prefer EU suppliers or EU-hosted options where available.
- Choose A when: you want a working store or lane in months and accept vendor lock-in and per-site pricing.
Path B — Build on cloud APIs
- Catalogue matching: multimodal embedding endpoints (Google Vertex AI Multimodal Embeddings / Vision Warehouse product search [2], Azure AI Vision multimodal embeddings, Amazon Bedrock Titan multimodal embeddings, Cohere Embed v4) with a vector database in an EU region.
- VLM fallback for ambiguous crops (Gemini, Claude, GPT, Mistral multimodal): useful for the long tail and for review assistance; too slow and costly for the main loop.
- Fit: pilots for kiosks, micro-markets and scale-side recognition. Real-time multi-camera tracking is not a good cloud-API workload because of latency and bandwidth.
Path C — Self-host (open source)
- Detection and tracking: Ultralytics YOLO (AGPL, commercial licence available), RT-DETR, Grounding DINO / YOLO-World for open-vocabulary, SAM 2 for segmentation, ByteTrack or BoT-SORT for tracking [3].
- Retrieval: DINOv2/DINOv3 or SigLIP 2 embeddings plus FAISS, Qdrant or pgvector for SKU matching [4].
- Hardware: a scale or kiosk runs on HW1 (Jetson Orin / Hailo); a whole store with dozens of cameras needs one HW3 server or several edge boxes. Calibrated multi-camera tracking is the hardest engineering part.
Path D — Custom training
Worth it when the assortment is distinctive (fresh bakery, local produce, returnable crates) or accuracy per SKU must be very high. Needs several images per SKU under real store lighting and occlusion (many hundreds of captured baskets for a first model), labelled baskets from pilot operations, and a pipeline to enrol new SKUs weekly. Continuous correction data from reviewed baskets is the best training source.
State of the art (2026)
- Retrieval-based recognition with foundation embeddings (DINOv2/v3, SigLIP 2, CLIP-class) made enrolment of new products cheap; few-shot SKU recognition is mainstream for scales and kiosks.
- Vision-language models serve as reviewers and for attribute reading (variant, size, text on pack), not yet as real-time trackers.
- Fusion (vision + weight + planogram) is what gets checkout-free to commercially acceptable accuracy; vision-only systems struggle with occlusion by crowds, identical items and items swapped back to wrong shelves.
- Limits and failure modes: children and groups sharing a basket, look-alike packs, reflective or transparent packaging, new seasonal packaging, adversarial item concealment, crowded peaks, lighting change, and manual review dependence. Report accuracy per basket, not per item, since one error spoils a basket.
Shared resources & combinations
- Same stack: VIS-009 (shelf monitoring uses the same SKU detector and embedding index), VIS-047 (self-checkout loss prevention), VIS-008 (footfall from the same cameras), VIS-022 (visual search shares embeddings), VIS-018 (OCR on pack labels).
- Product reference-image library and the vector index are reusable across all of them; camera calibration and edge GPUs (HW1/HW3) are shared infrastructure.
Effort, cost & KPIs
- Path A: weeks for a micro-market or smart fridge; months for a store build-out. Cost drivers are camera and sensor installation, per-store licence, and support. Vendor quotes vary widely, so benchmark several.
- Path C/D: 6-12+ months and a specialised vision team for store-scale; calibration, labelling and drift handling dominate effort.
- KPIs: basket accuracy (share of baskets charged correctly), item-level precision/recall, share of baskets needing manual review, receipt-dispute rate, shrinkage, throughput per hour, cost per transaction, time to enrol a new SKU.
People & change
Involve management first (budget, scope, decision on point recognition versus a full checkout-free store), then the department lead and the staff who work at the till, because their routine changes most. IT handles cameras, network and POS integration; the data owner of the article master must clear the product data. Camera coverage of the shop floor also records employees, so the works council has co-determination under §87(1) no. 6 BetrVG (objective suitability for monitoring is enough per the BAG) and must be informed under §90(1) no. 3; a works agreement should limit cameras to product tracking and exclude performance evaluation. Where no works council exists, still inform staff openly. Personal data (shopper images, app identity) means: involve the DPO early, run a DPIA (Art. 35 GDPR) for ceiling cameras, sign an AVV with the vendor, and display clear signage.
Support staff AI literacy under AI Act Art. 4 with a short session on what the system sees, where it errs and how to override it. Start with one counter or one store as a pilot group, with low-confidence items always routed to a human. Typical fears are surveillance and job loss; address them with a written purpose limitation and by redeploying freed time to customer service. See GUIDE-004, GUIDE-005 and GUIDE-008.
Risks, regulation & compliance
- EU AI Act: most deployments are minimal or limited risk. Art. 5 flag: the system must not perform emotion inference on staff, nor biometric categorisation of shoppers by sensitive attributes. Avoid real-time remote biometric identification; use app or card identity, not face recognition, for entry (see VIS-016). Check whether any component (for example employee monitoring) moves the system into Annex III. Always seek legal review; not legal advice.
- GDPR: tracking shoppers across a store processes personal data even with anonymous IDs. Needs a lawful basis (typically legitimate interest with documented balancing, or contract for app users), transparency signage, DPIA (likely required for systematic monitoring of a public-accessible area), data minimisation (process on the edge, delete video quickly, store only event data), and strict limits on secondary use such as marketing profiling. Staff cameras also raise works-council co-determination (Germany: BetrVG 87(1) no. 6) [6]. The EDPB guidelines on video devices are the main reference [7].
- Mitigations: edge processing, no biometric templates, short retention, visible signage, human review of disputed baskets, regular bias and error audits.
Getting started
- Pick the narrowest format that pays: a scale or bakery counter, a micro-market or a smart fridge, before any store-wide rollout.
- Run a DPIA and involve the data protection officer and works council before cameras go up.
- Photograph your top 200-500 SKUs under real conditions; build a reference library and test open-source embeddings against it.
- Request references and basket-accuracy data from two or three vendors from Path A; ask where the data is hosted and who reviews uncertain baskets.
- Pilot at one site for 8-12 weeks, measure basket accuracy and dispute rate, then decide go or no-go.
Sources
- Amazon - Just Walk Out technology (official page) - https://justwalkout.com/
- Google Cloud - Vertex AI Vision Warehouse and Multimodal Embeddings - https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-multimodal-embeddings
- Ultralytics YOLO documentation - https://docs.ultralytics.com/
- DINOv2 (Meta AI) - https://github.com/facebookresearch/dinov2
- Wei et al., RPC: A Large-Scale Retail Product Checkout Dataset (arXiv 1901.07249) - https://arxiv.org/abs/1901.07249
- Betriebsverfassungsgesetz section 87 (Mitbestimmung, Ueberwachung) - https://www.gesetze-im-internet.de/betrvg/__87.html
- EDPB - Guidelines 3/2019 on processing of personal data through video devices - https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en
- Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- AP via ABC7 - Amazon to close Amazon Go, Amazon Fresh locations to concentrate on Whole Foods (27 Jan 2026) - https://abc7.com/post/amazon-stores-closing-close-go-fresh-locations-concentrate-foods-grocery-delivery/18486641/