Retail shelf monitoring & planogram compliance

Empty shelves, misplaced items and wrong price labels cost retailers sales and irritate shoppers, yet shelf checks are still mostly manual walks with clipboards or handhelds. Vision models read shelf images, recognise products and price tags, compare them with the planogram and turn gaps into prioritised restocking tasks. Brands use the same technology to measure share of shelf and promotion execution across thousands of stores.

Problem & value

• Today: staff walk aisles, scan barcodes and eyeball facings; field-sales reps audit a few stores per day with paper or tablet checklists. Data is sparse, late and subjective. Replenishment systems trust book inventory, which is wrong whenever shrink, misplacement or a late delivery occurs ("phantom inventory").
• AI value: frequent, objective shelf state; on-shelf availability fixes within hours instead of days; verified promotion and planogram execution; fewer price-label errors; less labour for audits.
• Evidence: vendors publish case studies on availability and sales uplift, but figures are self-reported and depend on store format; run a controlled pilot (test vs. control stores) before relying on any benchmark [3][4].

Variants

• Retailer in-store operations: fixed cameras or autonomous robots scan aisles daily; tasks go to staff devices.
• Staff or crowd photo capture: employees photograph shelves with a phone; lowest hardware cost, suits SMB chains and discounters.
• CPG / brand field audit: merchandisers photograph shelves in third-party stores; AI computes share of shelf, facings, price and promo compliance.
• Price and label verification: OCR on shelf-edge labels matched against price file or electronic shelf labels (ESL).
• Small shops and pharmacies: simple cabinet or fridge cameras detecting empty slots (vending, beverage coolers).

How it works

1. capture (phone / fixed camera / robot / cooler camera)
2. image quality gate (blur, angle, coverage; stitching of panoramas)
3. shelf & product detection (object detection / open-vocabulary detector)
4. SKU recognition (embedding match against reference images + OCR of labels/price tags)
5. planogram alignment (map detections to shelf slots, count facings, find gaps)
6. rules & scoring (out-of-stock, wrong position, wrong price, missing promo)
7. task generation (restock, re-facing, relabel) with priority by sales impact
8. human confirmation on low-confidence items
9. KPI dashboards, feed to replenishment

• Detection: dense-scene detectors (YOLO-class, RT-DETR/RF-DETR) trained on crowded shelves; the SKU-110K benchmark popularised densely packed retail object detection [5].
• Recognition: thousands of near-identical SKUs are handled by two-stage matching: a class-agnostic detector proposes boxes, then an embedding model (CLIP/SigLIP/DINOv2-style) retrieves the nearest reference product from a vector index. New products are added by uploading reference images, no retraining.
• VLMs help with long-tail questions ("is the promotion sign present?", "which brand is this?") and with reading unusual labels, but are too slow and costly for every frame.
• Human-in-the-loop: reviewers confirm uncertain SKUs; store managers decide on action; the system never changes prices or orders autonomously without rules and approval.

Data preparation

Shelf recognition needs four data sets: shelf images, the planogram (positions, facings), product master data with reference images per SKU, and POS or stock data. They typically sit in the ERP or merchandise system, in space-planning tools or Excel files, in supplier PDFs and on staff phones. Typical gaps are missing or outdated reference images, EANs without clean product names, planograms that exist only as slides or PDFs, duplicate article numbers and shelf photos with customers or staff in view.

Preparation steps: export article master data with EAN, name, size and brand (see PREP-005); photograph each SKU from several sides, or obtain packshots from suppliers; convert planograms into tidy tables of shelf, position and facings (PREP-011); label 200 to 500 real shelf photos with the store team as a test set (PREP-018); blur faces and persons before storing images (PREP-022). Define a fixed photo routine (distance, angle, lighting) so image quality stays consistent.

Rough share of effort: about 30 to 40 percent of the project, mostly for the SKU image library and planogram clean-up. Assortment changes require ongoing upkeep of the reference images.

Implementation paths

Which path for whom

• SMB: Path A, because mature shelf-recognition apps work with ordinary phone photos at per-store prices and the process (planogram check) is standard.
• Enterprise: Path B, because chains need the results inside their own ERP, replenishment and planogram tools and have the SKU image volume to tune models on their own range.
• Developer: Path C, because open detectors (YOLO-class) plus embedding search on a HW2 edge box give full control over a custom product catalogue.

Path A — Buy (recommended)

• Shelf-intelligence platforms: Trax (retail execution, image recognition for CPG), Simbe (Tally shelf-scanning robot), Zebra (Retail Vision / fixed and mobile capture), Focal Systems (shelf cameras with real-time shelf data), Scandit (smartphone-based shelf scanning), Pensa Systems, Vusion/SES-imagotag (ESL with camera-based shelf sensing; Vusion is EU-based). EU-based or EU-rooted options include SES-imagotag/Vusion (France) and Pricer (Sweden) on the label side; verify data residency for image storage with each vendor.
• Choose A when: more than a handful of stores, you need integration with the planogram tool (e.g. Blue Yonder, Relex, JDA-style space planning) and a supported device ecosystem.

Path B — Build on cloud APIs

• Detection / OCR: Google Cloud Vision product search or Vertex AI Vision, Azure AI Vision and Custom Vision/Foundry, Amazon Rekognition Custom Labels and Textract, all with EU regions.
• VLM check: Gemini, Claude, GPT or Mistral multimodal models via EU endpoints for photo audits with structured JSON output; combine with a reference-image index (pgvector, Qdrant).
• Orchestration: mobile photo app, storage in an EU region, n8n or Power Automate for task creation in the store workforce app. Good for photo-based audits at modest volume.

Path C — Self-host (open source)

• Models: Ultralytics YOLO (AGPL, check licence), RT-DETR and RF-DETR (Apache-2.0 variants), Grounding DINO / OWLv2 for open-vocabulary detection, DINOv2 or SigLIP embeddings, PaddleOCR or Docling for labels, Qwen-VL class open VLMs for occasional queries.
• Stack: Roboflow-style annotation (CVAT, Label Studio), Qdrant/Milvus for SKU embeddings, NVIDIA DeepStream or Frigate-like pipelines for camera streams, ONNX/TensorRT export for edge devices.
• Hardware: HW1 edge (Jetson Orin, Hailo) for in-store inference; HW2 central GPU server serves hundreds of stores doing a few scans per day.

Path D — Custom training

Worth it when assortment is unusual (fresh produce, pharmacy, DIY, Asian-language packaging) or when scaling to many stores makes per-image API fees large. Needs several hundred labelled shelf images per format plus 5-20 reference images per SKU; active learning from reviewer corrections keeps the model current as packaging changes.

State of the art (2026)

• Two-stage detect-then-match with strong image embeddings is the practical standard, because assortments change weekly. Open-vocabulary detectors and foundation embeddings reduce labelling effort compared with training one classifier per SKU.
• Multimodal LLMs reasonably answer shelf questions but still miscount dense facings and confuse look-alike variants (flavours, sizes); use them as a second opinion, not as the counting engine.
• Edge shelf cameras and ESL-integrated sensing shift monitoring from periodic scans to near-real-time availability, while robots and phone scans stay cheaper per store for large formats.
• Failure modes: occlusion by shoppers and carts, reflective packaging and glass doors, low light, stacked or pushed-back products, near-identical SKUs, seasonal packaging changes, and planograms that are outdated or not digital in the first place.
• Data foundation matters: value appears only when alerts connect to a task process and to inventory systems (see OPT-005).

Shared resources & combinations

• Same stack (object-detection + embeddings + edge-runtime): VIS-017 visual product recognition and checkout-free retail, VIS-047 self-checkout loss prevention, VIS-019 warehouse inventory counting, VIS-026 visual counting and measurement; one SKU reference-image index can serve all of them.
• Downstream: OPT-005 replenishment and PRED-001 demand forecasting (use measured availability to correct lost-sales demand), OPT-004 pricing, OPT-016 promotion planning.
• Privacy tooling: VIS-030 anonymisation of faces in shelf images; VIS-008 footfall analytics if the same cameras are used (separate purpose, separate assessment).

Effort, cost & KPIs

• Path A: weeks to pilot a few stores; subscription per store, device or image; robots and fixed cameras carry capex or leasing.
• Path B: 2-4 months for a photo-audit app with integrations.
• Path C/D: 4-9 months incl. data collection, labelling and edge rollout; ongoing model upkeep for new packaging.
• Cost drivers: camera or robot hardware, store network, labelling and SKU reference data, change management.
• KPIs: on-shelf availability (%), out-of-stock duration, planogram compliance score, share of shelf, price-label error rate, SKU recognition precision/recall, task completion time, incremental sales in test vs. control stores, labour hours saved per audit.

People & change

Involve management for budget and scope, the department lead (store or category manager) as process owner, and IT for ERP and device integration from the start. Store staff who take the photos are the key end-users: bring them in at the pilot stage so tasks feel helpful rather than controlling. Data owners of article master data and planograms must confirm the data early.

Works council: depends. Phone photos and fixed cameras can capture employees, and task and compliance reports per store or person may be suited to monitor performance, which triggers co-determination under §87(1) no. 6 BetrVG; also inform it under §90(1) no. 3. Agree that reports show shelves, not people. DPO: check incidental capture of customers and staff, run a DPIA where fixed cameras film continuously, and sign an AVV with any cloud vendor (GUIDE-005, GUIDE-004).

AI literacy: the AI Act asks deployers to support staff in building AI literacy, so offer a short session on what the system sees, typical false detections and how to correct them (GUIDE-006). Pick two or three stores as pilot (GUIDE-009). Typical concerns are job loss and surveillance; address them with a clear purpose statement and by keeping staff in control: they confirm or dismiss every task and can report wrong detections (GUIDE-008).

Risks, regulation & compliance

• EU AI Act: product-recognition on shelves is typically minimal risk (AI literacy, Art. 4). Art. 5 caution: do not extend cameras to emotion inference, biometric categorisation or identification of shoppers or employees; those uses are prohibited or high-risk and are out of scope here [1].
• GDPR: shelf images can incidentally capture customers and staff. Aim for downward-facing or shelf-only framing, capture when aisles are empty (robot night scans), blur faces at the edge (VIS-030), short retention, and a DPIA where fixed cameras run continuously. Follow EDPB guidelines on video devices and German works-council co-determination (BetrVG §87) if staff performance could be inferred [2][6].
• Staff monitoring risk: compliance scores per employee create surveillance concerns; report at store or aisle level.
• Mitigations: privacy by design, vendor DPA and EU hosting, bias checks across store formats, human review of automatic orders.

Getting started

1. Pick 3-5 stores and 2 categories with high stock-out cost; measure current availability manually for two weeks.
2. Make sure planograms are digital and mapped to product master data (GTIN).
3. Run a phone-photo pilot or vendor proof-of-concept with your own shelves, including glass doors and promotions.
4. Define who receives each alert, within what time, and how completion is confirmed.
5. Do the DPIA and works-council agreement before any fixed camera goes live; compare test and control stores after 8-12 weeks.

Sources

1. Regulation (EU) 2024/1689 (AI Act), Art. 4, 5 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
2. EDPB Guidelines 3/2019 on processing of personal data through video devices — https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en
3. Trax Retail — shelf image recognition solutions — https://traxretail.com
4. Simbe Robotics — Tally shelf-scanning robot — https://www.simbe.com
5. Goldman et al., "Precise Detection in Densely Packed Scenes" (SKU-110K), CVPR 2019 — https://arxiv.org/abs/1904.00853
6. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj
7. Ultralytics YOLO documentation — https://docs.ultralytics.com
8. Focal Systems — real-time shelf digitisation — https://focal.systems