Catalog / Robotics, Autonomy & Physical AI / ROBO-001

Vision-guided bin picking & pick-and-place

Let industrial robot arms see and grasp randomly placed parts or mixed items from bins, totes and conveyors using 3D cameras and learned grasp planning, for manufacturers and warehouses.

Parts needed to run it yourselfOptimizationRoboticsVision

Parts in crates, mixed items in totes and goods on conveyors are still often handled by people because classic robots need exact positions. Vision-guided picking combines 3D cameras, learned object detection and grasp planning so a robot arm can pick unsorted items reliably, feed machines and fulfil orders. It relieves staff from repetitive, ergonomically harmful work and keeps lines running in the face of labour shortages.

Problem & value

  • Today: fixed-position feeders, vibratory bowls, trays and manual kitting; or people picking from bins and totes. Warehouses pick millions of different SKUs, which fixed programming cannot cover.
  • AI value: robots handle unknown or randomly oriented items, changeovers need new CAD or images instead of reprogramming, and exceptions can go to a remote human operator.
  • Benefits: vendors report pick rates of several hundred to over a thousand picks/hour per cell and success rates in the high 90s % on suitable items, but figures depend on item mix, gripper and cell design and are vendor-reported; validate with your own items [3][4]. The International Federation of Robotics tracks strong industrial robot installations in Europe, with logistics and electronics among the growth areas [1].

Variants

  • Machine tending / bin picking of known parts (CAD-based): metal, plastic and cast parts in crates, fed to CNC machines, presses or assembly lines (automotive, mechanical engineering).
  • Piece picking of mixed SKUs (e-commerce, retail, pharma): goods-to-robot cells picking from totes into order boxes; unknown items, deformable packaging.
  • Pick-and-place on conveyors / kitting: high-speed sorting, food handling, electronics assembly, packing.
  • Waste and recycling sorting: picking materials from belts (see also VIS-family sorting use cases).
  • SMB cells: cobot plus plug-and-play vision kit for small batches.

How it works

  1. pick order / trigger (PLC, WMS, MES)
  2. 3D image capture (point cloud + RGB) of bin or belt
  3. detection / segmentation / pose estimation (CAD match or learned, class-agnostic)
  4. grasp planning (candidates scored for gripper, reachability, collisions with bin walls)
  5. motion planning + execution (force/vacuum feedback, in-hand checks)
  6. verification (weight, vacuum pressure, camera)
  7. place / drop
  8. fail handling: retry other grasp
  9. remote assist
  10. manual fallback
  • Perception: CAD-based 6D pose matching for known parts; learned instance segmentation and open-vocabulary detectors for unknown items. Depth quality on shiny, transparent or black items is the classic weak spot.
  • Grasp planning: neural networks (e.g. GraspNet-style, Dex-Net lineage) predict grasp success maps for suction or parallel grippers; modern systems learn from fleet data and field failures.
  • Control/safety: deterministic robot controller and certified safety functions remain outside the learned model; AI only proposes targets (see ROBO-016).
  • Human-in-the-loop: operator or remote teleoperator resolves unsolvable picks; staff approve new item onboarding and cell changes.

Data preparation

Needed: CAD models or reference images of the parts, SKU master data (dimensions, weight, gripper class), pick orders from WMS/MES/ERP, and 3D scans of real bins. In an SMB these live in the ERP, in CAD/PDM folders, on engineers' file shares and partly only on paper drawings. Typical gaps: missing or wrong item dimensions, duplicate article numbers, no CAD for legacy parts, no labelled images of failed grasps and unclear access rights to supplier CAD data.

Steps: (1) clean the article master and export it as a table (see PREP-005 and PREP-011); (2) collect CAD or scan every part once, prioritising the top 20 percent of parts by volume; (3) record 100 to 500 real bin images per part family under production light, including bad cases such as reflections and tangled parts; (4) define success and failure labels together with the operators (PREP-019, PREP-018); (5) log grasp results from day one to build a feedback set (PREP-021). Tools: vendor calibration software, Open3D, CVAT, simple spreadsheets. Roughly 25 to 35 percent of total effort goes into data and cell calibration; vendor tooling reduces this for Path A. See also GUIDE-003.

Implementation paths

Which path for whom

  • SMB: Path A. Mature bin-picking packages (3D camera plus grasp software, often EU-built) exist at SMB prices and the task is standard.
  • Enterprise: Path B. Picking must tie into WMS/MES/PLC and many cells, so integrators combine vendor vision with own orchestration and cloud/API services.
  • Developer: Path C. Open-source grasp and perception stacks (ROS 2, MoveIt, open grasp models) on HW1 edge hardware allow own products.
  • Turnkey piece-picking and bin-picking systems: Mujin (JP), Pickit 3D (BE, EU), Photoneo (SK, EU; part of Brightpick Group), Nomagic (PL/EU), RightHand Robotics (US), Dexterity (US), Covariant technology (team joined Amazon), Plus One Robotics (US, supervision software), Ocado/Kindred-type systems for grocery.
  • Robot OEMs with vision offers: Fanuc iRVision / 3D Area Sensor, ABB, KUKA (DE), Yaskawa, Universal Robots (DK) ecosystem, Schunk (DE) grippers, Zivid (NO) cameras, SICK (DE) and IDS (DE) sensors, Roboception (DE).
  • Choose A when: the task is common (tote picking, machine tending), you lack robotics ML staff, and the integrator provides safety validation and service. Check: EU support, CE conformity, interface to your PLC/WMS, pick-rate guarantee on your items.

Path B — Build on cloud APIs

  • Cloud is rarely in the control loop (latency, availability). Useful for: offline analytics of pick failures, VLM-based item attribute checks and onboarding (Gemini, Claude, GPT, Mistral multimodal in EU regions), fleet monitoring and teleoperation backends.
  • Framework layer: ROS 2 with MoveIt 2 for motion planning, NVIDIA Isaac ROS/Isaac Manipulator (GPU-accelerated FoundationPose, cuMotion, grasp generation) with cloud-based simulation [2][5].
  • Choose B when: you build your own cell with an integrator and want cloud for monitoring, data and simulation.

Path C — Self-host (open source)

  • Stack: ROS 2, MoveIt 2, Open3D / PCL, Segment Anything (SAM 2) and open-vocabulary detectors (Grounding DINO, YOLO family), FoundationPose / MegaPose for 6D pose, Contact-GraspNet or AnyGrasp-class grasp networks (check licences), GraspGen from NVIDIA research; OpenCV for calibration.
  • Hardware: on-cell industrial PC with Jetson Orin-class module or a 16–24 GB GPU (HW1 to HW2); inference in tens to hundreds of ms.
  • Ops effort: hand-eye calibration, gripper selection and tuning dominate; plan robotics engineers, not just ML.

Path D — Custom training

Worth it for unusual items (deformable, transparent, very small), very high volumes or when the pick-rate target is not met by off-the-shelf models. Needs hundreds to thousands of labelled images or synthetic data from simulation (Isaac Sim, domain randomisation), plus real pick outcomes (success/failure labels) for fine-tuning grasp scoring. Fleet data from production is the best source.

State of the art (2026)

  • Class-agnostic perception (SAM-family segmentation, foundation pose estimators) removes much per-item CAD effort; new items can often be onboarded with a few images.
  • Learned grasping is production-proven for suction piece picking; dexterous grasping of fragile or deformable items is improving but less reliable.
  • Vision-language-action and robot foundation models (e.g. Physical Intelligence pi0 family, NVIDIA GR00T, Google Gemini Robotics) are demonstrated for general manipulation, mostly in research and pilots; for industrial picking, the specialised pipeline above remains more reliable and certifiable (see ROBO-004).
  • Remote assist (human decides a grasp point from a remote screen) is used to reach near-100% completion while the model improves.
  • Limits and failure modes: reflective/transparent/dark items, tangled parts, double picks, items deforming under suction, lighting changes, camera drift, long-tail SKUs, and distribution shift after packaging changes. Always monitor with in-hand checks.

Shared resources & combinations

  • Shared stack: 3d-vision + segmentation + robotics-control + simulation is reused by ROBO-013 (welding path generation), ROBO-014 (depalletising), ROBO-017 (disassembly), ROBO-003 (programming by demonstration).
  • Safety layer: ROBO-016 for person detection and speed-and-separation monitoring around the cell.
  • Upstream/downstream: ROBO-002 AMRs bring and take totes; OPT-006 slotting and picking optimisation; VIS-001/VIS-002 inspection can run in the same cell on the same camera; ROBO-010 simulation to train and validate.

Effort, cost & KPIs

  • Path A: typically 3–9 months from study to production for an integrated cell; capital cost is a robot, gripper, camera, safety and integration, and varies widely (low to mid six figures EUR for a full piece-picking cell is common; get quotes).
  • Path B/C: engineering-heavy, months of integration and tuning; lower licence cost, higher staff cost.
  • KPIs: picks per hour, first-attempt pick success %, mean picks between failures, intervention rate (per 1,000 picks), changeover/onboarding time per new item, damage rate, uptime, payback time.

People & change

Involve management early for budget and safety responsibility, the department lead as process owner, IT and IT security for network segmentation of camera and robot controllers, and the affected operators from the first workshop (GUIDE-001). Works council: depends. The cell itself rarely monitors people, but camera systems near workplaces, cycle-time and performance dashboards or logged operator interventions can be suited to monitor behaviour or performance, which triggers co-determination under §87(1) no. 6 BetrVG (objective suitability suffices); in any case §90(1) no. 3 requires timely information about AI-supported work processes. Where a works council exists, involve it before the pilot and agree that data is not used for individual performance evaluation. The DPO is needed only if cameras capture people; mask or avoid that. AI literacy: the AI Act asks employers to support staff literacy, so train operators on what the vision system can and cannot do and on exception handling (GUIDE-006). Concerns are job loss and safety: communicate that the robot takes the heavy repetitive tasks, name a pilot cell and AI champions (GUIDE-008), keep an emergency stop and human release for every exception, and run a pilot with go/no-go criteria (GUIDE-009).

Risks, regulation & compliance

  • EU AI Act: depends. A robot is not high-risk by itself. AI used as a safety component of machinery covered by the Machinery Regulation (EU) 2023/1230 is high-risk under Art. 6(1)/Annex I and needs third-party conformity assessment where applicable; the pure perception/grasping function (not a safety function) is generally minimal-risk. Person detection for safety is covered in ROBO-016. No Art. 5 prohibition is relevant, but do not add workplace emotion recognition or covert performance scoring of workers [6][7].
  • Machinery: the Machinery Regulation 2023/1230 applies from 20 January 2027 (replacing Directive 2006/42/EC) and addresses self-evolving behaviour; ISO 10218-1/-2 (industrial robots), ISO/TS 15066 (collaborative operation) and ISO 13849 define safety requirements [7]. Keep learned components out of the safety function or assess them explicitly.
  • GDPR: cameras in the cell may capture workers; use narrow fields of view, no recording of people, works-council agreement (Betriebsrat, in Germany co-determination under §87 BetrVG) and a data-protection assessment if footage is stored.
  • Cybersecurity: connected cells fall under NIS2 for relevant sectors and the Cyber Resilience Act for products; segment OT networks, signed updates.
  • Other risks: workforce acceptance, over-promised pick rates, vendor lock-in on proprietary data. Mitigate with pilot on your own items, contractual KPIs and data-export rights.

Getting started

  1. Collect the item or part list with dimensions, weight, material and volumes; rank by difficulty (shiny, deformable, tiny).
  2. Define KPIs (picks/h, success rate, intervention rate) and send 200–500 real items to two or three vendors for a test.
  3. Choose gripper and camera placement with the integrator; run a risk assessment per ISO 10218 and agree who is responsible for CE marking.
  4. Pilot one cell with remote-assist fallback; log every failed pick with images.
  5. Roll out in stages; use the failure log to improve models and layout.

Sources

  1. International Federation of Robotics — World Robotics reports and press releases — https://ifr.org/worldrobotics/
  2. NVIDIA — Isaac Manipulator documentation — https://developer.nvidia.com/isaac/manipulator
  3. Pickit 3D — robot vision for bin picking and depalletizing — https://www.pickit3d.com/
  4. Mujin — MujinOS and piece picking — https://mujin-corp.com/
  5. MoveIt 2 documentation — https://moveit.picknik.ai/
  6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  7. Regulation (EU) 2023/1230 (Machinery Regulation) — https://eur-lex.europa.eu/eli/reg/2023/1230/oj
  8. Photoneo — 3D vision and robotic picking — https://www.photoneo.com/
  9. Segment Anything 2 (Meta) — https://github.com/facebookresearch/sam2