Robot programming by demonstration & natural language

Programming an industrial robot classically needs a specialist, a teach pendant and days per task, which is why small series and frequent changeovers rarely get automated. Hand-guided teaching, no-code skill libraries and, increasingly, LLM/VLM-assisted program generation let a process expert define a task by showing or describing it. The generated program is validated in simulation and released by a human. AI shortens commissioning and changeover; it never replaces the safety validation of the cell.

Problem & value

• Today: robot programs are written by integrators in vendor languages (KRL, RAPID, URScript, Karel) or taught point by point. Every product variant or layout change needs the same expert again. SMEs with lot sizes of 1-100 therefore often skip automation, and a shortage of robot programmers aggravates it.
• AI value: faster setup of new tasks (palletising patterns, machine tending, screwing, gluing, simple assembly), re-use of skills across variants, lower dependence on external integrators, and onboarding of shop-floor staff as "robot teachers".
• Evidence: vendors report cobot applications deployed in days instead of weeks, but few independent, quantified studies exist; measure on your own task. The market is real: the IFR counts roughly 540,000 industrial robots installed worldwide in 2024 and sees easier programming as a key driver for SMEs [1].

Variants

• Lead-through / kinesthetic teaching: hand-guide the arm, record waypoints, tool actions and force profiles (mainstream in cobots).
• No-code skill graphs: drag-and-drop skills (pick, place, screw, inspect) with camera-based parameterisation (UR PolyScope X, Wandelbots, Intrinsic Flowstate, Franka Desk).
• Learning from demonstration (LfD / imitation learning): teleoperated or video demonstrations train a visuomotor policy (ACT, Diffusion Policy, VLA models) for contact-rich or variable tasks; see ROBO-004.
• Natural-language-to-program: an LLM maps "stack the grey boxes on the pallet, 4 layers" to calls of vetted skills or code, with simulation preview.
• Integrator / OEM engineering: offline programming and code generation from CAD for whole cells (see ROBO-013 welding paths).
• Education and private use: hobby arms and teaching robots (e.g. LeRobot SO-101) for learning; no regulatory burden beyond product safety.

How it works

1. demonstration (hand-guiding / teleop / video) or text/voice instruction
2. perception of workcell (CAD + depth camera + VLM scene understanding)
3. task parsing: LLM/VLM maps intent to a library of verified skills + parameters
4. program synthesis (skill graph, behaviour tree or constrained code)
5. static checks (allowed API, joint/speed limits) + simulation (collision, cycle time, reachability)
6. human review & approval
7. dry run at reduced speed with safety functions active
8. release to production; execution logging, correction by operator
9. skill update

• Constrain, do not trust: the LLM should only call a typed skill API (e.g. pick(object, grasp)), not emit free motion commands. Code-generation research (Code as Policies, ProgPrompt) and SayCan-style grounding show this pattern; direct low-level code is brittle [5].
• Demonstration to policy: imitation learning (ACT, Diffusion Policy) needs tens to hundreds of teleoperated demonstrations per task and yields learned policies that are harder to verify than deterministic programs [6].
• Human-in-the-loop: the programmer approves every generated program; safety parameters (speed and separation limits, tool centre point, payload, safety zones) are set and validated by qualified people, never by the model.

Data preparation

Needed: demonstrations (kinesthetic teaching, teleoperation or video) of the task, the work instruction or drawing, part and tool data, and process parameters. In an SMB the know-how sits in experienced workers' heads, in paper work instructions, PDF drawings, PLC programs and the existing robot programs on the controller. Typical gaps: undocumented variants and tolerances, inconsistent work instructions, no recording of good and bad runs, scattered robot backups and unclear ownership of the expert knowledge.

Steps: (1) pick one stable task and write down the target process with the expert (PREP-016); (2) digitise instructions and drawings (PREP-006) and collect existing robot programs; (3) record 20 to 100 clean demonstrations, labelled with part variant and outcome (PREP-019, PREP-020); (4) define a small test set of situations for acceptance (PREP-018); (5) keep a glossary of part and tool names for language prompts (PREP-015). Tools: vendor teaching apps, video capture, spreadsheets, CVAT. Data work is roughly 20 to 30 percent of effort and is mostly expert time; cell setup and safety validation are separate. See GUIDE-003.

Implementation paths

Which path for whom

• SMB: Path A. Cobot vendors offer demonstration and no-code programming tools out of the box, which fits small batch sizes without in-house robot experts.
• Enterprise: Path B. Many cells and product variants favour integrating cloud or LLM-based programming services with own skill libraries and MES data.
• Developer: Path C. Open-source imitation-learning and LLM planning stacks (for example LeRobot, ROS 2) on HW1/HW2 hardware allow own skills and data control.

Path A — Buy (recommended for most SMBs)

• Cobot vendors with no-code or AI tooling: Universal Robots (PolyScope X, AI Accelerator with NVIDIA; DK), Franka Robotics (DE, Franka Desk and FR3 with app-based skills), KUKA (DE; iiQKA ecosystem), ABB (GoFa/SWIFTI, Wizard Easy Programming), FANUC (CRX with tablet programming), Doosan, Techman.
• Hardware-agnostic platforms: Wandelbots (DE; Wandelbots NOVA, teach by demonstration with a handheld device, now with code/AI interfaces), Intrinsic (Alphabet; Flowstate, skill-based), Mech-Mind (vision-guided no-code), Neura Robotics (DE; MAiRA cognitive robot), Sereact (DE; language/vision-driven picking), Micropsi Industries (DE; learned skills).
• Choose A when: the task is standard (pick-place, machine tending, palletising, screwing) and an integrator or vendor already certifies the cell.

Path B — Build on cloud APIs

• Use a frontier LLM/VLM (Claude, Gemini, GPT, Mistral; EU regions or EU providers where possible) for task parsing and program drafting against your own typed skill API, with structured outputs/tool calling.
• Vendor interfaces: ROS 2 with MoveIt 2 (open), UR RTDE/URScript, Wandelbots NOVA API, Intrinsic Flowstate SDK, Isaac Sim for validation.
• Cloud is acceptable for planning (non-real-time). Never put a cloud call in the real-time or safety loop.

Path C — Self-host (open source)

• Stack: ROS 2, MoveIt 2, BehaviorTree.CPP/py_trees, Gazebo or NVIDIA Isaac Sim, Hugging Face LeRobot (ACT, Diffusion Policy, SmolVLA, pi0 open weights) for demonstration learning [6][7].
• Models: open-weight coding LLM (Qwen-Coder, Mistral/Devstral class) and a VLM (Qwen-VL) served via vLLM/Ollama.
• Hardware: HW2 for LLM + VLM planning and small policy training; real-time control on an industrial PC; HW3 for larger VLA fine-tuning.

Path D — Custom training

Worth it for high-volume, contact-rich or variable tasks where scripted skills fail (cable insertion, deformable objects). Needs 50-500+ teleoperated demonstrations per task variant, a calibrated cell and an evaluation protocol with success rates over many trials. Plan significant ML and robotics expertise.

State of the art (2026)

• Skill-based programming plus LLM front-ends is the practical pattern in production; vendors add natural-language assistants that generate skill graphs or scripts for human review.
• Imitation learning (ACT, Diffusion Policy) is robust for single tasks; VLA models (RT-2, OpenVLA, pi0, Gemini Robotics, NVIDIA GR00T) promise generalisation but remain research-to-pilot grade for industrial use [6][7].
• Failure modes: hallucinated objects or skills, wrong spatial reasoning (left/right, units), unsafe trajectories, sensitivity to lighting or fixture changes, and policies that fail silently out-of-distribution. Simulation-to-real gaps remain (see ROBO-010).
• Not solved: certifying learned behaviour for safety functions. Safety remains deterministic and separate.

Shared resources & combinations

• Same stack: ROBO-001 (vision-guided picking), ROBO-013 (welding path generation), ROBO-014 (palletising), ROBO-017 (disassembly) all use the skill library, camera calibration and simulation twin.
• Safety layer: ROBO-016 provides AI person detection and speed and separation monitoring.
• Simulation: ROBO-010 digital twins; ROBO-004 for VLA/humanoid policies; DOC-013 reads drawings that feed task parameters; AGENT-003 shares the "LLM operates a tool through a typed API" pattern.

Effort, cost & KPIs

• Path A: days to weeks per application; cobot cell plus software licences, often cheaper than an integrator project.
• Path B/C: 2-4 months for a validated skill API, simulation pipeline and approval workflow; ongoing skill maintenance.
• Path D: quarters; demonstration collection and evaluation dominate cost.
• KPIs: time from task definition to production release, changeover time, share of programs created by non-specialists, first-pass simulation-to-real success rate, cycle time vs. expert program, stops/interventions per shift.

People & change

Involve management for the business case, the department lead and the skilled workers who teach the robot (they are the key data source), IT and IT security for controller access, and later the programmers or maintenance staff who own the skills (GUIDE-001). Works council: depends. Recording demonstrations captures a worker's movements, and logs of who programmed or corrected what can be suited to monitor behaviour, which engages §87(1) no. 6 BetrVG even without monitoring intent; §90(1) no. 3 requires timely information about AI-planned work processes. Agree in writing that demonstration recordings serve only skill creation, are deleted or anonymised afterwards and are not used for performance review (GUIDE-004). If video shows identifiable people, involve the DPO (GUIDE-005). AI literacy: the AI Act asks employers to support staff literacy, so train workers to teach, test and correct skills and to read the robot's limits (GUIDE-006). Typical concerns: being replaced by one's own recorded know-how and loss of expertise. Respond with a clear statement that the goal is relief from repetitive tasks, recognition and qualification for teachers, a pilot group of volunteers with AI champions (GUIDE-008), and human approval of every new skill before production release.

Risks, regulation & compliance

• Machinery safety: the new Machinery Regulation (EU) 2023/1230 applies from 20 January 2027 and explicitly addresses machinery with self-evolving behaviour/AI-based safety functions; ISO 10218-1/-2 (2025) and ISO/TS 15066 cover robot and collaborative safety. Every program change that alters hazards needs risk assessment and CE-conformity consideration [2][3].
• EU AI Act: the programming assistant alone is minimal risk. If AI is a safety component of machinery covered by Annex I (Machinery Regulation), the product is high-risk, with obligations originally applying from August 2027 (Art. 6(1)); the Digital Omnibus on AI (Reg. (EU) 2026/1744) postponed this to 2 August 2028. Art. 4 AI literacy applies to staff [4].
• Art. 5 note: not applicable to the core function; do not combine with worker emotion inference or covert performance monitoring (prohibited at work, Art. 5(1)(f)).
• GDPR / works council: cameras in cells capture employees; demonstrations may record the teacher. Mask or avoid persons, define purpose and retention, involve the works council (BetrVG §87).
• Security: prompt injection or manipulated generated code; sandbox the skill API, sign released programs, restrict network access (IEC 62443, NIS2, Cyber Resilience Act).
• Mitigations: typed skills only, simulation gate, reduced-speed first run, four-eyes release, immutable audit log.

Getting started

1. Pick one repetitive, low-variance task with clear success criteria and a safe, fenced or cobot cell.
2. Try hand-guided teaching on a cobot from Path A and time it against the integrator baseline.
3. Define 5-10 vetted skills as a typed API and let an LLM compose them in simulation only.
4. Write the approval workflow: simulation report, risk assessment update, reduced-speed trial, sign-off.
5. Collect corrections and failures per shift to extend the skill library before trying learned policies.

Sources

1. IFR — World Robotics 2025 (installations) — https://ifr.org/worldrobotics/
2. Regulation (EU) 2023/1230 (Machinery Regulation) — https://eur-lex.europa.eu/eli/reg/2023/1230/oj
3. ISO 10218-1:2025 Robots and robotic devices — Safety requirements for industrial robots — https://www.iso.org/standard/73933.html
4. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. Liang et al., Code as Policies — https://arxiv.org/abs/2209.07753
6. Hugging Face LeRobot — https://github.com/huggingface/lerobot
7. Physical Intelligence, pi0 / openpi — https://github.com/Physical-Intelligence/openpi
8. Wandelbots — https://www.wandelbots.com/
9. Universal Robots — PolyScope X — https://www.universal-robots.com/products/polyscope-x/
10. MoveIt 2 — https://moveit.ai/