Simulation & digital twins for robot training (sim-to-real)

Collecting robot experience on real hardware is slow, expensive and sometimes unsafe. Physics simulators and digital twins let teams run thousands of parallel virtual robots, randomise conditions, generate labelled synthetic data and test software before a machine is built or stopped. The trained policy or perception model is then transferred ("sim-to-real") and refined with a small amount of real data. Typical benefit: faster development cycles, fewer physical crashes and earlier validation of edge cases.

Problem & value

• Today: robot behaviours are hand-programmed or trained on real demonstrations. Each real hour of data costs operator time, wear and cell downtime; rare failure cases (dropped parts, odd lighting, people stepping in) are hard to collect and unsafe to provoke on purpose.
• AI value: simulation provides cheap, parallel and safe experience. GPU-accelerated simulators run thousands of environments at once, so reinforcement learning (RL) for locomotion and manipulation converges in minutes to hours instead of weeks of real operation [1][3]. Generative world models and 3D generation add visual diversity and new scenarios [2].
• Digital twins: a calibrated virtual copy of a line or warehouse enables virtual commissioning (testing PLC and robot code before installation), layout what-if analysis and continuous regression testing of software updates.
• Evidence: legged and dexterous in-hand policies trained purely in simulation have run zero-shot on hardware [3][4]; no independent average cost-saving figure exists, so measure against your own baseline.

Variants

• Policy training (RL / imitation): locomotion, manipulation, whole-body control for humanoids and arms (see ROBO-004, ROBO-003).
• Synthetic perception data: domain-randomised images, depth and point clouds for pick detection and segmentation (ROBO-001, GEN-020).
• Virtual commissioning / digital twin of a cell: test robot programs, PLC logic and safety layouts before installation.
• Fleet/facility twins: AMR traffic and layout simulation (ROBO-002); AV/drone scenario tests (ROBO-005/006).

How it works

1. CAD / URDF / USD + scans of the real cell
2. asset creation & scene build (3D generation, photogrammetry, Gaussian splats)
3. physics simulation (GPU-parallel, contacts, sensors, actuators)
4. domain randomisation (friction, mass, lighting, textures, latency, sensor noise)
5. training (RL, imitation learning, VLA fine-tuning) + synthetic data generation
6. evaluation in held-out simulated scenarios (regression suite)
7. transfer to real robot (zero-shot or with real-data fine-tuning, system identification)
8. monitoring on real robot
9. logs feed back to calibrate the twin

• Physics engines: rigid-body and contact simulators (PhysX, MuJoCo/MJX, Newton, Genesis) running on GPUs for massive parallelism.
• Sim-to-real techniques: domain randomisation, system identification of the real robot, actuator models learned from real data, privileged-teacher/student distillation, residual learning on top of classical controllers.
• Human-in-the-loop: engineers define tasks, rewards and safety limits; every policy passes a simulated and then a supervised real-world test (reduced speed, safety-rated monitoring) before release. Safety functions are never delegated to a learned policy alone (see ROBO-016).

Data preparation

Simulation needs accurate geometry and behaviour data: CAD/STEP models of cells, robots and products, kinematics and payload data from the robot vendor, layout drawings, cycle times and sensor or camera data from the real cell for calibration. In an SMB these typically sit in the CAD/PDM system, vendor PDFs, Excel cycle-time lists, PLC projects and on engineers' laptops.

Typical gaps: simplified or outdated CAD, missing physical parameters (friction, mass, tolerances), no recorded real-world reference runs to validate against, scattered knowledge in individual engineers, and unclear rights to supplier CAD files.

Preparation steps: (1) collect one verified model per cell and convert to simulation formats (USD, URDF); (2) record real reference data (cycle times, camera images, joint logs) for calibration (see PREP-005, PREP-011); (3) define success metrics and a test set of scenarios with the domain experts (PREP-018); (4) clarify IP and licence rights for CAD data (PREP-025); (5) generate synthetic variations (lighting, positions) with domain randomisation. Tools: CAD converters, Isaac Sim/Omniverse, ROS 2 bag recording.

Rough share of effort: 40-50 percent, since model fidelity and calibration decide whether sim-to-real works. Run GUIDE-003 first.

Implementation paths

Which path for whom

• SMB: Path B — a managed simulation or integrator service fits best, because the twin must connect to the company's own robot cell and CAD data and no off-the-shelf product covers that.
• Enterprise: Path C — a shared internal simulation platform on own GPU servers serves many robot and line projects, keeps CAD and process data in-house and avoids high per-seat and cloud GPU costs.
• Developer: Path C — open-source simulators and RL frameworks (e.g. Isaac Lab, MuJoCo, Gazebo) are the standard toolchain for sim-to-real work.

Path A — Buy (off-the-shelf)

• Robot-vendor and automation suites: Siemens Process Simulate / Tecnomatix and Siemens Xcelerator (DE, EU), Dassault Systemes DELMIA (FR, EU), Visual Components (FI, EU, now part of KUKA), ABB RobotStudio, FANUC ROBOGUIDE, Universal Robots simulator, NVIDIA Omniverse-based factory twin blueprints. Strongest for virtual commissioning and offline programming.
• Choose A when: you integrate standard robots into cells and need commissioning and layout validation, not novel learned skills.

Path B — Build on cloud APIs / managed simulation (recommended)

• Managed GPU simulation: NVIDIA Isaac Sim and Isaac Lab on cloud GPU instances (EU regions of AWS, Azure, Google Cloud; EU providers such as OVHcloud, Scaleway, STACKIT, IONOS for GPUs).
• Generative augmentation: NVIDIA Cosmos world foundation models, cloud video/image generation APIs for scene variation [2].
• Orchestration: Kubernetes/Ray training jobs, MLflow, ROS 2 bridge to the robot.
• Choose B when: you need learned policies or perception but want to avoid owning a GPU cluster from day one.

Path C — Self-host (open source)

• Simulators: MuJoCo and MJX/MuJoCo Playground (Google DeepMind, Apache 2.0) [3][5], NVIDIA Isaac Lab (BSD-3; Isaac Sim has its own licence) [1], Newton physics engine (Linux Foundation, open source, co-developed by Disney Research, Google DeepMind and NVIDIA), Genesis, Gazebo (Open Robotics), PyBullet, ManiSkill, Drake.
• Learning stack: PyTorch/JAX, RL libraries (rsl_rl, skrl, Stable-Baselines3, RLlib), LeRobot (Hugging Face, imitation learning and datasets) [6], open VLA models (GR00T N1.x open weights, OpenVLA, pi0 open weights).
• Hardware: HW2 is enough for MuJoCo/MJX experiments and small Isaac Lab tasks (RTX GPU with RT cores required for Isaac Sim rendering); HW3-HW4 for large-scale parallel RL, vision-based training and VLA fine-tuning. Ops effort: moderate, mostly GPU and driver maintenance.

Path D — Custom training

Standard in this use case: policies and perception models are trained on your robot, parts and cell. Needs an accurate robot model, actuator/sensor identification data from the real machine (minutes to hours of logs), a task and reward design and a few hundred real demonstrations if combined with imitation learning. Fine-tuning a robot foundation model on 10s to 100s of demonstrations is the common route for manipulation.

State of the art (2026)

• GPU-parallel simulation is the default: Isaac Lab, MJX/MuJoCo Playground and Newton train locomotion policies in minutes on one GPU and have shown zero-shot transfer for legged and humanoid robots [1][3][5].
• Zero-shot dexterity: heavy domain randomisation transferred in-hand manipulation [4]; contact-rich assembly remains harder.
• Generative simulation: world foundation models (e.g. NVIDIA Cosmos) and video generation augment or replace parts of the renderer; they add visual realism but are not physically exact [2].
• Robot foundation models (VLA, see ROBO-004) are increasingly pre-trained on mixes of real, teleoperated and simulated data, with simulation supplying scale for rare cases.
• Limits and failure modes: the sim-to-real gap (contact, friction, deformables, cables, liquids, soft parts, sensor noise, actuator latency); reward hacking that exploits simulator quirks; a twin that drifts from reality after maintenance or retooling; false confidence because the test scenarios share the training distribution; photorealistic but physically wrong generative scenes. A simulation pass is evidence, not certification.

Shared resources & combinations

• Same stack (simulation + reinforcement-learning + robotics-control): ROBO-003 programming by demonstration, ROBO-004 humanoids/VLA, ROBO-001 bin picking (synthetic training images), ROBO-002 fleet and layout simulation.
• Data pipeline: GEN-020, GEN-025 (Gaussian-splat site scans) and DATA-005 share rendering tools.
• Safety and validation: ROBO-016 AI safety functions, ROBO-005 scenario-based AV testing.
• Hardware: one GPU cluster (HW3-HW4) serves simulation, RL, VLA fine-tuning and image generation; SCI-005 simulation surrogates use similar infrastructure.

Effort, cost & KPIs

• Path A: weeks to a few months for commissioning workflows; licences per seat, typically the main cost.
• Path B: 3-6 months for a first transferred policy or synthetic-data pipeline; GPU-hour cost and engineering time dominate.
• Path C: 3-9 months incl. team ramp-up; one or more GPU servers and robotics-RL expertise.
• KPIs: success rate sim vs. real (sim-to-real gap), real-robot hours saved, real data needed per skill, time from CAD to first working program, defects found in virtual commissioning, scenario coverage, commissioning time on site.

People & change

Involve management for scope and budget, the department lead (production or robotics engineering) as owner, IT and IT security for GPU servers, licences and network separation, and data owners of CAD and PLC data. Operators and maintenance staff who later work next to the robot must be involved early in defining scenarios.

Works council: depends. A pure engineering simulation with CAD data usually does not touch employee data. It applies once operator movements or performance data are recorded, or digital twins model workers or shift performance (§87(1) no. 6 BetrVG). Robot introduction also changes work processes, so inform the works council in good time under §90(1) no. 3 and agree on safety and no-performance-monitoring rules.

A DPIA is usually not needed unless people are recorded; involve the DPO if camera data of staff enters the twin. Support AI literacy (Art. 4 AI Act) with training on simulation limits and the reality gap. Start with one cell and a small engineering pilot group, communicate that simulation shortens commissioning rather than replaces staff, and address concerns about job changes. People stay in control: every policy trained in simulation passes human-supervised tests with safety functions before real operation.

Guides: GUIDE-008, GUIDE-009, GUIDE-004.

Risks, regulation & compliance

• EU AI Act: depends. Simulation itself is minimal risk. A trained policy that acts as a safety component of machinery is high-risk under Art. 6(1) and Annex I (Machinery Regulation (EU) 2023/1230 applies from 20 January 2027), with high-risk obligations for such products applying from August 2028 (Digital Omnibus, Reg. (EU) 2026/1744) [7][8]. Pure R&D use before placing on the market is largely out of scope (Art. 2(8)). Workplace emotion recognition and other Art. 5 practices are prohibited and must not be added to robot or twin data pipelines.
• Machinery safety: the Machinery Regulation, ISO 10218 (industrial robots), ISO/TS 15066 (collaborative operation) and ISO 13849 still require safety functions that are verifiable; learned policies need deterministic safety layers and documented verification. Simulation can support, not replace, validation.
• GDPR: twins of workplaces can capture workers' movements, faces or voices (scans, camera logs); remove or anonymise them, involve the works council (BetrVG in Germany), run a DPIA where worker monitoring is possible. Synthetic data reduces but does not remove this duty if generated from real people.
• IP and licences: check CAD, scan, asset, simulator and model licences (e.g. non-commercial terms).
• Cyber-security: twins hold detailed plant data; NIS2 and the Cyber Resilience Act may apply; segment the environment.
• Mitigations: held-out and adversarial test scenarios, staged rollout, speed and force limits, independent safety controller, drift monitoring of twin vs. reality.

Getting started

1. Pick one narrow task with a clear success metric (e.g. pick a part class, a locomotion gait).
2. Build or import an accurate model of the robot and cell (URDF/USD); log real data to fit actuator and friction parameters.
3. Prototype in MuJoCo/MJX or Isaac Lab on one GPU; add domain randomisation step by step.
4. Test the policy on the real robot under reduced speed with a certified safety layer; record the sim-to-real gap.
5. Feed real logs back into the twin; scale to more tasks only after the gap is measured and stable.

Sources

1. NVIDIA Isaac Lab (open-source robot learning framework) - https://github.com/isaac-sim/IsaacLab
2. NVIDIA Cosmos world foundation models - https://github.com/nvidia-cosmos
3. Google DeepMind - MuJoCo Playground (paper, arXiv 2502.08844) - https://arxiv.org/abs/2502.08844
4. OpenAI et al. - Solving Rubik's Cube with a Robot Hand - https://arxiv.org/abs/1910.07113
5. MuJoCo / MJX documentation - https://mujoco.readthedocs.io/
6. Hugging Face LeRobot - https://github.com/huggingface/lerobot
7. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
8. Regulation (EU) 2023/1230 (Machinery Regulation) - https://eur-lex.europa.eu/eli/reg/2023/1230/oj