Classic motion capture needs suits, markers, a calibrated studio and cleanup artists. AI motion capture estimates a 3D skeleton, hands and facial expression from plain video and retargets the result onto a rigged character, so a small team can animate with a phone or webcam. Output is production animation data (FBX, BVH, USD), not just a pretty clip, which separates this from video generation (GEN-002) and static 3D asset generation (GEN-008).
Problem & value
- Today: optical mocap (Vicon, OptiTrack) and inertial suits cost from tens of thousands of euros and need a studio; keyframing by hand is slow. Indie studios, agencies, educators and VTubers often cannot afford either, so characters are animated by hand or not at all.
- AI value: capture anywhere with a phone, iterate in hours, reuse one performance on many characters, and replace an actor in a live-action plate with a CG character without a mocap stage.
- Benefit: no independent cost-saving figures are published; measure animator-hours per finished second in a pilot.
Variants
- Game and indie animation: gameplay animations, cutscenes, retargeting to Unreal MetaHuman or Unity humanoid rigs.
- Film, TV and ad VFX: character replacement in a shot with camera track, clean plate and lighting match (see GEN-021).
- VTubing, social and live avatars: realtime face and body tracking driving a VRM or Live2D-like avatar (see CONV-018 for customer-facing avatars).
- VR/AR training and simulation: record expert movements, animate virtual instructors.
- Sports and biomechanics: this use case only outputs animation; analysis of posture or ergonomics is VIS-024.
- Private use: hobby animation, game modding, avatar content.
How it works
- video (phone / multi-cam / film plate)
- person detection + tracking (multi-person IDs)
- 3D human mesh/pose recovery (SMPL-family body, hands, face), world-grounded with camera estimation
- temporal smoothing + foot-contact / physics clean-up (anti-jitter, anti-foot-slide)
- retargeting to the target rig (bone mapping, proportions, IK)
- [optional] scene pipeline: camera track, clean plate, relighting, CG render, composite
- human clean-up in DCC tool (Blender, Maya, MotionBuilder, Unreal, Unity)
- export (FBX, BVH, USD, glTF) / render
- Pose recovery: transformer models regress SMPL/SMPL-X body parameters per frame and recover global trajectory despite a moving camera; hand and face models add finger and blendshape curves.
- Generative clean-up: learned motion priors fill occlusions, reduce jitter and sliding and can extend or blend clips; text-to-motion models create motion without any video.
- Human-in-the-loop: an animator reviews the clip, fixes contact errors and adds performance polish. Raw output is typically previz or mid-tier quality, not final hero animation.
Data preparation
The input data is mostly new material: video of performers, the target character rig (FBX, GLB, VRM) and, for scene work, camera metadata. Rigs and character models usually sit on designers' workstations or file shares; footage lives on phones, SD cards or a NAS. Typical gaps are inconsistent rig standards and bone names, missing performer consent records, uncontrolled raw footage on private devices, and old character models without clean topology or licence documentation. There are rarely labels or training sets to build, so effort is low.
Preparation steps: - Fix one rig standard (Mixamo-like humanoid, Unreal Mannequin or VRM) and convert legacy characters once. - Define a capture checklist: tripod, full body in frame, plain background, 30-60 fps, lens notes; this matters more than any model setting. - Create a consent and retention register per performer and a clip naming scheme in a shared folder or DAM. - Put source footage and finished clips in structured, access-controlled storage; see PREP-009 and PREP-010 for restructuring and tagging, and DATA-003 for asset management ideas.
Rough share of effort: about 10-15 percent of the project, mainly rig standardisation and capture discipline. For personal data in footage, see PREP-022 and PREP-024.
Implementation paths
Which path for whom
- SMB: Path A, because mature EU-usable mocap products (Rokoko, Move.ai, Flow Studio) deliver usable clips within days at modest subscription prices.
- Enterprise: Path B, because studios with many shots need an automated pipeline with own retargeting and EU-region GPUs, while the process stays non-sensitive.
- Developer: Path C, because open pose models (GVHMR, WHAM) plus Blender give full control and no per-minute fees, if the SMPL licence is cleared.
- Private: Path A, because free or cheap phone-based tools (Rokoko Vision, Plask) cover hobby, VTuber and game-jam needs.
Path A — Buy (recommended)
- Video-to-animation and CG replacement: Autodesk Flow Studio (formerly Wonder Studio; character replacement, camera track, clean plate, exports to Maya, Blender, Unreal).
- Markerless mocap: Move.ai (single- and multi-camera, Move One / Move Pro), Rokoko Vision (free single-camera tier) and Rokoko suits for hybrid capture (Rokoko is Danish, EU), DeepMotion Animate 3D, Plask, Radical, Kinetix (France, EU, emotes for games), Cascadeur (AI-assisted physics-based keyframing).
- Face/live: Apple ARKit-based tools such as Live Link Face with Unreal MetaHuman Animator; webcam VTubing tools (VSeeFace, VTube Studio).
- Choose A when: you need results in days, volume is modest and cloud upload of footage is acceptable.
Path B — Build on cloud APIs
- Vendor REST APIs (Move.ai, DeepMotion) integrate capture into game or content pipelines; combine with a general VLM to QA clips (flag foot sliding, limb flips) and with cloud video models for stylised rendering (GEN-002).
- Run pose models on EU-region GPU instances (AWS Frankfurt, Azure West Europe, EU GPU clouds) to keep footage in the EU.
- Choose B when: you want an automated, high-volume pipeline with custom retargeting.
Path C — Self-host (open source)
- Models: 4D-Humans/HMR2.0, WHAM, GVHMR (world-grounded, gravity-aware), TRAM, SMPLer-X and HaMeR (hands), SAM 2 for masks, MediaPipe Pose/Holistic (light, realtime), RTMPose/MMPose for 2D keypoints. Note: SMPL/SMPL-X body models carry a non-commercial research licence; commercial use needs a licence from the rights holder (Meshcapade, Germany, EU) or a model not based on SMPL.
- DCC and engines: Blender (with retargeting add-ons), Godot, Unreal, Unity; export via glTF/FBX.
- Hardware: HW2 handles offline clips and 30 fps realtime body tracking; HW3 for multi-person and 4K plates.
- Effort: research code needs integration work; plan engineer time for retargeting and clean-up tooling.
Path D — Custom training
Worth it for non-human characters (animals, creatures), unusual rigs, or a studio style. Needs motion libraries (hundreds of clips from licensed mocap datasets such as AMASS, plus own captures) and ML engineers. Fine-tuning retargeting or motion priors is more realistic than training pose recovery from scratch.
State of the art (2026)
- World-grounded monocular recovery (WHAM, GVHMR, TRAM) gives plausible global trajectories from moving-camera footage; benchmarks include 3DPW, EMDB, RICH and Human3.6M.
- Video-to-scene products now bundle camera solve, clean plate and character replacement.
- Text-/music-to-motion diffusion models complement capture by generating or editing motion.
- Limits and failure modes: occlusion, loose clothing, fast spins and crowds cause limb flips and jitter; foot sliding and ground contact errors; weak finger and facial subtlety versus optical mocap; depth ambiguity in single-camera setups; scale drift; characters with very different proportions retarget badly. Multi-camera capture and a polish pass remain the quality route for hero animation.
Shared resources & combinations
- Same stack: pose-estimation and segmentation shared with VIS-024; 3D tooling shared with GEN-008 and GEN-025 (camera/scene reconstruction); one HW2/HW3 GPU box can serve all.
- Combinations: GEN-008 (generate the character, then animate it), GEN-013 (game content), GEN-021 (previs/VFX), GEN-003 and CONV-018 (avatars), GEN-012 (edit footage).
Effort, cost & KPIs
- Path A: hours to days; subscription or per-minute pricing, check current vendor plans.
- Path B: 1-3 months for an integrated pipeline; cost drivers are GPU minutes and storage.
- Path C: 2-4 months with an engineer; hardware plus maintenance.
- KPIs: animator-hours per finished second, share of clips usable without rework, foot-slide/jitter rate (QA metric), turnaround time, cost per clip.
People & change
Who and when. Involve the department lead (animation or marketing) and end-users (animators, video editors) from the first test, because they judge clip quality and own the clean-up. Bring in legal-compliance and the data protection officer before capturing anyone other than yourself: performer contracts must cover digital-replica use, and footage of people is personal data (Art. 6 and possibly Art. 9 GDPR if face data identifies a person). IT sets up storage, accounts and GPU access; management decides budget and policy (GUIDE-007).
Works council. Depends. A tool that only processes actors' footage for animation is usually not suited to monitor employee behaviour. If employees are captured regularly or motion data could be evaluated, §87(1) no. 6 BetrVG applies (objective suitability suffices, per BAG), and §90(1) no. 3 requires timely information about AI-supported work processes. Use of motion data for monitoring or emotion inference is not acceptable; exclude it in writing. Public sector: staff council.
DPO and DPIA. Sign an AVV (Art. 28 GDPR) with every cloud vendor; a DPIA is likely only for regular capture of staff or large-scale face data.
Literacy, communication, pilot. Support AI literacy (Art. 4 AI Act) with a short hands-on session on limits (foot sliding, jitter); see GUIDE-006. Pilot with 2-3 animators on five clips (GUIDE-009). Typical concern: "AI replaces animators." Address it by positioning output as previz and first pass, with the animator keeping final quality control and credit (GUIDE-008).
Risks, regulation & compliance
- EU AI Act: the tool itself is minimal-risk, but synthetic or manipulated content of real people (deepfake-like replacement) triggers Art. 50 transparency labelling (limited). Art. 5 prohibits emotion recognition in workplaces and schools; do not use facial-capture data to infer emotions of staff or students. Using motion data for worker monitoring would be a different use case with its own classification (see VIS-024).
- GDPR: video of actors and employees is personal data; motion and face data can be biometric in nature, so special-category rules (Art. 9) may apply if used to identify a person. Obtain explicit consent, define retention, and prefer EU hosting and a DPA with the vendor.
- Rights: performer contracts must cover digital-replica use (e.g. SAG-AFTRA and national guild rules); check training-data and output licences of tools; SMPL licensing as above.
- Mitigations: consent forms for performers, watermark/label synthetic output, keep raw footage deletion schedules, human review before release.
Getting started
- Record 5 representative performances (slow, fast, occluded) with a phone on a tripod.
- Test Rokoko Vision or Move One and Flow Studio on the same clips; score jitter, foot slide, retargeting.
- Set up a rig standard (Mixamo-like humanoid, Unreal Mannequin or VRM) and a Blender clean-up workflow.
- Draft performer consent and replica-use terms before capturing others.
- If volume justifies it, trial GVHMR on a HW2 GPU and compare cost per clip.
Sources
- Autodesk Flow Studio — https://www.autodesk.com/products/flow-studio/overview
- Move.ai — https://www.move.ai/
- Rokoko Vision — https://www.rokoko.com/products/vision
- GVHMR (SIGGRAPH Asia 2024) — https://github.com/zju3dv/GVHMR
- WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion — https://github.com/yohanshin/WHAM
- 4D-Humans (HMR2.0) — https://github.com/shubham-goel/4D-Humans
- MediaPipe Pose Landmarker — https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker
- SMPL body model and licence — https://smpl.is.tue.mpg.de/
- Regulation (EU) 2024/1689 (AI Act), Art. 5 and 50 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj