Pose & gesture analysis (ergonomics, fitness, rehab)

Human pose estimation turns ordinary video into skeleton keypoints, from which joint angles, repetitions and movement quality can be computed. Employers use it to find awkward postures in manual work, gyms and apps use it for form feedback, and physiotherapists use it to supervise home exercises. The technology is mature; the hard parts are measurement validity from a single camera, worker trust and (in the EU) data-protection and AI Act boundaries.

Problem & value

• Today: ergonomic assessments (REBA, RULA, OWAS, EAWS; ISO 11226 / EN 1005-4 for static postures) are done by trained specialists observing a few minutes of work with checklists or wearable sensors. Coverage is thin, snapshots are subjective and re-assessment after a workplace change is rare. Fitness form feedback needs a coach; rehab exercises at home are often done wrongly or not at all, and clinicians see little of what happens between visits.
• AI value: markerless pose estimation lets a phone or existing camera produce repeatable joint-angle measurements at scale, flag high-risk tasks for expert review, count repetitions and give instant feedback.
• Why it matters: work-related musculoskeletal disorders are among the most common occupational health problems in the EU, which is why EU-OSHA treats prevention as a priority [4]. No sourced, vendor-neutral ROI figures are quoted here; measure your own baseline.

Variants

• Workplace ergonomics (assembly, logistics, healthcare, offices): task-level risk scoring from video, before/after comparison of workstation changes; assessor-assisted rather than continuous individual monitoring.
• Fitness & sports coaching: rep counting, squat/deadlift form, running gait, golf/tennis swing; consumer apps, connected gym equipment, trainer tools (team sport tracking: VIS-015).
• Rehabilitation & physiotherapy: tele-rehab exercise supervision, range-of-motion tracking after surgery, stroke and orthopaedic home programmes; may be a medical device.
• Gesture interfaces: hand/body gestures for kiosks and sterile-environment control.
• Cycle-time analytics from the same keypoints: see VIS-034; safety/PPE: VIS-007.

How it works

1. camera / phone video
2. person detection + tracking (multi-person, ID persistence)
3. 2D keypoints (17-33+ joints, optional hands/face/feet)
4. 3D lifting or depth fusion (monocular lifting, multi-camera triangulation, depth sensor)
5. temporal smoothing + joint angles / kinematics
6. rules or models: ergonomic score, rep segmentation, deviation from reference movement
7. feedback (on-screen cues, report for assessor/physio) with human review

• Pose models: top-down (detector then per-person keypoints, e.g. RTMPose, ViTPose) or one-stage multi-person (e.g. RTMO, YOLO-pose). MediaPipe Pose Landmarker returns 33 body landmarks in image and 3D world coordinates and runs on phones and browsers [1].
• 3D and kinematics: 2D-to-3D lifting or multi-view triangulation, then biomechanical fitting (e.g. OpenSim-based pipelines such as OpenCap) for research-grade joint angles.
• Scoring layer: deterministic rules (REBA/RULA tables, angle thresholds) are preferred over an opaque neural score because they are auditable and match the standards assessors know.
• Human-in-the-loop: ergonomists review flagged tasks; clinicians set exercise targets and review adherence; no automated sanctions or HR decisions from raw scores.

Data preparation

Data needed: video or camera frames of people performing tasks or exercises, plus reference knowledge such as ergonomic assessments (e.g. RULA or REBA scores), exercise definitions and physiotherapist guidance. In an SMB this is usually not yet available as data: videos sit on phones or tablets, assessments in paper forms or Excel sheets, and exercise plans in PDFs.

Typical gaps: poor camera angles, occlusion by machines, inconsistent lighting, no labelled examples of good and bad posture, no consent records, and assessments that exist only on paper.

Preparation steps: 1. Define the setup: camera position, which movements, which thresholds, agreed with the department and the works council. 2. Digitise paper assessments and checklists (PREP-008, PREP-011). 3. Collect a small set of test clips with expert ratings (PREP-018), using volunteers who gave informed consent. 4. Process video locally and store only skeleton keypoints instead of images, which counts as pseudonymisation (PREP-022). 5. Document purpose and legal basis (PREP-025).

Effort: low in data terms (about 10 to 20 percent); the larger work is camera set-up, consent and agreeing rules for use.

Implementation paths

Which path for whom

• SMB: Path C, because body-pose data of employees is sensitive and open-source pose models such as MediaPipe run on-device on simple hardware (HW1).
• Enterprise: Path C, since keeping video on site avoids transfer of employee images and fits a works agreement.
• Developer: Path C, because open pose-estimation models run locally and give full control over privacy and latency.
• Private: Path A, because finished fitness and rehab apps process video on the phone and need no set-up.

Path A — Buy (off-the-shelf)

• Ergonomics: video-based assessment and EHS analytics (e.g. Intenseye, Protex AI, both non-EU) and digital-human simulation in Siemens Tecnomatix (Jack / Process Simulate). Verify EU hosting and that continuous individual monitoring can be switched off.
• Fitness / rehab: exercise apps and platforms such as Kaia Health (DE, EU), Physitrack (SE-listed), Sword Health and Kemtai; rehab claims may need medical-device certification.
• Choose A when: standard assessments or exercise programmes, no custom camera integration, and the vendor can provide a DPIA template, works-council documentation and EU hosting.

Path B — Build on cloud APIs

• Pose: Google MediaPipe Pose Landmarker (on-device, Apache-2.0 licence, mobile/web) [1]; managed video AI APIs for object tracking; VLMs (Gemini, Claude, GPT, Mistral Pixtral/Medium with vision) for narrative feedback on short clips, hosted in EU regions.
• Pattern: run pose extraction on the device so only keypoints leave the camera (data minimisation).
• Choose B when: you build a product or an internal tool and want fast iteration.

Path C — Self-host (open source)

• Models: MMPose with RTMPose / RTMO / RTMW (whole-body incl. hands) [2], ViTPose, Ultralytics YOLO-pose [3] (check AGPL/commercial licence terms), MediaPipe, OpenPose (non-commercial licence, mostly historical), MotionBERT or similar for 3D lifting, OpenCap for biomechanics research.
• Runtime: ONNX Runtime, TensorRT, OpenVINO, TFLite; NVIDIA Jetson or Hailo for edge cameras; tracking with ByteTrack-class trackers.
• Hardware: HW1 suffices for a few streams (Jetson Orin class, modern phone NPU); HW2 for dozens of streams or high-resolution multi-camera 3D capture.

Path D — Custom training

Worth it when the setting differs from public datasets (heavy PPE, occluded workstations, wheelchairs, children, unusual tools or exercise equipment). Fine-tune a keypoint model with a few thousand annotated frames from the target scene, or train a classifier on keypoint sequences (graph-convolution or transformer action recognition) for specific movements.

State of the art (2026)

• 2D multi-person pose is a solved-enough, real-time commodity: lightweight RTMPose/RTMO and YOLO-pose variants run in real time on CPUs, embedded GPUs and phones; transformer models (ViTPose family) lead accuracy benchmarks such as COCO and Human3.6M-style 3D sets.
• Limits and failure modes: single-camera depth ambiguity (flexion/rotation in the viewing axis is estimated poorly), occlusion by machines, loose clothing and PPE, unusual postures (lying, crouching, bending over), camera angle, children and bodies underrepresented in training data, and jitter. Reported agreement with expert or marker-based references varies strongly with setup; validate on your own task before trusting scores.

Shared resources & combinations

• Same pose/tracking stack serves VIS-007 (PPE and unsafe-posture alerts), VIS-034 (manual cycle times), VIS-015 (sports tracking), VIS-028 (in-cabin driver monitoring) and HLTH-015 (fall detection). One edge GPU box or Jetson fleet can run several of them.
• Anonymisation at the edge: VIS-030 (blur faces; transmit skeletons only).
• Wearable fusion with PERS-002 (IMU data) and exoskeleton planning with ROBO-018.

Effort, cost & KPIs

• Path A: days to weeks for pilots; subscription per user, site or camera.
• Path B: 1-3 months for a mobile or internal app with validation.
• Path C: 2-4 months incl. camera setup, calibration and validation study.
• KPIs: agreement with expert assessment (e.g. angle error in degrees, rating kappa), share of assessed tasks versus before, number of workstation improvements implemented, repetition-count accuracy, exercise adherence and completion in rehab, false alert rate, user acceptance.

People & change

Who and when: management, HR and the department lead agree on purpose first; the works council joins before any camera is installed; the DPO assesses legal basis and DPIA; IT and occupational safety set up cameras and storage. End users are involved from the first test, because acceptance decides success.

Works council: yes. Cameras and pose analysis are suited to monitor behaviour and performance, so §87(1) no. 6 BetrVG applies; the Federal Labour Court holds objective suitability is enough. Also §90(1) no. 3 (information on AI-supported work processes) and §80(3) (AI expert). A works agreement should restrict use to ergonomics and exclude performance evaluation (GUIDE-004). Public sector: Personalrat.

Data protection: a DPIA (Art. 35 GDPR) is likely, since systematic observation of employees may touch health data; employee consent is rarely truly voluntary, so rely on a works agreement and data minimisation (GUIDE-005).

AI literacy: support staff with short training on what the system measures and its error rates (Art. 4 AI Act, GUIDE-006).

Pilot and concerns: pilot with volunteers in one area. Fears of surveillance are legitimate: process locally, store no video, show employees their own results first, and never use data for appraisals (GUIDE-008).

Risks, regulation & compliance

• EU AI Act, Art. 5 prohibition (flag): inferring emotions of people in the workplace or education settings is prohibited (except for medical or safety reasons). Posture, gesture or facial-expression models sold as "stress", "engagement", "mood" or "motivation" detection for employees are in the prohibited zone; the Commission's Art. 5 guidelines clarify scope [5][6]. Biometric categorisation inferring sensitive traits (e.g. health, religion) is also prohibited. Pose keypoints for ergonomics are not emotion recognition and are not prohibited per se.
• Classification otherwise: ergonomics tools used only for workplace design are usually minimal risk. If outputs feed decisions on performance, promotion, termination or task allocation of workers, they may fall under Annex III (employment, worker management) as high-risk. Rehab and physio software that diagnoses or guides therapy can be a medical device (MDR; software rule 11) and then high-risk through Annex I. Remote biometric identification must be kept out of the system.
• GDPR: video of identifiable workers is personal data; with any health-related inference it approaches special-category data (Art. 9). Employee consent is rarely valid because of power imbalance; rely on legitimate interest or collective agreements with DPIA, strict purpose limitation, and works-council co-determination (in Germany §87(1) no. 6 BetrVG). Prefer on-device keypoint extraction, no raw video storage, aggregation to team or task level, short retention and no individual ranking.
• Risks and mitigations: measurement errors (validate), bias across body types (test diverse cohorts), function creep into productivity surveillance (written use limits, audit), chilling effect on staff (transparency, participation), injury risk from erroneous exercise feedback (conservative limits, clinician sign-off, stop-and-seek-help messages).
• AI literacy (Art. 4) applies to deployers.

Getting started

1. Define the decision and what will not be done (no individual ranking, no emotion inference).
2. Involve works council and DPO; complete a DPIA.
3. Pilot MediaPipe or RTMPose on 20-50 clips from your real setting; compare angles with an expert.
4. Run keypoint extraction on the device and store only keypoints or aggregates.
5. Add rule-based scoring (REBA/RULA or exercise-specific angle limits), with expert review.

Sources

1. Google AI Edge — MediaPipe Pose Landmarker guide — https://ai.google.dev/edge/mediapipe/solutions/vision/pose_landmarker
2. OpenMMLab — MMPose (RTMPose, RTMO, RTMW) — https://github.com/open-mmlab/mmpose
3. Ultralytics — Pose estimation docs — https://docs.ultralytics.com/tasks/pose/
4. EU-OSHA — Musculoskeletal disorders — https://osha.europa.eu/en/themes/musculoskeletal-disorders
5. European Commission — Guidelines on prohibited AI practices (AI Act Art. 5) — https://digital-strategy.ec.europa.eu/en/library/commission-publishes-guidelines-prohibited-artificial-intelligence-ai-practices-defined-ai-act
6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. OpenCap — markerless biomechanics from smartphone video — https://www.opencap.ai/