Manual assembly and packing stations are still timed with stopwatches, REFA/MTM studies and sporadic audits, which capture a few cycles on a few days. Temporal action segmentation on station cameras splits every cycle into its work steps automatically, so engineers see cycle-time distributions, waiting time and bottlenecks for every shift and variant. The result feeds line balancing and Kaizen. The same data can easily become worker surveillance, so works-council agreements, anonymisation and a station-level (not person-level) design are preconditions.
Problem & value
- Today: industrial engineers measure cycle times by stopwatch or manual video review (REFA time study, MTM planned-time systems). Samples are small, the Hawthorne effect distorts them, and variant-specific effort, micro-stops and waiting are hardly visible. Line balancing therefore rests on planned rather than observed times.
- AI value: continuous, objective step-level timing across all cycles; fast detection of the true bottleneck station; variance analysis between shifts and product variants; evidence for work-instruction redesign; detection of skipped or reordered steps that cause quality escapes.
- Evidence: vendors such as Drishti (acquired by Atlas Copco) publish deployments in automotive supply (e.g. HELLA) and electronics assembly. Reported gains are vendor-stated, so run your own baseline [1]. Academic benchmarks on assembly video confirm the technique is viable but not solved [2][3].
Variants
- Discrete assembly (automotive, machinery, electronics): step-level segmentation with torque tools and fixtures as additional signals.
- Packing and pick stations in logistics: coarse actions (pick, scan, pack, seal); see VIS-019 and VIS-048 for turnaround milestones.
- Line-level digital twin input: observed times feed simulation for balancing and layout.
- Training mode: step-completeness feedback to new operators at the station, with data deleted after the session.
- Lab/work-study mode: short recording campaigns (days) replacing manual stopwatch studies, with consented participants; the lowest-risk variant.
How it works
- station camera (overhead, fixed)
- privacy filter at the edge (blur faces / crop to workspace / hands-only)
- detection + hand/body keypoints + object tracking (parts, tools, fixtures)
- temporal action segmentation (frame-wise labels -> step segments)
- cycle detection (start/end by fixture, part or step grammar)
- alignment with MES order / variant data
- analytics: cycle time per step and station, waiting, takt gap, variance
- balancing dashboard + exception clips -> engineers / team leaders
- Segmentation: a video backbone (I3D, VideoMAE/V-JEPA-class or pose + object features) feeds a temporal model (MS-TCN/ASFormer-class) that labels every frame with a step class; post-processing enforces the step grammar of the work instruction.
- VLMs: zero-shot VLMs help to bootstrap labels and describe anomalies but are still too coarse and slow for exact timing; typically used for annotation and clip explanation.
- Human-in-the-loop: engineers define the step taxonomy, review low-confidence segments and approve every finding before it is used for staffing or work-instruction changes. Results are aggregated per station and shift; individual-level views are blocked by design.
Data preparation
Video of the stations is the core data, plus context from the MES or ERP (orders, variants, takt times) and the work instructions, which often exist as PDF, Word or paper and describe steps inconsistently. Typical gaps: no agreed step taxonomy, camera views blocked by fixtures or people, variants that are not tagged in the MES, work instructions that differ from actual practice, and no consent or agreement on what is recorded.
Preparation steps: - Agree the step vocabulary and step grammar with engineers and team leaders; digitise the work instructions (PREP-006, PREP-008). - Install fixed cameras with an edge privacy filter (blur faces, crop to the workspace) before any footage is stored. - Export orders and variants from the MES or ERP (PREP-005). - Label several hundred cycles per station and variant with CVAT or Label Studio, ideally bootstrapped by a VLM and corrected by engineers (PREP-018, PREP-021). - Pseudonymise or aggregate everything that could identify a person (PREP-022). - Define retention and purpose in writing (PREP-025).
Effort: high, about 50-60 percent of the project, mostly labelling, camera placement and agreement on privacy rules. A short data readiness check (GUIDE-003) helps to scope the pilot to one or two stations.
Implementation paths
Which path for whom
- enterprise: C - the footage shows employees, so privacy filtering and processing should stay on-premises at the edge; step vocabularies are product-specific and the capability serves many lines (Baukasten effect).
- developer: C - open stacks (pose, tracking, temporal segmentation, CVAT) run on HW1 edge devices and give full control of what leaves the station.
- smb: A - with a handful of stations and no in-house vision team, a vendor with a mature privacy concept and onboarding is the pragmatic route.
Path A — Buy (recommended)
- Dedicated manual-work analytics: Drishti (US, Atlas Copco group), Retrocausal (US, Pi-Assembly and process-flow analytics), Invisible AI (US); for EU options run a market scan of MES vendors and Siemens Industrial Edge apps. Always verify EU hosting, on-premises edge processing and a configurable anonymisation layer.
- Choose A when: more than a handful of manual stations, no in-house vision team, and the works council prefers a vendor with a mature privacy concept.
Path B — Build on cloud APIs
- Cameras with on-prem edge box; clips or keypoints (not raw identity-bearing video) sent to cloud endpoints. Video-capable VLMs (Gemini, Qwen-VL via EU hosting, Pixtral/Mistral for stills) can label step boundaries semi-automatically for training data. Azure AI Video/Vision or Google Video Intelligence are weak at fine-grained industrial steps; plan custom models.
- Dashboards in Power BI/Grafana; integration with MES via OPC UA or REST.
Path C — Self-host (open source)
- Stack: NVIDIA DeepStream or Frigate-like RTSP ingestion, Ultralytics YOLO (check AGPL licence) or RT-DETR for objects, RTMPose/MMPose or MediaPipe for hands/keypoints, MMAction2 and open temporal-segmentation code (MS-TCN++, ASFormer, DiffAct) for step labelling, Label Studio/CVAT for annotation, Grafana/Superset for dashboards.
- Hardware: HW1 edge (Jetson Orin class) per 2-4 cameras for inference; one HW2 GPU for training and batch re-processing.
Path D — Custom training
Usually needed even with Path A/B/C: step vocabularies are product-specific. Plan for labelled cycles per step class and variant (hundreds of cycles per station is a realistic order of magnitude, to be confirmed in a pilot). Use few-shot/self-supervised video pre-training to reduce labelling.
State of the art (2026)
- Temporal action segmentation on assembly video has dedicated public benchmarks (IKEA ASM, Assembly101) with multi-view and hand-pose data [2][3]; accuracy is good for coarse steps but drops for fine-grained, short, occluded or similar-looking steps and for unseen variants.
- Hand-object understanding and egocentric/multi-view models improved robustness; foundation-model video backbones reduce labelled data needs.
- Commercial systems mostly combine detection, pose and segmentation with per-station tuning rather than a single universal model.
- Limits: occlusion by body or fixtures, camera placement, lighting changes, new product variants (re-labelling), step ambiguity, and mixed manual/machine steps. Cycle boundaries with batching or interruptions need rules. Do not treat model output as exact MTM-grade times without validation against a stopwatch study on a sample.
Shared resources & combinations
- Same stack: VIS-002 (assembly completeness), VIS-024 (pose/ergonomics), VIS-007 (PPE, safety), VIS-031 (VLM camera search) share cameras, edge GPUs, tracking and annotation tooling.
- Privacy layer: VIS-030 anonymisation before storage.
- Process analytics: PRED-024 (predictive process monitoring) and OEE/MES data; VIS-048 for logistics milestones.
- Governance: workforce-analytics topics are outside this entry; coordinate with HR and the works council.
Effort, cost & KPIs
- Path A: pilot on 3-5 stations in 2-3 months; cost drivers are cameras, edge boxes, per-station licences and engineering time for step definition.
- Path B/C: 4-9 months incl. labelling, privacy review and MES integration; labelling and per-variant re-tuning dominate cost.
- KPIs: segmentation accuracy vs stopwatch sample, share of cycles correctly segmented, takt-time gap per station, line-balance efficiency, waiting/idle share, rework from missed steps, engineer hours per time study saved.
People & change
This is a people-sensitive use case: start with management and the works council together, before any camera is bought. Further involve the production or industrial-engineering lead as process owner, IT and IT security (edge boxes, network), the DPO, and the team leaders and operators at the pilot stations.
Works council: yes. The system is objectively suited to monitor behaviour and performance, so co-determination under §87(1) no. 6 BetrVG applies (the Federal Labour Court holds that suitability is enough); §90(1) no. 3 requires early information about AI-planned work processes, and engaging an AI expert counts as necessary under §80(3). Agree a works agreement: purpose is line balancing, not individual evaluation; results only per station and shift; defined retention; no use for disciplinary action (GUIDE-004). Public sector: staff council.
DPO and DPIA: very likely required (systematic observation of employees, GDPR Art. 35); sign an AVV with every vendor (GUIDE-005). AI literacy: train operators and engineers on what the system sees and where it errs; AI Act Art. 4 asks employers to support staff literacy (GUIDE-006). Communicate openly with posted notices, workshops at the station and an AI champion from the shop floor (GUIDE-008). Pilot one station for 6-8 weeks. Typical fears are surveillance and speed-up; address them with the works agreement, anonymised views and by letting operators review the findings. Engineers approve every finding before staffing changes.
Risks, regulation & compliance
- EU AI Act: classification depends on use. Pure process timing aggregated at station level is likely minimal risk. Systems used to monitor and evaluate worker performance or behaviour or to allocate tasks are high-risk under Annex III point 4 (employment and workers management). Art. 5(1)(f) prohibits inferring emotions of workers: do not add emotion, fatigue-mood or "engagement" inference [4]. Art. 4 AI literacy applies to users.
- GDPR: video of identifiable employees is personal data. Needs a legal basis (Art. 6, in Germany § 26 BDSG as interpreted after the CJEU ruling on § 23 HDSG), purpose limitation, DPIA (Art. 35), minimisation, short retention and no secondary use for discipline [5].
- Labour law (DE): co-determination under § 87(1) Nr. 6 BetrVG applies to any technical system suited to monitor behaviour or performance; a works agreement (Betriebsvereinbarung) is needed before the pilot. Comparable rights exist in other member states [6].
- Mitigations: station cameras framed on hands/workspace, edge blurring (VIS-030), aggregated reporting with minimum group sizes, no personal ranking, defined retention, worker access to their own data, transparent communication, and an independent review of the purpose.
Getting started
- Agree purpose, scope and a draft works agreement with the works council and DPO before any recording; run the DPIA.
- Pick 2-3 representative manual stations and run a classic stopwatch study as ground truth.
- Define the step taxonomy with industrial engineers; install fixed cameras with privacy framing.
- Run a 4-6 week pilot with a vendor or a small open-source pipeline; compare against ground truth.
- Use station-level findings for balancing, then decide on roll-out with the works council.
Sources
- Drishti — manual assembly line analytics — https://drishti.ai
- Ben-Shabat et al., "The IKEA ASM Dataset" — https://arxiv.org/abs/2007.00164
- Sener et al., "Assembly101: A Large-Scale Multi-View Video Dataset" — https://arxiv.org/abs/2203.14712
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj
- Betriebsverfassungsgesetz § 87 — https://www.gesetze-im-internet.de/betrvg/__87.html
- MMAction2 — https://github.com/open-mmlab/mmaction2
- MS-TCN++ temporal action segmentation — https://github.com/yabufarha/ms-tcn