SOP & work instruction generation from screen recordings and expert videos

Processes live in people's heads and hands: the senior clerk who knows the ERP workaround, the maintenance technician who is retiring next year. Writing SOPs by hand is slow, so they are missing or outdated. Capture tools record the work as it happens; a vision-language model (VLM) plus speech recognition turns the recording into numbered steps with annotated screenshots or video frames, which a subject-matter expert reviews, translates and publishes — for onboarding, ISO 9001 "documented information" and shop-floor delivery.

Problem & value

• Today: SOPs are written in Word after the fact, screenshots are pasted manually, and the result is outdated after the next software update or process change. On the shop floor, work instructions are paper, slide decks or tribal knowledge; experienced staff leave and take tacit know-how with them.
• AI value: the author does the work once while it is recorded; the draft appears in minutes instead of hours; translation and re-versioning are cheap, so documents can stay current. Experts review instead of writing.
• Vendor-reported figures (not independently verified): Guidde claims video documentation "11x faster" [3]; Dozuki publishes customer claims such as an 80 % cut in SOP authoring time at Mack Trucks and a 40 % average reduction in training time [4]; Azumuta states productivity and quality gains for customers without describing the methodology [5]. Treat all as marketing numbers and measure your own baseline (time to author one SOP, time-to-competence).
• Boundary to neighbours: KNOW-011 generates knowledge-base articles from existing text/tickets; KNOW-014 captures lessons learned as text; EDU-006 produces training courses; VIS-034 measures cycle times from video. This use case is specifically demonstration (screen/video) → structured procedure.

Variants

• Software/office SOPs: click-path capture in browser or desktop (finance close, HR onboarding, CRM or ERP tasks) → step list with screenshots; the dominant SMB variant.
• Shop-floor work instructions: video of assembly, setup/changeover, maintenance or inspection → steps with key frames, short clips, tool/part lists, torque or safety notes; delivered on tablets, terminals or in AR/smart glasses.
• Expert-offboarding capture: structured interview plus demonstration of a retiring expert's tasks (spoken commentary is transcribed and aligned with the video).
• Customer-facing help: the same recordings become video how-to guides for customers (see KNOW-011).
• Public sector: Fachverfahren (specialist application) procedures for new case workers; usually self-hosted and with works/staff council involvement.
• Private use: family IT help guides or hobby/repair instructions; low stakes, free tiers suffice.

How it works

1. capture (browser extension / desktop recorder / phone or headcam video + voice)
2. pre-processing (key-frame extraction, scene/step segmentation, ASR transcript with timestamps)
3. redaction (blur/mask PII, passwords, patient or customer data, faces; before anything leaves the device)
4. step understanding (VLM: what changed on screen / which action on which part; click events + UI text if available)
5. drafting (LLM: numbered steps, titles, warnings, tools, expected result, quality checkpoints, in a house template)
6. human review (SME corrects, adds "why", safety and compliance notes; approver signs off)
7. versioning, translation, voiceover/subtitles
8. publication (wiki/DMS, work-instruction or MES system, LMS, AR) + change detection

• Capture quality drives output quality. Click-capture tools record the exact UI element, text and screenshot per step (more reliable than guessing from pixels). Pure video needs a VLM to segment actions; spoken commentary ("now I tighten the M6 screws to 8 Nm") disambiguates what the video cannot show.
• Key components: ASR (Whisper-class), a VLM that reads long video or sampled frames, an LLM that writes to a template with schema-constrained output, OCR/layout parsing for on-screen text and existing documents, a translation step with a controlled glossary.
• Human-in-the-loop (mandatory): a domain expert verifies every step, numeric value and safety warning before release. A VLM can mis-order steps, miss off-camera actions or invent plausible values.
• Staying current: re-record or let a computer-use agent replay the documented steps against the current UI and flag mismatches (see AGENT-003).

Data preparation

The raw material is new rather than historic: screen recordings or click captures of software tasks, phone or headcam video of experts on the shop floor, and their spoken commentary. Existing SOPs typically sit in Word files on file shares, in the DMS or QM system, as slide decks or on paper; templates, glossaries and safety rules are often scattered or only in people's heads.

Typical gaps: poor audio in noisy plants, off-camera actions, no standard template, outdated or duplicate SOP versions, passwords, customer or patient data visible on screen, and faces of staff in video.

Preparation steps: (1) set a capture standard (resolution, narration while working, one task per recording); (2) agree a house template and a short glossary (PREP-015, PREP-016); (3) collect the existing SOP versions as gold examples and retire duplicates (PREP-013, DOC-014); (4) configure redaction or blurring before anything leaves the device (PREP-022); (5) set a review and release workflow with named approvers; (6) rescan important paper instructions with OCR (PREP-006).

Rough share of effort: low, about 10 to 20 percent. Most of it is capture discipline and the template, not data cleaning, because the data is created fresh during the pilot.

Implementation paths

Which path for whom

• SMB: A — mature products (Scribe, Tango, Guidde, Dozuki) exist at SMB prices and SOP creation is a standard process; run a pilot on 5 to 10 procedures and check EU hosting and the DPA.
• Enterprise: B — SOPs must follow your own templates and flow into MES, LMS or a controlled-document workflow, and many languages and plants justify a small pipeline on EU-hosted APIs.
• Developer: C — Qwen3-VL, faster-whisper and Docling let you build a product or integration without sending customer video to third parties.
• Public: C — raw video of staff and citizen-related screens should stay inside the authority's network, and procurement favours sovereign hosting.

Path A — Buy (off-the-shelf; recommended for SMBs)

• Screen/software SOP tools: Scribe (click-capture to step-by-step guides; free basic plan, paid plans with redaction, enterprise SSO and auto-redaction of PII/PHI) [1], Tango (record workflows → guides with screenshots, voice transcription, SOC 2 Type II) [2], Guidde (recorded workflows → AI narrated video guides, 50+ languages, 200+ voices) [3]. Iorad and similar tools exist in this category; verify current features yourself. Most are US vendors: check EU hosting, DPA and sub-processors.
• Shop-floor/connected-worker platforms: Dozuki (digital work instructions with AI drafting, SOC 2, ISO 27001) [4], Azumuta (digital work instructions, MES, skill matrix; European vendor, check hosting and contract terms) [5], plus Tulip, Augmentir and similar; for DACH also look at established MES/Industrie-4.0 vendors with work-instruction modules.
• Document/QM suites: QM systems and DMS (DocuWare, d.velop, ELO, SharePoint) increasingly add AI drafting; useful when the controlled-document workflow (approval, release, revision) is the priority.
• Choose A when: you have dozens to a few hundred procedures, no strict on-prem need, and want results in weeks. Run a pilot on 5–10 real procedures, including a safety-relevant one.

Path B — Build on cloud APIs

• Video-capable multimodal APIs: Google Gemini API accepts video directly (File API up to 20 GB paid tier, configurable frame sampling, per-second timestamps; Gemini 2.5 and later accept several videos per request) [6]; frame-sampling pipelines also work with Claude, GPT or Mistral vision models. Choose EU regions/EU-hosted endpoints (Vertex AI or Azure EU, Mistral on its EU platform) and sign a DPA.
• ASR: EU-hosted ASR APIs or Whisper-class models; the transcript with timestamps is aligned to key frames.
• Orchestration: n8n / Power Automate / small service: upload → redact → ASR + VLM → draft → review task in Confluence/SharePoint/MES → publish. Schema-constrained JSON output for steps, tools, warnings.
• Choose B when: you need your own template, ERP/MES integration, or a pipeline across many languages, but no data-residency constraint on raw video.

Path C — Self-host (open source)

• Models: Qwen3-VL (Apache-2.0; sizes from 2B to 235B-A22B, long-video understanding with timestamp alignment, GUI/computer-use cookbooks) [7]; faster-whisper (MIT) for ASR [8]; Docling (MIT, LF AI & Data) can already turn MP4/MOV/WebM into an ASR transcript plus representative keyframes [9].
• Serving: vLLM or Ollama for the VLM/LLM; Docling or ffmpeg for key frames; n8n for workflow; Markdown/Git or a wiki (BookStack, Wiki.js) as the SOP store with review via merge request.
• Hardware: HW2 (one 24–32 GB GPU) for 8B-class VLM + Whisper on a few hours of video per day; HW3 for 32B-class VLMs and better long-video quality. Raw video stays inside the plant/network — a good fit for IP-sensitive production processes and public authorities.
• Ops effort: moderate; the hard part is capture standards and review workflow, not the models.

Path D — Custom training

Rarely worth it. Consider fine-tuning a small VLM on your own annotated demonstrations only for very domain-specific vocabulary (specialised machinery, unusual UIs) or high volumes. Data: a few hundred recordings with reviewed step lists — a by-product of the review process when corrections are logged. Prompting with a house template and glossary plus retrieval of existing SOPs usually suffices.

State of the art (2026)

• Two proven capture modes: click/DOM-assisted screen capture (Scribe, Tango, Guidde) is mature and reliable; free-form video understanding of physical work is newer and less reliable.
• Video models: current multimodal models take hours of video with timestamps [6][7]; frame sampling at 1–2 fps is typical, so fast hand movements, small parts and off-camera actions are the weak spot.
• Agentic upkeep: computer-use agents can replay SOPs to detect UI drift [7]; this is early-adopter, not turnkey.
• Connected-worker platforms add AI drafting ("starting point for SMEs to refine") rather than autonomous authoring [4], reflecting that human review remains a core part of the workflow.
• Limits / failure modes: hallucinated torque values, tool names or order of steps; missing "why" and tacit judgement ("listen for the sound"); screenshots that leak personal or confidential data; generated SOPs that look polished but are not validated. Safety-relevant instructions (machine safety, LOTO, medical, GMP) need formal validation by the responsible person.

Shared resources & combinations

• Shared building blocks: asr + vlm + llm-general + ocr-docparse + translation — the same self-hosted VLM/Whisper server (HW2/HW3) can serve KNOW-016 (multimodal archive search), VIS-031 (video search and summaries), DOC-014 (document parsing for RAG) and meeting-notes use cases.
• Upstream/downstream: KNOW-011 (publish SOPs into the knowledge base and FAQ), KNOW-004 (technical documentation assistant uses the SOPs as retrieval corpus), KNOW-014 (lessons learned and expert offboarding), EDU-006 (microlearning and courses from the same recordings), GEN-003 (avatar/voice presenter for narrated versions).
• Operations: VIS-034 (cycle-time analytics on the same shop-floor video), AGENT-002 (workflow engine for review/approval and publishing), AGENT-003 (computer-use agent that replays SOPs and can later automate the documented task), ROBO-003 (robot programming by demonstration reuses demonstration capture).
• Data pipeline: shared redaction step (faces, personal data, screen secrets) and shared glossary/ terminology store for translation consistency.

Effort, cost & KPIs

• Path A: days to set up; per-seat SaaS pricing (Scribe lists free and per-seat paid plans [1]); effort is mostly capture discipline and review time.
• Path B: 1–2 months for a pipeline with template, review workflow and integration; API costs scale with video minutes (long video is token-heavy; trim and sample before sending).
• Path C: 2–3 months; one GPU server plus ops; shop-floor video infrastructure if not yet present.
• KPIs: time to produce a validated SOP (hours per SOP, before/after), share of processes with a current SOP, review edit rate (how much the SME changes), time-to-competence for new hires, error/rework rate on documented tasks, documents out of date (age since last verification), audit findings on documented information.

People & change

Start with management and the department lead, who pick the first procedures. Involve the experts being recorded and the future SOP users from the start: they review every step and will own the documents. IT covers accounts, hosting and integration. The data protection officer checks recordings of employees (voice, faces, screens with customer data), the legal basis, redaction and the AVV with the tool vendor; a DPIA is usually not needed unless performance is evaluated, but check it.

Works council: depends. Recording staff at work, with their voice and image, and tools that log who recorded what are suited to monitor behaviour (§87(1) no. 6 BetrVG), so co-determination is likely where a council exists; §90(1) no. 3 requires early information. Agree in writing that recordings are used only for creating SOPs, not for evaluating individuals. In public bodies the Personalrat is responsible.

AI literacy: support staff in learning to review AI drafts critically (Art. 4 AI Act; GUIDE-006), especially numeric values and safety warnings. Communicate openly that experts are not being replaced but relieved of writing documents. Pilot with volunteers on 5 to 10 procedures, including one safety-relevant one. Typical concerns are fear of being recorded and of making oneself redundant; address them with voluntary participation, clear deletion rules and credit for the expert. A human always signs off before release. Further reading: GUIDE-008, GUIDE-009, GUIDE-004.

Risks, regulation & compliance

• EU AI Act: typically minimal risk; documentation generation is not an Annex III use. Art. 4 AI literacy applies to staff using the tools [10]. Caveats: do not use the same video to infer emotions of workers (prohibited in the workplace under Art. 5) and do not repurpose recordings for individual performance scoring — that changes the classification (see VIS-034 for the cycle-time boundary). Synthetic voiceovers or avatars in training material can trigger transparency duties under Art. 50 in some settings (see GEN-003).
• GDPR / labour law: recordings contain employees' faces, voices and screens, and can expose customer, patient or credential data. Use purpose limitation, DPA, minimisation (record the task, not the person), automatic redaction, retention limits. In Germany a works council has co-determination where technical systems are suited to monitor behaviour or performance (§ 87(1) no. 6 BetrVG) [11]; agree a works agreement stating that recordings are not used for performance evaluation.
• Quality management: ISO 9001 requires controlled documented information (creation, review, approval, versioning); keep AI draft, reviewer, approver and date in the revision history.
• Sector rules: pharma/GMP, medical devices (MDR), food safety (HACCP) and machinery safety require formal validation of instructions; AI drafts do not replace that validation.
• IP/security: raw video of production processes is trade-secret material; prefer EU/on-prem processing, check vendor training-on-customer-data terms.

Getting started

1. Pick 5–10 recurring procedures (mix of software task and a physical task) and record baselines: authoring time, age of the current SOP, onboarding time.
2. Define a standard template (purpose, scope, prerequisites, steps, warnings, quality checks, revision history) and the redaction rules; agree with data protection officer and works council.
3. Pilot one Path A screen-capture tool and, for the shop floor, one work-instruction platform or a simple Path B/C pipeline (phone video + voice commentary → ASR + VLM → draft).
4. Have the SMEs review the drafts; log every correction to measure edit rate and failure patterns.
5. Roll out with an owner and review interval per SOP (e.g. re-verify every 6–12 months or at software release) and link the SOPs to your knowledge base (KNOW-011) and onboarding.

Sources

1. Scribe — plans and features (redaction, SSO, export) — https://scribe.com/pricing
2. Tango — product overview (guides from recorded workflows, SOC 2 Type II) — https://www.tango.ai
3. Guidde — AI video documentation platform (Magic Capture, translation, claims) — https://www.guidde.com
4. Dozuki — connected worker platform, CreatorPro AI and customer claims — https://www.dozuki.com
5. Azumuta — digital work instructions and MES — https://www.azumuta.com
6. Google AI for Developers — Gemini API video understanding — https://ai.google.dev/gemini-api/docs/video-understanding
7. Qwen3-VL repository (models, Apache-2.0, video and computer-use cookbooks) — https://github.com/QwenLM/Qwen3-VL
8. faster-whisper (SYSTRAN) — https://github.com/SYSTRAN/faster-whisper
9. Docling project (MIT; audio/video to transcript and keyframes) — https://github.com/docling-project/docling
10. EU AI Act, Article 4 AI literacy — https://artificialintelligenceact.eu/article/4/
11. § 87 BetrVG (Mitbestimmungsrechte des Betriebsrats) — https://www.gesetze-im-internet.de/betrvg/__87.html
12. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj