Live-camera visual assistance & remote guidance (smartphone, smart glasses)

A broadband customer cannot tell which cable goes where; a technician faces a machine he has never serviced. Instead of a phone call that tries to describe the problem in words, the person points a camera at it. A streaming vision-language assistant recognises the device, cables, LEDs and parts, talks the user through the procedure and tracks which step is done. If it gets stuck, a human expert joins the same live view and draws annotations.

Problem & value

• Today: call-centre agents try to diagnose hardware by voice ("what colour is the light?"); field staff phone colleagues or wait for a specialist visit. Result: long calls, repeat contacts, unnecessary truck rolls, wrong parts.
• AI value: the camera replaces verbal description; the assistant handles routine cases (router set-up, error LEDs, filter changes, standard maintenance rounds) at any hour and frees experts for the hard ones.
• Evidence: TechSee reports its remote visual assistance in use at Vodafone across ~10 markets since 2016, where customers open a link and the phone camera starts without an app [1]. TechSee's own March 2026 platform launch cites early deployments with 5-15% higher digital containment and 5-20% fewer truck rolls; these are vendor-reported, not independent figures [2]. TeamViewer positions Frontline Assist as "see-what-I-see" expert support for hands-free field work [3].

Variants

• Customer self-service (B2C): telecom, utilities, appliances, consumer electronics; web link or in-app camera.
• Field-worker guidance (B2B): maintenance, installation, inspection rounds with smart glasses or rugged tablets.
• Assisted live support: the human agent sees the stream and an AI copilot suggests the next step (see CONV-007).
• Retail/in-store and unboxing: product set-up, returns triage before a device is shipped back.
• Private use: general multimodal assistants in a phone app ("what is wrong with this tap?"); consumer-grade, no procedure tracking.

How it works

1. camera/glasses stream (WebRTC) + user voice
2. consent + AI-disclosure screen, privacy masking (faces, screens, documents)
3. frame sampling (~1 fps default; burst on request) + speech recognition / native audio
4. vision-language model: identify device/part/state (model, LED pattern, port, fault)
5. retrieval of the right procedure from manuals / KB (KNOW-004 style RAG, device-specific)
6. dialogue manager: procedure state machine (current step, expected visual result, retry)
7. spoken instruction + on-screen/AR overlay ("plug into the yellow port")
8. verification: check frame for step completion
9. fallback: warm handover to expert with shared annotated view, snapshots, transcript
10. ticket / work order update

• Perception: the VLM interprets sampled frames, not continuous video; small cheap detectors (LED state, port, label/OCR, barcode) run alongside for reliability. Gemini's Live API, for example, takes video as ~1 fps JPEG frames, recommended at 768x768 [4].
• State tracking: the key difference from a chatbot. Procedures are defined as steps with visual success criteria; the LLM decides, a deterministic controller enforces order and safety stops.
• Human-in-the-loop: the assistant must hand over on low confidence, user frustration, any safety-relevant step (mains voltage, gas, pressurised systems) and whenever the user asks for a person.

Data preparation

The assistant is only as good as the procedures behind it. Source material typically lives in PDF manuals, wiring guides and service bulletins on file shares or a DMS, in the CRM or field-service system (device models, past tickets, work orders) and in the heads of senior technicians. Typical gaps: scanned or outdated manuals, no step lists with a visual success criterion, inconsistent model and part numbers, missing photos of LED patterns and ports, and no outcome labels on past sessions.

Preparation steps: OCR and clean the manuals (PREP-006, PREP-007), build a device and part master list (PREP-010), rewrite the 5-10 most frequent procedures as short steps with an expected visual result (PREP-014, PREP-016), collect labelled photos and videos per device class and session outcomes from tickets (PREP-020), and build an evaluation set with experts on real footage (PREP-018). Pull device and customer context from the CRM or field-service system (PREP-005). Define masking rules for faces, screens and documents before any recording (PREP-024). Tools: Paperless-ngx or an OCR pipeline, a labelling tool, pgvector/Qdrant.

Rough share of effort: 40-50 percent, mostly procedure authoring and device coverage, more than the model integration itself. GUIDE-003 helps to size it.

Implementation paths

Which path for whom

• smb: A, because mature EU-capable products (e.g. TeamViewer Frontline, TechSee) cover WebRTC, expert handover and FSM integration at per-seat or per-session prices.
• enterprise: B, because custom procedures, brand voice and tight CRM/field-service integration justify building on realtime APIs once volumes are high; A stays valid for commodity device types.
• developer: B, because LiveKit or Pipecat with a realtime multimodal API is the fastest way to build and test; C only for strict data residency.

Path A - Buy (recommended for most SMBs)

• Visual customer service: TechSee (Sophie AI, visual AI agent; customers include Vodafone, Samsung, Electrolux) [1][2]; similar remote-visual-support tools from vendors such as Streem, Zoom-based add-ins and ServiceNow/Salesforce field-service integrations. Check EU data residency and recording controls.
• Industrial/field: TeamViewer Frontline (EU, Germany) with smart-glasses support [3]; other EU-relevant options include Ubimax/Frontline lineage, Vuzix/RealWear hardware ecosystems. Verify each vendor's current AI features and glasses list.
• Choose A when: you need production-grade WebRTC, expert escalation, CRM/FSM integration and have standard device types.

Path B - Build on cloud APIs

• Realtime multimodal models: Gemini Live API (audio + 1 fps video over WebSocket) [4]; OpenAI Realtime API and Anthropic/Claude or Mistral vision models for turn-based image understanding. Prefer EU regions and data-processing terms that disable training use.
• Transport/orchestration: LiveKit Agents (open source, has a Gemini Live plugin with video) or Daily/Pipecat, plus your FSM/CRM via tool calls.
• Smart glasses: Meta Wearables Device Access Toolkit (developer preview) gives apps camera streaming and photo capture on Ray-Ban Meta and Oakley Meta, but publishing is limited to selected partners and the camera requires explicit user permission [5]. Industrial glasses (RealWear, Vuzix) run Android apps and are usually the safer enterprise choice.
• Choose B when: you want custom procedures, brand voice and tight integration, and can accept non-EU-provider risk assessments.

Path C - Self-host (open source)

• Models: open-weight VLMs such as Qwen-VL family and Gemma-class multimodal models served with vLLM; Whisper-class ASR and open TTS; small YOLO-class detectors for LEDs/ports.
• Stack: LiveKit (self-hosted) or Pipecat for WebRTC and agent loop; Qdrant/pgvector for manuals.
• Hardware: HW3 for multi-user concurrency with larger VLMs; a single HW2 GPU serves pilots. Latency target is conversational (about 1-2 s to first spoken response), so keep GPUs close to users.

Path D - Custom training

Worth it for recurring device families: fine-tune a detector/classifier on labelled photos of your products (LED patterns, port layouts, wear states) and optionally a small VLM on annotated sessions. Needs hundreds to thousands of labelled frames per device class; past support sessions with outcomes are the natural data source.

State of the art (2026)

• Native realtime multimodal models accept audio and camera frames in one session with sub-second to ~1 s latency, replacing the older chain ASR -> LLM -> TTS plus separate image calls [4].
• Vision-grounded agents combine perception with tool use (ticketing, parts lookup, scheduling) and procedure state machines.
• Smart-glasses ecosystems are opening to third-party apps, though consumer glasses remain restricted for production use [5].
• Limits and failure modes: hallucinated component identification (similar-looking models), poor lighting/glare, LED colour confusion, 1 fps sampling missing fast blinking patterns, spatial-direction errors ("left" from whose view?), long-session drift, and over-confident instructions on safety-critical steps. Always show the identified model and let the user confirm it.

Shared resources & combinations

• Same stack as CONV-007 (agent assist), CONV-001 (service bot, shared KB and handover), KNOW-004 (manual RAG), SPEECH-011 (hands-free control) and PERS-007 (scene description; different goal and safety profile). Overlaps with VIS-002 (fixed-camera checks) in detector models but not in deployment.
• Downstream: OPT-019 field-service dispatch (avoid visits, send the right part), CONV-026 for in-app embedding.

Effort, cost & KPIs

• Path A: weeks to a few months; typically per-session or per-seat pricing. Path B: 3-6 months incl. procedure authoring and integrations. Path C: 4-8 months plus GPU and MLOps. Biggest cost is not the model but procedure authoring, device coverage and testing.
• KPIs: resolution/containment rate without expert, first-contact resolution, truck-roll and return avoidance, average handling time, escalation rate, step-completion accuracy, CSAT, safety incidents (target zero).

People & change

Start with management and the department lead (service or maintenance) to pick low-risk procedures and define the stop rules for safety-relevant steps. Field technicians and service agents as end users should co-write the procedures; they know where customers fail. IT handles WebRTC, glasses or phone rollout and CRM integration; the data protection officer assesses live video that captures bystanders, addresses and documents, prepares notices, AVV with every vendor and a DPIA, especially for glasses in the workplace (GUIDE-005).

Works council: yes in practice. Cameras, session logs, transcripts and analytics on technicians are objectively suited to monitor behaviour or performance, so §87(1) no. 6 BetrVG applies (BAG: suitability suffices); §90(1) no. 3 and §80(3) (expert) support early involvement. Agree on a works agreement covering recording, retention and no performance evaluation (GUIDE-004). Public sector: Personalrat.

AI literacy: train staff on what the assistant gets wrong (similar-looking models, LED colours) and when to take over; the Act requires measures that support literacy (GUIDE-006). Pilot with a small team in one region, name AI champions, and show that experts are freed for hard cases, not replaced (GUIDE-008, GUIDE-009). The human can always stop the assistant and call an expert, and safety steps always go to a person.

Risks, regulation & compliance

• EU AI Act: usually limited risk: Art. 50 requires telling users they interact with an AI system [6]. Emotion inference from faces or voice must not be used on employees (Art. 5 prohibition at the workplace). Guidance for machinery or safety components, or use in worker management, can raise the classification; assess per deployment. AI literacy (Art. 4) applies.
• GDPR: live video routinely captures bystanders, family members, faces, addresses, documents and screens. Needs a lawful basis (consent or contract), clear notice before the camera starts, data minimisation, no recording by default, short retention, a DPA with every vendor and a DPIA for glasses in the workplace; employee monitoring needs works-council involvement (Germany: BetrVG).
• Safety/liability: no autonomous guidance on electrical, gas or pressure work; define stop rules and expert handover.
• Mitigations: blur or mask non-relevant regions, on-device pre-processing where possible, explicit confirmation of identified device, audit logs.

Getting started

1. Pick 5-10 high-volume, low-risk procedures (router set-up, filter change) and write them as step lists with visual success criteria.
2. Prototype with a WebRTC browser link and a realtime multimodal model on real user footage; measure identification accuracy.
3. Add consent/AI-disclosure screen, masking and retention rules; complete DPIA.
4. Wire a warm handover to a human expert with shared view and ticket context.
5. Pilot with one product line; track containment, truck rolls and safety escalations before widening.

Sources

1. TechSee / Fierce Network - Vodafone and TechSee remote visual assistance - https://www.fierce-network.com/telecom/vodafone-keeps-customers-connected-during-covid-19-by-using-techsee
2. TechSee - Sophie AI multisensory agent - https://techsee.com/ai-virtual-agent/
3. TeamViewer - Frontline remote assistance - https://www.teamviewer.com/en/products/frontline/
4. Google Cloud - Live API: send audio and video streams - https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/live-api/send-audio-video-streams
5. Meta - Wearables Device Access Toolkit - https://developers.meta.com/wearables/device-access-toolkit/
6. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj