A washing machine shows error "E18", a dripping tap needs a new cartridge, a bicycle derailleur will not shift. Today people search forums and videos and often guess. A multimodal assistant sees the device, reads the type plate, retrieves the matching manual or repair guide and talks the user through each step, with explicit safety gates (mains voltage, gas, water, load-bearing parts) and a clear "stop and call a professional" rule. The EU Right to Repair Directive makes this more relevant, because repair is being pushed ahead of replacement.
Problem & value
- Today: consumers search manuals by hand (often missing for older devices), watch generic videos for the wrong model, or discard working-but-faulty goods. Wrong guidance can cause injury (electric shock, gas, scalding) or a larger failure.
- AI value: (1) device identification from a photo of the device or type plate, (2) matching the exact model's documentation, (3) interactive diagnosis with follow-up questions, (4) hands-free guidance while the user's hands are busy, (5) consistent safety escalation.
- Policy tailwind: Directive (EU) 2024/1799 on common rules promoting the repair of goods adds a manufacturer repair obligation beyond the legal guarantee, a European online repair platform and repair information duties; Member States must apply it from 31 July 2026 [1].
Variants
- Appliance error-code decoder: white goods, boilers' user-level checks only, heat pumps, printers.
- Electronics and gadgets: phones, laptops, consoles (teardown guides, parts).
- Home maintenance: taps, silicone seals, drywall, simple plumbing; renters get help documenting defects for the landlord (see PERS-015).
How it works
- camera / photo / voice
- device identification (type plate OCR, VLM, barcode)
- manual & guide retrieval (RAG over manuals, repair-guide sites, parts catalogues; web search)
- symptom dialogue + diagnosis (reasoning LLM, asks for a photo or measurement)
- step plan with safety classification of every step
- live guidance (step-by-step, re-checks progress via camera)
- escalation gate: stop / call professional / contact manufacturer repair service
- Identification: OCR of the rating plate gives brand, model and serial number, which is much more reliable than recognising the appearance; the VLM is the fallback.
- Grounding: answers should cite the manual page or guide step. Without a source for the exact model, the assistant must say so rather than improvise.
- Safety layer: a rule set classifies each step (low / needs power-off and verification / regulated trade). Steps on gas, mains-fixed wiring, fuse boxes, pressurised refrigerant circuits, swollen lithium batteries or structural elements are not instructed; the assistant refers to a registered professional. This layer is deterministic (policy and keyword/classifier rules), not left only to the LLM.
- Human-in-the-loop: the user confirms each safety precondition (device unplugged, water shut off) before the next step; the user makes the final stop/continue decision.
Data preparation
Quality depends on the manuals and service knowledge behind the camera. For private users these are public PDFs from manufacturers; for an SMB they sit in the DMS, on file shares, in mail attachments, in the ERP (spare parts, serial numbers) and partly on paper in the van or workshop. Typical gaps: scanned manuals without text layer, several versions of the same manual, model numbers missing in file names, error-code lists buried in images, and tacit knowledge of senior technicians that is written nowhere. Steps: (1) inventory manuals per device family and model (PREP-001); (2) OCR old scans (PREP-006, DOC-014); (3) remove duplicates and tag each document with manufacturer, model, year and language (PREP-010); (4) split manuals by chapter and error code for retrieval; (5) capture technician know-how as short fix cards (PREP-013) and collect real photos with correct diagnoses as a test set (PREP-018); (6) give the assistant read access only to released documents (PREP-023). Rough share of effort: 30-40 percent for an SMB, almost none for private users. See GUIDE-003 for a quick readiness check.
Implementation paths
Which path for whom
- SMB: Path B. Service and facility-management firms need answers from their own manuals, spare-part lists and ticket history, and no off-the-shelf product combines these with their own systems.
- Private: Path A. Consumer apps with camera input and manual lookup are mature and cheap, and a household has no data worth the effort of self-hosting.
- Developer: Path B. Vision-language APIs plus a retrieval layer over manufacturer manuals are the fastest route to a product; self-hosting only pays off at volume.
Path A — Buy (recommended for private users)
- General multimodal assistants with live camera: Gemini Live (camera and screen sharing), ChatGPT advanced voice with video, Claude mobile apps with image input. Fast, no setup, but not grounded in the exact manual.
- Repair-specific: iFixit's repair guide library and its AI assistant (FixBot); manufacturers' own apps (e.g. Bosch Home Connect, Miele, Samsung SmartThings with diagnosis), plus repair-community sites such as Repair Café networks (volunteer help).
- EU angle: the EU online repair platform required by Directive 2024/1799 will list repairers [1]. Choose providers with EU processing if photos could show the home interior.
- Choose A when: one-off repairs; accept that general assistants can be wrong on model-specific details.
Path B — Build on cloud APIs
- Models: multimodal frontier models via API (Claude, Gemini, GPT; Mistral's vision models as an EU provider) with image input and tool use; the Gemini Live API and OpenAI Realtime API support streaming audio and video for hands-free guidance.
- Retrieval: index manuals (PDF via document parsing) and open repair guides in a vector database with hybrid search; add a parts-catalogue lookup tool.
- Orchestration: an agent with tools
identify_device,search_manual,lookup_error_code,find_part,find_repairer; guardrails for the safety classification. - Choose B when: a manufacturer, retailer, insurer or repair-network operator wants a branded service assistant.
Path C — Self-host (open source)
- Models: open-weight VLMs such as Qwen2.5-VL/Qwen3-VL, Gemma 3 (multimodal), Pixtral or Llama vision variants served via Ollama or vLLM; Whisper for speech; Piper for TTS.
- Retrieval: Docling or PaddleOCR for manual parsing, an open embedding model, Qdrant or pgvector.
- Hardware: HW2 (one 16-32 GB GPU, or an Apple-silicon machine at home) for a 7-30B VLM; small VLMs run on phone-class devices for simple identification (HW1). Fits the home-server setup in PERS-005.
- Choose C when: privacy of home photos matters, or for community repair projects.
Path D — Custom training
Worth it for a manufacturer: fine-tuning a VLM on its own service documentation, fault photos and technician resolutions, or an image classifier for component/part recognition. Needs curated labelled images per model family plus resolved service cases. For consumers: not needed.
State of the art (2026)
- Live multimodal sessions (streaming camera + voice with low latency) are available in the major consumer assistants; this is the new interaction mode for "show me the problem".
- Strengths: reading error messages and type plates, explaining a visible part, translating manuals, generic fault trees, parts lookup.
- Limits and failure modes: hallucinated model-specific steps, confusing similar-looking models or revisions, poor judgement of electrical or structural hazards from a photo, inability to see inside sealed housings, over-confident "it is safe" statements, and stale manuals. Videos give little depth information, so measurements (voltage, dimensions) must come from the user with a tool.
Shared resources & combinations
- Same stack as: KNOW-004 (technician manual assistant: same RAG over manuals, but professional users), DOC-014 (document parsing for RAG), PERS-007 (scene description and voice-first interaction), PERS-006 (image indexing).
- Combines with: PERS-011 (find the replacement part or compare repair vs. buy), PERS-015 (warranty and consumer rights when repair is refused), PERS-005 (local home assistant), VIS-023 (damage documentation for insurance claims), PERS-009 (household maintenance schedules).
Effort, cost & KPIs
- Path A: minutes; subscription or free tier.
- Path B: 1-3 months for a branded assistant with manual indexing and safety rules; cost driven by image/video tokens and manual coverage.
- Path C: weeks for a hobby setup; the main effort is curating manuals.
- KPIs: correct device identification rate, first-time fix rate (user-reported), safety-escalation precision/recall on a test set, share of answers citing a source, avoided service call-outs or returns, user satisfaction.
People & change
For an SMB, involve management (budget, liability for wrong advice), the service lead as process owner, the technicians who will use it daily, IT for access and the data owners of the DMS and ERP. Bring in the technicians early: they supply the best fix knowledge and decide whether the tool is trusted. Works council: usually no, because the assistant answers questions rather than recording behaviour; but if usage logs per technician are kept or analysed, section 87(1) no. 6 BetrVG applies, and section 90(1) no. 3 requires timely information when work processes change (GUIDE-004). A DPIA is rarely needed, but photos can show people or customer homes, so involve the DPO, sign a processor agreement and blur faces (GUIDE-005). AI literacy: Art. 4 AI Act requires measures that support staff's AI literacy, so run a short training on photo quality, limits and safety (GUIDE-006). Start a pilot with three to five technicians on one device family for four to six weeks (GUIDE-009). Typical concerns are being replaced or checked: position it as a second pair of eyes, never auto-closing tickets. Humans stay in control: the assistant must advise stopping and calling an expert for gas, mains voltage and safety-relevant work.
Risks, regulation & compliance
- EU AI Act (Reg. 2024/1689): minimal risk for a consumer information assistant; no Annex III use. It would only become relevant if embedded as a safety component of a regulated product (then via product legislation). AI literacy (Art. 4) applies to providers and deployers. No Art. 5 practice is involved.
- GDPR: camera streams of a home can capture people, documents, addresses and children; minimise (blur or crop background, on-device pre-processing), no retention by default, EU processing where possible, clear notice for a branded service.
- Product liability and safety: the revised Product Liability Directive (EU) 2024/2853 covers software including AI; providers of branded assistants should document safety rules, disclaimers and escalation logic [3]. Regulated trades (gas, electrical installation) have licensing rules by Member State.
- Typical risks: electric shock, gas leaks, fire from battery handling, falls, and voided warranty. Mitigations: deterministic hazard gate, mandatory power-off confirmation, refusal on regulated trades, citation of sources, visible "not a substitute for a qualified professional" notice, test set with hazardous scenarios.
Getting started
- Pick 5 devices you own and test a general assistant with live camera; note where it is wrong or unsafe.
- Collect the manuals for them (manufacturer sites, iFixit) and build a small RAG test with citations.
- Write a safety policy: which repair classes the assistant may guide, which it refuses.
- Add a confirmation step before each hazardous action and a clear escalation message with how to find a repairer.
- Measure on 30 real cases: fix rate, wrong-model errors, safety violations; iterate.
Sources
- Directive (EU) 2024/1799 on common rules promoting the repair of goods — https://eur-lex.europa.eu/eli/dir/2024/1799/oj
- Regulation (EU) 2024/1781, Ecodesign for Sustainable Products — https://eur-lex.europa.eu/eli/reg/2024/1781/oj
- Directive (EU) 2024/2853 on liability for defective products — https://eur-lex.europa.eu/eli/dir/2024/2853/oj
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- iFixit repair guides — https://www.ifixit.com
- Gemini Live API documentation — https://ai.google.dev/gemini-api/docs/live
- Docling project — https://github.com/docling-project/docling
- Qwen-VL models — https://github.com/QwenLM/Qwen2.5-VL