Smart homes have long meant rigid rules ("if motion, then light") that are tedious to set up and brittle in daily life. AI adds natural-language and voice control, automations generated from a sentence, routines learned from behaviour, price- and forecast-aware energy control, and anomaly alerts. The key design question in the EU is where the intelligence runs: locally on a hub, or in a vendor cloud that sees everything that happens in the home.
Problem & value
- Today: households and small landlords juggle several vendor apps, hand-written rules and fragile cloud integrations. Voice assistants handle simple commands but fail on compound or context-dependent requests. Heating and electricity use are rarely optimised against dynamic tariffs.
- AI value: (1) conversational control of many devices through one interface, (2) automations authored in plain language and explained back to the user, (3) optimisation of heat pump, PV, battery and EV charging against tariffs and forecasts, (4) early warning (water leak, device failure, unusual night-time activity), (5) accessibility for people with limited mobility or vision.
- Evidence status: vendors publish energy-saving claims for learning thermostats and optimisation features, but these depend heavily on the home and baseline; measure your own consumption before and after rather than relying on headline figures. No generally accepted independent benchmark exists for assistant quality in homes.
Variants
- Private household: hub-based setup with local voice and LLM, or a mainstream ecosystem assistant.
- Energy-focused: home energy management (HEMS) for PV, battery, heat pump and wallbox; see OPT-007, PRED-031.
- Assisted living / elderly: routine-deviation alerts and voice control; see CONV-016 (sensitive, consent and dignity issues).
- Landlords, holiday lets, small offices: remote access, leak and occupancy-based heating, guest check-in; tenant privacy rules apply.
- Security cameras and doorbells: person/package detection on the edge. Facial recognition of visitors is a biometric topic, treated separately and not recommended here.
How it works
- sensors & devices (Matter / Zigbee / Z-Wave / Thread / MQTT)
- hub / gateway (state model, local rules engine)
- perception: wake word
- speech-to-text
- intent
- LLM agent with tool calls restricted to an exposed-entity allowlist
- planner / energy optimiser (tariff, weather, PV forecast, comfort constraints)
- device commands + confirmation to user
- logs, anomaly detection, user feedback
- Voice and language: on-device wake word and speech-to-text, a small LLM (or deterministic intent matcher with LLM fallback) maps requests to device actions via tool calling.
- Automation authoring: LLM drafts a YAML/JSON automation from a sentence; the user reviews a readable preview before activation.
- Learning routines: statistical or reinforcement-learning models infer schedules and preheating times; keep them bounded by hard comfort and safety limits.
- Anomaly detection: time-series models on power, water and temperature data.
- Human-in-the-loop: confirmation for security-relevant actions (unlocking doors, disabling alarms, opening garage doors); physical override always available; deterministic safety logic (smoke, gas, frost protection) never delegated to an LLM.
Data preparation
Smart home AI needs a device inventory, sensor and actuator states, schedules, energy meter and tariff data, presence information and user preferences. At home or in a small business these live in manufacturer apps, the router, the energy meter portal, spreadsheets and installer documents.
Typical gaps: devices from different vendors and protocols (Zigbee, Matter, KNX, Wi-Fi) with inconsistent names, no room or zone assignment, missing history, cloud-only devices without local API, and weak access control and default passwords.
Preparation steps: (1) inventory all devices and assign room, function and clear names (see PREP-011, PREP-015); (2) prefer local protocols such as Matter, Zigbee or KNX and connect everything to one hub; (3) start recording history (energy, temperature, presence) for a few weeks before expecting useful predictions; (4) decide which data may leave the building and which is processed only locally (PREP-024); (5) set up accounts, network segments and updates. Tools: Home Assistant, MQTT broker, installer documentation.
Rough share of effort: 10-20 percent, mostly naming and integration. Wider data checks are covered by GUIDE-003.
Implementation paths
Which path for whom
- Private: Path C — Home Assistant running locally with a small local model keeps cameras, presence and voice data at home and avoids vendor lock-in.
- SMB: Path A — mature building-automation products (e.g. KNX, Loxone, Shelly ecosystems via an installer) cover standard needs for offices and small properties with support and warranty.
- Developer: Path C — open-source Home Assistant, MQTT and local LLMs offer full control for custom integrations.
Path A — Buy (off-the-shelf)
- Ecosystem assistants: Google Gemini for Home, Amazon Alexa+, Apple Home with Siri, Samsung SmartThings; broad device support but cloud processing and vendor data policies.
- Hub and energy products: Homey (EU, Netherlands/Denmark), Loxone (EU, Austria), Home Assistant Green/Yellow (Nabu Casa, open source), Shelly (EU, Bulgaria), tado° (EU, Germany) for heating, Bosch Smart Home (EU, Germany), Hager/Busch-Jaeger/Gira KNX systems (EU), plus inverter and heat-pump vendor energy managers (e.g. SMA, Viessmann, Stiebel Eltron).
- Choose A when: you want low effort, accept vendor cloud, and use mainstream devices. Check Matter support, local-control options, EU data hosting and how long the vendor promises updates.
Path B — Build on cloud APIs
- LLM conversation agent on top of an open hub (Home Assistant integrations for OpenAI, Anthropic, Google, Mistral and others): expose only selected entities, restrict tools, and avoid sending camera images or presence history.
- Mistral (EU) or EU-region cloud endpoints where a cloud LLM is wanted; strip names and addresses from prompts.
- Orchestration: Home Assistant, Node-RED, MQTT broker, n8n for notifications and summaries.
- Choose B when: you need stronger language understanding than a small local model gives, and can accept that command text (not raw sensor streams) leaves the home.
Path C — Self-host (open source)
- Platform: Home Assistant with its Assist voice pipeline; openHAB or ioBroker as alternatives; Zigbee2MQTT, Z-Wave JS, Matter server, Mosquitto.
- Speech: Whisper-family or Piper/Wyoming-protocol components for speech-to-text and text-to-speech (Open Home Foundation tooling).
- LLMs: small open-weight models (Llama, Qwen, Gemma, Mistral families, 3-8B) via Ollama or llama.cpp for conversation and tool calling; deterministic intent matching first, LLM as fallback to save latency.
- Vision: Frigate NVR with object detection on a Coral or Hailo accelerator for person/vehicle/package events, local only.
- Hardware: HW1 (mini PC or Raspberry Pi-class for the hub; an N100 mini PC or small GPU for a 3-8B model). Weak CPUs make local LLM replies slow, so many users split hub and inference box.
Path D — Custom training
Rarely needed for a household. Worth considering for manufacturers and large property operators: fine-tuning small models for intent-to-device-action mapping in their own device ontology, wake-word models in local languages, or occupancy and load-forecasting models from their fleet data (needs consent-based data collection).
State of the art (2026)
- Matter (Connectivity Standards Alliance) over Thread/Wi-Fi is the interoperability layer; it has pushed local control and multi-ecosystem support, though coverage of device types and feature parity still varies by vendor [1].
- LLM-based assistants replaced intent-only voice control in major ecosystems; open platforms provide local-first voice pipelines and LLM conversation agents with controlled tool access [2].
- Edge inference: quantised small LLMs and accelerators make fully local voice control feasible, but latency and multi-step reasoning remain weaker than cloud models.
- Limits and failure modes: misunderstood commands in noisy rooms; hallucinated device names or actions; prompt injection via text in calendar entries, notifications or web content that the agent reads; over-broad permissions (an LLM able to unlock doors); learned routines that misfire after a life change; vendor cloud shutdowns that strand devices; fragmented standards.
Shared resources & combinations
- ROBO-011 (TinyML sensors) shares the edge runtime and sensor firmware; ROBO-018 shares the on-device inference stack in a different form factor.
- CONV-014 / CONV-015 / CONV-016: voice assistants, personal assistants and elderly companions reuse the same ASR, TTS and tool-calling pipeline.
- OPT-007, PRED-031, PRED-007, PRED-048: building energy optimisation, smart-meter disaggregation, load forecasting and building performance.
- Same hub hardware (HW1) can also host local vision and MQTT analytics.
Effort, cost & KPIs
- Path A: hours to days; device cost dominates, subscriptions for premium assistants or cloud video.
- Path B: days to weeks of configuration; API usage cost is small for text commands.
- Path C: weeks of initial setup and ongoing maintenance (updates, backups, Zigbee mesh tuning); hardware from a small hub to a mini PC or GPU.
- KPIs: command success rate and latency, false-trigger rate, energy use and cost vs. baseline (kWh, EUR), self-consumption share of PV, manual overrides per week, alert precision, share of processing kept local.
People & change
Involve the owner or management for budget and priorities, the people who live or work in the building as end users, IT (or the installer) for network and accounts, and suppliers such as the installer and device vendors. The DPO is relevant when cameras, presence or voice data cover employees, tenants or visitors.
Works council: depends. In a private home, no. In a company building, presence sensors, access logs, camera or occupancy analytics can be suited to monitor employees (§87(1) no. 6 BetrVG, objective suitability suffices), so involve the works council before introducing them and agree on purposes, retention and no individual evaluation. Inform it also under §90(1) no. 3 where work processes change.
A DPIA is likely for cameras or presence tracking of people at work or in shared spaces; sign an AVV with cloud vendors. Support AI literacy (Art. 4 AI Act) with a short briefing on what the assistant does, what it records and how to override it. Start with one room or function (e.g. heating), involve a small pilot group, communicate openly about sensors, and address privacy fears with local processing and visible off switches. Manual control must always remain possible and take priority over automation.
Guides: GUIDE-005, GUIDE-007, GUIDE-008.
Risks, regulation & compliance
- EU AI Act: typically minimal risk (AI literacy duty, Art. 4, for professional operators). Becomes sensitive if the system infers emotions (emotion recognition is prohibited in workplaces and education under Art. 5(1)(f); not in the private home as such, but avoid it), uses biometric identification, or is deployed as monitoring of employees or tenants. Safety components of regulated products can fall under high-risk rules via sectoral law. Chat-style assistants need transparency that users interact with AI (Art. 50) [3].
- GDPR: home data reveals habits, presence, health and relationships. The household exemption covers purely personal use, but not landlords, employers, care providers or manufacturers processing the data. Apply data minimisation, local processing, retention limits, and clear information for guests, tenants and visitors, especially with cameras and microphones [4].
- Cyber Resilience Act and Radio Equipment Directive: connected products sold in the EU face security requirements (updates, vulnerability handling) phased in through 2027 [5]. Prefer vendors with support commitments.
- Data Act: since September 2025 users have rights to access data generated by connected products, which helps portability and independent energy management [6].
- Security risks: exposed hubs, weak credentials, malicious integrations, prompt injection. Mitigate with network segmentation (IoT VLAN), no port forwarding (use VPN), least-privilege entity exposure, confirmation for locks and alarms, local backups.
Getting started
- List devices and goals (comfort, energy, security); check Matter support and local-control capability before buying more.
- Set up a local hub (e.g. Home Assistant) on a segmented network; migrate devices one room at a time.
- Add voice control locally first; add a cloud LLM only for selected entities if local quality is insufficient.
- Add energy data (smart meter, PV, heat pump) and measure a baseline for 4 weeks before enabling optimisation.
- Define hard safety rules and a permission list for the AI agent; review logs monthly.
Sources
Accessed 2026-10. Fact-checked 2026-10 (regulatory dates and vendor names); links are primary sources. 1. Connectivity Standards Alliance — Matter — https://csa-iot.org/all-solutions/matter/ 2. Home Assistant — Assist voice control and conversation agents — https://www.home-assistant.io/voice_control/ 3. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj 4. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj 5. Regulation (EU) 2024/2847 (Cyber Resilience Act) — https://eur-lex.europa.eu/eli/reg/2024/2847/oj 6. Regulation (EU) 2023/2854 (Data Act) — https://eur-lex.europa.eu/eli/reg/2023/2854/oj 7. Frigate NVR — https://github.com/blakeblackshear/frigate 8. EDPB Guidelines 01/2020 on connected vehicles and mobility-related apps / Guidelines 3/2019 on video devices — https://www.edpb.europa.eu/our-work-tools/general-guidance/guidelines-recommendations-best-practices_en