In-car & embedded voice assistant

Classic car voice control understands a few dozen fixed commands and fails as soon as drivers talk naturally. Hybrid embedded-plus-cloud assistants combine on-device speech recognition with large language models that understand free-form requests, call vehicle functions as tools, answer manual questions and chat with context. The gain is safer, less distracting operation and a differentiating brand experience; the hard parts are latency, offline behaviour, safety and data protection.

Problem & value

• Today: touchscreens demand visual attention while driving. Legacy voice systems cover a limited set of predefined tasks and break down on off-script phrasing; many users therefore stop using them or fall back to smartphone mirroring (Apple CarPlay, Android Auto), which gives the OEM no control over the experience or the data.
• AI value: natural, multi-turn dialogue ("it is too cold for the kids in the back, and find a charger with a cafe on the way"), multi-domain requests in one sentence, manual and troubleshooting answers without leafing through a handbook, proactive suggestions, and multilingual support.
• Evidence: Mercedes-Benz describes its previous MBUX as handling roughly 20 predefined tasks and moved to a generative, agentic virtual assistant first shown in the CLA, with Cerence providing speech input/output across about 25 languages and Google Cloud's automotive agent adding conversational navigation based on Google Maps data [1][2][3]. No independent figures on accident reduction exist; treat distraction benefits as plausible but unproven.

Variants

• Passenger cars: full assistant with vehicle control, navigation, charging, media, manual Q&A.
• Commercial vehicles & trucks: hands-free dispatch, tachograph/hours-of-service queries, load info.
• Two-wheelers, boats, agricultural and construction machines: noisy environments, gloves and helmets, often offline-only command sets.
• Other embedded devices: smart home appliances, coffee machines, medical and industrial HMIs, e-bikes, smart speakers without cloud dependency.
• Private use / retrofit: aftermarket assistants and smartphone-based assistants (CarPlay, Android Auto, Gemini or Siri); open-source home-grown voice satellites for hobbyists.

How it works

1. microphone array (multi-zone, echo/noise cancellation, wake word)
2. on-device ASR (embedded) ⇄ cloud ASR (when connected)
3. intent routing: embedded NLU for safety-relevant/offline commands | LLM for open requests
4. LLM agent with tool calls (vehicle-function API, navigation, charging, calendar, RAG over manual)
5. policy/guardrail layer (what may be actuated while driving, confirmation rules)
6. response generation
7. on-device or streaming TTS
8. speaker zone
9. logging with consent; feedback to OEM backend

• Hybrid architecture: wake word and basic commands run fully on the head unit so that they work in tunnels and without consent to cloud processing; the LLM runs in the cloud or, increasingly, a small on-device model handles simple cases. Routing decides per utterance.
• Tool use: the LLM never talks to the CAN bus directly. It emits calls against a whitelisted, versioned vehicle-function API; the gateway checks driving state, user authorisation and safety class. Safety-critical functions (steering, braking, drive modes while moving) are excluded.
• Realtime speech: end-to-end speech-to-speech models and streaming ASR/TTS cut latency, supporting natural barge-in; the cascaded pipeline remains the more controllable and certifiable option.
• Human-in-the-loop: confirmation for consequential actions (purchases, calls, messages), a visible fallback to touch and physical buttons, and workshop/service escalation.

Data preparation

Data comes from several worlds: the owner's manual and service knowledge (PDFs, authoring systems, DMS), vehicle signals and function catalogues (engineering specs, signal databases), navigation and POI data from suppliers, and voice recordings for tuning. Typical gaps: manuals written for print rather than retrieval, inconsistent function names across model variants, missing synonyms and dialect phrasings, little real in-cabin speech with noise, and unclear consent for voice data.

Preparation steps: restructure manuals into short, variant-tagged passages (PREP-014, PREP-010), maintain a glossary of functions and colloquial synonyms (PREP-015, PREP-012), define the command and intent catalogue (PREP-019), collect test utterances across dialects, ages and noise conditions with domain experts (PREP-018), and pseudonymise recordings before use (PREP-022). Document the legal basis and retention for voice data (PREP-025). Tools: document converters, a vector database, speech datasets, simulation rigs. Rough share of effort: about 30 percent, with function catalogue and test data as the main parts.

Implementation paths

Which path for whom

• Enterprise (OEMs, tier-1 suppliers): Path B, because the voice experience is a brand differentiator, must integrate with vehicle signals and own backends, and a hybrid of cloud LLM plus on-device fallback keeps control over data and cost per vehicle.
• Developer (device and embedded builders): Path C, because open ASR, TTS and small models run on edge hardware (HW1) offline, avoid per-request fees at fleet volume and keep voice data in the device.
• Private (drivers, e-bike owners, makers): Path A, because the assistant ships built into the vehicle or a certified app; hobbyists may use local open-source voice stacks.

Path A — Buy (for device makers without own voice team; consumers get it built in)

• Automotive voice platforms: Cerence (xUI / CaLLM family; powers Mercedes-Benz MBUX), SoundHound (OEM voice AI), Amazon Alexa Custom Assistant / Alexa Built-in, Google Gemini and Google built-in (Android Automotive OS), Apple CarPlay for smartphone projection. EU-relevant options: Cerence (US-headquartered, large engineering site in Aachen), OEM in-house stacks such as MB.OS, and Mistral-based custom deployments; Chinese stacks (Baidu, iFlytek) matter for export markets.
• Embedded device voice: Picovoice (wake word and on-device speech-to-intent, Canada), Sensory, Fluent.ai (offline), DSP Concepts for audio front-ends.
• Choose A when: you need production-grade multilingual coverage, certified audio front-end, and a roadmap; check data residency, who owns the data and training rights, and OEM brand control.

Path B — Build on cloud APIs

• Models: Mistral (EU) or other LLMs with function calling; Azure OpenAI or Google Vertex AI in EU regions; Claude via EU-region cloud endpoints; realtime speech APIs (e.g. OpenAI Realtime, Gemini Live) or Deepgram/ElevenLabs/Azure Speech for cascaded pipelines.
• Orchestration: agent framework with strict tool schemas, a vehicle-API gateway, RAG over the owner's manual (see KNOW family), telemetry via the OEM cloud. Use Android Automotive OS's built-in voice-interaction services as the system integration point [4].
• Choose B when: you are an OEM or tier-1 with an engineering team and want control of dialogue design; keep an embedded fallback for offline use.

Path C — Self-host (open source)

• On-device/edge: whisper.cpp or Whisper-turbo variants, Vosk, sherpa-onnx, Piper TTS, openWakeWord; small LLMs (Qwen 3 small, Llama 3.2 1-3B, Phi, Gemma 3n, Mistral Ministral) quantised via llama.cpp, ExecuTorch or ONNX Runtime on automotive SoCs (Qualcomm Snapdragon Cockpit, NVIDIA DRIVE/Jetson Orin, Ambarella, Renesas).
• Server side: vLLM with a 7-30B open model for harder queries; Home Assistant's Assist / Wyoming protocol is a practical open reference for private embedded voice pipelines.
• Hardware: HW1 for embedded inference; HW2/HW3 for the backend model.

Path D — Custom training

Worth it for OEM-specific vocabulary, dialects, in-cabin acoustics and tool-calling over proprietary vehicle APIs: fine-tune a small model for function calling and routing, and adapt ASR to cabin noise and accents. Needs thousands of annotated in-cabin utterances plus synthetic dialogues; collect with explicit consent and privacy-by-design.

State of the art (2026)

• From commands to agents: OEMs are shipping LLM-backed assistants (Mercedes-Benz MBUX with Cerence and Google, plus announcements by other European and global manufacturers) that call vehicle functions and external services as tools [1][2][3].
• Hybrid inference: small on-device models for latency and offline, big cloud models for knowledge and reasoning; dynamic routing is the key engineering topic.
• Speech-to-speech and proactivity: natural interruptions and context-aware suggestions are arriving; voice-only operation still lags behind touch for precision tasks.
• Limits and failure modes: hallucinated answers about the vehicle (wrong tyre pressures or fuse data), mis-recognition in noise or with accents, latency spikes on poor mobile coverage, prompt injection via web content or passenger speech, unintended actuation by other occupants or media audio, and cognitive load from verbose answers. Keep answers short and ground manual answers in retrieved text.

Shared resources & combinations

• Same stack: CONV-002 (voicebot ASR/TTS and dialogue management), SPEECH-003 (real-time translation), SPEECH-006 (TTS), CONV-015 (personal assistant: calendar, mail, tasks), KNOW family (manual RAG).
• Combine with: CONV-001 (service chat after the drive), CONV-024 (child-directed voice in rear seats), predictive maintenance on telemetry (see PRED family) for proactive workshop booking.
• Hardware: the embedded SoC (HW1) is shared across ADAS and cockpit workloads; backend serving shared with other voice products.

Effort, cost & KPIs

• Path A: 6-18 months in a vehicle programme (integration, homologation, language packs); licence typically per vehicle plus cloud usage.
• Path B: 6-12 months for a production pipeline; cloud token/audio-minute cost scales with fleet size.
• Path C: 9-18 months; embedded optimisation and validation dominate cost.
• KPIs: task completion rate, word/intent error rate in cabin conditions, end-to-end latency (speech end to first audio), offline coverage, voice-use share vs. touch, user satisfaction, safety-related override events, cost per active vehicle.

People & change

Involve management and the product owner first, then engineering/IT, IT security (remote command safety), the DPO and legal (product liability, UNECE rules, driver distraction) and suppliers of ASR, maps and head units. Users are customers, so run clinics and field tests with real drivers early.

Works council: depends. The product itself faces customers, but internal test drives, fleet data and voice logging of employees can monitor behaviour; then §87(1) no. 6 BetrVG applies. §90(1) no. 3 also covers AI in the development process. A DPIA is likely for voice and location data; use AVV with every processor. Offer opt-in for voice storage and make local processing the default.

AI literacy: support engineers, testers and service staff with training (AI Act Art. 4; GUIDE-006). Communicate transparently that users talk to an AI (Art. 50) and that safety-critical functions stay under deterministic control. Pilot with a small vehicle fleet or one device line. Typical concerns: privacy ("is the car listening") and distraction; address with wake-word design, visible indicators and clear data settings. See GUIDE-005, GUIDE-008 and GUIDE-013.

Risks, regulation & compliance

• EU AI Act: typically limited risk: users must be informed they interact with an AI system and synthetic speech must be identifiable where Art. 50 applies; AI Literacy duty (Art. 4). The assistant itself is not a safety component. Art. 5 caution: do not add emotion recognition of drivers inferred from voice or face in a workplace context (e.g. company fleets) — prohibited in workplaces; driver-monitoring for safety is regulated separately, and biometric categorisation or identification of occupants via voice needs a separate legal assessment [5]. If AI controls safety functions, vehicle type-approval law and high-risk rules via Annex I may apply instead.
• GDPR / ePrivacy: voice recordings, location, driving behaviour and contact data are personal data; vehicle owners, passengers and previous owners are distinct data subjects. EDPB guidelines on connected vehicles require local processing where possible, clear consent for each purpose, and easy deletion; ePrivacy Art. 5(3) applies to access to terminal equipment [6]. Always-listening wake-word detection must remain on-device.
• Vehicle regulation: UNECE R155 (cybersecurity management) and R156 (software updates) are conditions of EU type approval; EU General Safety Regulation 2019/2144 includes driver distraction requirements [7]. The EU Data Act gives users access to vehicle-generated data.
• Mitigations: vehicle-API whitelist, driving-state locks, confirmation prompts, injection filtering, grounded answers with source, on-device-first processing, retention limits, red-team tests.

Getting started

1. Define the scope: which functions may be voice-controlled while driving, which offline.
2. Collect 200-500 real in-cabin utterances (consented) per language and measure baseline error rates.
3. Build a hybrid prototype: on-device wake word + ASR, cloud LLM with a mock vehicle-function API.
4. Run a safety/privacy assessment (DPIA, TARA under R155) before in-vehicle trials.
5. Pilot with a small fleet and track KPIs; widen tool access only after review.

Sources

1. Cerence — Cerence AI powers in-car experience in Mercedes-Benz's new electric sedan — https://www.cerence.com/newsroom/press-releases/cerence-ai-powers-in-car-experience-in-premier-german-automakers-new-electric-sedan
2. Voicebot.ai — Mercedes-Benz will keep using Cerence to run revamped MBUX Virtual Assistant — https://voicebot.ai/?p=21059
3. eWeek — Mercedes adds conversational navigation with Gemini AI — https://www.eweek.com/news/mercedes-gemini-ai-navigation/
4. Android Developers — Android Automotive OS — https://source.android.com/docs/automotive
5. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
6. EDPB — Guidelines 01/2020 on connected vehicles — https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-012020-processing-personal-data-context_en
7. UNECE — UN Regulations 155 (cybersecurity) and 156 (software updates) — https://unece.org/transport/vehicle-regulations
8. Home Assistant — Assist voice control — https://www.home-assistant.io/voice_control/