Private local AI assistant on own hardware (home server)

Cloud chatbots are convenient, but every prompt, scanned letter or health question is sent to a third party. A home-server assistant runs open-weight language, speech and vision models on hardware the household owns: a gaming PC, a small GPU server, a mini-PC or a unified-memory Mac. With a few containers it offers chat, "ask my documents", dictation, voice control and automations, with data staying in the house and no per-token cost. The price is hardware, some upkeep and a quality gap to the biggest cloud models.

Problem & value

• Today: people paste sensitive material (tax letters, medical findings, contracts, family photos) into cloud assistants, or avoid AI for exactly those tasks. Consumer cloud terms differ on retention and training use; data leaves the EU in many cases.
• AI value: privacy by architecture (no third-party processing), offline capability, no subscription or usage fees, full control over models, memory and integrations, and a shared assistant for the whole household.
• Why now: open-weight models in the 8-30B class (Qwen3, Gemma 3, Mistral, gpt-oss-20b) run on consumer GPUs and are good enough for drafting, summarising, translation, extraction and document Q&A [1][3][4]. Frontier-level reasoning and very long, complex agentic tasks are still stronger in the cloud.
• No reliable aggregate benefit figures exist for this use case; value is mostly qualitative (privacy, cost, autonomy).

Variants

• Chat and writing hub: one web UI for the family with separate accounts (Open WebUI, LibreChat).
• Private document brain: local RAG over scans, PDFs and notes (paperless-ngx plus embeddings) - see PERS-008, DOC-014, KNOW-002.
• Voice assistant: local wake word, speech-to-text, LLM and TTS via Home Assistant Assist - see ROBO-012 [5].
• Photo and media intelligence: local tagging and search - see PERS-006.
• Family-safe tutor / child profile: restricted models and filters - see PERS-019.
• Micro-SMB / freelancer: same stack for client documents under confidentiality duties (lawyers, doctors, tax advisers) on a small office server.
• Hybrid: local by default, a privacy-filtered call to an EU cloud model only for hard questions.

How it works

clients (phone, laptop, browser, voice satellite)
  -> reverse proxy + VPN (WireGuard/Tailscale/Headscale), auth, TLS
  -> UI / orchestration (Open WebUI, Home Assistant, n8n)
  -> inference server (Ollama / llama.cpp / vLLM) serving chat LLM, embedding model, VLM
  -> tools: document index (vector DB), speech (Whisper-class ASR, Piper-class TTS), smart-home API, calendar
  -> answers with citations; actions only after confirmation
  -> encrypted local storage + backups

• Inference engine: llama.cpp / Ollama run quantised GGUF models on CPU, GPU or Apple Silicon; vLLM suits multi-user throughput [2].
• RAG: documents are parsed (OCR), chunked, embedded and stored in a local vector store (built into Open WebUI, or Qdrant/pgvector); the model answers with citations.
• Human-in-the-loop: write actions (sending e-mail, unlocking doors, deleting files) require explicit user confirmation; answers over legal, medical or tax material are drafts, not advice.

Data preparation

The assistant is only as useful as the documents it can read. For a household these are scanned letters, contracts, PDFs, notes and exported e-mail; in a small business they sit in file shares, the DMS, mailboxes, the ERP and often still on paper. Typical gaps: scans without text layer, duplicates and old versions, cryptic file names, knowledge that exists only in people's heads, and access rights that are too open (everyone can read the HR folder).

Concrete steps: - Decide the scope: start with one folder tree, not "everything" (see GUIDE-003 and PREP-001). - Scan and OCR paper archives with Paperless-ngx or a comparable tool (PREP-006, DOC-014), convert legacy formats and mail archives (PREP-007). - Remove duplicates, give files meaningful names and basic tags (PREP-009, PREP-010). - Mirror the source permissions in the index or build one index per group, and keep personal and confidential data in separate collections (PREP-023, PREP-024). - For businesses, pseudonymise where possible (PREP-022) and write a short list of terms and abbreviations (PREP-015).

Rough share of effort: about 20-30 percent of the total project for a household, 30-40 percent for a business, because the infrastructure itself (one container stack) is quick to set up while clean documents are not. Plan the index refresh as a recurring task.

Implementation paths

Which path for whom

• private: C, because a gaming PC or mini-PC with Ollama and Open WebUI keeps letters, health and family data in the house at no per-token cost.
• developer: C, because the open stack (llama.cpp/Ollama, Qdrant, n8n) is the natural lab and can be extended freely.
• smb: C, because a small team with sensitive documents (contracts, customer or HR data) can serve several use cases from one HW1/HW2 box; no product replaces this local platform at SMB prices (Baukasten effect).

Path A - Buy (appliance / consumer product)

• Apps with on-device models: desktop apps such as LM Studio, Jan, GPT4All; Apple Intelligence and Windows Copilot+ PCs run small models on-device.
• Plug-and-play boxes: home-server OS and appliance offerings (e.g. Olares, Umbrel, Unraid or TrueNAS app stores, Synology/QNAP AI packages) bundle Ollama and Open WebUI [6].
• EU angle: Mistral (Le Chat, EU provider) and Nextcloud Assistant (German, runs on own Nextcloud) are privacy-leaning EU options, though the former is still cloud-processed.
• Choose A when: one user, little tech interest, light use.

Path B - Build on cloud APIs

• Hybrid only: a local gateway (LiteLLM, Open WebUI connections) routes simple prompts to local models and hard ones to EU-hosted APIs (Mistral, Aleph Alpha/ Schwarz STACKIT AI Model Serving, OVHcloud AI Endpoints, Scaleway) or global providers with EU data residency.
• Strip or pseudonymise personal data before any external call; log what leaves the house.
• Choose B when: you want frontier quality occasionally but keep sensitive files local.

Path C - Self-host (recommended)

• Models (open weights): Qwen3 family, Gemma 3, Mistral Small, Llama 3.x, gpt-oss-20b / gpt-oss-120b (Apache 2.0), DeepSeek-R1 distills; embeddings such as bge-m3 or nomic-embed; VLMs such as Qwen-VL or Gemma 3 for scans and photos. Check each licence.
• Stack: Ollama or llama.cpp server; Open WebUI as front-end [1]; paperless-ngx plus Docling/PaddleOCR for documents; faster-whisper / whisper.cpp for ASR; Piper for TTS; Home Assistant Assist for voice [5]; n8n for automations; Docker Compose; Restic/Borg encrypted backups.
• Hardware: HW1 (mini-PC, 16-32 GB RAM, small 3-8B models, slow but usable); HW2 (one 16-32 GB VRAM GPU, 14-30B quantised models at interactive speed, recommended minimum); HW3 (Mac Studio or 48-96 GB GPU) for ~70-120B models. Memory bandwidth and VRAM, not raw compute, decide speed.
• Ops effort: a weekend for first setup; roughly an hour a month for updates, backups and model changes. Power draw is a real cost (idle 20-60 W for a GPU box).

Path D - Custom training

Rarely sensible at home. LoRA fine-tuning of a small model on personal writing style or a niche domain is possible on a 24 GB GPU, but retrieval (RAG) and good prompts solve nearly all household needs. Never train on third-party personal data without a legal basis.

State of the art (2026)

• Capable small models: 20-30B open-weight models now handle everyday drafting, summarising, translation and tool calling; mixture-of-experts models give large-model quality at modest active compute, though they still need the memory to hold all weights.
• Quantisation (4-bit GGUF, AWQ) makes 20-30B models fit in 16-24 GB at small quality loss; OpenAI states gpt-oss-20b runs within 16 GB memory and gpt-oss-120b on a single 80 GB GPU [4].
• Tooling matured: one-command model management (Ollama), OpenAI-compatible local APIs, MCP tool connectors and local voice pipelines.
• Limits: weaker than frontier cloud models on hard reasoning, long-context, coding agents and multilingual nuance; hallucination remains (RAG with citations reduces it); small models are more prompt-injection-prone when reading untrusted e-mail or web content; no built-in web knowledge (add a local SearXNG search only deliberately).
• Security is on you: exposed Ollama/Open WebUI ports, default passwords and unpatched containers are the realistic risk, not the model.

Shared resources & combinations

• Same GPU and inference server serve PERS-008 (official letters), PERS-016 (data-rights agent), PERS-006 (photo search), PERS-014 (medical explainer), PERS-019 (child safety) and PERS-017 (lifelogging).
• Shares capabilities (ocr-docparse, embedding-text, vector-db, llm-general) with DOC-014, KNOW-002, SPEECH-001; the same stack scales to a small office (HW2-HW3).
• Smart-home voice and automation link to ROBO-012; energy-aware scheduling of GPU load to OPT-032.

Effort, cost & KPIs

• Path A: hours; free to low subscription.
• Path C: EUR 600-1,500 for a used or new 16-24 GB GPU system, EUR 2,000-5,000 for Mac Studio / 48 GB-class (typical market price ranges, check current prices); electricity EUR 3-15/month depending on use; setup 1-2 days.
• KPIs: tokens/s and time to first token, answer quality on a personal test set of 20-50 real questions, share of tasks handled locally vs. escalated, uptime, monthly power cost, backup restore test passed.

People & change

Private households need no formal process, but share the setup with family members who use it and agree what may be stored (children's data, guests, voice recordings). In an SMB, involve management first (scope, budget, AI policy, GUIDE-007), then IT and IT security (hosting, VPN, backups, patching, GUIDE-013), the data owners of the document sources, and the data protection officer early: personal data in documents usually requires a legal-basis check and, for HR or customer data at scale, a DPIA; no AVV is needed when nothing leaves your own server, but any cloud fallback needs one (GUIDE-005).

Works council: depends. If the assistant logs prompts per user, or analyses employee documents or voice, §87(1) no. 6 BetrVG applies (objective suitability for monitoring suffices), and §90(1) no. 3 requires timely information; a short works agreement with a no-performance-monitoring clause removes most concerns (GUIDE-004). In public bodies the Personalrat is involved.

AI literacy: provide a short introduction on what the assistant can and cannot do, hallucinations and what not to enter, in support of the Art. 4 AI Act duty (GUIDE-006). Start with a pilot group of 3-5 curious users and name an AI champion (GUIDE-008). Typical concerns are job loss, surveillance and wrong answers; address them with transparency, citations in every answer and confirmation for all write actions, so humans stay in control.

Risks, regulation & compliance

• EU AI Act: minimal risk for private use. The Regulation applies to deployers acting in a professional capacity; purely personal non-professional use is outside deployer duties (Art. 2(10) excludes obligations of natural persons using AI in a purely personal non-professional activity). A micro-business using it for client work is a deployer: AI literacy (Art. 4) and transparency (Art. 50) if it generates content shown to others. Open-weight model providers carry GPAI obligations, not the household [7]. No Art. 5 practice is involved unless the assistant is built for prohibited uses (e.g. emotion recognition of employees, biometric categorisation).
• GDPR: the household exemption (Art. 2(2)(c)) covers purely personal use, but processing third parties' data (guests recorded by voice satellites, employees' data, client files) outside it needs a legal basis, information duties and security (Art. 32). Local processing removes the processor and third-country transfer issues.
• Professional secrecy: lawyers, doctors and tax advisers gain confidentiality by keeping processing local.
• Security risks: open ports, prompt injection via ingested documents, model-file supply chain (use safetensors/GGUF from trusted publishers, verify hashes), unencrypted backups.
• Mitigations: VPN-only access, per-user accounts, full-disk encryption, no autonomous destructive tools, confirmation prompts, regular updates, tested backups.

Getting started

1. Inventory your hardware; install Ollama (or LM Studio) and pull one 8B and one 20-30B model.
2. Add Open WebUI via Docker and create accounts for household members [1].
3. Index a folder of non-critical documents for RAG, and test it on 20 real questions with citations.
4. Add voice (Whisper + Piper, or Home Assistant Assist) and one confirmed-action automation.
5. Secure it: VPN only, disk encryption, backup, update schedule; then decide which tasks (if any) may escalate to a cloud model.

Sources

1. Open WebUI documentation - Starting with Ollama - https://docs.openwebui.com/getting-started/quick-start/connect-a-provider/starting-with-ollama.md
2. llama.cpp project - https://github.com/ggml-org/llama.cpp
3. Ollama project and model library - https://github.com/ollama/ollama
4. OpenAI - gpt-oss open-weight models announcement - https://openai.com/index/introducing-gpt-oss/
5. Home Assistant - Assist, local voice control - https://www.home-assistant.io/voice_control/
6. Olares documentation - Open WebUI - https://docs.olares.com/one/open-webui.html
7. Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
8. Regulation (EU) 2016/679 (GDPR), Art. 2(2)(c) household exemption - https://eur-lex.europa.eu/eli/reg/2016/679/oj