Private Smart Home: Running a Local LLM Voice Assistant in 2026

Private Smart Home: Running a Local LLM Voice Assistant in 2026

Direct answer. To run a private local LLM for smart home use in 2026, host a small language model (2B–8B parameters) on a machine you own — typically a Raspberry Pi 4 8GB for lightweight assistants or a Windows or Linux box with a 12GB GPU like the MSI GeForce RTX 3060 12GB for full-fidelity Alexa or Google-class voice. Route Home Assistant intents through the local model via the Ollama or llama.cpp API. No voice data leaves the network.

Why the smart-home privacy story is finally credible

Until 2024, the private-voice pitch was mostly aspirational. Speech-to-text was heavy, natural-language understanding was heavier, and even a competent home lab could not answer "what is the weather" in under three seconds without cloud help. That changed in the last twelve months. Whisper's small and turbo variants run in real time on modest CPUs. Small language models (2B–4B) are now genuinely useful for smart-home intent routing. Home Assistant's Assist voice pipeline treats local wake-word, STT, LLM, and TTS as first-class components.

The remaining question is not "is this possible" — it is "what hardware do I actually need, and how does the assistant behave when the model gets a request it does not understand." This synthesis pulls from Home Assistant's official documentation, community measurements on r/homeassistant and r/LocalLLaMA, and the LLM runtime docs to give a realistic build tier map for 2026. It is aimed at the builder who already has Home Assistant, has one or two ESP32-based voice pucks or a repurposed Alexa unit, and wants to know what to buy and what to expect.

Key takeaways

• Wake-word + STT + intent routing on 8B-class LLM is achievable end-to-end in under 2 seconds on a 12GB desktop GPU. Per Home Assistant benchmarks and community measurements, the LLM is not the slow step; TTS often is.
• A Raspberry Pi 4 8GB can host Home Assistant, wake-word, STT, and a 2B LLM — but not fluently. Response latency lands in the 4–8 second range for typical queries.
• The privacy story is only as strong as your wake-word layer. OpenWakeWord and its clones run entirely offline; cloud wake-word services do not qualify.
• Storage matters. A Crucial BX500 SSD or WD_BLACK SN770 NVMe will keep model load times fast and STT model swaps sane.

What does "private local LLM smart home" actually mean?

The phrase covers three layers that all have to be local for the privacy claim to hold:

1. Wake-word detection. Something on the home network listens for the trigger phrase without ever sending audio out. OpenWakeWord is the current reference project.
2. Speech-to-text (STT). Once the wake word fires, the microphone stream goes to a local Whisper model or equivalent. Per OpenAI's Whisper repository, the small and base variants are usable on CPU, and the turbo variant runs fast on any modern GPU.
3. Intent routing / conversation LLM. The transcribed text goes to a local language model that either matches it against Home Assistant's intent registry or generates a free-form response. Home Assistant's conversation integration supports Ollama-hosted models directly.

Text-to-speech (TTS) closes the loop. Piper is the current local-first choice; it runs on modest CPUs and produces natural-sounding output.

The reason each layer matters for the privacy claim is straightforward: any layer that reaches out to the cloud creates the exact microphone-recording that vendors like Amazon and Google have been criticized for over the last decade. A partial-local setup is still a cloud setup.

How much hardware do you actually need?

The right tier depends on how you use voice. Someone asking "turn off the kitchen lights" thirty times a day has completely different needs from someone asking "read the news headlines from The New York Times."

Tier 1 — Raspberry Pi 4 8GB only. Home Assistant Green or an equivalent RPi-based install can host wake-word, STT (Whisper base), a 2B LLM (Gemma 2B or Llama 3.2 3B at Q4), and Piper TTS. Response latency for basic intents lands around 4–8 seconds per Home Assistant community reports. That is workable for command-style voice ("lights off") but tiring for conversation.

Tier 2 — Mini PC or NUC-class. An Intel N100 or Ryzen 5 5600G box runs Whisper small comfortably and a 4B–7B model at Q4. Response latency drops into the 2–4 second range. Add a WD_BLACK SN770 NVMe for fast model swaps.

Tier 3 — Desktop with 12GB GPU. A MSI RTX 3060 12GB paired with a Ryzen 7 5700X hosts Whisper turbo and an 8B model at Q4 with room for a moderate context window. This is where the assistant starts to feel like a real product — sub-2-second response on the LLM step, with TTS usually the bottleneck.

Tier 4 — 16GB or 24GB GPU. For 13B and 30B-quantized conversational models. Overkill for smart-home intent routing; useful if you want a personal-assistant experience that summarizes email or drafts messages.

Setting up the pipeline on Home Assistant

Home Assistant's Assist pipeline is the anchor. Per its voice control overview, you configure four stages — wake, STT, conversation, TTS — and can swap any of them for a local implementation. The critical settings:

1. Wake word: Use the openWakeWord add-on or a custom-trained wake word from the Wyoming protocol project.
2. STT: Whisper via Wyoming, running on the same host or a network peer. Model size drives latency; base is the sweet spot on a Pi 4, small on a mini PC, turbo on any GPU host.
3. Conversation: Ollama add-on, pointed at a model like llama3.2:3b (small hosts) or llama3.1:8b-q4_K_M (GPU hosts). The Home Assistant conversation integration passes the transcribed text and Home Assistant's context — including which devices are available — to the model.
4. TTS: Piper via Wyoming. Voice quality varies by language and voice pack. English medium voices are pleasant enough for daily use.

The specific commands used in a smart home are typically deterministic — "turn on X," "set the thermostat to Y" — which means the LLM's job is mostly parsing and routing, not creative generation. That is good news for latency because the response is short and the model can be small.

Latency budget on a Raspberry Pi 4 vs 12GB desktop

The perceived-fast threshold for a voice assistant is often cited around 1.5–2.0 seconds from end-of-speech to start-of-response. Voice-UX papers and product-team writeups converge on the same target. Below the numbers land where public reports and Home Assistant's own Assist benchmarks put them.

The Pi 4 build produces a working private assistant. It does not produce a "faster than Alexa" private assistant. The 12GB-GPU build does, especially when Whisper turbo is used and the LLM is capped at 4B parameters or below.

Which models to run on 12GB

For intent routing plus casual chat, three families cover most of the useful surface:

• Llama 3.2 3B-Instruct — small enough for a Pi, capable enough for a mini PC. Understands typical smart-home phrasing well.
• Gemma 2 9B-Instruct — the strongest per-parameter choice under 10B for conversational tasks; runs at Q4 on the RTX 3060 12GB with room for KV cache.
• Qwen 2.5 7B-Instruct — strong multilingual coverage. If your household is bilingual, Qwen tends to outperform Llama 3 at that size.

All three ship at 4-bit and 8-bit quantizations that fit in 12GB with margin. Q4 is a reasonable default for smart-home use; Q5 or Q6 buys back small quality improvements for a real VRAM cost. The llama.cpp discussion on quantization captures the trade-offs well.

Storage and networking notes

Storage. A private assistant pipeline consumes tens of gigabytes of models over time — Whisper variants, TTS voice packs, LLM weights at different quantizations. Plan for at least 100 GB of dedicated storage on the assistant host. A WD_BLACK SN770 NVMe keeps model swaps fast. A Crucial BX500 SATA SSD is a reasonable secondary target for less-frequently-used voice packs or archived Whisper checkpoints.

Networking. Wired Ethernet from the assistant host to the LAN switch is the honest choice. Wi-Fi introduces variable latency that shows up as inconsistent response time. The wake-word puck can be Wi-Fi (it is uplinking short audio streams intermittently), but the LLM host should be wired if possible.

Backup. The privacy claim is only worth anything if the setup keeps running. Snapshot the assistant host to a NAS or an external drive weekly. Home Assistant's built-in backup makes this trivial for its own state; the Ollama model directory needs its own backup policy.

Multi-user considerations

Households with two or three regular users add real complexity to the assistant. Voice-print recognition is not first-class in Home Assistant yet; the assistant will happily respond to whoever spoke last. For most households that is fine — command-style voice ("lights off") does not care who said it. For assistant-style voice ("read me my calendar"), user identity matters.

The current pragmatic answer is per-room wake-word pucks and per-room contexts. The kitchen puck's assistant knows the kitchen. The office puck's assistant knows the office. Voice-print identification will land in Home Assistant eventually; until then, the room is the identity.

Extending the pipeline

Once the core loop works, the natural extensions are:

• RAG (retrieval augmented generation) over your Home Assistant state or personal notes. The LlamaIndex documentation and community RAG guides walk through the pattern.
• Tool use — the LLM calls specific Home Assistant services rather than free-generating text. The conversation integration supports this in stages.
• Multi-model routing — a small model for intents, a larger model when the intent is "conversation."
• Latency budgets per intent type — commands go through a fast small model; open questions go through a slower larger model.

Each of these is a real project. The core private-voice loop is the foundation; the rest is optional depth for users who want a more capable assistant.

Common pitfalls

• Skipping wake-word offloading. Running openWakeWord on the same Pi that hosts Home Assistant + STT + LLM produces cascading load. Move wake-word to an ESP32-based voice puck.
• Choosing Whisper large on a CPU host. Whisper large exceeds the throughput a Pi 4 can supply in real time. Use base or small on the Pi.
• Cloud-first Home Assistant integrations. Some skills reach out to the internet for what looks like a local answer (weather, news headlines). If the LLM answers the weather question by asking a cloud API, the privacy claim is degraded.
• Under-provisioning storage. Whisper + a 7B LLM in Q4 is around 5 GB on disk. Multiply by the number of models you want available and it adds up. A Crucial BX500 SSD is enough headroom for a typical build.
• Ignoring thermals. Small-form-factor mini PCs throttle on sustained LLM load. If your assistant handles frequent multi-sentence queries, keep the case ventilated.

When NOT to build this

If you use voice primarily to control a phone or car — a domain the assistant vendors have optimized for — the on-device Siri and Google models remain hard to beat. If you have one smart plug and one lightbulb, the payoff on a full-local pipeline is small; a physical switch or a hard-coded automation is simpler. And if your household includes users who cannot tolerate any latency regression on voice commands, budget for a GPU tier — the Pi 4 build is not a drop-in Alexa replacement, it is a private-first alternative that trades a couple of seconds for the privacy claim.

Related guides

• vLLM on Windows in 2026: What Actually Works on a 12GB Card
• Ollama vs llama.cpp on a 12GB GPU in 2026: Which Local Runtime to Pick
• Build a Self-Hosted AI Home Lab on a Raspberry Pi 4 8GB in 2026

Bottom line

A private voice assistant is now buildable end-to-end with commodity hardware. If you accept a 4–7 second response window, a Raspberry Pi 4 8GB and a wake-word puck cover the full pipeline. If you want an experience that feels comparable to a cloud vendor, plan on a desktop with a 12GB RTX 3060 and a fast NVMe like the WD_BLACK SN770. Either build keeps voice data on your network.

Citations and sources

• Home Assistant voice control documentation
• Home Assistant conversation integration
• openWakeWord project
• OpenAI Whisper repository
• Piper TTS project
• Wyoming protocol project
• Ollama documentation
• llama.cpp quantization discussion
• r/homeassistant community

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.