Can a Raspberry Pi 4 8GB Run a Local LLM in 2026? Real tok/s and What Fits

Yes — a Raspberry Pi 4 8GB can run a local LLM. Not comfortably, not quickly for interactive chat, but usably for background tasks. Community measurements on llama.cpp show roughly 2–3 tokens per second on Llama 3.1 8B q4 and 8–14 tok/s on smaller 1B–3B models. That's slow enough that you should think of the Pi 4 as an always-on background assistant, not a snappy chat companion. When speed matters, an RTX 3060 12GB is the ~15× upgrade.

The appeal and limits of edge LLMs on an SBC

There's something genuinely magical about running an AI model on a $75 board. No cloud dependency. No API key. No usage cap. Silent, low-power, sits on a shelf and answers questions when asked. For hobbyists building home assistants, tinkerers exploring what "local AI" actually feels like, and anyone with privacy requirements that rule out cloud APIs, the appeal is obvious.

The limits are also obvious. The Pi 4's ARM Cortex-A72 cores are old, its memory bandwidth is a fraction of any GPU's, and it has no matrix acceleration to speak of. LLM inference is memory-bandwidth-bound; the Pi 4's ~7 GB/s of LPDDR4 is 50× less than an RTX 3060 12GB's 360 GB/s. That ratio roughly predicts the throughput difference you'll actually observe.

Still, "slow" and "unusable" are different things. Small models like Llama 3.2 1B, Qwen 2.5 1.5B, and Gemma 3 2B produce output at 8–14 tok/s on a Pi 4 — snappy enough for many use cases. Even 8B models produce output at 2–3 tok/s, which is fine for background tasks that don't need real-time response. This piece walks through what actually fits, how fast it runs, and where the real ceiling is.

Key takeaways

• What fits: Up to ~8B parameters at q4 quantization on 8GB Pi 4 RAM.
• Real speeds: Llama 3.2 1B q4 at ~14 tok/s; Qwen 2.5 3B q4 at ~7 tok/s; Llama 3.1 8B q4 at ~2.5 tok/s.
• Best use: Background classification, summarization, home-automation LLM triggers. Not interactive chat with larger models.
• Cooling: Non-negotiable. Passive heatsink minimum; active fan strongly preferred under sustained load.
• Best model choice: Llama 3.2 1B for interactive; Qwen 2.5 3B for balance; Llama 3.1 8B for one-shot summaries.
• Perf-per-watt: The one metric where the Pi 4 legitimately wins vs a GPU.

What you'll need checklist

• Raspberry Pi 4 Model B 8GB — 4GB works for 3B models; 8GB is the practical minimum for anything at 7B+
• Fast storage: USB SSD like the Crucial BX500 1TB via USB 3.0 is much faster than microSD for model loading
• Official Pi 4 5V/3A power supply — cheap PSUs cause undervoltage throttling, which tanks performance
• Active cooling — a fan-cooled case or Argon ONE. Sustained inference will thermal-throttle passive setups
• Raspberry Pi OS Lite (64-bit) — the 64-bit build unlocks proper LLM performance
• Optional: Coral USB accelerator (not helpful for LLMs directly; useful for vision workloads on the same Pi)

Which models fit in 8GB?

Model + quant vs memory budget on 8GB Pi 4 (leaving ~1.5GB for OS + framework overhead):

Practical guidance: use q4_K_M for the sweet spot between speed, memory, and quality. Only drop to q3 or q2 if you need faster tok/s and don't mind quality loss.

How do you install llama.cpp / Ollama on Raspberry Pi OS?

Steps for llama.cpp on Raspberry Pi OS Lite 64-bit:

1. Update: sudo apt update && sudo apt upgrade -y
2. Install build tools: sudo apt install -y build-essential cmake git
3. Clone llama.cpp: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
4. Build with NEON acceleration: make -j4 (uses all four Cortex-A72 cores)
5. Download a GGUF model: wget https://huggingface.co/.../Llama-3.2-1B-Instruct.q4_K_M.gguf
6. Run: ./main -m Llama-3.2-1B-Instruct.q4_K_M.gguf -p "Hello" -n 100

Ollama on Pi 4 is similar — install with the official one-line script, then ollama run llama3.2:1b. Ollama simplifies management but adds some framework overhead vs raw llama.cpp.

Store models on a USB 3.0 SSD like the Crucial BX500 1TB. Loading a 5GB model from a microSD takes 45+ seconds. From a USB SSD it takes 8–12 seconds.

How fast is it, really?

Community measurements on Pi 4 8GB with active cooling, running Raspberry Pi OS Lite 64-bit, llama.cpp built with NEON acceleration:

Numbers vary ±20% depending on cooling, thermal state, and background CPU load. Passive cooling drops these numbers by 30–50% under sustained use as the Pi throttles.

For context: comfortable reading speed for most people is 4–6 tok/s. So Llama 3.2 1B feels fast, Qwen 2.5 3B feels normal, and Llama 3.1 8B feels slower than reading. That's the practical dividing line for "interactive vs background."

Where prefill latency hurts

Prefill (processing the prompt) is where the Pi 4 hurts most for RAG or long-context workloads. Prefill tok/s on Llama 3.1 8B q4 is roughly 15–20 — meaning an 8K-token prompt takes 400–500 seconds. Nearly 10 minutes just to start answering.

For any workload that involves stuffing significant context in — codebase RAG, long documents, chat history — the Pi 4 is impractical. For short prompts and short answers, it's fine.

When to reach for a Pi 5, a Coral / Hailo, or a real GPU

• Stay on Pi 4 8GB for 1B–3B interactive assistant work, home automation triggers, background summarization of small texts, and anywhere silence + low power matter more than tok/s.
• Upgrade to Pi 5 8GB if you want ~2× tok/s at the same power envelope. Same ARM architecture, faster memory, higher clocks.
• Add a Coral USB Accelerator if your workload is vision inference (image classification, object detection). Coral does NOT accelerate transformer LLM generation — those are two different silicon problems.
• Add a Hailo M.2 accelerator on Pi 5 for supported vision workloads. Same caveat: not a general LLM accelerator.
• Move to a discrete GPU like the RTX 3060 12GB if you want 40+ tok/s on the same 8B model — see our LLM GPU leaderboard for the current best budget option.
• Full-fat mini-PC (N100/N305) if power draw matters but you want 5–10× the Pi's LLM throughput. Trade cost for perf.

Perf-per-watt: the one metric the Pi actually wins

Rough throughput per watt on Llama 3.1 8B q4:

The Pi 5 comes out ahead on perf-per-watt. The Pi 4 is competitive with a discrete GPU on watts per tok/s, and clearly wins on absolute power draw. For an always-on background LLM assistant running 24/7, the Pi is genuinely cheap to operate: ~$10/year in electricity vs a GPU's ~$220/year at heavy use.

Bottom line: useful assistant or novelty?

Useful assistant — for specific workloads:

• Home Assistant natural-language triggers ("turn off all lights in the study")
• Background summarization of daily emails or RSS feeds
• Classification of incoming photos, documents, or notes
• Voice command interpretation with STT + tiny LLM
• Personal search over notes with a small model + RAG cache

Novelty — for these workloads:

• Interactive chat where snappy response matters
• Long-context anything (RAG over documents, codebase understanding)
• Anything involving 13B+ models
• Real-time voice assistant that must feel instant

If your workflow is "background AI while I'm doing other things," a Pi 4 is a genuinely great platform. If your workflow is "chat interactively with a smart model," pair a small Ryzen 7 5800X system with an RTX 3060 12GB and enjoy the 15–20× throughput uplift.

Real-world example: home assistant LLM box

Build recipe for an always-on Pi 4 running a small LLM as a Home Assistant helper:

1. Pi 4 8GB in an Argon ONE case with active fan
2. USB SSD like the Crucial BX500 1TB for OS + models
3. Raspberry Pi OS Lite 64-bit
4. Ollama installed via the one-line script
5. Pull llama3.2:1b for fast interactive tasks
6. Home Assistant integration via the Ollama API — HA sends structured prompts, Pi replies in under 2 seconds
7. Monitor power draw and thermals; Pi 4 sits at ~7W idle, ~9W under LLM load
8. Cost of build: ~$130 total (Pi + SSD + case + PSU + SD)

Ongoing electricity cost: about $10–$12/year. Cheaper than any cloud API for the same use.

Common gotchas

• Passive cooling. Under sustained LLM load, a passive Pi throttles hard. Get active cooling.
• microSD bottleneck. Loading a 5GB model from SD takes 45+ seconds. USB SSD is a night-and-day upgrade.
• 32-bit OS. The default Raspberry Pi OS 32-bit build tanks LLM performance. Use the 64-bit build.
• NEON not enabled at build. If you build llama.cpp on the Pi, confirm NEON acceleration is on — vector instructions are the difference between 2.5 and 1.0 tok/s on 8B.
• Undervoltage warnings. Use the official 5V/3A power supply. Cheap PSUs cause silent throttling.

FAQ

Can I run bigger models with more RAM? No. The Pi 5 with 16GB variants can fit 13B models at q4, but generation is still slow (~1.5 tok/s). 8B is the practical cap for both speed and memory on Pi-class hardware.

Would a Pi 5 be worth the upgrade? Yes if you want faster LLM tok/s. Roughly 2× the Pi 4's throughput at the same power envelope. Same software works.

How does this compare to a Mac Mini? A base M3 Mac Mini destroys the Pi 4 on LLM tok/s at ~5× the price. Worth it if you want Mac ecosystem; overkill if you just want a small LLM box.

Can I fine-tune models on the Pi 4? Not practically. Training or fine-tuning transformer LLMs on ARM CPUs is technically possible but glacially slow. Fine-tune on a GPU box; deploy the tuned model to the Pi.

Should I use Ollama or llama.cpp directly? Ollama for convenience and API-style deployment; llama.cpp for maximum performance and control. Both work fine on Pi 4.

Related guides

• Can a Raspberry Pi 4 8GB Run a Local LLM? Ollama Tiny-Model Benchmarks
• Build a Local Voice Assistant on a Raspberry Pi 4 8GB
• Best GPU for Local LLMs Under $400
• Raspberry Pi 5 vs Pi 4 8GB for a 2026 Homelab
• Best NVMe SSD for Local LLM Model Storage in 2026

Citations and sources

• Raspberry Pi 4 Model B product page
• llama.cpp on GitHub
• Phoronix

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.