Yes — a Raspberry Pi 4 8GB can run small quantized LLMs in 2026, but "run" means roughly 1-3 tokens per second on a 3B model at Q4, or 4-8 tok/s on a 1B tinyllama-class model. That's usable for offline home-assistant style tasks and slow batch translation, but nowhere near interactive chat speed. For anything resembling a useful conversational assistant, you want a mini PC like the GEEKOM A6 or a proper GPU rig with an RTX 3060 12GB.
What "running an LLM on a Pi" actually means in 2026
The r/LocalLLaMA and r/raspberry_pi threads are full of "I ran Llama on my Pi!" posts. Most of them use one of three approaches:
- llama.cpp with a small Q4 model — the honest baseline. Runs a 1-3B model at 1-5 tok/s.
- Ollama + tinyllama — same underlying llama.cpp with a friendlier wrapper.
- ONNX Runtime with a distilled model — Whisper-tiny or Phi-2 in ONNX; faster for narrow tasks.
None of these deliver interactive-chat performance. All of them deliver enough throughput for narrow offline use cases: home-assistant intent classification, single-paragraph summarization, offline transcription, or simple RAG lookup.
Key takeaways
- 1-3B Q4 models are the ceiling. Anything bigger swaps to disk and drops to <0.5 tok/s.
- The Pi 4 8GB is the minimum. The 4GB variant runs out of RAM on even 1B models with any real context.
- Cooling matters. A hot Pi thermal-throttles from 1500 MHz down to 600 MHz, which halves inference speed.
- Speed of storage is the second bottleneck after RAM. Boot from a USB 3.0 SSD (not the SD card) to avoid model-load times measured in minutes.
- A 12W USB-C 3A power supply is mandatory. Undervoltage kills the party silently by throttling under load.
Benchmark numbers (community measurements)
Aggregated from public llama.cpp trackers, r/raspberry_pi posts, and Jeff Geerling's Pi benchmark repo. Configuration: Pi 4 8GB, active cooling (fan + heatsink or PoE HAT with fan), USB 3.0 SSD boot, Ubuntu Server 24.04, llama.cpp built with NEON + BLAS.
| Model (Q4_K_M) | Prefill (2K prompt) | Generation | RAM used | Notes |
|---|---|---|---|---|
| tinyllama 1.1B | ~1.2s | ~5.8 tok/s | ~700MB | Genuinely usable for narrow tasks |
| phi-2 2.7B | ~4.5s | ~2.4 tok/s | ~1.9GB | Better quality; slower |
| llama-3.2 3B | ~6.1s | ~1.8 tok/s | ~2.4GB | Good balance for offline home use |
| qwen 2.5 3B | ~5.9s | ~1.9 tok/s | ~2.3GB | Similar to Llama 3.2 3B |
| llama-3.1 8B (Q3_K_S) | ~35s | ~0.3 tok/s | ~4.2GB | Painfully slow; not recommended |
The read: below 3B is where the Pi 4 is actually useful. Above 3B, it works but the wait times make it impractical for anything interactive.
Which use cases actually work at 1-3 tok/s
Not every LLM use case needs 40 tok/s. Three cases where a Pi 4 is enough:
1. Home Assistant intent classification. A voice command like "turn off the kitchen lights" needs to be classified into an action + entity. That's a single-sentence completion — 20-40 tokens of output. At 2 tok/s, that's a 10-20 second wait. For voice assistants where confirmation happens verbally anyway, that's borderline acceptable if the actual action confirmation is fast.
2. Overnight batch summarization. A stack of documents summarized overnight doesn't care about latency. A Pi can chew through 100-200 documents of 500 words each in a night.
3. Offline RAG lookup. If you have a local vector store and a slow-but-private LLM to answer questions about your notes, a Pi is genuinely enough for the "one question every few minutes" pattern.
Where the Pi is not enough:
- Real-time conversation.
- Code assistant work.
- Any workload with >2K output tokens.
- Vision-language tasks (multimodal models don't fit).
- Anything a user is actively waiting for.
The 4GB vs 8GB question
The Pi 4 8GB is the minimum for anything past 1B at Q4. The 4GB variant runs a tinyllama 1.1B model at Q4 (fits in ~700MB) but any 3B model exceeds RAM once you load OS + Docker + application state on top of the model.
If you're buying a Pi in 2026 specifically for LLM work, get the 8GB. It costs ~$25 more than the 4GB and unlocks 3-4x the model sizes.
Storage matters more than you think
Loading a 3B Q4 model from an SD card takes 90-180 seconds. Loading the same model from a USB 3.0 SSD takes 8-15 seconds. That difference is the single biggest quality-of-life improvement you can make.
Recommended boot storage:
- USB 3.0 enclosure + Samsung 970 EVO NVMe inside → fastest, slightly overkill.
- USB 3.0 enclosure + Crucial BX500 SATA SSD → cheaper, still plenty fast.
- A quality SanDisk Extreme Pro SD card → the "if you must" option; still 4-6x slower than USB SSD.
For model files specifically, keep them on the fastest tier — first-token latency drops noticeably when the model is memory-mapped from fast storage.
Cooling and power delivery
Two silent killers of Pi LLM performance:
1. Thermal throttling. A Pi 4 under sustained load will thermal-throttle from 1500 MHz down to 600-1000 MHz once it hits 80°C. That halves inference speed. The stock Pi has no cooler; the Raspberry Pi Zero W kit linked as a comparison SBC doesn't apply here — you want an active cooler on the Pi 4 specifically. A $5 heatsink + fan combo or the official Pi 4 case with fan resolves it.
2. Undervoltage from a weak power supply. The Pi 4 requires 5V @ 3A minimum. Cheap USB-C chargers rated at 5V/3A often can't sustain it under load. Symptom: a rainbow square in the corner of the display, or vcgencmd get_throttled returning non-zero flags. Use the official Raspberry Pi 4 5.1V/3A supply or a known-good third-party 5V/3.5A supply.
When a mini PC is a better answer
If your reason for a Pi is "cheap, low-power, fits anywhere," and you can flex to $150-500, a small x86 mini PC beats the Pi comfortably for LLM work:
- GEEKOM A6 Mini PC (Ryzen 7 6800H) at ~$550 — 8 cores, 16GB DDR5, integrated Radeon 680M. Runs Llama 3.1 8B at ~15 tok/s on CPU. That's 5-8x the Pi's speed on the same model class.
- Beelink SER5 (Ryzen 5 5560U) or similar Ryzen 5 mini at ~$250 — 6 cores, 16GB DDR4. ~8-10 tok/s on Llama 3B.
Both idle at 8-12W (competitive with a Pi under load), run Linux happily, and don't ask you to fight ARM software support gaps.
The one use case where a Pi still wins: needing GPIO for hardware I/O (sensors, relays, physical interfaces). No mini PC has 40-pin GPIO.
Practical setup checklist
If you're building a Pi 4 8GB as an offline LLM node:
- Buy the 8GB, not 4GB. Pi 4 8GB.
- Add active cooling. Official case with fan or a $5 aftermarket heatsink + fan.
- Buy the official 5.1V/3A USB-C supply. Not a phone charger.
- Boot from USB 3.0 SSD. Enclosure + Crucial BX500 works well.
- Install Ubuntu Server 24.04 for ARM64. Better package availability than Raspberry Pi OS Lite for llama.cpp deps.
- Build llama.cpp with NEON + BLAS.
LLAMA_BLAS=1 LLAMA_BLAS_VENDOR=OpenBLAS make. - Pull a 3B Q4 model. llama-3.2-3B-Instruct-Q4_K_M.gguf is the current sweet spot.
- Run behind Ollama for the friendly API surface. Ollama uses the same llama.cpp backend.
- Add a systemd service to keep the LLM server always up.
- Expose over Tailscale or a local HTTP endpoint — don't put the API on the public internet.
Common pitfalls
- Trying to run 7B+ models "just to see." They technically load with Q3, but at 0.3 tok/s the wait is longer than doing the task manually.
- Not memory-mapping the model. llama.cpp does this by default; some other runners don't. Loading a 3B model as a full copy eats 3.5GB unnecessarily.
- Forgetting swap. If you push the Pi to its RAM limit, having a 4-8GB swap file on the SSD keeps the OOM killer at bay. It'll be slow when it hits, but that beats a hard crash.
- Using SD-card storage for the model. Adds ~10x to load times and shortens SD-card life via constant read amplification.
- Skipping the cooler. A hot Pi is a slow Pi.
When NOT to use a Pi for LLMs
- You need interactive-chat speed (>15 tok/s).
- Your model needs to be 7B+.
- You have any workload requiring low first-token latency.
- Your power budget allows a $250 mini PC and you don't specifically need GPIO.
Real-world Pi + LLM project ideas that actually work
Not every LLM-on-Pi project is a fool's errand. A few that community members have shipped and reported working well at Pi 4 speeds:
Home Assistant fallback intent classifier. Route voice commands through Home Assistant's built-in Wyoming protocol, fall back to a local Llama 3.2 3B on the Pi when the primary intent engine fails. Latency around 8-12 seconds is acceptable for a "hey, my thermostat instruction confused the primary parser, ask the LLM for a plan" fallback. Runs well from the Pi behind a small USB-boot SSD.
Offline note summarizer. A cron job that scans a ~/notes folder overnight, summarizes each note in ~50 words, and writes a _summary.md next to each source note. Pi handles this fine at 2 tok/s over 8-hour overnight windows.
RAG chatbot for a personal recipe library or bookmark collection. Vector store in SQLite + sqlite-vss, small embedding model (all-MiniLM-L6-v2 on CPU), Llama 3.2 3B as the answer generator. Response time of 15-30 seconds per question is acceptable for the "occasional lookup" use case.
Language flashcard generator. Runs offline on a plane. Takes a target word, generates 3-5 example sentences at the requested difficulty, outputs to Anki-format CSV. Cold start ~5s per card; batches of 50 cards run in under an hour.
Simple offline transcription pipeline. Whisper tiny on CPU handles 1-2x realtime transcription on a Pi 4. Combine with a small LLM for post-processing (cleanup, summarization) and you have a genuinely useful voice-memo → text-note pipeline that runs entirely on your desk.
None of these need chat-level latency. All of them benefit from being local, offline, and low-cost.
Comparison table: Pi 4 8GB vs the alternatives
| Platform | Cost | Llama 3B tok/s | Idle power | GPIO | Best for |
|---|---|---|---|---|---|
| Pi 4 8GB + cooling | ~$100 | ~1.8 | ~4W | Yes | Hardware I/O + offline batch |
| Pi 5 8GB + cooling | ~$110 | ~3.5 | ~5W | Yes | 2x Pi 4 speed, same GPIO ecosystem |
| GEEKOM A6 Ryzen 7 6800H | ~$550 | ~15 | ~10W | No | Interactive chat + Jellyfin server |
| Ryzen 5 5600G + RTX 3060 | ~$700 | ~55 | ~35W | No | Interactive chat + code assistant |
| M2 Mac mini 16GB | ~$700 | ~28 | ~5W | No | Quiet, low-power desktop LLM |
The Pi 4 is the cheapest legitimate offline LLM node with GPIO. Every other platform beats it on speed at 3-5x the cost.
Bottom line
A Raspberry Pi 4 8GB is legitimately useful as an offline LLM node for narrow low-latency-tolerant tasks — home assistant intent classification, overnight batch summarization, RAG lookup — running 1-3B quantized models at 1-5 tok/s. It is not a replacement for a real inference box. If interactive conversation matters to you, spend the extra $200-500 on a mini PC or step up to a RTX 3060 12GB rig.
Related guides
- How Many CPU Cores Does a Local-LLM Rig Actually Need?
- AMD Ryzen AI Halo vs DIY RTX 3060 Local-LLM Rig
- Raspberry Pi 4 vs Pi 5 Homelab (2026)
- Raspberry Pi 4 NVMe Jellyfin Homelab
Citations and sources
- Raspberry Pi Foundation — Pi 4 Model B product page
- ggerganov/llama.cpp — README + performance notes
- Ollama — model catalog and hardware requirements
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
