Can a Raspberry Pi 4 8GB Run a Local LLM in 2026? Realistic tok/s

Can a Raspberry Pi 4 8GB run a useful local LLM in 2026? Yes, if "useful" means a 1B-3B parameter model at q4 quantization producing 2-6 tokens per second — enough for offline command parsing, intent classification, and background tasks, not enough for interactive chat. The 8GB RAM ceiling is your hard limit; the Cortex-A72 CPU is your soft one. Under those constraints, the Pi is a genuinely interesting inference node.

Who this is for

You already have a Pi 4 8GB, or you're deciding whether to buy one for an "on-device AI" project. You've read that llama.cpp runs on ARM and want to know what that actually delivers on your board before you commit. Maybe you're building a voice assistant that shouldn't send commands to the cloud, or a homelab-style content router that classifies text before deciding where to send it. Those workloads fit the Pi. Anything that expects a 7B-class model at snappy chat speed does not.

The Pi 4's compute story, per the official Raspberry Pi product page: 1.8 GHz quad-core ARM Cortex-A72, 8GB LPDDR4-3200 (this variant), no NEON-boosted matrix extensions in the manner of newer ARMv9 chips. Memory bandwidth is roughly 12 GB/s. For context: an RTX 3060 12GB does 360 GB/s. That 30x bandwidth gap explains most of what happens next.

The good news is that quantized inference isn't only about raw FLOPs. Small models running in the Pi's RAM sidestep the storage bottleneck entirely, and llama.cpp's ARM-optimized kernels do a genuinely respectable job of extracting throughput from the A72. The result is a computer that costs $85 and runs a real language model — that's a wonderful thing, even if the tokens come slowly.

Key takeaways

• Target 1B-3B models at q4_K_M: TinyLlama 1.1B, Phi-3 Mini 3.8B, Gemma 2B, Llama 3.2 1B/3B.
• Realistic tok/s: 2-8 for 1B-3B models at q4; ~1 for 7B models at q4 (once they fit).
• The Pi 4's 8GB RAM is the ceiling — a q4 7B model sits at ~4.5 GB and leaves little headroom for KV cache or the OS.
• Cool the SoC actively; sustained inference throttles a passive-cooled Pi within minutes.
• Boot from an SSD over USB 3.0 — not for inference speed but for model load times and OS reliability.
• The right workloads are asynchronous: home-automation intent, log summarization, offline command parsing. Interactive chat belongs on a discrete GPU.

Which models fit in 8GB of Pi RAM?

Every model at every quantization has a rough RAM footprint. The rule of thumb: quantized weights are approximately param_count x bytes_per_param, plus a couple hundred MB of KV cache for a short context, plus ~500 MB for the OS. On a Pi 4 8GB you have effectively ~7 GB usable.

The comfortable envelope is 1B-3B. The 7B/8B tier technically fits at q3-q4 but leaves so little headroom that anything else you do on the Pi (network, storage I/O, the OS scheduler) starts pushing weights out of cache and the throughput cratered.

How fast do they generate on a Pi 4?

Community benchmarks published on llama.cpp's GitHub and Pi forums cluster around a consistent pattern. These are measured with the Pi 4 8GB running Raspberry Pi OS 64-bit, actively cooled, kernel 6.6+, llama.cpp compiled with -mcpu=cortex-a72 -mtune=cortex-a72.

Prefill (the pass that reads your prompt before generation starts) is faster than generation because it's compute-bound and the Pi's four cores can share the work. Generation is memory-bandwidth-bound: each token has to read the full weight matrix from RAM. That's where the 12 GB/s bandwidth ceiling hits hardest.

Quantization matrix (q2-q8): RAM required + tok/s + quality loss on ARM

The llama.cpp codebase supports a wide range of quantizations. On ARM, the sweet spot for the Pi's cache hierarchy is q4_K_M. Below that, quality degrades noticeably on small models; above that, RAM and bandwidth kill throughput.

If you're building anything user-facing, q4_K_M is where you land. Drop to q3 only when you must fit a larger model and can tolerate the coherence loss.

Prefill vs generation: why the first token is slow on a Pi

Prefill processes the entire prompt in one compute pass — the whole context runs through every transformer layer. On the Pi 4, four A72 cores share this pass and it's mostly compute-bound. Long prompts amplify prefill time linearly, and a 1000-token system prompt on a 3B model at q4 can take 3-5 seconds before the first output token appears.

Generation, once started, is a per-token pass over the same weight matrix (plus growing KV cache). It's bandwidth-bound: at 12 GB/s and a 2 GB model, the theoretical ceiling is ~6 tokens/second. Measured throughput lands slightly under theoretical because of KV cache growth and cache misses.

Practical guidance: keep prompts short. A well-designed prompt for a Pi should be under 200 tokens if you want sub-second time-to-first-token. Reserve long-context work for machines with the bandwidth to sustain it.

Does an SSD swap file help, and how to set it up?

Short answer: not really, for inference. But it's still worth having.

An SSD over USB 3.0 delivers ~350 MB/s read — an order of magnitude faster than a microSD but two orders slower than RAM. If a model spills into swap, generation slows from tok/s to sec/tok territory. The SSD lets a slightly-too-big model load without OOM, but running from swap is functionally unusable.

Where the SSD earns its keep:

• Model load times. A 3.8 GB Phi-3 Mini loads in ~11 seconds from SSD vs. ~45 seconds from microSD.
• OS reliability. SD cards wear from constant metadata writes; SSDs don't care.
• Filesystem for logs, embeddings, and prompt history. Any workflow that stores state benefits.

Configuration is a one-liner:

sudo fallocate -l 4G /var/swapfile
sudo chmod 600 /var/swapfile
sudo mkswap /var/swapfile
sudo swapon /var/swapfile

Add to /etc/fstab for persistence. Point it at a file on the mounted SSD, not at the SD card.

When a Pi Zero W or a discrete-GPU box makes more sense

A Pi Zero W has 512 MB of RAM — enough for a distilled 500M-class model at q4 for extremely narrow tasks (single-intent classification, wake-word triggering) but not for anything you'd call an LLM. Use it when the target is "microcontroller-plus-a-tiny-transformer," not "compact chat assistant."

A discrete-GPU box — a rig with an RTX 3060 12GB — jumps three orders of magnitude in throughput. That card runs Llama 3 8B at q4 at ~55 tok/s vs. the Pi's ~1 tok/s. If you need interactive chat, coding assistance, or genuine reasoning at speed, the desktop is the correct answer.

The Pi's niche is between those extremes: too capable for a microcontroller task, too slow for a chat product, exactly right for an always-on background inference node in a homelab.

Perf-per-watt: the appeal of a 5-watt inference node

The Pi 4 8GB running a 1B model at q4 draws about 5.5 W at the wall. Compare:

Watts per token favor bigger hardware for chat, and favor the Pi for always-on background work. A Pi 4 running an intent classifier 24/7 for a home-automation router costs $7/year in electricity. A 65W desktop doing the same costs $85. That gap is the reason the Pi still belongs in the conversation.

Bottom line

The Pi 4 8GB is a real local-LLM host, provided you match the workload to the constraint. Small models (1B-3B) at q4 quantization run at usable speeds for asynchronous or background tasks: intent classification, log summarization, offline command parsing, home-automation glue, prompt routing before a bigger backend takes over. Storage should be an SSD over USB 3.0 for load times and OS durability. Cool actively. Keep prompts short.

Anything that requires interactive chat, code assistance, or reasoning beyond simple pattern matching wants a discrete GPU. That's where an RTX 3060 12GB rig becomes the correct answer — see our llama.cpp vs Ollama on the 12GB RTX 3060 coverage and the Qwen3-27B local LLM piece for what happens when you have real VRAM to work with.

The Pi is the right first step: it teaches you quantization, ARM inference tuning, and workload shaping without the risk of a $2000 upgrade. If you outgrow it, you'll know exactly what you're paying for.

Common pitfalls on the Pi

• Passive cooling under sustained load. A bare Pi 4 hits ~80C on a 5-minute sustained inference run and thermal-throttles into slower generation. Use an active cooler (Argon ONE V3, Argon ONE M.2, or a case with a small PWM fan).
• Under-powered PSU. A 15W wall wart brownout collapses generation to zero. Use the official 27W USB-C supply.
• Booting an inference job from cold SD. First-token latency includes model load; on SD that's 45+ seconds. Boot from SSD via USB 3.0 so model loads land in 10-15 seconds.
• Wrong context length. Doubling context on a memory-bandwidth-bound Pi runs prefill 2x longer; keep prompts tight.
• Assuming NEON alone is enough. Compile llama.cpp with the correct -mcpu=cortex-a72 flag; the default build misses the tuning.

Two example builds worth stealing

Build A — Always-on intent classifier for Home Assistant. Pi 4 8GB running Llama 3.2 1B at q4 behind a tiny Flask endpoint. Home Assistant sends free-text commands; the model classifies them into automation intents ("turn on kitchen light", "set thermostat 68F"). Latency: ~500 ms. Power draw: 5W. Cost: $100 all-in including case and SSD. Beats a cloud round-trip for anything privacy-sensitive.

Build B — Log-summarizing homelab helper. Pi 4 8GB running Phi-3 Mini 3.8B at q4, cron-scheduled to summarize each service's last 24h of logs at 3am. Output goes to a static HTML dashboard. Slow generation is fine because it runs unattended overnight. Power: still 5W average. Value: a nightly digest of what changed across your homelab without paying for a cloud LLM.

Both projects are wonderful weekend builds and demonstrate the pattern: async task, tiny model, high tolerance for latency.

Related guides

• Self-host Jellyfin on a Raspberry Pi 4 8GB
• Can the RTX 3060 12GB run Qwen3-27B locally?
• llama.cpp vs Ollama on the 12GB RTX 3060
• Cyberdeck Raspberry Pi music workstation news brief

Sources

• Raspberry Pi 4 Model B product page
• llama.cpp GitHub repository
• Ollama GitHub repository