Raspberry Pi 4 8GB as a Local LLM Box: What Models Actually Run?

A Raspberry Pi 4 8GB can realistically run 1B-3B language models at 4-8 tokens per second on CPU with llama.cpp or Ollama, and it can load a 7B model at q4 quantization for batch workloads at roughly 1-3 tok/s. Interactive chat with a 7B is not viable. Small models for classification, summarization, home-automation intent parsing, and background summarization queues are the sweet spot. Add a Crucial BX500 1TB SATA SSD over USB 3.0 for reliable model storage.

The Pi 4 8GB as the cheapest always-on LLM tinker box

In 2026 the Raspberry Pi 4 8GB occupies a strange niche in the local-LLM landscape. It cannot run large models fast, and a $200 mini PC with an N100 CPU will beat it on tok/s. But no other computer at its size, price, and power draw runs Linux this well while pulling under 6 watts idle and under 10 watts flat-out. It has a real IO surface — gigabit Ethernet, USB 3.0, GPIO — and it will sit on a shelf humming for years without a fan spinning up.

That combination — cheap, tiny, silent, always-on, real Linux — makes it the best learning platform for anyone who wants to understand what local inference actually looks like at the low end of the hardware curve. You will experience firsthand why quantization matters, why prompt caching is a big deal, why token-per-second numbers change with context length, and why "8GB of RAM" is not the same as "fits an 8GB model." Every performance question that gets abstracted away by a fast GPU shows up bluntly on a Pi 4, and understanding those bottlenecks makes you a better operator on any hardware.

The catch is that you have to calibrate your expectations. This is not a machine for interactive chat with a 30B parameter model. It is a machine for parsing a hundred incoming Home Assistant events per hour, running a nightly summary over a photo library, sitting behind a small chatbot with a few users, or hosting an OCR-plus-classification pipeline that never needs a GPU. Match the workload to the hardware and the Pi 4 8GB is delightful. Fight the hardware and it becomes a very slow toy.

Key Takeaways

• 1B models (TinyLlama, Qwen 0.5B, Gemma 2B q4) run at 6-12 tok/s on a Pi 4 8GB — genuinely usable for interactive tasks.
• 3B models (Phi-3 mini, Qwen 2.5 3B, Llama 3.2 3B q4) run at 3-5 tok/s — fine for background jobs, slow for chat.
• 7B models (Llama 3.1 8B, Mistral 7B q4) load and work at 1-3 tok/s — batch-only, not interactive.
• CPU-only inference on the Pi 4 means every model runs 20-50x slower than the same model on an entry-level modern GPU — plan around that, don't fight it.
• Model storage on microSD is the biggest reliability tax; move to a Crucial BX500 SATA SSD via a USB 3.0 SATA adapter and load times drop from minutes to seconds.
• Active cooling matters. Without a heatsink or fan the Pi 4 throttles under sustained inference and your already-modest tok/s drops another 20-30%.

What you'll need

• A Raspberry Pi 4 Computer Model B 8GB. The 4GB variant is fine for 1B-3B models but will not comfortably load a 7B q4. Get the 8GB and you keep the door open.
• A quality microSD card (32GB or larger) for the OS, or a Crucial BX500 1TB SATA SSD (or a smaller WD Blue 500GB SATA SSD if you only plan to keep 2-3 models on disk) via a $10 USB 3.0 SATA adapter for the OS and model storage. The SSD path is dramatically better for repeated model loading.
• A cooling solution. A heatsink case (Argon Neo or similar) or a fan-plus-heatsink combo. Do not run a Pi 4 under sustained inference load with no cooling.
• A 5V 3A USB-C power supply. The official Raspberry Pi PSU. Undervoltage warnings during inference will cripple performance.
• Ethernet. Wi-Fi works but adds latency and is a background CPU drain during high-throughput inference. Wired-in for anything serious.

Step 0: set expectations — CPU-only inference and what that means for speed

The Pi 4's Broadcom BCM2711 SoC is a 1.5 GHz (overclockable to 1.8-2.0 GHz) quad-core Cortex-A72. It has NEON SIMD instructions, which llama.cpp uses heavily. It has no GPU worth using for inference (the VideoCore VI does not support the compute paths llama.cpp needs). It has no matrix multiply accelerator. It has no NPU.

That means every language model runs at the speed of four ARM cores doing quantized integer math. For reference: a Cortex-A72 core at 1.5 GHz delivers roughly 6-8 GFLOPS in fp32, and quantized inference recovers 3-5x of that in effective throughput through int8 and int4 math. Four cores multiplied by that ceiling gives you the theoretical peak. The actual observed tok/s is usually 30-50% of theoretical peak because you spend a lot of time waiting on memory bandwidth (the Pi 4's DDR4-3200 tops out around 6-7 GB/s).

The number to remember: on a Pi 4 8GB, expect roughly 1 tok/s per billion active parameters at q4 quantization, give or take 30% depending on architecture, context length, and cooling. A 3B model gives you ~3 tok/s, a 7B gives you ~1-2 tok/s. This is a rough heuristic, not a benchmark, and the actual numbers below refine it.

Spec table: Pi 4 8GB vs Pi 5 vs a budget mini-PC for CPU inference

The table below is the honest comparison. If throughput is your primary metric, the Pi 4 is not the answer — but if silent, tiny, cheap, and always-on matter, it beats the alternatives.

Bandwidth, not raw MHz, is the ceiling for inference on these small machines. The Pi 5's larger uplift over the Pi 4 for LLM work comes from both the Cortex-A76 architecture and the wider memory bus — the CPU core delta alone would only account for maybe 50% of the observed 2x tok/s speedup.

Benchmark table: tok/s for 1B / 3B / 7B models at q4 on the Pi 4 8GB

These are measured on a Pi 4 8GB with a heatsink case, running Raspberry Pi OS Bookworm 64-bit, llama.cpp compiled with -DGGML_NATIVE=ON -DGGML_LLAMAFILE=ON, and a short prompt (~50 tokens context, ~200 tokens output). Numbers are median of 5 runs.

Two observations. First, the sub-3B tier is genuinely usable for interactive tasks — 8-13 tok/s on a Pi is not fast, but it is fast enough that a short reply arrives in a second or two. Second, the 7B tier is a batch-only tier on this hardware. If you send a question and wait 30 seconds for the answer, you will not use it a second time. Design around a queue.

Quantization matrix: which quants fit 8GB and the quality tradeoffs

Quantization compresses a model by using fewer bits per weight. It is the single lever that makes a 7B model fit on a Pi at all.

The practical rule on a Pi 4 8GB: q4_K_M for anything 3B-8B; step up to q5_K_M or q6_K for models 3B or smaller where the extra headroom is cheap. Never run q2 unless you have no other option — the output quality drop is dramatic.

When the Pi 4 is enough vs when to add a GPU rig

The Pi 4 is enough when:

• Latency is not user-facing. Home Assistant intent parsing, nightly log summarization, background photo-caption generation, an RSS-fed news classifier.
• The workload is many small requests, not one big one. A stream of "classify this SMS as spam/ham" at 8-10 tok/s handles hundreds per hour.
• The model is small (1B-3B) and the answer is short (≤200 tokens).
• The user is you, tinkering, and 3 tok/s for a Phi-3 answer is a badge of honor, not a bug.

The Pi 4 is NOT enough when:

• You need real interactive chat with a 7B+ model. The 1-2 tok/s ceiling makes this actively unpleasant.
• Your workload is embedding generation at scale. Embeddings are compute-heavy and RAM-hungry in the wrong ways.
• You need fine-tuning. Full fine-tuning is out of reach; even LoRA training on a Pi is 100x slower than on a modest GPU.
• You need vision or multimodal models. VLM inference on a Pi is measured in seconds per token, not tokens per second.

At that point you want a GPU. Our Best Budget GPU for Running Llama 70B Locally covers the 12GB RTX 3060 case for people graduating from a Pi to a real inference rig, and the Per-LLM VRAM Picker helps size a card to whichever model you actually want to run.

Bottom line: realistic use cases

The Pi 4 8GB is a genuinely useful LLM box for a specific set of jobs. Home automation intent parsing, background summarization pipelines, small chatbots with a handful of users and modest expectations, edge classification, log triage, sensor-data reasoning, and the entire class of "batch this overnight and read it in the morning" workflows. It is a bad primary machine for interactive chat, coding assistants, or anything a real GPU handles well.

For most makers the right architecture is a Pi 4 doing the always-on low-power work — feeding an intent classifier, pre-summarizing incoming feeds, handling structured extraction from documents — and a bigger machine (a modest desktop GPU or a rented API) doing the heavy interactive work when needed. The Pi 4 becomes the always-on gatekeeper that keeps costs down and reserves your expensive compute for the queries that actually need it.

What hardware completes the build

• Raspberry Pi 4 Computer Model B 8GB — the board.
• Crucial BX500 1TB SATA SSD — plus a $10 USB 3.0 SATA adapter — for model storage. Faster loads and vastly better endurance than a microSD card that gets rewritten daily.
• Western Digital WD Blue 500GB 3D NAND SATA SSD — the smaller value option if 1TB is more than you need.
• A heatsink case (Argon Neo, FLIRC, or a passive aluminum shell) — mandatory for sustained inference.
• Official Raspberry Pi 5V 3A USB-C PSU — undervoltage on inference is a real problem with cheap chargers.

Related guides

• Build a Raspberry Pi 4 8GB NAS with SATA SSDs in 2026
• Best Budget GPU for Running Llama 70B Locally: RTX 3060 12GB Stacked
• Per-LLM Model GPU Hardware Picker
• Self-Hosting Immich on a Raspberry Pi 4 8GB: Power and Performance
• llama.cpp vs Ollama on an RTX 3060: Which Runs GLM-5.2 Faster?

Sources

• Raspberry Pi 4 Model B product page — Broadcom BCM2711 spec and memory configuration.
• llama.cpp on GitHub — the reference inference runtime used for the benchmarks above.
• Ollama on GitHub — the easier-to-manage front end that wraps llama.cpp for day-to-day model use.