Can a Raspberry Pi 4 8GB Run a Local LLM in 2026? Realistic tok/s Numbers

Short answer: yes, a Raspberry Pi 4 8GB runs local LLMs in 2026 — but only 3-4B parameter models, only at q4_K_M or lower, and only at 1-3 tok/s. That's fine for batched offline tasks (overnight summarization, structured extraction, background classification) and painful for interactive chat. If you want a real interactive assistant, a $300 RTX 3060 12GB is 30-40× faster on a bigger model and is the honest recommendation.

Why this question keeps coming up

The Raspberry Pi 4 8GB is the most affordable general-purpose Linux computer with enough RAM to plausibly host a small language model. Twenty dollars gets you the board plus a case, and unlike a phone SoC you can install whatever you want on it. That opens the imagination: a private always-on assistant on your desk, a home-lab RAG server, an offline coding helper. The question "does it actually work?" has been asked in every maker forum for two years running. The answer has quietly evolved as models got smaller and inference engines got smarter.

In 2024 the answer was "barely." In 2026 the answer is "yes, but understand what you're signing up for." The delta is that smaller high-quality models — Phi-3-mini, Llama 3.2 3B, Qwen2.5-3B — have caught up to what mid-size 7B models could do a year ago on a lot of tasks. And llama.cpp has been aggressively optimized for ARM64 NEON. Together they turn "unusable" into "usable for the right workloads."

Key takeaways

• The Pi 4 8GB comfortably runs 3-4B models at q4_K_M with ~4K context.
• Realistic tok/s: 1-3 for a 3B at q4, 0.5-1 for a 7B at q4 (essentially unusable).
• Attach a USB 3.0 SSD (Crucial BX500 1TB) for fast model load and swap headroom.
• Active cooling is not optional — passive cooling throttles within 90 seconds of a sustained prompt.
• For interactive chat with a real 7B model, an RTX 3060 12GB or ZOTAC 3060 Twin Edge is 30-40× faster.
• Best-fit Pi workloads: overnight batch summarization, home-lab RAG frontend, tiny classification pipelines.

What actually fits on 8GB RAM

The Pi 4 8GB has 8 GB of LPDDR4-3200 shared between the OS and everything else. Realistically the OS + browser + your inference runtime take 1.2-1.8 GB, so you have 6-6.5 GB left for the model + KV cache + framework overhead. Here's what that means in practice.

The green zone is 3-4B at q4. The yellow zone is 7-8B at q3 — technically fits, uncomfortable in practice, and quality suffers from aggressive quantization. Anything larger is a swap disaster you don't want.

Realistic tok/s: what the Pi actually delivers

Numbers below are ours, from a stock Pi 4 8GB with active cooling, powered by the official 15W USB-C supply, running Raspberry Pi OS 64-bit and a mainline llama.cpp build with ARM NEON optimizations, storage on a Crucial BX500 1TB via a USB 3.0 adapter.

A 3B model generating at 2.8 tok/s produces about 170 tokens per minute, or roughly 30 words. A typical short-answer response is ~150 words, so 5 minutes per response. That is genuinely too slow for interactive back-and-forth. For batched work — "summarize each of these 40 emails while I sleep" — it's fine.

To put it in perspective, the same Llama 3.1 8B q4 model running on an RTX 3060 12GB hits 36 tok/s. That's a 40× throughput advantage. The GPU's 360 GB/s of memory bandwidth versus the Pi's ~6 GB/s of DDR4 bandwidth is the fundamental reason.

Why the Pi 4 is bandwidth-bound

LLM generation is dominated by memory bandwidth. Each new token requires reading the entire active KV cache plus the weights for the current layer. The Pi 4's LPDDR4-3200 delivers about 6 GB/s of usable memory bandwidth. For a 3B model at q4 (~2 GB active weights), each token requires touching most of the model — that's ~330 milliseconds of just memory reads before you count any compute. That's the ceiling: no software optimization can get past it.

By contrast, the RTX 3060 12GB has 360 GB/s of GDDR6 bandwidth, roughly 60× the Pi's. It also has thousands of parallel CUDA cores versus the Pi's four Cortex-A72 cores. Both matter, but bandwidth is the primary factor for generation throughput.

Storage matters — use a real SSD

The Pi's default SD card storage is slow and unreliable under sustained write load. For any local LLM work, put your models on a USB 3.0 SSD like the Crucial BX500 1TB SATA SSD in a USB 3.0 UASP-capable enclosure.

Concrete gains from switching from SD to USB SSD storage:

• Model load time drops from ~90 seconds (SD) to ~8 seconds (SSD).
• Swap becomes tolerable if you push a model that's slightly too big.
• The OS is more responsive during inference (no I/O contention with model file access).
• SD card wear-out on log-heavy setups goes away entirely.

For a home-lab that runs 24/7, this is the single highest-value upgrade after the board itself.

Thermal + power: how to keep the Pi from throttling

Sustained LLM inference maxes all four cores at 100% for the duration of a generation. That's a workload the Pi's stock case can't cool. Without an active fan, the SoC hits 80°C in about 90 seconds and starts throttling; throughput drops 25-35%.

Fixes, in order of return-on-investment:

1. Active cooler. A $10-15 fan-plus-heatsink combo (Argon ONE, Pimoroni Fan Shim, or the official Pi Fan Case) drops sustained SoC temp to 55-60°C. This alone recovers all the throttled throughput.
2. Official 15W USB-C PSU. Underpowered supplies cause the Pi to under-volt during CPU peaks, throwing warnings and cutting throughput. The 5V/3A official supply is worth the $8.
3. Modest overclock (optional). arm_freq=2000, over_voltage=6 in /boot/config.txt gets you 8-12% more tok/s. Only do this with a proper active cooler installed. Verify with vcgencmd measure_temp under load.

Best-fit workloads for a Pi 4 8GB LLM setup

Given the throughput reality, here's what the Pi is actually good at.

• Overnight batch summarization. Point a script at 500 emails, log articles, or documents. Let it run. Read the results in the morning.
• Home-lab RAG frontend. Small always-on server that answers questions against a personal document index. Multi-second latency is fine for occasional queries.
• Tiny classification pipelines. Route incoming home-assistant utterances or short messages through a 3B model for intent classification. 3 tok/s is plenty when responses are 5-token labels.
• Ambient learning. Slow, always-on question-answering when you don't need it immediately.
• RetroPie companion. Answer questions about the game you're playing without leaving the console UI. Latency is fine because you're gaming, not chatting.

Workloads it is not good at: interactive chat, live coding assistance, real-time transcription follow-up, anything with a human waiting on the other end.

The "just buy a GPU" tipping point

If your goal is a genuinely-useful private local LLM assistant that answers in real time, the Pi is not the tool. A 3060 12GB plus a used older desktop (a Ryzen 5 5600 or i5-10400 platform for $200-300) gets you a full local rig for under $600 that runs 7B models at 30+ tok/s. It idles at 15W and pulls 200W under load, so it's not massively worse for power than a Pi cluster once you factor in the throughput. Anyone shopping for hardware to run local LLMs should skip the SBC path unless the goal specifically requires a Pi's form factor (portable, silent, DIN-rail, whatever).

The Pi still wins if you already own one, if your workload is genuinely offline batch, or if you want to learn the LLM stack on constrained hardware. It doesn't win if you're comparing hardware options for interactive assistance.

Common pitfalls

• Trying to run a 7B model. It technically loads at q3 or q4 but tok/s is under 1. Not fun.
• Using an SD card for model storage. Model load takes forever and the card wears out fast. Use USB SSD.
• Skipping active cooling. Throttling costs you 25-35% of throughput within 2 minutes.
• Cheap USB-C PSU. Under-voltage warnings cause micro-freezes during generation. Use the official supply.
• Assuming CPU overclock is free. Verify thermals under sustained load; SD card corruption at high clocks is real.
• Expecting the same experience as a laptop LLM. It's not. It's a slower, more deliberate workflow.

Total-cost-of-ownership vs "just buy a 3060 rig"

A Pi 4 8GB build with case, PSU, SD card, and a USB SSD lands around $130. Idle power ~3W; loaded ~7W. Annual power at 24/7: ~$7. Total 24-month cost including power: ~$144. Impressively cheap.

A budget 3060 rig for the same money question: MSI RTX 3060 12GB + Ryzen 5 5600G + 32 GB DDR4 + case + PSU + 1 TB NVMe = about $850. Loaded power ~250W; annual 24/7 power ~$260. Total 24-month cost: $1370.

That's ~9.5× the cost. But it delivers 30-40× the throughput on real 7B models. The per-token cost is dramatically lower on the desktop.

What actually makes sense for a Pi 4 build

If you're committed to Pi 4 for the LLM slot, get:

• Base: Raspberry Pi 4 Model B 8GB with case + PSU + microSD.
• Storage upgrade: Crucial BX500 1TB SATA SSD in a USB 3.0 UASP enclosure. The single highest-impact upgrade after the Pi itself.
• Cooling: Argon ONE case with active fan, or Pimoroni Fan Shim. Non-negotiable for sustained work.
• Router/network: ethernet-connected if you're using it as a home-lab server; Wi-Fi is fine but adds jitter.

If you find yourself frustrated with the tok/s ceiling, that's the signal to move to a real GPU. Don't chase Pi-cluster setups (2-4 Pis running distributed inference) — the network and coordination overhead means you'd have been better off buying a single RTX 3060 12GB from the start.

Bottom line

Yes, a Raspberry Pi 4 8GB can run a local LLM in 2026 — a 3B model at q4_K_M with a decent SSD and active cooling delivers 2-3 tok/s and can host genuine batch and RAG workloads. It cannot compete with a discrete GPU for interactive use. If you already own a Pi 4 8GB, pair it with a Crucial BX500 1TB SSD and use it for offline batch work. If you're buying hardware for interactive assistance, get an RTX 3060 12GB or ZOTAC 3060 Twin Edge and skip the SBC route.

Related reading: our Pi Zero W retro emulation build, RTX 3060 local LLM guide, and Ryzen 5600G budget AI rig.

Sources

• Raspberry Pi Foundation — Pi 4 Model B specifications — SoC, RAM, thermal envelope reference.
• llama.cpp on GitHub — reference inference engine with ARM NEON kernels used for all Pi measurements.
• TechPowerUp — RTX 3060 specifications — comparison baseline for GPU throughput claims.