Running Local LLMs on the Raspberry Pi 4 8GB: 2026 Reality Check

Running Local LLMs on the Raspberry Pi 4 8GB: 2026 Reality Check

The short version on raspberry pi 4 8gb local llm 2026: yes you can run a small language model, no it will not replace your desktop GPU, and the practical ceiling is a 3-billion-parameter model at q4_K_M quantization yielding 2 to 4 tokens per second. Below is the troubleshooting and benchmark deep-dive for anyone who wants to actually ship a usable assistant on a Pi 4 8GB.

Why the Pi 4 8GB is the floor for local LLMs (and what won't work)

The Raspberry Pi 4 Computer Model B 8GB (ASIN B0899VXM8F) sits at the entry point of the local-LLM-on-edge category. The 4GB and 2GB variants do not have the memory headroom to load anything larger than TinyLlama at q2 quantization, and even that is unstable in practice. The Pi 4 8GB gives you roughly 6.5GB of usable RAM after the OS overhead (Raspberry Pi OS Lite, 64-bit, no desktop), which is enough for a 3B parameter model at q4 quantization with a small context window.

What will not work on the Pi 4 8GB: any 7B model in fp16 (14GB minimum), any 13B model regardless of quantization (the smallest q2 13B is around 5GB and runs at less than 1 token per second), any model that requires a CUDA backend (the VideoCore VI GPU has no usable LLM acceleration path in 2026), and any LangChain or LlamaIndex stack that loads embeddings into memory simultaneously with a 3B+ model.

What will work: TinyLlama 1.1B at q4 (1.5 to 3 tok/s, conversational), Phi-3-mini 3.8B at q4_K_M (2 to 4 tok/s, capable but slow), Qwen2.5-1.5B at q4 (3 to 5 tok/s, surprisingly good), Gemma 2B at q4 (2 to 4 tok/s), and StableLM Zephyr 3B at q4 (1.5 to 3 tok/s). For pi local llm troubleshooting purposes, TinyLlama and Qwen2.5-1.5B are the most forgiving starting points.

Models that actually fit: TinyLlama, Phi-3-mini, Qwen2.5-1.5B at q4

Per llama.cpp GitHub discussion threads and r/LocalLLaMA Pi-4 benchmark posts, the practical ceiling for a Pi 4 8GB is a 3B parameter model at q4_K_M. These fit in approximately 2.5GB RAM and yield 2 to 4 tokens per second on the Cortex-A72 cores. 7B models technically load (Mistral 7B at q2 fits in ~3GB) but generate at less than 1 token per second, which is below conversational usability for most users.

Our recommended starting model is TinyLlama 1.1B at q4_K_M. It loads in under 800MB RAM, runs at 2.5 to 4 tok/s on a Pi 4 8GB with passive cooling, and delivers coherent (if simple) chat responses. For coding assistance, Qwen2.5-1.5B-Instruct at q4 is the better pick. For reasoning-heavy queries, Phi-3-mini 3.8B at q4_K_M is the ceiling of what fits.

Benchmark table: tok/s across 5 models on Pi 4 8GB

All numbers from llama.cpp built with -mcpu=cortex-a72 -mtune=cortex-a72 -mfpu=neon-fp-armv8, Pi 4 8GB on a heatsink with active fan, ambient 22 degrees C.

Quantization matrix: q2/q3/q4/q5 — VRAM, tok/s, quality loss

Recommendation: use q4_K_M as the default. q5_K_M if you have headroom (3B models only). Avoid q2 unless you are explicitly testing the quality floor.

Common errors: OOM kills, swap thrashing, thermal throttling

The three most common pi local llm troubleshooting failure modes are out-of-memory kills, swap thrashing, and thermal throttling.

OOM kills typically appear as Killed in the llama.cpp output with no other context. Cause: model plus context plus KV cache exceeds 7.5GB. Fix: reduce context window with -c 1024, switch to a smaller quantization, or use a smaller model. Confirm with dmesg | grep -i oom and watch free -h during generation.

Swap thrashing manifests as initial generation that runs at expected tok/s, then drops to <0.3 tok/s and the SD card LED stays on solid. Cause: enabled swap on SD card while the model marginally exceeds RAM. Fix: disable swap entirely with sudo swapoff -a and reduce context window. SD card swap is unusable for LLM workloads and will also wear out the card in days.

Thermal throttling appears as gradually decreasing tok/s over a multi-minute generation with no error message. Cause: Cortex-A72 throttles at 80 degrees C. Fix: install a heatsink and active fan, or accept the throttle and run shorter sessions. vcgencmd measure_temp and vcgencmd get_throttled (return value 0x0 means no throttle history) are the diagnostic tools.

llama.cpp build flags that actually help on ARM

For raspberry pi llama.cpp builds, the flags that move the needle on a Pi 4 8GB are: -mcpu=cortex-a72, -mtune=cortex-a72, -mfpu=neon-fp-armv8, -O3, -DGGML_USE_OPENBLAS=ON (with libopenblas-dev installed), and -DGGML_NATIVE=OFF (to avoid mis-detection of ARM extensions). With these flags the Phi-3-mini benchmark above improves by approximately 25 to 35 percent over a default make build.

Avoid: -DGGML_USE_CUBLAS (no GPU on the Pi), -DGGML_USE_METAL (Apple-only), and any quantization downgrading at runtime. Build the GGUF model file at the target quantization once and load it directly.

Pi 4 vs Pi 5 vs Jetson Nano for LLM workloads

The Pi 5 8GB delivers approximately 2.0 to 2.5x the LLM throughput of the Pi 4 8GB at the same quantization, per Hackster.io benchmarks and r/LocalLLaMA posts. The Cortex-A76 cores and improved memory bandwidth are the differentiators. If you are starting fresh, Pi 5 is the better buy for LLM work.

The Jetson Nano (4GB) and the Orin Nano (8GB) are different beasts. The Orin Nano with TensorRT-LLM hits 15 to 25 tok/s on Phi-3-mini, which is 5 to 7x the Pi 4. The Orin Nano costs roughly 5x the Pi 4 8GB ($249 vs $75 in 2026), so it is the right pick when LLM is the primary workload, not a side project.

For pi 4 ollama users specifically: ollama is wrapping llama.cpp under the hood, so the same model and flag recommendations apply. Ollama defaults are conservative and work fine on the Pi 4 8GB out of the box, just expect slower performance than a direct llama.cpp build with the ARM-tuned flags above.

Verdict matrix: when the Pi is enough, when it isn't

The Pi 4 8GB is enough when: you want a learning project, your queries are short and infrequent, latency tolerance is high (5 to 30 second responses), and you are running offline or on a battery-powered device. The Pi 4 8GB is not enough when: you need real-time chat (streaming under 1s first token), you want any model larger than 4B parameters, you need vision or multi-modal support (use a Jetson Orin Nano), or you need throughput for multiple concurrent sessions.

FAQ (5 Q&A)

What's the largest model that actually runs usably on a Pi 4 8GB? A 3B parameter model at q4_K_M (Phi-3-mini, Qwen2.5-3B). 7B models load but run at sub-1 tok/s, which is below conversational usability.

Should I use Ollama or llama.cpp directly on the Pi 4? Ollama for ease of setup, llama.cpp for maximum performance. The cost of using ollama is roughly 15 to 25 percent throughput compared to a tuned llama.cpp build.

Do I need a heatsink and fan to run LLMs on the Pi 4? Yes. Without active cooling the Pi 4 throttles within 1 to 2 minutes of sustained inference and tok/s drops to half.

Is swap useful for running larger models on the Pi 4 8GB? No. SD card swap is so slow that a model that thrashes is functionally unusable. NVMe-over-USB3 swap helps marginally but still costs more than dropping a quant level.

Can I fine-tune on a Pi 4 8GB? Not in any practical sense. LoRA fine-tuning of even TinyLlama takes 24+ hours per epoch on the Pi 4. Use Colab or a cloud GPU for training and deploy the result to the Pi.

Citations and sources

• llama.cpp GitHub repository and ARM build threads
• r/LocalLLaMA Pi 4 and Pi 5 benchmark megathreads
• Hackster.io single-board computer LLM benchmarks
• Raspberry Pi Foundation thermal and throttling documentation
• Hugging Face Phi-3, Qwen2.5, and TinyLlama model cards

Related guides

• Best Gaming GPUs for 1080p High-Refresh Rate (2026)
• Best Internal SSDs for Gaming PCs (2026)
• Best CPU Cooler for Ryzen 7 5800X Gaming Builds (2026)
• Best PlayStation 5 Racing Wheels for Sim Racing (2026)