Running a Local LLM on a Raspberry Pi 5 With llama.cpp: Real tok/s on 1B-8B Models

If you want one line: a Raspberry Pi 5 (16 GB) running llama.cpp with Q4_K_M-quantized GGUF models pushes about 14 tok/s on a 1.1B model (TinyLlama), 9-10 tok/s on a 1B-class model (Llama-3.2 1B, Gemma 3 1B), ~5-6 tok/s on a 3-3.8B model (Phi-3 Mini, Llama-3.2 3B), and ~1-2 tok/s on a 7-8B model (Llama-3.1 8B, Mistral-7B v0.3). Numbers are measured on Raspberry Pi OS Bookworm 64-bit, an actively-cooled Pi 5, four threads, and a build with NEON SIMD + OpenMP enabled. The Pi 5's quad-core Arm Cortex-A76 at 2.4 GHz — a real cellphone-class CPU, not the toy 1.5 GHz A72 in the Pi 4 — is fast enough for usable interactive output on anything up to 3-4B parameters. Above 7B you are batch-only.

Why this is worth measuring in 2026

The Pi 5 launched in October 2023 with a roughly 2-3× CPU performance uplift over the Pi 4 and a real PCIe lane. In the years since, llama.cpp has shipped roughly weekly with kernel optimizations specifically for Arm — including hand-tuned NEON SIMD paths that target the A76's vector units. The result: an $80 board now runs small LLMs at usable interactive speed. Not GPU-class. Not server-class. But usable for personal assistants, document Q&A, code-snippet completion, and a long list of edge applications where shipping a query to OpenAI is the wrong answer for cost, privacy, or latency reasons.

What you do not get on a Pi 5 is fast 7B+ inference. The community headline numbers from late-2024 of "5-10 tok/s on 7B models" were almost always quoted on cherry-picked Q4_0 builds with overclocked SoCs and tiny context windows. With a stock Pi 5 at default clocks and a 4K context, a 7-8B Q4_K_M model lands at 1-2 tok/s — fine for an overnight summarization job, painful as an interactive assistant. This article is written to give you those numbers honestly, not the inflated ones.

Hardware setup that matters

1. Pi 5 16 GB. The 16 GB variant launched January 2025 at $120. For LLMs above 3B parameters, that's the model to buy. The 8 GB Pi 5 ($80) handles up to ~3-4B comfortably; the 4 GB Pi 5 chokes past 1.5B because the OS + llama.cpp + KV cache eat ~2 GB before the model loads. The 2 GB Pi 5 ($50) is for IoT, not LLMs.
2. Active cooling. The Pi 5's BCM2712 SoC throttles from 2.4 GHz to 1.5 GHz when it hits 80 °C, and reaches that threshold within ~60-90 seconds of sustained LLM inference without a fan. The official Active Cooler ($5), the Argon NEO 5 case with built-in fan ($25), and the Pimoroni NVMe Base + active cooler combo all keep the SoC at 65-72 °C under sustained 100% CPU. Buy one of them. This is not optional — without it, your decode speeds quietly drop 25-35% after the first minute and you won't know why.
3. NVMe SSD via the M.2 HAT. The Pi 5 has a single PCIe 2.0 ×1 lane (5 GT/s line rate, ~500 MB/s after 8b/10b overhead). An NVMe SSD on a $25 M.2 HAT loads a 4 GB GGUF in ~9 seconds vs ~28 seconds from a Class 10 microSD card. Critical for iterating on prompts or swapping models.
4. 27 W USB-C PSU. The Pi 5's official PSU delivers 5 A at 5.1 V (27 W). Under sustained LLM inference plus NVMe plus USB peripherals, an undersized 3 A supply triggers brownouts that show up as random llama.cpp crashes with no obvious cause. Use the official 5.1 V / 5 A supply or a quality third-party equivalent.

Total recommended kit: Pi 5 16 GB + active cooler + 256 GB NVMe + M.2 HAT + official PSU = roughly $220-240 USD.

Software setup

Use Raspberry Pi OS Bookworm (64-bit). The 32-bit OS does not work for any LLM heavier than 1B — pointer math caps virtual address space at 3 GB.

# Update + tooling
sudo apt update && sudo apt install -y build-essential cmake git curl

# llama.cpp clone + build with NEON + OpenMP
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_NATIVE=ON -DGGML_OPENMP=ON
cmake --build build --config Release -j4

-DGGML_NATIVE=ON tells the build to detect host CPU features. On a Pi 5 this enables NEON, the Cortex-A76's dot-product extension, and FP16 vector arithmetic. Skipping this flag costs 30-40% in tokens/sec — verify the compile output mentions NEON, DOTPROD, and FP16 in its CPU-feature dump before you trust your numbers.

-DGGML_OPENMP=ON enables the OpenMP runtime so llama.cpp can launch one inference thread per core. Pin to four threads (-t 4) — the Pi 5 has four cores, no SMT, and benchmarking shows pinning to fewer than four reduces throughput linearly, while pinning to more than four introduces context-switch overhead that costs another 8-12%.

Models that fit and where to get them

All numbers below assume Q4_K_M quantization (4-bit weights, K-quant family, mixed precision per layer). Pull GGUF weights from Bartowski on Hugging Face — they are the best-maintained community uploads in 2026 (TheBloke, the previous standard, stopped uploading in mid-2024 and the older repos no longer track upstream tokenizer changes). For first-party models check the model author's own GGUF release, when one exists.

Download a model:

mkdir -p models
curl -L -o models/llama-3.2-3b-q4km.gguf \
 https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF/resolve/main/Llama-3.2-3B-Instruct-Q4_K_M.gguf

Run interactively:

./build/bin/llama-cli \
 -m models/llama-3.2-3b-q4km.gguf \
 -t 4 -c 4096 --temp 0.7 -p "Explain GraphQL in two paragraphs"

(If you were on an older clone where the binary was named main, the upstream rename to llama-cli happened mid-2024. Pull and rebuild.)

Real-world benchmarks

Tested on Raspberry Pi 5 16 GB, active cooler, Raspberry Pi OS Bookworm 64-bit, llama.cpp built from master in May 2026, four threads, Q4_K_M quantization, 2K context window. Numbers are the mean of five llama-bench runs each generating 128 tokens after a 16-token prefill, cross-checked against published numbers from Stratosphere Labs' Pi 5 LLM evaluation and the llama.cpp issue tracker.

Reading the table: the decode column is what feels like "typing speed" to a user. Above 10 tok/s is comfortable to read in real time. Around 5 tok/s is noticeably slow but still usable for short responses. Below 3 tok/s is impractical for interactive use — fine for queued summarization, painful as a chat partner.

Compared with a modern desktop Ryzen 9 (16 cores, no GPU offload) running the same llama.cpp build, the Pi 5 is roughly 4-5× slower at 1B, 8× slower at 3B, and 12× slower at 7-8B. The gap widens with model size because the Pi 5's LPDDR4X-4267 memory bandwidth (~17 GB/s peak, ~12 GB/s sustained) becomes the limit as the weight matrix gets bigger — desktop DDR5-6000 dual-channel delivers 90+ GB/s. LLM inference is overwhelmingly a memory-bandwidth workload at this scale, and that ratio is the single biggest predictor of Pi-vs-desktop performance.

Compared with a Pi 4 (8 GB, same llama.cpp build), the Pi 5 is roughly 2.5-3× faster at every model size. Most of the gain is the A76 cores and the LPDDR4X memory upgrade; the rest comes from the wider memory bus.

Quantization choice

Q4_K_M is the right default for the Pi 5. It's the smallest quant that holds quality at 7-8B model size and the fastest quant that fits within the Pi 5's compute budget. Other options:

• Q8_0 — 8-bit weights, twice the disk and memory, and about half the decode speed because memory bandwidth saturates faster. Use only if your application demands the slight quality bump (mostly visible on instruction-tuned models with subtle reasoning chains).
• Q3_K_M — 3-bit weights, smaller, faster, but visibly degrades on 3B+ models. Notable quality regression on Llama 3.x; OK on TinyLlama.
• Q5_K_M — 5-bit, marginal quality improvement over Q4_K_M, marginal speed cost. Worth trying if you are inference-quality-limited and not memory-bandwidth-limited.
• IQ4_XS / IQ3_XS (importance-matrix quants) — newer quants that preserve quality at smaller sizes by spending more bits on "important" weights. On the Pi 5's NEON path these are 10-15% slower than K-quants due to less-optimized SIMD kernels, but they save 10-20% of RAM. Worth experimenting with on the 16 GB Pi for 7B+ models.

Run a comparison yourself:

./build/bin/llama-bench \
 -m models/llama-3.2-3b-q4km.gguf \
 -m models/llama-3.2-3b-q5km.gguf \
 -m models/llama-3.2-3b-iq4xs.gguf \
 -t 4

llama-bench prints a clean comparison table and is the cleanest way to validate your own setup before you build anything on top of it.

Use cases that actually work

1. Personal assistant on the LAN. Run a 1-3B model on the Pi 5 wired into your home network. Expose an OpenAI-compatible API endpoint via llama-server. Now your home apps (Obsidian plugins, Raycast extensions, anything that speaks OpenAI) can hit a local model with zero per-token cost. Llama-3.2 1B is the sweet spot for "answers within a few seconds, decent at common-sense follow-ups."
2. Document Q&A with RAG. A Pi 5 running a 3B model plus a small embedding model (BGE-small, ~125 MB) handles a 5,000-document RAG corpus comfortably. Embedding throughput is around 600 chunks/min on the BCM2712, and query latency stays under 3 seconds on Top-K = 5.
3. Code-snippet completion. Phi-3 Mini at 5-6 tok/s is workable for a VS Code autocomplete-on-demand pattern (not always-on ghost text — too slow for that). Not Copilot-fast, but it is free, offline, and never trains on your code.
4. Edge IoT inference. Drive a Pi 5 from a smart-home hub and run intent classification with a 1B model locally. Total power draw under load: 7-10 W. Easily runs on a USB-PD battery for days.
5. Air-gapped tools. Hospitals, manufacturing plants, and security-sensitive teams have use cases where data cannot leave premises. A Pi 5 cluster is the cheapest air-gapped LLM platform on the market.

Common pitfalls

1. Skipping -DGGML_NATIVE=ON. Easy to forget on a fresh clone. Costs about 30% throughput. Always verify the build log shows NEON, DOTPROD, and FP16 in its detected CPU features.
2. Running five or more threads. The Pi 5 has four physical cores. -t 5 introduces a context-switch storm that drops throughput 8-12%. Always -t 4 unless you are deliberately running a daemon next to inference.
3. No active cooling. Without a fan, the BCM2712 throttles within 60-90 seconds of sustained inference. You will see decode speed quietly drop 25-35% and not know why. The Pi-5 idle-only "tiny heatsink" trick from the Pi-4 era does not work here.
4. microSD boot. Class 10 microSD cards have unpredictable random-read latency, especially for large sequential reads of GGUF weights. Model load times triple. Use NVMe.
5. Trying to fit a 13B model on 16 GB. Q4_K_M of a 13B is ~7.5 GB on disk, ~9 GB at inference. With OS + llama.cpp + KV cache you are at 11 GB tight, and longer contexts OOM-kill. Cap at 7-8B on the 16 GB Pi.
6. Inflating context window to 32k. The KV cache scales linearly with context length × hidden-state size. A 3B model at 4096 tokens takes ~600 MB KV; at 32768 it takes ~5 GB. Throughput drops because memory bandwidth saturates. Keep context to 4-8k on the Pi 5.
7. Forgetting OpenMP. A llama.cpp build without -DGGML_OPENMP=ON falls back to a single-thread inner loop. Decode speed drops by roughly a factor of three.
8. Cross-compiling on x86 with wrong CPU flags. If you cross-compile on a desktop, pass -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16; otherwise the binary will not use the A76's dot-product extension and silently underperforms.

When NOT to use a Pi 5 for LLMs

If your usage pattern is "burst high-throughput inference for hours at a time," a Pi 5 is a poor fit. The CPU and memory-bandwidth ceiling caps you well below what a used $250 GPU like an RTX 3060 12 GB handles. The Pi 5 is the right answer for:

• Always-on, low-rate, edge inference (1-10 queries/min) where total power matters
• Air-gapped deployments
• Hobbyist LLM exploration without a beefy desktop
• LAN-local OpenAI-compatible endpoints for personal projects
• Models in the 1-3B range that don't need fast 7B+ throughput

It is not the right answer for:

• A team's shared coding-assistant backend (latency-sensitive, high-throughput — buy a used 4060 Ti 16 GB instead)
• Image-generation diffusion models (no GPU, hours per image)
• Anything requiring sustained > 10 tok/s on a 7B+ model

Real-world numbers — comparing to alternatives

The Pi 5 is the cheapest entry to credible LLM inference — credible meaning "1-3B model in real time, 7B+ as a batch job." The Jetson Orin Nano Super (NVIDIA dropped the price to $249 at the end of 2024) is the best $/tok-per-second under $300 if you want real-time 7B inference at low power. The Mac Mini M4 is the best $/tok-per-second above $500 thanks to unified memory and Metal acceleration. A used RTX 3060 12 GB on Facebook Marketplace remains the best raw $/perf for anyone who already owns a PC chassis with a free PCIe slot and a 500 W PSU.

Bottom line

A Pi 5 16 GB with active cooling, an NVMe SSD, and llama.cpp built with NEON + OpenMP gets you usable interactive LLM inference at 5-14 tok/s on 1-3B models for about $220-240 of hardware and 8 W of sustained power. That's the cheapest credible LLM platform available in 2026. Pair it with a local API endpoint via llama-server, wire it into your home network, and you have personal AI infrastructure that costs nothing per query, runs offline, and stays out of vendor lock-in. Just calibrate expectations: 7-8B models on the Pi 5 are a batch tool, not an interactive one.