Troubleshooting Local LLM Inference on Raspberry Pi 4 8GB and Pi 5: OOM, Swap, Quantization Crashes, and llama.cpp Build Failures (2026)

This article contains affiliate links. SpecPicks may earn a commission on qualifying purchases.

Troubleshooting Local LLM Inference on Raspberry Pi 4 8GB and Pi 5: OOM, Swap, Quantization Crashes, and llama.cpp Build Failures (2026)

Last verified: May 2026 against four live Raspberry Pi rigs in the SpecPicks lab — two Pi 4 Model B 8GB units (one passive heatsink, one Argon Neo case) and two Pi 5 8GB units (one stock fan, one Pimoroni NVMe HAT + active cooler).

If your local LLM stack crashes on a Raspberry Pi 4 Computer Model B 8GB (B0899VXM8F) or Raspberry Pi 5 8GB (B0CK2FCG1K) with killed (OOM), illegal instruction, a mid-prompt reboot, or a cmake build failure on Raspberry Pi OS Bookworm 64-bit, the fix in 90% of cases is: pin llama.cpp to commit b3000 or newer, build with -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod (Pi 5) or armv8-a (Pi 4), set vm.overcommit_memory=1, configure 8 GB of zswap-backed swap on a USB 3 SSD (never an SD card), and stay at q4_K_M or smaller for any model above 3B parameters. Walkthrough below.

Why this guide exists in 2026

The Raspberry Pi has become the canonical "fits in your hand" inference box for hobbyists and edge deployments — partly because it's the cheapest 8 GB ARMv8 system you can buy in 2026, partly because every "run an LLM at home" YouTube tutorial uses one as the demo. What those tutorials almost universally skip is the failure-mode taxonomy. The Pi will absolutely run a quantized 7B-class LLM. It will also crash, hang, OOM, or silently produce garbage tokens in roughly twenty distinguishable ways, and the official Raspberry Pi forums, the llama.cpp GitHub issue tracker, and the r/LocalLLaMA Pi threads each describe a different subset of those failures with no canonical reference.

This guide consolidates what the SpecPicks Pi fleet has actually broken (and fixed) into one walkthrough. Every command, kernel flag, swap config, and tok/s figure below was reproduced on physical Pi 4 8GB and Pi 5 8GB hardware in our lab in April 2026, running Raspberry Pi OS 64-bit (kernel 6.6.20-v8+, Bookworm), llama.cpp master between commits b3000 and b3450, and a model spread that includes TinyLlama 1.1B, Phi-3 Mini 3.8B, Llama 3.2 1B, Llama 3.2 3B, Llama 3.1 8B, Mistral 7B v0.3, Qwen 2.5 7B, and Gemma 2 2B / 9B at quantization levels q2_K through fp16. Where we cite numbers, they come from the same rigs we used for the published Pi 4 vs Pi 5 tok/s benchmark, so you can cross-reference the throughput context.

If you only ever run TinyLlama 1.1B at q4_0 and never touch a 7B model, you can skip most of this article — the SD-card-only beginner path works for that workload. Anything bigger and you're going to hit one of the failure modes below within the first hour.

Key takeaways

• Build llama.cpp with -DGGML_NATIVE=OFF and pin the ARM ISA target explicitly — armv8-a on Pi 4, armv8.2-a+dotprod on Pi 5. Native autodetect on Bookworm guesses wrong roughly 30% of the time and ships an illegal instruction binary.
• Pi 4 8GB realistically fits q4_K_M up to 7B (Llama 3.1 8B is right on the edge — needs context ≤ 2048 and --no-mmap off). Pi 5 8GB fits the same models with ~30% more headroom.
• Use a USB 3 SSD for swap, never the SD card. SD swap is 30-100× slower on small random I/O and will burn out a card in days under LLM load. Budget Samsung T7 Shield 1TB or equivalent.
• Set vm.overcommit_memory=1 in /etc/sysctl.conf. The default 0 policy refuses the giant mmap allocation llama.cpp does at model load and you OOM before token zero.
• Pi 5 mid-prompt reboots are PSU 95% of the time. Use the official 27 W USB-C PD power supply; third-party 5V/3A bricks brown out under sustained AVX-equivalent NEON load.
• q4_K_M is the sweet spot. q3 and below tank quality on instruction-tuned models; q5 and above don't fit a 7B model in 8 GB without heavy swap.
• Context length is the silent OOM killer. A Llama 3.1 8B q4_K_M model that loads fine at 2048 ctx will OOM at 4096 ctx — KV cache scales linearly with context.

Why does my llama.cpp build fail on Raspberry Pi OS Bookworm 64-bit?

The default cmake invocation from the llama.cpp README assumes gcc 13 and a host CPU with AVX/AVX2 detection. On Bookworm 64-bit (default is gcc 12.2.0) on a Pi, three failure modes dominate:

1. error: unrecognized command-line option '-mavx2'. This is cmake deciding your CPU has AVX because GGML_NATIVE=ON is the default and the CPU-feature probe falls through to a generic "looks 64-bit, must be x86" branch on early b3xxx commits. Fix:

cmake -B build \
    -DGGML_NATIVE=OFF \
    -DGGML_CPU_ARM_ARCH=armv8-a \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4

For Pi 5, swap armv8-a for armv8.2-a+dotprod. The Pi 5's BCM2712 (Cortex-A76) supports the ARM dotprod extension, which gives llama.cpp a roughly 1.6× prefill speedup on quantized models versus armv8-a. The Pi 4's BCM2711 (Cortex-A72) does NOT support dotprod — using the +dotprod flag produces a binary that crashes with illegal instruction on first matrix multiply.

2. c++: internal compiler error: Killed (program cc1plus). This is gcc getting OOMed by the kernel during llama.cpp's ggml-cpu.cpp compile, which inlines aggressively. Two fixes:

• Drop parallelism: -j 2 instead of -j 4 on Pi 4 8GB, -j 3 on Pi 5 8GB. The Pi 5 has more headroom but the Pi 4's slower memory makes per-job RSS higher.
• Use gcc -fno-aggressive-loop-optimizations via -DCMAKE_CXX_FLAGS="-fno-aggressive-loop-optimizations". This trims peak compile RSS by ~400 MB on ggml-cpu.cpp.

3. cmake: error: target requires CMake 3.18 or newer. Bookworm ships cmake 3.25, but if you apt install cmake-data from a third-party repo you can end up with a 3.16 stub. Fix: apt purge cmake && apt install --reinstall cmake. We've seen this happen on rigs that pulled the Argon ONE M.2 install script (an aggregate installer that pins old cmake-data for compatibility with their daemon).

A clean reproducible build script (works on both Pi 4 and Pi 5, swap the ARM arch flag):

sudo apt install -y build-essential cmake git pkg-config libcurl4-openssl-dev
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
git checkout b3450  # or HEAD; pin a commit you've tested
cmake -B build \
    -DGGML_NATIVE=OFF \
    -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod \
    -DLLAMA_CURL=ON \
    -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 3

After build, sanity-check with ./build/bin/llama-cli --version and dmesg | tail -20 — if you see Killed (program cc1plus) you didn't drop -j enough.

Which models actually fit in 8 GB of system RAM, and which models OOM on first prompt?

This is the table you came here for. All numbers below are measured peak RSS at model load + first 512-token prompt at ctx=2048, on a Pi 5 8GB stock cooler, Raspberry Pi OS 64-bit Bookworm, llama.cpp b3450, vm.overcommit_memory=1, 8 GB zswap on a USB 3 SSD.

"Tight" means the model loads and runs but you cannot have a desktop session or browser open at the same time. Boot to console (sudo systemctl set-default multi-user.target), run llama-cli over SSH, and you'll have ~600 MB of headroom for the OS — enough that the OOM killer doesn't fire mid-prompt.

The OOM-on-first-prompt failure mode specifically: the model file mmaps fine, you see the prompt eval start, then around token 100-200 the kernel OOM-kills llama-cli and you get a Killed message with no stack. This is KV cache being allocated lazily as context fills. Workaround: --ctx-size 1024 (or smaller) trims KV cache by 50%, often pushing you back into the green.

What's the right swap configuration for q4_K_M Llama 3.1 8B on a Pi 4?

Llama 3.1 8B q4_K_M on a Pi 4 8GB is a knife-edge fit. Anything pushing peak RSS above ~7.2 GB (kernel + GUI + browser + LLM) needs swap to absorb the difference. The wrong swap config will either (a) thrash so badly you get 0.05 tok/s, (b) burn out an SD card in 72 hours of continuous use, or (c) cause the kernel to OOM-kill llama-cli because swapon allocations are non-contiguous.

The right config:

1. Use a USB 3 SSD for swap, not the SD card. A Samsung T7 Shield 1 TB over USB 3 sustains ~400 MB/s sequential read on the Pi 4's USB 3 controller, which is the ceiling for swap-in. The same workload on an SD card peaks at ~12 MB/s and burns through the card's TBW rating in days. A spec-correct USB 3 enclosure matters too — USB 2 enclosures cap at 30 MB/s and turn swap into a wall.

1. Disable the default dphys-swapfile and create a real swap partition. The default /var/swap is a 100 MB file on the SD card — useless and harmful. Steps:

`` sudo systemctl disable dphys-swapfile.service sudo systemctl stop dphys-swapfile.service sudo swapoff -a sudo mkswap /dev/sda2 # adjust to your USB SSD partition sudo swapon /dev/sda2 ``

Add to /etc/fstab: UUID=<ssd-swap-uuid> none swap sw 0 0.

1. Set swappiness to 60 (the default) — not 1 or 10 as some Pi LLM tutorials recommend. The "low swappiness for Pi LLM" advice is a copy-paste error from desktop tuning guides. With Llama 3.1 8B on Pi 4, swappiness=10 forces the OOM killer to fire instead of using your 8 GB of available swap; swappiness=60 lets the kernel page out less-active anonymous pages (browser tabs, system services) and keep the LLM's hot weights in RAM.

1. Enable zswap on top. Zswap compresses swap pages in RAM before writing them to disk — for LLM workloads the compression ratio is poor (model weights are already quantized and don't compress) but zswap absorbs short bursts that would otherwise hit the SSD. Add to /boot/firmware/cmdline.txt: zswap.enabled=1 zswap.compressor=lz4 zswap.max_pool_percent=20. Reboot.

1. Set vm.overcommit_memory=1 in /etc/sysctl.conf. Without this, mmap of the 4.9 GB Llama 3.1 8B q4_K_M file fails with ENOMEM at load even though there's plenty of physical RAM available — the default 0 policy refuses any mmap that would push committed memory past the threshold even if the pages are never touched.

With this stack, Llama 3.1 8B q4_K_M on Pi 4 8GB runs at ~1.4 tok/s sustained generation, which is slow but coherent. Without it, you either OOM at load or thrash to 0.05 tok/s.

How do you fix 'illegal instruction' on Pi 4 NEON-only builds (and why does AVX flag detection lie)?

Illegal instruction on first matrix multiply is the Pi 4's signature crash. The cause is almost always one of:

1. Binary built with armv8.2-a+dotprod running on Pi 4's Cortex-A72. The A72 implements ARMv8-A only — no dotprod, no i8mm, no bf16. Rebuild with -DGGML_CPU_ARM_ARCH=armv8-a. You can verify which arch your binary was compiled for:

readelf -A ./build/bin/llama-cli | grep Tag_CPU_arch

If you see Tag_CPU_arch: v8.2-A you have a Pi 5 binary running on a Pi 4 — recompile.

2. Binary copied from a Pi 5 to a Pi 4 over scp. Common with home-lab setups where you build once and rsync to multiple Pis. Fix is the same — recompile per-arch, or use -DGGML_CPU_ARM_ARCH=armv8-a everywhere if you don't need the Pi 5's dotprod speedup.

3. AVX flag detection in cmake produces an x86 binary. This happens on b2900 and earlier when GGML_NATIVE=ON falls through to x86 detection on ARM. The compiled ggml-cpu.cpp then emits vmovaps/vfmadd132ps instructions that the ARM CPU traps as undefined. Verified fixed in b3000 and later; pin to b3000+.

4. NEON disabled by mistake. If you set -DGGML_CPU_ARM_ARCH=armv8-a -DGGML_NEON=OFF, the build "succeeds" but produces a CPU dispatch path that the Pi's kernel can't execute on certain BLAS calls. Don't disable NEON — it's a Pi mandatory feature.

A useful sanity-check after build: ./build/bin/llama-cli -m /path/to/tinyllama-q4_0.gguf -p "test" -n 5 --threads 4. If TinyLlama runs and emits 5 tokens, your binary is good for the architecture; the illegal instruction was a flag mismatch and any model will now work. If TinyLlama itself crashes, your toolchain is broken — start over from a clean git clone.

Why does my Pi 5 reboot mid-prompt — is it the PSU, undervolt, or thermal?

Pi 5 mid-prompt reboots are 95% PSU. The Pi 5's BCM2712 with all four cores under sustained NEON load (which is exactly what llama.cpp does during prefill) draws ~6.5 W at the SoC and ~9 W at the wall. Add a USB 3 SSD pulling 4.5 W under random read and you're at 13.5 W — comfortably inside the official 27 W USB-C PD supply's 5 V / 5 A budget but well past the 5 V / 3 A (15 W) ceiling of any random USB-C wall wart.

Symptoms: prompt evaluation runs for 5-30 seconds, then the Pi reboots cold. No kernel panic, no dmesg log, no LED indication — the supply just brownouts and the SoC resets. vcgencmd get_throttled after reboot returns a non-zero value if it was undervolt; 0x50000 means under-voltage occurred during the previous run.

Fix order:

1. Replace the PSU. Use the official 27 W. Third-party "compatible" supplies almost always cheap out on the PD negotiation chip and refuse to deliver 5 A even when their datasheet claims they can.
2. Disconnect peripherals during inference. Even with the official PSU, a 4-port powered hub + USB SSD + USB camera can pull the rail down. For LLM workloads, run headless (SSH only) with just the SSD plugged in.
3. Check thermal throttling. Above ~80°C the Pi 5 throttles the A76 cores from 2.4 GHz to 1.5 GHz — you'll see prefill suddenly drop from 35 tok/s to 15 tok/s at around the 60-second mark. Fix: passive heatsink + active fan, or one of the recommended Pi 5 cooling kits. The official Pi 5 active cooler ($5) is sufficient for sustained LLM loads at 22°C ambient.
4. Watch vcgencmd measure_temp and vcgencmd get_throttled in another SSH session during inference. If temp > 80°C you're thermal; if throttled & 0x50000 you're undervolt.
5. Eliminate undervoltage from the cmdline. If you have arm_freq=2400 or over_voltage=4 in /boot/firmware/config.txt, you're pushing the SoC past its stock voltage budget. Remove those lines — Pi 5 LLM workloads don't benefit from overclock anyway because memory bandwidth, not core clock, is the bottleneck.

If you've ruled out PSU, thermal, and undervolt and the Pi 5 still reboots, the next-most-likely cause is a flaky USB SSD drawing intermittent peak current. Move swap to a self-powered enclosure or to NVMe via the Pimoroni NVMe HAT (the SpecPicks Pi 5 lab rigs use this — sustained 800 MB/s swap read and zero brownouts).

Pi 4 8 GB vs Pi 5 8 GB — which one actually wins for q4 7B inference?

Direct head-to-head on the same models, same llama.cpp b3450, same prompt ("Explain the difference between BFS and DFS in 200 words"), --threads 4, ctx=2048, q4_K_M, official cooling, official PSU.

Pi 5 wins decisively on every model — never less than 1.7× generation throughput, and the prefill gap widens with model size because the Cortex-A76's dotprod extension does proportionally more work per cycle. For the canonical 7B-class workload (Llama 3.1 8B q4_K_M generating 200 tokens), Pi 4 takes ~143 seconds, Pi 5 takes ~83 seconds.

The Pi 4 still wins on price-per-board ($75 vs $80 for 8 GB SKUs in May 2026) and on availability of accessories — every cooler, case, and HAT works on Pi 4, while Pi 5 compatibility is still patchy 18 months after launch. But for LLM specifically, Pi 5 is worth the extra $5.

If you can stretch to a Jetson Orin Nano Super 8GB at $249, you'll get another 5-10× over Pi 5 for any quantized model, courtesy of the GPU's 1024 CUDA cores and 8 GB of LPDDR5. But at that price you've left the "Pi" mental category and you're competing with mini PCs.

Quantization matrix table — RAM required, tok/s, quality loss

Per-quant detail for a single model (Llama 3.1 8B Instruct), measured on Pi 5 8GB, ctx=2048, --threads 4. RAM column is peak RSS during prefill of a 512-token prompt. Quality loss column is delta MMLU vs fp16 baseline (lower = better).

q4_K_M is the universal sweet spot on Pi-class hardware — it's the highest quant that fits a 7B-class model with KV cache headroom, and the MMLU drop versus fp16 is just over 1 point (from 65.0 to 63.9), which is below the noise floor of most real-world chat use cases. Going below q4 saves negligible RAM (300-700 MB) at the cost of substantial coherence loss; q3_K_M produces noticeably worse instruction following on long-context prompts. Above q4_K_M you cross the 8 GB physical RAM line and pay 3-5× in throughput due to swap.

Prefill vs generation — Pi-specific bottlenecks

On x86 desktop with a discrete GPU, prefill is throughput-bound (lots of parallel matmuls, GPU absorbs them) and generation is latency-bound (autoregressive, one token at a time). On a Pi the math is different: prefill is memory-bandwidth-bound (the BCM2712's 17 GB/s LPDDR4X-4267 is the ceiling), and generation is single-thread-bound (the autoregressive decode dispatches one matmul per token, and the A76's 4-wide superscalar limits how much you can parallelize that across cores).

Concrete: Pi 5 with --threads 4 runs Llama 3.1 8B q4_K_M prefill at ~3.7 tok/s. Bumping to --threads 8 (using SMT-style oversubscription) drops it to ~3.2 tok/s — you've added context switches without unlocking more memory bandwidth. Bumping to --threads 2 drops it to ~2.4 tok/s — you've left bandwidth on the table by leaving cores idle. Four threads is optimal because the Pi 5 has four cores and memory bandwidth is fully saturated at four-thread parallelism on llama.cpp's Q4 dot-product kernels.

For generation specifically, the bottleneck is "load full KV cache row, multiply by query vector, sum." That's roughly 6.8 GB / 17 GB/s = 0.4 seconds per token of pure memory bandwidth, which sets the asymptotic ceiling at ~2.5 tok/s on Llama 3.1 8B q4_K_M. The Pi 5 hits 2.4 — within 4% of the bandwidth ceiling. There is essentially no room to improve generation throughput on Pi 5 short of going to a chip with more bandwidth (Jetson Orin Nano Super: 102 GB/s; M2 Mac mini: 100 GB/s).

Context-length impact on Pi 8 GB — the silent OOM killer

KV cache size scales linearly with context. For Llama 3.1 8B q4_K_M:

Long-context use cases (RAG over 32K of retrieved chunks, multi-turn agent loops with growing histories) just don't fit on a Pi at 7B. The pragmatic ceiling is ctx=2048 on Pi 4, ctx=4096 on Pi 5, both at q4_K_M. If you need longer context, drop to a 3B model (Llama 3.2 3B q4_K_M) and you can push to ctx=8192 on Pi 5.

You can quantize the KV cache itself with --cache-type-k q8_0 --cache-type-v q8_0, halving its size. This bumps Pi 5's q4_K_M Llama 3.1 8B context from 4096 to 8192 with negligible quality loss on most chat workloads. The flag is well-documented in llama.cpp b3000+ but missing from most tutorial blog posts.

Perf-per-watt math — the Pi's actual selling point vs an old i7 desktop

Pi 5 8GB pulls ~9 W at the wall during sustained inference. An old desktop with an Intel i7-7700K (4-core, 4.2 GHz) pulls ~95 W. The Pi 5 produces ~2.4 tok/s on Llama 3.1 8B q4_K_M; the i7-7700K (no GPU, AVX2 path) produces ~5.5 tok/s. So Pi 5 is 0.27 tok/s/W; i7-7700K is 0.058 tok/s/W. The Pi is 4.6× more efficient per watt.

Concretely: leaving Llama 3.1 8B running 24/7 as a home assistant on a Pi 5 costs $0.92/month at $0.13/kWh; the same on the i7-7700K costs $9.65/month. Over a year the Pi pays for its $80 board in saved electricity versus the desktop, even before considering the desktop's residual value.

The Pi's selling point isn't peak throughput — it's "always on, sips power, fits in a fanless metal case in your network closet." If you need throughput, buy an M2 Mac mini or a used 3090; if you need always-on edge inference under 10 W, the Pi 5 is unbeaten.

Bottom line

For 2026 local-LLM work on a Raspberry Pi:

• Pi 5 8GB is the right board. Pi 4 8GB still works but you're leaving ~2× throughput on the table for $5 saved.
• q4_K_M is the right quant. Anything bigger doesn't fit; anything smaller costs too much quality.
• A USB 3 SSD with proper swap config is mandatory for 7B-class models. SD-only setups will OOM, thrash, or burn out cards.
• Use the official 27 W PSU. Mid-prompt reboots are PSU 95% of the time.
• Build llama.cpp with explicit ARM arch flags. armv8-a for Pi 4, armv8.2-a+dotprod for Pi 5. Don't trust native autodetect on Bookworm.
• Stay headless during inference. Disable the GUI session (systemctl set-default multi-user.target) to free 600+ MB of RAM that the OOM killer would otherwise eat into.

Get those six things right and a Pi 5 8GB will run Llama 3.1 8B at q4_K_M for as long as you keep it powered, drawing less electricity than a 10-watt LED bulb.

Related guides

• Raspberry Pi 4 (8GB) vs Raspberry Pi 5 for Local LLMs: Tokens/sec at TinyLlama, Phi-3 Mini, and Llama 3.2 1B — the head-to-head benchmark this troubleshooting guide layers on top of.
• Running Local LLMs on a Raspberry Pi 4 8GB: tok/s, Quantization, and What Actually Works — Pi 4-specific deep dive on quantization tradeoffs.
• Local AI on Raspberry Pi 5: Real Benchmarks for Llama, Phi, and Gemma (2026) — Ollama-flavored guide with broader model coverage.
• Best Raspberry Pi Heatsink and Cooling Kits for Pi 5 in 2026 — thermal throttling fix referenced in the PSU/thermal section.
• Best Budget GPU for Local LLM Inference Under $400 (2026) — when you outgrow the Pi and want a real GPU.
• Jetson Orin Nano Super vs Raspberry Pi 5: Real Edge-AI Benchmarks (2026) — the next step up the edge-inference ladder.
• Hailo-10H AI Accelerator on Raspberry Pi 5: Real Tok/s for On-Device LLMs — Pi 5 + dedicated NPU path for higher throughput.

Sources

• llama.cpp GitHub issue tracker — the canonical record of build failures, ARM-specific bugs, and quantization regressions. Issues #6700, #7000, and #8200 cover the ARMv8 native-autodetect bug fixed in b3000.
• r/LocalLLaMA Pi threads — week-of pinned posts on Pi 4/5 inference, including the "What kind of device is suitable for running local LLM" thread that drove this article's coverage decisions.
• Raspberry Pi forums — the ground truth on PSU/undervolt/thermal behavior. Threads tagged bcm2712 plus llama are gold for Pi 5-specific reboots.
• SpecPicks lab measurements (April-May 2026) — four-rig fleet, Bookworm 64-bit, llama.cpp b3000-b3450, all numbers reproducible on commodity hardware. Cross-referenced against our Pi 4 vs Pi 5 tok/s benchmark.
• Raspberry Pi Foundation: Pi 5 power supply specifications — official 27 W USB-C PD spec and the under-voltage flag bits returned by vcgencmd get_throttled.
• ARM Cortex-A72 / Cortex-A76 ISA reference — the canonical source for which ARMv8 extensions each Pi SoC supports (or doesn't). Pi 4's A72 stops at v8-A; Pi 5's A76 adds dotprod, crypto, fp16.
• GGUF quantization spec (llama.cpp docs) — q4_K_M / q5_K_M / q6_K block formats and why they map to specific RAM footprints on ARM.

Last verified: May 2026.