Can a Raspberry Pi 4 Run a Local LLM? Ollama tok/s Tested

Yes, a Raspberry Pi 4 Model B 8GB can run a local LLM through Ollama, but slowly. Practical throughput for interactive use tops out around 3-5 tok/s on a 3B q4 model. For anything larger or faster, you want a discrete GPU rig — the featured MSI RTX 3060 12GB delivers 30-40× more tokens per second.

Why anyone tries this in the first place

The Raspberry Pi 4 8GB has 8 GB of shared RAM, a quad-core Cortex-A72 at 1.5-1.8 GHz, and no dedicated GPU compute. Objectively it is not a suitable LLM host. But subjectively, it is the cheapest computer many people already own, and "can I run a chatbot on my Pi?" is a rite of passage. The honest answer is: yes, with caveats, and here are the numbers so you can decide if it is worth setting up.

Public reports on r/LocalLLaMA converge on the same conclusion — TinyLlama and Phi-3-mini work; anything larger than 3B parameters is where interactive use falls off a cliff. This piece puts numbers on that intuition and explains where a Pi ends and a discrete GPU rig begins.

What you'll need

• Board: Raspberry Pi 4 Model B 8GB. The 8 GB variant is required; smaller RAM configurations do not fit larger models comfortably.
• Storage: Samsung 970 EVO Plus with a USB 3.0-to-NVMe adapter, or a similar fast SSD. Boot off the SSD, not the SD card.
• Cooling: any active cooler that keeps the SoC below 70 °C under sustained load. Sustained inference will otherwise thermal-throttle within minutes.
• Optional GPU baseline: MSI RTX 3060 12GB with a Raspberry Pi Zero W kit as an edge accessory to compare against.

Key takeaways

• 1B-3B q4 models work interactively on Pi 4 8GB. TinyLlama, Phi-3-mini, and smaller Qwen variants are the practical ceiling.
• 7B q4 loads but is not interactive. Roughly 1-2 tok/s — fine for offline batch, painful for chat.
• The bottleneck is memory bandwidth, not raw CPU. Pi 4 shared LPDDR4 is the limiter.
• A discrete GPU rig delivers 20-40× the throughput at maybe 30× the price. Perf-per-watt is where the Pi wins.
• Skip the Pi entirely for real LLM work. Buy the MSI RTX 3060 12GB if you want a serviceable local LLM host.

Spec table: Pi 4 8GB memory bandwidth vs what an LLM needs

Public specs from the Raspberry Pi 4 product page:

Memory bandwidth is the single number that predicts LLM inference throughput on non-GPU platforms. The Pi 4 has roughly 1/85th the memory bandwidth of the RTX 3060. That is why the Pi is slow — not because its CPU is bad, but because inference is bandwidth-bound and the Pi's bus is narrow.

Benchmark table: tok/s for 1B/3B/7B models at q4 on the Pi 4 via Ollama

Community measurements from r/LocalLLaMA at typical Pi 4 8GB clocks with active cooling:

Generation tok/s in the 3-5 range is at the edge of "usable for interactive chat" — you can read the streaming output roughly as fast as it prints. Below 2 tok/s the experience is batch, not chat.

Quantization matrix

Above 7B, the Pi 4 runs out of headroom regardless of quant. That is the ceiling.

Which tiny models are actually usable on a Pi 4?

• TinyLlama 1.1B q4_K_M — 8-12 tok/s. Usable for simple tasks, weak reasoning.
• Phi-3-mini 3.8B q4_K_M — 3-5 tok/s. Best quality-per-second on the Pi. Reasonable for basic Q&A.
• Qwen 3B q4_K_M — 4-6 tok/s. Solid alternative to Phi.
• Gemma 2B q4_K_M — 5-7 tok/s. Decent for classification and simple summarization.
• Llama 3.2 1B q4 — 8-10 tok/s. Fast but limited.

Under 3 tok/s stops being interactive. Above 5 tok/s starts feeling responsive.

Where a discrete GPU rig leaves the Pi far behind

For every generation-tok/s number in the table above, the MSI RTX 3060 12GB does 20-40× better. Concretely, running Llama 3 8B q4_K_M:

• Pi 4 8GB: ~1.5 tok/s (slow batch).
• RTX 3060 12GB: ~35 tok/s (comfortable interactive).

That is why almost every "serious local LLM" thread on r/LocalLLaMA points at a used consumer GPU rather than a bigger Pi. It is not close. The Pi's role in an LLM context is not "small server" — it is "edge sensor plus small classifier" or "curiosity project."

Perf-per-watt: the one metric where the Pi wins

Idle to load the Pi 4 draws about 3-7 W. On TinyLlama at 10 tok/s that is roughly 1.4-2 tok/s per watt. On a 3060 rig at 34 tok/s and ~250 W under load that is roughly 0.14 tok/s per watt. The Pi wins perf-per-watt by an order of magnitude.

That matters if you are running an always-on edge classifier — a Pi 4 running Phi-3-mini as a small assistant for home automation is fine. It does not matter if you are trying to replace a cloud API for coding or chat — you would need 40 Pis at $190 each to match one 3060, and you would consume more total power and spend $7,600 doing it. The math is only interesting at very specific edge use cases.

Bottom line: valid edge use cases vs when to step up

Valid use cases for a Pi 4 8GB LLM host:

• Simple home-automation classifiers (route intents to specific handlers).
• Small always-on translation or transcription with a tiny model.
• Edge sensor devices where a 1-3B model is the whole ML stack.
• Learning and experimentation — this is a great platform to understand quantization and Ollama internals cheaply.

Step up to a discrete GPU rig when:

• You want interactive chat with any 7B+ model.
• You want coding assistance.
• You want an agent to run tools and loop for you.
• You need real throughput.

The MSI RTX 3060 12GB build is the honest answer, paired with the Samsung 970 EVO Plus for models and a modest Raspberry Pi Zero W kit if you still want a tiny edge sensor node on the side.

Common pitfalls

• Running from SD card. Model files load painfully slowly and SD cards die under 24/7 use. Use an SSD.
• No active cooling. Sustained inference cooks the SoC and clocks fall. Add a fan.
• Buying the 4 GB or 2 GB Pi 4. 8 GB is the floor for anything larger than 1B.
• Expecting the Pi to replace ChatGPT. It cannot. It is an edge device, not a workstation.
• Ignoring load times. A 7B model load takes 30-45 s off SSD; you feel that on every context switch.

Real-world numbers: three profiles

Home automation intent classifier: Phi-3-mini 3B q4 at 4 tok/s classifying voice-transcribed commands. Good fit; the Pi 4 handles it inline with Home Assistant.

Learning project: load various quant formats, benchmark them, understand kv-cache. Cheap way to learn; the Pi 4 does not disappoint here.

Personal chatbot: TinyLlama at 10 tok/s. Works but the model itself is too limited for most use cases; graduate to the MSI RTX 3060 12GB once you outgrow it.

Offline note-taking assistant: Phi-3-mini paired with a small summarization pipeline that runs once a night against a Markdown vault. Batch use hides the Pi's throughput ceiling, and the model is capable enough for one-paragraph summaries and section headings. Set it up once and it just runs.

When NOT to run an LLM on a Pi

• Any interactive coding, agent, or heavy chat use case. Get the GPU.
• Anything requiring large-context prompts (16k+). Memory pressure crushes throughput.
• Any workload where you value your time more than a few hundred dollars. The GPU rig pays back in productivity fast.

Ollama-on-Pi setup notes

If you decide to try it anyway, notes from public reports for a smoother experience:

• Install Ollama via the official install script. It detects ARM64 and downloads the right build.
• Boot from the SSD, not the SD card. Model loads are painful otherwise; the Samsung 970 EVO Plus via USB 3.0 is the most reliable path.
• Set OLLAMA_NUM_PARALLEL=1 to keep memory pressure predictable. Concurrent requests OOM the box.
• Use htop to watch memory during load. If you see swap activity you have chosen too big a model.
• Prefer q4_K_M over lower quants. Quality-per-quant drops fast below q4.
• Avoid long-context prompts. KV cache scales linearly with context; keep prompts short.
• Do not run other services concurrently. Home Assistant plus Ollama on the same Pi is not a good time.

Edge patterns that actually work on Pi

Instead of trying to make the Pi a chatbot, use it as an edge classifier:

• Intent classification for voice assistants. Small model routes utterances to specific handlers on a beefier box.
• Simple summarization on-device. Read a paragraph and return a one-liner. Useful for local RSS or note-taking.
• Sentiment or topic tagging. Batch pipeline overnight for a personal notes archive.
• Command generation for shell tools. Constrain to a small vocabulary; the Pi handles it.
• Chunked translation. One paragraph at a time — the Pi is slow but not embarrassing at translation of short passages.

The pattern: use tiny models for narrow tasks, not general chat.

Alternative Pi-class SBCs for LLM work

• Raspberry Pi 5 8GB/16GB. Faster CPU, more RAM ceiling. Not a game-changer for LLM tok/s — bandwidth is still the throttle.
• Orange Pi 5. Rockchip-based; has an NPU for select AI workloads. Software support is spotty.
• Radxa Rock 5B. Similar Rockchip SoC. Same caveats.
• Jetson Orin Nano. Not a Pi but plays in the same physical envelope. NVIDIA Tegra with a real GPU — much faster for LLMs. Costs more.

If the Pi 4 is not enough and you want a small board, the Jetson Orin Nano is the interesting one. Otherwise step up to a proper desktop rig.

Related guides

• Building a silent Pi 4 NVMe homelab (cooling, VESA, SSD boot).
• Arc A770 16GB vs RTX 3060 12GB for local Llama 3.
• Best SSD for a local LLM model library in 2026.

Citations and sources

• Raspberry Pi — Model B product page
• Ollama — GitHub repo
• Phoronix — Raspberry Pi 4 benchmarks

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.