Raspberry Pi AI Module in 2026: Can a Pi + Hailo Run Local LLMs?

Yes, but with sharp caveats. A Raspberry Pi 4 8 GB paired with a Hailo AI accelerator via the Raspberry Pi AI Kit runs small language models — 1-3B parameters at 4-bit quantization — at low single-digit to low double-digit tokens per second. The Hailo accelerator is optimized for vision workloads (object detection, classification) and gives a modest boost to LLM inference at best; most of the LLM work still runs on the Pi's ARM cores. If you want conversational speed on a real LLM, you want a desktop GPU. If you want an always-on edge inference box that sips 5-10 W and runs vision plus a small text model, the Pi + Hailo combination is genuinely capable.

What the Pi AI Module/Hailo accelerator is for

Raspberry Pi's AI Kit — the M.2 HAT+ that mounts a Hailo-8 or Hailo-8L accelerator to a Pi 5 — arrived in 2024 and has continued to see maker adoption through 2026. The pitch was straightforward: an inexpensive add-on that gives the Pi meaningful neural-network inference throughput for real-time vision, aimed at makers who want to build camera-based projects (security cameras, wildlife detectors, industrial vision) without the power and heat budget of a GPU.

The Hailo-8 hits roughly 26 tera-operations per second (TOPS) in the compact form factor Raspberry Pi packaged, and the Hailo-8L variant offers a lower-cost 13 TOPS tier. Both are optimized for the kind of dense integer matrix math that convolutional neural networks and vision transformers do. They are not optimized for the memory-bandwidth-limited work large language models do at generation time.

That distinction is the whole story of "can a Pi run LLMs." The Hailo accelerates the compute-heavy parts of vision models efficiently. The Pi CPU handles LLM inference. The two coexist, but the Hailo does not turn the Pi into a small language-model powerhouse.

This piece walks through what the Pi + Hailo combination can actually do at each model size, where the Raspberry Pi 4 8 GB sits relative to newer Pi 5 boards, and where the Raspberry Pi Zero W fits as an auxiliary sensor node. Public references come from the Raspberry Pi AI Kit product page, the Hailo-8 product page, and the llama.cpp release notes that document ARM-CPU LLM performance.

Key takeaways

• The Pi AI Module (Hailo) is optimized for vision, not language. LLM throughput gains from the accelerator are limited.
• On a Pi 4 8 GB, a 1-3B model at q4 runs on CPU at low single-digit to low double-digit tok/s.
• The practical LLM ceiling for a Pi 4 8 GB is roughly a 3B model at q4_K_M — larger models slow to a crawl.
• Storage matters more than you'd think: booting from a USB SSD like the Crucial BX500 is a real quality-of-life upgrade over microSD.
• Use the Pi for always-on edge inference; use a desktop GPU for real conversational LLM work.

What does the Pi AI Kit actually accelerate?

The Hailo-8 (26 TOPS) and Hailo-8L (13 TOPS) are neural network accelerators specialized for the dense multiply-accumulate operations that dominate convolutional neural network inference. Their compiler toolchain converts models from ONNX, TensorFlow, and PyTorch into the accelerator's binary format, with best support for computer-vision families like YOLO, MobileNet, and various vision transformers.

Approximate workload characterizations:

The uncomfortable truth for LLM enthusiasts: language-model generation is a memory-bandwidth story, not a compute story. Each token's forward pass reads the entire model weight file. The Hailo helps with the per-layer math but doesn't fundamentally change the memory-bandwidth ceiling. On a Pi, that ceiling is set by LPDDR4 at roughly 12-15 GB/s — orders of magnitude below a desktop GPU.

Prefill, by contrast, benefits more from the accelerator because it's compute-dense. If your workload is short-prompt short-completion (a keyboard-agent style call), the Hailo helps the prefill piece meaningfully. If it's long generation, the CPU-bound memory read dominates and the accelerator sits mostly idle.

Which small LLMs fit on Pi-class hardware?

Approximate performance on a Raspberry Pi 4 8 GB using llama.cpp on the 4 ARM Cortex-A72 cores, with a Hailo accelerator handling compatible layers where possible:

Numbers are estimates from the llama.cpp release notes and community benchmarks; actual performance depends on Pi power/cooling and llama.cpp build flags.

The takeaway: a Pi 4 8 GB can host small LLMs at low tens of tok/s at best. That's fast enough for automation, background workers, and simple chat, but far too slow for interactive conversational use. A newer Pi 5 8 GB improves these numbers meaningfully thanks to the faster Cortex-A76 cores and LPDDR4X memory, but the ceiling remains "sensible for edge tasks, wrong for chat."

Quantization matrix for small models on Pi

Quantization on ARM Pi hardware works essentially the same as on desktop, but with different tradeoffs because ARM's SIMD instructions favor certain quant formats.

On ARM specifically, q4_0 is often the best-optimized quant in llama.cpp — it maps cleanly to NEON SIMD instructions and hits its theoretical throughput reliably. q4_K_M is preferred on desktop GPU but slightly slower on Pi CPU. For a Pi build, test both q4_0 and q4_K_M and pick the one your specific model implementation runs fastest.

Prefill vs generation: why RAM bandwidth dominates on Pi

Every token generated in an LLM requires reading the full weight file once. On a Pi 4 with LPDDR4 at ~12-15 GB/s, a 2 GB weight file at q4_K_M takes roughly 130-170 ms to read fully. That's your absolute minimum per-token time regardless of any compute optimization.

Compare to a desktop card like the RTX 3060 12 GB with 360 GB/s of GDDR6: the same 2 GB weight file reads in ~6 ms, and the actual generation is much faster still because the GPU can hide latency with parallel work.

Prefill is different. For prefill, all the tokens of the prompt run through the model at once, and the work is dense enough that compute (not memory) becomes the bottleneck. This is where the Hailo can help: a 200-token prompt prefills roughly 3-5x faster with the Hailo accelerator handling compatible transformer layers than on Pi CPU alone.

For a keyboard-agent-style workload where the prompt is 100-300 tokens and the response is 20-100 tokens, the Hailo speedup on prefill matters. For a chat workload where the response is 500+ tokens, generation dominates and the Hailo helps less.

Context length impact on RAM-limited inference

The Pi 4 8 GB has ~7 GB usable after the OS. A 2B model at q4_K_M weights uses ~1.2 GB, leaving ~5.8 GB for everything else — KV cache, application code, and OS overhead.

KV cache scales linearly with context length:

Practical ceiling on a Pi 4 8 GB is roughly 8k-16k context. Beyond that, KV cache pushes into swap and generation slows dramatically. Quantizing the KV cache (q8_0 or int8) roughly halves these numbers and buys you another 2x context if you need it.

What to buy for a Pi local-inference bench

For a first-time builder wanting to try LLMs on a Pi:

1. Raspberry Pi 4 Computer Model B 8 GB — the reference SBC for edge inference. 8 GB is the practical minimum for LLM work.
2. Raspberry Pi AI Kit (Hailo-8 or Hailo-8L) — the accelerator itself. Sold separately by Raspberry Pi and authorized resellers.
3. Crucial BX500 1 TB SATA SSD — boot device via USB 3.0 enclosure. Massively better than microSD for model loading and general responsiveness.
4. Active cooling — a fan case is essential. Sustained inference workloads push the Pi's SoC hard enough to throttle without active cooling.
5. Raspberry Pi Zero W Basic Starter Kit — optional. Useful as an auxiliary sensor node, Wi-Fi bridge, or lightweight peripheral controller if your project spans multiple devices.

Total cost for a functional Pi 4 + AI Kit LLM bench lands around $300-400 in 2026 depending on the AI Kit price and enclosure choices. That's cheaper than any desktop GPU, and appropriate for the target workload — always-on edge inference, not desktop-speed chat.

When a Pi is the right host, and when to step up

Pi is the right host when...

• Power budget matters. The whole system draws 5-10 W under load — a desktop GPU exceeds that at idle.
• The workload is edge inference — camera-based detection, sensor triggering, voice keyword spotting — not conversational LLM.
• You need 24/7 uptime. A Pi runs continuously for years on cheap USB-C power.
• You're building a distributed sensor network where each node needs local inference.

Step up to a desktop GPU when...

• You want real conversational LLM speed. 40-60 tok/s on a mid-range card versus 3-6 tok/s on a Pi is a qualitative difference, not just quantitative.
• You want to run 7B+ models. Pi RAM caps you at 3B; 7B needs 24 GB+ of accessible memory for comfort.
• You want to iterate on prompts or fine-tune. Pi is too slow for interactive development.
• You need reliable long-context inference. Pi RAM limits meaningful context.

The most common pattern is to use both: a Pi as an always-on edge node running vision and simple triggers, and a desktop GPU as an "expensive brain" that the Pi calls into for heavy language work. That split lets each device do what it's best at.

Common pitfalls

• Booting from microSD. SD card I/O is slow enough to make model loading painful. USB SSD is a real upgrade.
• No active cooling. Sustained LLM inference on Pi CPU pushes the SoC to thermal throttling in minutes.
• Trying 7B+ models on Pi. The math doesn't work. Stay at 3B or under.
• Assuming the Hailo speeds up all LLM work. It helps prefill more than generation. Understand the workload first.
• Skipping the AI Kit for LLM-only projects. If you only care about LLM tok/s (not vision), the Hailo's marginal help doesn't justify its cost — a Pi 4 8 GB alone is close to the same performance.

When NOT to build this

If your workload is real-time conversational chat with an LLM, a Pi will disappoint. Buy a desktop GPU. If your workload is a computer vision pipeline, the Pi + Hailo combination is excellent and worth every dollar. If you're straddling both — a small language model plus a vision model — the Pi + Hailo is workable, but the LLM piece will feel slow.

Bottom line

A Raspberry Pi 4 8 GB with the AI Kit runs 1-3B language models at low double-digit tok/s under ideal conditions — usable for automation and edge inference, not for real-time chat. The Hailo accelerator's real strength is vision, not LLMs; if language models are your primary interest, save the AI Kit budget and put it toward a desktop GPU. If you need a small always-on edge inference node that sips power and handles both vision and modest LLM work, the Pi 4 8 GB is genuinely capable and the Crucial BX500 SSD upgrade is well worth the small extra spend.

Related guides

• On-Device AI Keyboards: What a Sub-2GB LLM Needs to Run Local
• Building a Local AI-Agent Eval Rig After AISI's Benchmark Warning
• Oomwoo: The Open-Source, 3D-Printed Robot Vacuum That Skips the Cloud

Citations and sources

• Raspberry Pi — AI Kit product page
• Hailo — Hailo-8 AI Accelerator
• llama.cpp — release notes and benchmarks

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.