IPEX-LLM + Ollama in Docker: Running Local Models on Intel GPUs

How do you run Ollama on an Intel GPU with IPEX-LLM in Docker? Pull Intel's prebuilt intelanalytics/ipex-llm-inference-cpp-xpu container, mount /dev/dri for GPU passthrough, expose port 11434, and set OLLAMA_INTEL_GPU=1. The result is a drop-in Ollama server that offloads inference to Intel Arc, Iris Xe, or Battlemage GPUs via oneAPI + SYCL — 20-40 tokens/second on a 7B-8B model at Q4, meaningfully behind an MSI RTX 3060 12GB on CUDA but usable enough to matter for Intel-first builds.

Why Intel-GPU users need IPEX-LLM and where this fits vs CUDA

Local LLM inference on consumer hardware has, for years, effectively meant "buy an NVIDIA GPU." CUDA has been the default backend for llama.cpp, Ollama, vLLM, and most other inference engines, and the tooling assumes a green box. That has left Intel GPU owners — Arc A770 A750 A580, the newer Battlemage B580 and B570, Iris Xe integrated graphics on 12th-gen and newer Core CPUs — in an awkward spot. The hardware is capable, sometimes competitive, but the software wasn't there.

IPEX-LLM (Intel PyTorch Extension for LLMs) closes that gap. It is Intel's official acceleration library that wraps oneAPI + SYCL and provides drop-in compatibility for llama.cpp and Ollama. The intelanalytics/ipex-llm-inference-cpp-xpu Docker container ships everything preconfigured — Intel's compute runtime, level-zero, oneAPI base kit, IPEX-LLM's forked llama.cpp with Intel SYCL backend, and Ollama itself — all in a single 6GB image. Pull, run, GPU-inference. No compilation, no driver-version wrestling.

For an Intel-first builder, this changes the calculus meaningfully. An Arc B580 12GB at $270 street runs Llama 3.1 8B at Q6 at roughly 30 tokens/second. That is real performance — good enough for casual chat, RAG, and coding assistance. It costs less than an equivalent-VRAM RTX 3060 12GB (which runs the same model at ~45 tok/s on CUDA), and if you already own an Intel GPU or an Intel laptop with Iris Xe, it costs nothing extra to try.

The CUDA path on an RTX 3060 12GB with an AMD Ryzen 7 5800X and a WD_BLACK SN770 NVMe SSD is still smoother — better driver stability, faster inference, larger ecosystem — but it is no longer the only consumer path. This guide walks through the IPEX-LLM Ollama setup end to end, then compares against a CUDA baseline so you can decide which side of the fence to be on.

Key takeaways

• IPEX-LLM's Docker container wraps oneAPI + Ollama into a one-command setup for Intel GPUs.
• Real performance: Arc A770/B580 at Q4 8B lands around 30-40 tok/s, versus 45 tok/s on a CUDA RTX 3060 12GB.
• Requires kernel 6.2+ and Intel Compute Runtime — Ubuntu 22.04 LTS or newer works cleanly.
• Iris Xe integrated graphics can run this too — slower, but functional.
• Not every model is supported; check IPEX-LLM's compatibility table before committing.

What you will need

Parts and prerequisites:

• Intel GPU: Arc A380 (6GB, slowest), Arc A580 (8GB), Arc A750 (8GB), Arc A770 (16GB, best value for LLM), Battlemage B570/B580, or Iris Xe integrated graphics (2GB-4GB shared, entry-level).
• CPU: Intel 12th-gen or newer, or any AMD. Older Intel CPUs work but the platform driver stack is less proven.
• Reference alt-hardware for CUDA comparison: MSI GeForce RTX 3060 12GB — the mainstream CUDA reference and the direct alternative to an Arc A770 at similar price.
• Motherboard / system: Anything modern with PCIe 3.0 x16 or better. An AMD Ryzen 7 5800X works well as a host.
• Storage: A WD_BLACK SN770 NVMe SSD or equivalent. Model weights are 4-15GB per model.
• OS: Ubuntu 22.04 LTS or 24.04 LTS. Debian 12 works. Windows 11 works via WSL2 with more setup. Fedora works.
• Docker Engine: 20.10+.
• Intel Compute Runtime: install via apt install intel-opencl-icd libigc-dev intel-level-zero-gpu.

Total setup time: 30-60 minutes from Ubuntu install to first token if it all works. First-run driver install can take 60-90 minutes if you have to fight kernel modules.

Step-by-step: pulling the IPEX-LLM Ollama container

Assuming you have Docker and the Intel Compute Runtime installed on Ubuntu 22.04+:

1. Verify GPU visibility:

sudo apt install intel-gpu-tools
sudo intel_gpu_top

You should see your GPU listed. Ctrl+C to exit.

1. Pull the IPEX-LLM container:

docker pull intelanalytics/ipex-llm-inference-cpp-xpu:latest

Image is ~6GB. First pull takes a few minutes on a fast connection.

1. Run Ollama with GPU passthrough:

docker run -it -d --net=host \
 --device=/dev/dri \
 --shm-size=16g \
 -v /path/to/ollama/models:/root/.ollama/models \
 -e OLLAMA_INTEL_GPU=1 \
 -e no_proxy=localhost,127.0.0.1 \
 --name ollama-ipex \
 intelanalytics/ipex-llm-inference-cpp-xpu:latest \
 /bin/bash -c "ollama serve"

Key flags: --device=/dev/dri gives the container access to the Intel GPU device node, --shm-size=16g provides scratch space for model loading, and OLLAMA_INTEL_GPU=1 toggles IPEX-LLM's oneAPI/SYCL codepath instead of the CPU fallback.

1. Pull and run a model:

docker exec -it ollama-ipex ollama pull llama3.1:8b-instruct-q4_K_M
docker exec -it ollama-ipex ollama run llama3.1:8b-instruct-q4_K_M "Hello, world."

You should see the model download, load onto the GPU, and generate output. On an Arc A770 the first response returns in about 2 seconds; subsequent tokens flow at 30-45 tok/s.

1. Optionally expose the Ollama API for external clients (e.g. Open WebUI, LibreChat, VS Code Continue plugin):

curl http://localhost:11434/api/tags

The API is Ollama-standard — anything that talks to a stock Ollama server works unchanged with this container.

Performance table — Intel Arc vs RTX 3060 CUDA reference

Tested with the standard llama.cpp benchmark suite, 512-token generation, 30-second warmup:

Read the pattern: CUDA on the RTX 3060 is consistently 20-40% faster than IPEX-LLM on Arc across every model tested. That is not a small gap, but it is not a wall either. An Arc A770 at 42 tok/s on Llama 8B Q4 is a genuinely usable speed for interactive chat and coding assistance.

The Arc B580 12GB — the newer Battlemage midrange — sits close to the A770 on most models but pulls ahead on smaller ones (7B and below) thanks to better matmul throughput. It is also the current price/performance sweet spot at $270 street for anyone building an Intel-only LLM box.

Quantization matrix on Intel vs the CUDA baseline

The Intel oneAPI/SYCL backend supports the same quantization tiers as CUDA — Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0. Quality is identical because the quantization happens at the model weights, not at the backend. What differs is throughput per quant:

The Intel penalty grows at higher quants — the SYCL backend has less-optimized codepaths for the more bit-per-weight formats. For most users this argues for Q4_K_M as the daily driver on Intel GPUs, versus Q6_K on CUDA. The quality difference between Q4 and Q6 is small enough that most people cannot reliably tell them apart in blind side-by-side.

Common gotchas: driver versions, SYCL device selection, container permissions

The IPEX-LLM Docker path is meaningfully more forgiving than trying to build llama.cpp with SYCL from source, but there are still real setup pitfalls:

1. Kernel version. Intel Compute Runtime for Arc requires Linux kernel 6.2 or newer. Ubuntu 22.04 ships kernel 5.15 by default; run apt install linux-generic-hwe-22.04 to get a newer kernel, or upgrade to Ubuntu 24.04.
2. /dev/dri device permissions. Docker needs read/write on /dev/dri. On stock Ubuntu, render and video groups own it. Either run Docker as root (works but not ideal) or add your user to both groups.
3. SYCL device selection. By default IPEX-LLM picks the first oneAPI device. If you have both an Intel iGPU (Iris Xe) and an Arc discrete card, the container may pick the wrong one. Set ONEAPI_DEVICE_SELECTOR=level_zero:1 (or :0) to force the correct card.
4. Model compatibility. Not every Ollama model works. Check IPEX-LLM's compatibility table on GitHub before pulling exotic models. Llama, Qwen, Mistral, Gemma, and Phi work reliably. Some MoE models and vision-language models have partial support.
5. VRAM ceiling on Arc A380. Only 6GB — enough for 7B Q4 with a small context window, not enough for anything larger without CPU offload.

Prefill vs generation behavior on Intel GPUs

Something Intel-GPU users notice quickly: prefill (processing the initial prompt) is often faster per-token than generation on IPEX-LLM. This is opposite to CUDA, where generation and prefill are typically within 30% of each other. The reason: Intel's SYCL backend has better-optimized matmul for the large batched matrix operations that prefill uses; generation involves smaller sequential matmuls where CUDA's cuBLAS still holds an advantage.

Practical effect: long-context tasks (RAG, code review over large files) run at reasonable throughput on Intel. Short-prompt interactive chat feels a little slower because the per-token generation rate is the visible metric. This is a real user-experience difference, not a benchmarking quirk.

When CUDA on an RTX 3060 12GB is still the smoother choice

Even accounting for IPEX-LLM's genuine improvements, an MSI RTX 3060 12GB on CUDA remains the smoother path for most first-time local LLM users. Concrete reasons:

• Broader model compatibility. Every quantized model published on Ollama or HuggingFace works on CUDA. Intel SYCL support is narrower.
• Better driver stability. NVIDIA's Linux driver has 15 years of production hardening. Intel's Arc Linux driver is 2 years mature.
• Faster raw throughput. 20-40% more tokens per second on the same model at the same quant.
• Larger community. Every Reddit r/LocalLlama tutorial assumes CUDA. Intel GPU tutorials exist but are much smaller in number.
• Rock-solid Ollama support. Ollama's CUDA path has been the reference implementation since day one. IPEX-LLM is a fork.

The Intel path wins on: better VRAM/dollar on the A770 16GB (16GB for the price of a 12GB 3060), lower total system power draw, and — importantly — for anyone who already owns an Intel GPU or is building a Team-Blue-only system for other reasons.

Common gotchas expanded

• Docker Desktop on Windows via WSL2: works, but requires Windows Docker Desktop 4.28+ and the Intel WSL GPU driver. Native Linux is smoother.
• AMD systems with Intel GPUs: work fine, but ensure intel-opencl-icd is installed and the AMD system doesn't have mesa-opencl-icd conflicting.
• Multi-GPU setups: IPEX-LLM currently favors single-GPU. Multi-GPU inference is possible via llama.cpp's tensor-split, but the SYCL backend's multi-device support lags CUDA.
• Model download to correct volume: Mount your models volume with correct paths — Ollama defaults to ~/.ollama/models. In Docker map it: -v ~/ollama-models:/root/.ollama/models.

Verdict matrix

Use IPEX-LLM on Intel GPU if:

• You already own an Arc A770, A750, A580, or Battlemage B570/B580.
• You are building an Intel-only system for other reasons (Team-Blue preference, workload compatibility).
• You value 16GB VRAM on the A770 for the price of a 12GB RTX 3060.
• You are comfortable troubleshooting occasional driver/kernel edge cases.

Use CUDA on an RTX 3060 12GB if:

• You are building a first LLM box and want the smoothest software stack.
• You use exotic or vision-language models with partial SYCL support.
• Raw tokens/second is a priority and you accept the $50 price premium over an Arc B580 at 12GB.
• You want to run Stable Diffusion or other GPU-accelerated ML alongside LLM inference (CUDA has better cross-workload support).

Do NOT try IPEX-LLM if:

• Your GPU is pre-Arc (older Intel Iris on 11th-gen Core or below) — insufficient compute for meaningful LLM performance.
• Your OS is Ubuntu 20.04 or older — kernel and driver requirements will fight you.
• You need <1s first-token latency on every prompt — the SYCL backend has more variable prefill timing.

Bottom line

IPEX-LLM makes Intel GPUs a genuine option for local LLM inference in 2026 — no longer just a CUDA world. If you have an Arc A770 or a Battlemage B580 (or even Iris Xe integrated graphics as a starting point), the Docker container path gets you a working Ollama server in an evening. Pair it with an AMD Ryzen 7 5800X or any modern CPU and a fast WD_BLACK SN770 NVMe SSD to load models quickly and you have a viable Intel-first LLM stack. For most first-time builders who are shopping fresh, the RTX 3060 12GB on CUDA is still the smoother path — but it is no longer the only path. Read our best local LLM GPU guide and Arc B580 review for further reading.

Sources

• IPEX-LLM — Intel's official GitHub repository
• Intel oneAPI — Toolkit overview
• Phoronix — Intel LLM scaling and vLLM coverage