Ollama on Intel + AMD via IPEX-LLM: Docker Compose Guide

To run Ollama with IPEX-LLM in Docker Compose, pass the Intel GPU render device through into an IPEX-LLM Ollama image, mount a persistent model volume, and set the Intel Level Zero environment variables inside the container. The full docker-compose.yml is at the bottom of this guide. A CUDA-side comparison against a MSI GeForce RTX 3060 12GB rig follows.

Why Compose is the right shape for this stack

docker run works for a single Ollama instance. It falls apart the moment you want reproducible upgrades, versioned config, or a second container (like Open WebUI) hanging off the same backend. Compose gives you all of that in one file that lives next to the project. The stack this guide targets:

• One IPEX-LLM Ollama container as the LLM server.
• Persistent model volume so ollama pull results survive restarts.
• Intel GPU render device passthrough so the container actually accelerates.
• Environment variables that tell IPEX-LLM to use GPU and pick the right device.
• Restart policy so the box comes back up cleanly after reboots.

The Intel oneAPI runtime is what makes IPEX-LLM work on Arc silicon, and its container variant is shipped as part of the IPEX-LLM Ollama image. That is the piece that lets you skip installing oneAPI on the host — a big quality-of-life win.

Key takeaways

• Compose is the clean way to pin an off-CUDA Ollama stack. All the fiddly Intel bits (device passthrough, env vars, image tag) live in one file you commit.
• GPU passthrough is the single hardest thing to get right. Verify with sycl-ls inside the container before you trust throughput numbers.
• Persistent model volume is not optional. Ollama's model registry is many GB; store it on an SSD mounted into the container.
• The featured MSI RTX 3060 12GB is the CUDA reference. If IPEX-LLM's install effort feels like too much, that card is the escape hatch.
• Restart policy matters for always-on rigs. Set restart: unless-stopped so the daemon comes back cleanly.

What does IPEX-LLM change in the Ollama container?

Stock Ollama containers ship with CUDA and CPU back-ends. IPEX-LLM's Ollama container ships with an Ollama build linked against Intel's oneAPI compute runtime, plus SYCL kernels for the popular quant formats. Concretely:

• Base image: Intel-maintained intelanalytics/ipex-llm-inference-cpp-xpu variant (or the newer ipex-llm-ollama tag depending on the release).
• Extra libs: oneAPI Level Zero, SYCL runtime, IPEX-LLM Python and native libs.
• Init script: init-ollama populates the Ollama binary from IPEX-LLM sources.
• Env vars: OLLAMA_NUM_GPU=999, ONEAPI_DEVICE_SELECTOR, SYCL_CACHE_PERSISTENT=1.

Everything else is stock Ollama. ollama pull, ollama run, and the HTTP API on port 11434 work as expected.

Prerequisites table

The docker-compose.yml walkthrough

services:
 ollama-ipex:
 image: intelanalytics/ipex-llm-inference-cpp-xpu:latest
 container_name: ollama-ipex
 restart: unless-stopped
 devices:
 - /dev/dri:/dev/dri
 group_add:
 - "video"
 - "render"
 volumes:
 - ollama_models:/root/.ollama
 environment:
 OLLAMA_HOST: 0.0.0.0
 OLLAMA_NUM_GPU: "999"
 ONEAPI_DEVICE_SELECTOR: "level_zero:0"
 SYCL_CACHE_PERSISTENT: "1"
 ZES_ENABLE_SYSMAN: "1"
 ports:
 - "11434:11434"
 command: >
 bash -lc "
 source /opt/intel/oneapi/setvars.sh &&
 init-ollama &&
 ./ollama serve
 "

volumes:
 ollama_models:
 driver: local

Notes on each block:

• devices — passes the DRI render node into the container. Without this Ollama falls back to CPU inference.
• group_add — the container process must be in video/render to actually use the DRI device. Not everyone needs both; both is safe.
• volumes — persists the model registry between restarts. Point this at an NVMe partition; using /root/.ollama inside a slow storage tier turns cold-starts painful.
• OLLAMA_NUM_GPU=999 — tells Ollama to offload every layer to GPU. IPEX-LLM interprets this as "use the Arc device."
• ONEAPI_DEVICE_SELECTOR=level_zero:0 — pin to the first Level Zero device. If you have multiple Arc cards, use :1, :2, etc.
• SYCL_CACHE_PERSISTENT=1 — persists the SYCL kernel cache across restarts, saving a couple of seconds per cold start.
• command — sources setvars.sh to bring oneAPI into the shell, runs init-ollama to lay down the binary, then starts the server.

Benchmark table: in-container tok/s

Same disclaimers as elsewhere in this cluster — treat as directional pending your own runs.

CUDA container remains the more predictable choice on 8B; A770 flips the 14B row because the 3060 spills to CPU.

How do I verify GPU passthrough is actually working?

Three commands, in order, inside a running container shell:

1. ls -la /dev/dri — must show card0, card1, renderD128, renderD129 depending on your host.
2. sycl-ls — must list your Arc device with its Level Zero backend. If you see only CPU, the passthrough failed.
3. curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3","prompt":"hi","stream":false}' — first token should return in under three seconds on a warm model. If it takes 30 seconds, you are on CPU.

Common failure modes and how to fix them

Perf-per-watt with a Ryzen 7 5800X host

The AMD Ryzen 7 5800X is the featured host CPU for good reason: 8 cores means the container's Ollama process, tokenization, and any adjacent Open WebUI service all have room. Under load:

• Arc A770 draws ~180-220 W at generation.
• Ryzen 7 5800X draws ~40-60 W handling non-GPU work.
• Board + memory + fans add ~40-60 W.
• Total under load: ~280-340 W.

Perf-per-watt at Llama 3 8B q4 on the Arc-in-Compose stack lands around 0.10 tok/s per watt. The CUDA reference on the same host lands closer to 0.20-0.22 tok/s per watt. Compose does not change the ratios; it just makes the setup reproducible.

Bottom line: who should containerize this

• Yes: anyone running Arc silicon full-time as an Ollama host. Compose makes the stack reproducible and upgrades safe.
• Yes: anyone running multiple containers (Open WebUI, LiteLLM, LangChain agents) against a shared Ollama backend.
• No: anyone running Ollama on CUDA (e.g. the featured MSI RTX 3060 12GB) who wants "just Ollama" — the stock CUDA install is one line and fine.
• Maybe: anyone building a portable homelab that migrates between machines. Compose helps but Kubernetes plus device plugins scales further.

For the pragmatic upgrade path: if you find yourself fighting the IPEX-LLM install repeatedly, the MSI RTX 3060 Ventus 3X 12G plus the featured Ryzen 7 5800X host is the escape hatch. Pair with the Samsung 970 EVO Plus for models and the Crucial BX500 1TB SATA for cold archive.

When NOT to use Compose

• You only run one container and never plan to add a second. Plain docker run with the same env vars is simpler.
• You are on Kubernetes already. Compose is redundant; use a Deployment + Intel GPU device plugin instead.
• Your host is macOS. Metal cannot pass through to Linux containers; the whole Arc-in-Compose story does not apply. Run stock Ollama natively instead.

Extending the stack: Open WebUI + LiteLLM in the same Compose file

Adding a friendly UI and a proxy is a two-service extension of the same compose file. Append the following:

 webui:
 image: ghcr.io/open-webui/open-webui:main
 container_name: open-webui
 restart: unless-stopped
 depends_on:
 - ollama-ipex
 environment:
 OLLAMA_BASE_URL: http://ollama-ipex:11434
 ports:
 - "3000:8080"
 volumes:
 - open_webui:/app/backend/data

 litellm:
 image: ghcr.io/berriai/litellm:main-stable
 container_name: litellm
 restart: unless-stopped
 depends_on:
 - ollama-ipex
 ports:
 - "4000:4000"
 volumes:
 - ./litellm-config.yaml:/app/config.yaml:ro
 command: ["--config", "/app/config.yaml"]

volumes:
 ollama_models:
 driver: local
 open_webui:
 driver: local

Now the Compose stack exposes:

• Ollama on port 11434 (direct API).
• Open WebUI on port 3000 (browser).
• LiteLLM on port 4000 (OpenAI-compatible proxy for any tool that expects OpenAI's API shape).

All three share the same Arc GPU via the Ollama backend. Restart policies keep the stack alive across reboots.

Sizing the box for a small team

For a two-to-four person team using the same Ollama backend, size for concurrency, not throughput:

• Model resident memory: each concurrent request holds VRAM. A single Arc A770 16GB handles two concurrent 8B q4 sessions comfortably; a third saturates.
• CPU: Ryzen 7 5800X is ample for two-to-four users. Above that consider a 5900X or 5950X for more tokenization headroom.
• RAM: 32 GB is the floor. Ollama caches, WebUI state, and LiteLLM logs add up.
• Storage: a 1 TB Samsung 970 EVO Plus sibling drive at 2 TB for larger model libraries is a modest upgrade.
• Network: 1 GbE is fine for text; anything file-heavy (RAG document upload) benefits from 2.5 GbE.

Related guides

• Ollama on Intel + AMD via IPEX-LLM: benchmarks vs a CUDA RTX 3060 rig
• Best SSD for storing a large local LLM model library
• Building a silent Pi 4 NVMe homelab for lightweight self-hosting

Backup and upgrade discipline

The whole point of Compose is that this stack is a single file plus a couple of persistent volumes. A basic backup and upgrade pattern:

1. Snapshot the volume weekly. docker run --rm -v ollama_models:/from -v $PWD/backup:/to alpine tar cf /to/models-$(date +%Y%m%d).tar /from — tar the volume out on a schedule.
2. Pin the image tag. Do not use :latest. Choose a dated tag; upgrade by editing the tag, running docker compose pull, then docker compose up -d.
3. Test the upgrade off-hours. IPEX-LLM's Ollama build occasionally regresses on specific quant formats. Have a rollback tag ready.
4. Track upstream release notes. The IPEX-LLM release notes list breaking changes ahead of the tags.

Common pitfalls

• Using restart: always instead of restart: unless-stopped. With always, docker compose down still relaunches the container on host reboot. unless-stopped respects manual stops.
• Forgetting to add SYCL_CACHE_PERSISTENT=1. Cold starts stay slow forever without it.
• Pinning the image tag to :latest. Non-deterministic upgrades break stuff at random. Pin to a dated tag.
• Running the container as root without a persistent volume. Any restart wipes the model registry. Painful.
• Mounting ~/.ollama from the host directly. Ownership drift between host and container users breaks writes. Use a named volume like the compose above.
• Skipping the sanity check on sycl-ls after every image upgrade. Kernel or driver drift can silently break passthrough; re-verify on every upgrade rather than assuming the container is still on the GPU.

Citations and sources

• Intel Analytics — IPEX-LLM GitHub repo
• Docker — Docker Compose overview
• Intel — oneAPI overview

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.