Intel Arc LLM Inference: A770 & B580 Qwen3 tok/s Guide

Intel Arc LLM Inference: The Case for a Budget 16 GB AI GPU

As of October 2026, the Intel Arc A770 16GB generates Qwen3-8B at 21.0 tok/s through llama.cpp's SYCL backend and at 45.5 tok/s through OpenVINO INT4, per the arc-inference-bench results (tested March 2026). The 12 GB Arc B580 runs the same model at Q4_K_M at 30.39 tok/s on llama.cpp's Vulkan backend, per a July 2026 post in the llama.cpp benchmark thread. Neither card leads on throughput. Both fit current 8B–14B models entirely in VRAM at 4-bit, which keeps them credible entry-level inference cards when budget matters more than raw speed.

This guide synthesizes public sources, community benchmarks and Intel's own documentation on running large language models on Arc GPUs. It covers software stacks, compatible model sizes, build considerations and realistic cost expectations.

What changed — October 2026

• Cited tok/s rows for current models. The earlier version quoted no tok/s figures at all. It now carries Qwen3-8B and Qwen3-14B decode rates for both cards, in the current-generation table below. Sources: arc-inference-bench and llama.cpp discussion #12570.
• The runtime choice now has a measured gap. On the same A770, OpenVINO INT4 decoded Qwen3-8B 2.2× faster than llama.cpp SYCL at Q4_K_M (45.5 vs 21.0 tok/s). It decoded Qwen3-14B 1.6× faster (23.3 vs 14.7 tok/s), per arc-inference-bench.
• IPEX-LLM is archived. The intel/ipex-llm repository is read-only on GitHub, and its last push was on January 28, 2026. New model architectures now have to come through llama.cpp or OpenVINO.
• Spec links corrected. The earlier version linked TechPowerUp pages for unrelated GPUs. The A770 now cites TechPowerUp's A770 entry: 16 GB GDDR6 at a 2000 MHz memory clock on a 256-bit bus, which works out to 512 GB/s. That replaces an unsupported 560 GB/s figure. The B580 now cites TechPowerUp's B580 entry: 12 GB GDDR6 at 2375 MHz on a 192-bit bus, which works out to 456 GB/s.
• VRAM table corrected. A 13B model at Q8_0 (~14 GB) was marked "tight" on the 12 GB B580, but it does not fit at all. Rows for real Qwen3 GGUF file sizes from Hugging Face were added (Qwen3-8B, Qwen3-14B).

---

Why Intel Arc Attracts LLM Hobbyists

The Arc A770's headline advantage is its 16 GB GDDR6 frame buffer. Per TechPowerUp's Arc A770 entry, the 16 GB card pairs GDDR6 at a 2000 MHz memory clock (16 Gbps effective) with a 256-bit bus, which works out to 512 GB/s of bandwidth. That number bounds generation speed, because each new token streams the model's full weight set from VRAM.

When the A770 launched, 16 GB was more VRAM than any NVIDIA consumer GPU below the RTX 3090 tier. That positioning gave Arc an early foothold with local LLM hobbyists who needed to fit 13B-class models in GPU memory without resorting to the used workstation market. For the full picture on what works today, see Intel Arc LLM Support: What Works in 2026.

The 2024 Battlemage generation brought the Arc B580 on Intel's Xe2 architecture. At 12 GB GDDR6, the B580 trades 4 GB of capacity for improved XMX (matrix extension) units and meaningfully better driver maturity, per TechPowerUp's B580 entry. That entry lists GDDR6 at 2375 MHz on a 192-bit bus, or 456 GB/s. For quantized inference workloads, those dedicated INT8/INT4 matrix multiply units matter: they handle the dot-product-heavy operations that dominate transformer forward passes without consuming general shader resources.

---

The Software Stack: oneAPI, llama.cpp, and IPEX-LLM

Intel Arc inference runs on three parallel software paths. Choosing the right one for a given workload is the first decision any prospective Arc LLM user must make.

llama.cpp SYCL Backend

The most widely community-tested path is llama.cpp's SYCL backend, maintained under Intel's oneAPI toolkit. Active Arc users on the llama.cpp GitHub repository report usable throughput for GGUF-quantized models ranging from Q4_K_M through Q8_0. The SYCL path requires the Intel oneAPI Base Toolkit installed and compiled against before building llama.cpp — a notably higher setup bar than CUDA's plug-and-play experience on NVIDIA hardware, but well-documented within the community.

Per Intel's oneAPI documentation, the SYCL runtime supports all Arc discrete GPUs from the Alchemist (A-series) generation onward, including Arc Pro variants.

IPEX-LLM (Intel Extension for PyTorch)

Intel's IPEX-LLM (formerly BigDL-LLM) takes a higher-level approach: it wraps HuggingFace Transformers and LangChain models with Arc-optimized inference kernels, supporting INT4 and INT8 quantization via XMX units. Coverage from Tom's Hardware's B580 review identifies IPEX-LLM as the more accessible route for Python-ecosystem users who want to avoid manual GGUF conversion workflows. The library also exposes Intel AMX acceleration for hosts with 4th/5th-gen Xeon or Core Ultra CPUs, enabling hybrid CPU+GPU inference for models that exceed VRAM capacity.

As of October 2026, though, the intel/ipex-llm repository is archived on GitHub (last push January 28, 2026). Existing installs keep working for the models they already support. New architectures will not get Arc kernels there, so treat llama.cpp and OpenVINO as the maintained paths.

OpenVINO

For deployment scenarios — fixed models, batched API endpoints, edge inference — Intel's OpenVINO toolkit offers the most optimized path. OpenVINO's neural network compression framework (NNCF) applies INT8 post-training quantization and exports models as static intermediate-representation (IR) graphs, reducing per-call runtime overhead. OpenVINO is the natural fit for production services on Arc Pro hardware.

Related: Intel Arc B580 LLM Inference: 12GB Battlemage Benchmarks | Intel Arc A770 LLM Performance: Local Inference Guide

---

VRAM and Model Size Compatibility

VRAM capacity is the hard constraint for LLM inference. A model that does not fit in GPU memory degrades to CPU offloading, which collapses throughput to a fraction of full-GPU rates. The table below uses community-reported GGUF memory footprints as reference points:

Generic-size rows are estimates derived from community-reported GGUF memory footprints across r/LocalLLaMA and llama.cpp GitHub discussions. Qwen3 rows list the actual Hugging Face GGUF file size. The KV cache for context comes on top, so actual usage varies with context length, implementation and overhead allocation.

The A770's 16 GB headroom is genuinely useful: it accommodates 13B models at Q8 quality without offloading and leaves headroom for extended context windows (8K–16K tokens) that would otherwise squeeze a 12 GB card. For a direct head-to-head of the two Arc generations on these workloads, see Arc A770 vs B580 for Local LLM Inference: 2025 Comparison.

---

Current-generation model rows (October 2026)

Each row cites its measurement. The ceiling column is the card's memory bandwidth divided by the GGUF file size from Hugging Face. Decode streams the full weight set from VRAM once per generated token, so an honest decode figure cannot exceed that ceiling. Every measured figure below sits well under it. The OpenVINO INT4 rows use OpenVINO's own export rather than a GGUF file, so their ceiling is approximated with the Q4_K_M file size.

Three things stand out. First, on Arc the runtime matters as much as the card: OpenVINO roughly doubles A770 decode on Qwen3-8B. Second, context length costs speed. In the same arc-inference-bench run, A770 OpenVINO decode on Qwen3-8B falls from 38.2 tok/s at 1K context to 26.8 tok/s at 8K and 16.4 tok/s at 16K. Third, the A770 and B580 rows come from different testers on different backends (SYCL/OpenVINO versus Vulkan), so they are not a like-for-like card comparison. In the same run, Mistral Small 3.1 24B at Q4_K_M ran out of memory on the A770 under both runtimes. That marks the practical upper bound for a single 16 GB Arc card.

---

Performance in Context: What Community Benchmarks Show

No standardized Intel-published benchmark directly compares Arc against NVIDIA RTX or AMD consumer cards on production LLM inference workloads. Community measurements on r/LocalLLaMA and the llama.cpp issue tracker provide a less formal but actively updated picture:

• Arc A770 users have reported Llama 3 8B Q4_K_M generation throughput broadly comparable to lower-tier NVIDIA RTX 30-series cards with smaller VRAM (e.g., RTX 3060 12 GB), with variance depending on driver version and oneAPI build configuration.
• B580 (Battlemage) users report improved throughput over the A770 at equivalent model sizes and quantization levels, attributed to the Xe2 XMX unit refinements documented in Intel's Battlemage architecture materials.
• Prompt-processing throughput (prefill speed) on Arc lags generation throughput relative to NVIDIA equivalents more than might be expected from raw specs — the SYCL prompt-processing path in llama.cpp has historically been a weaker point compared to the mature CUDA implementation.

NVIDIA cards with CUDA remain the throughput reference for local LLM inference. Intel Arc's competitive argument is capacity-per-dollar, not throughput parity. Per ongoing TechPowerUp pricing coverage, the A770 has delivered 16 GB at a lower street price than the RTX 4060 Ti 16 GB for much of 2024–2025. For deeper benchmark analysis, see Intel Arc A770 for Local LLMs: 2026 Benchmark Guide and Intel Arc A770 16GB for Local LLMs: 2025–2026 Guide.

---

Building an AI Inference Rig Around Intel Arc

An Arc GPU performs only as well as the system feeding it. Intel's oneAPI hardware requirements documentation and community build threads point to several practical considerations.

System RAM

Fast system DDR4 or DDR5 RAM matters for CPU-side prompt tokenization and any CPU-layer offloading on larger models. For a 16 GB Arc build targeting 13B models with minimal offloading, 32 GB of system RAM is a comfortable floor:

• The CORSAIR Vengeance LPX DDR4 32 GB (2×16 GB) kit at $214.59 provides DDR4-3200 capacity well-matched to an inference host that will also run tokenization and preprocessing on-CPU.
• Budget builds targeting 7B-only workloads can start with the Crucial 16 GB DDR4 Kit (2×8 GB) at $127.00 and upgrade later if CPU offloading becomes a bottleneck.

Storage for Model Files

LLM weight files accumulate quickly. A Llama 3 8B Q4_K_M GGUF runs approximately 4.6 GB; a 13B Q4_K_M is around 8.4 GB; a Mistral 7B plus several fine-tuned variants can easily consume 30–50 GB. A minimum of 1 TB fast local storage is advisable for any serious model library:

• The Samsung 980 NVMe SSD 1 TB at $255.00 offers PCIe 3.0 ×4 throughput — more than sufficient bandwidth for model loading, which is a one-time per-session cost rather than a bottleneck during inference.
• For larger model collections, the Samsung 870 EVO in 2 TB or 4 TB SATA configurations provides high-density storage at a lower cost per gigabyte where raw sequential bandwidth is less critical than capacity.

CPU and Motherboard

Intel Arc does not require an Intel host CPU — a PCIe 4.0 ×16 slot on any modern AMD or Intel consumer platform is sufficient for GPU-attached inference. However, for IPEX-LLM hybrid workflows where models partially exceed VRAM and spill to system RAM, an Intel platform with Core Ultra or 4th-gen Xeon unlocks AMX (Advanced Matrix Extensions) for accelerated CPU-side inference layers. For pure GPU inference within VRAM, CPU choice is secondary.

---

Cost Comparison: Arc vs Consumer Alternatives

Intel Arc's value argument centers on upfront cost and VRAM density. The table below synthesizes mid-2025 retail pricing observed across community price-tracking sources:

Street prices sourced from community price trackers and retailer listings as of mid-2025. Prices fluctuate — verify current pricing before purchasing.

For running 7B–13B models at home, the Arc A770 delivers the lowest entry cost for 16 GB VRAM among new-retail consumer options. The tradeoffs are real: driver maturity, CUDA-ecosystem compatibility, and prompt-processing throughput all favor NVIDIA. Users who depend on CUDA-only tooling (certain custom kernels, TensorRT-LLM) cannot run those tools on Arc without porting to SYCL or OpenVINO. For a look at Intel's higher-end inference offering, see Intel Arc Pro B60 Review 2025: Xe2 Workstation GPU Guide and Intel Arc Pro B60 Price: Is $999 Worth It in 2025?.

---

Practical Setup: Running llama.cpp on Intel Arc

For readers evaluating the software path before committing to hardware, the community-documented setup per llama.cpp's official SYCL backend documentation involves four key steps:

1. Install Intel oneAPI Base Toolkit — available for Linux and Windows from Intel's developer portal. The icpx SYCL compiler is the critical component.
2. Compile llama.cpp with SYCL enabled — pass -DGGML_SYCL=ON to the CMake build. The build process links against the installed oneAPI runtime.
3. Target the Arc GPU — set ONEAPI_DEVICE_SELECTOR=level_zero:0 in the environment before running inference to direct computation to the discrete Arc device rather than the integrated GPU.
4. Run inference with standard flags — llama.cpp's -ngl 99 flag offloads all model layers to GPU VRAM; reduce the value for partial CPU offloading on models larger than VRAM capacity.

Windows support is functional but Linux is more stable per community consensus on the llama.cpp repository. The setup is meaningfully more involved than CUDA, but the process is well-documented and the community troubleshooting base has grown substantially through 2024–2025.

---

Frequently Asked Questions

Can Intel Arc A770 run 70B parameter models? A 70B model at Q4_K_M quantization requires approximately 39 GB of memory — well beyond the A770's 16 GB. Running 70B on a single A770 requires heavy CPU offloading, which reduces token generation throughput dramatically. Models up to 13B at Q4–Q8 and 34B at Q2 are practical targets for the A770's 16 GB buffer.

Is llama.cpp the only software option for Arc? No. Intel's IPEX-LLM provides a HuggingFace-compatible Python API with Arc-optimized kernels. OpenVINO supports static deployment workflows optimized for Arc Pro and embedded scenarios. llama.cpp (SYCL or Vulkan) is the most community-tested path. OpenVINO is the fastest measured one: it decoded Qwen3-8B at 45.5 tok/s on an A770, versus 21.0 tok/s for llama.cpp SYCL, per arc-inference-bench. The IPEX-LLM repository was archived in January 2026, so it no longer receives active development.

Does Intel Arc work on Windows for LLM inference? Yes, with caveats. Community reports consistently note that Windows inference performance lags Linux on the same hardware due to driver and runtime overhead. The oneAPI toolkit ships for both platforms, but Linux is recommended for latency-sensitive workloads.

How does Arc A770 compare to AMD RX 7900 GRE for local LLM? Both offer 16 GB VRAM at comparable prices. AMD's ROCm ecosystem on Linux is considered more mature than Intel's SYCL path for llama.cpp, and community measurements on r/LocalLLaMA generally show the RX 7900 GRE achieving higher token-per-second rates at equivalent quantization levels. The A770 typically undercuts the RX 7900 GRE on street price.

Does the B580 outperform the A770 for LLM inference? The B580's Xe2 XMX units offer improved INT4/INT8 matrix-multiply throughput versus the A770's first-generation Xe XMX, per Intel's architecture documentation. At 12 GB versus 16 GB VRAM, however, the B580 is capacity-constrained for 13B Q8 models. Community reports suggest the B580 is faster per GB of VRAM but the A770's larger buffer wins when model retention in full GPU memory is the priority.

How many tokens per second does an Arc A770 or B580 get on Qwen3-8B? Per arc-inference-bench, the A770 decodes Qwen3-8B at 21.0 tok/s with llama.cpp SYCL at Q4_K_M and at 45.5 tok/s with OpenVINO INT4. A B580 owner in llama.cpp discussion #12570 measured 30.39 tok/s at Q4_K_M and 14.03 tok/s at Q8_0 on the Vulkan backend. Both cards stay far below their bandwidth ceilings, so newer drivers and runtimes still have room to improve these figures.

Will Intel improve Arc LLM inference support going forward? Intel has publicly committed to deeper oneAPI and OpenVINO integration with popular inference stacks, with driver improvements shipping on roughly a quarterly cadence. The trajectory per Intel's developer communications points toward continued narrowing of the gap with CUDA — though the maturity gap remains substantial as of October 2026.

---

Citations and sources

• https://github.com/diunox/arc-inference-bench
• https://github.com/ggml-org/llama.cpp/discussions/12570
• https://www.techpowerup.com/gpu-specs/arc-a770.c3914
• https://www.techpowerup.com/gpu-specs/arc-b580.c4244
• https://huggingface.co/Qwen/Qwen3-8B-GGUF
• https://huggingface.co/Qwen/Qwen3-14B-GGUF
• https://huggingface.co/unsloth/Qwen3-14B-GGUF
• https://github.com/ggerganov/llama.cpp
• https://github.com/ggerganov/llama.cpp/blob/master/docs/backend/SYCL.md
• https://github.com/intel/ipex-llm
• https://github.com/openvinotoolkit/openvino
• https://www.intel.com/content/www/us/en/developer/tools/oneapi/overview.html
• https://www.intel.com/content/www/us/en/developer/tools/oneapi/system-requirements.html
• https://www.tomshardware.com/pc-components/gpus/intel-arc-b580-review
• https://www.reddit.com/r/LocalLLaMA/

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.