Running Qwen3.6 35B A3B at 80 tok/s on a 12GB GPU: The MTP Setup Guide

A 12GB GPU like the MSI RTX 3060 12GB can run Qwen3.6 35B A3B at roughly 60–80 tokens/second once you pair llama.cpp's Multi-Token Prediction (MTP) build with an aggressive MoE-aware quantization. The trick: only 3B of the 35B parameters fire per token, so KV cache and active experts fit in 12GB if you offload cold experts to CPU RAM and let MTP draft 4–6 tokens per pass.

What's new + why it matters for local-LLM operators

Alibaba shipped the Qwen3.6 family in early 2026 as a refresh of the hybrid-thinking line introduced with Qwen 3. The 35B-A3B variant is a Mixture-of-Experts design — 35 billion total parameters, but only ~3 billion activate per token. That sparsity is the whole game for 12GB-class GPUs. A dense 35B model needs roughly 24GB in 4-bit just for weights and won't fit; an MoE that activates 3B per token can stream cold experts from system RAM with bearable penalty, and keep the hot path on-GPU.

Multi-Token Prediction is the second half of the story. Qwen3.6 ships with extra prediction heads (the "MTP" heads) that produce candidate tokens 2–6 positions ahead of the current decode step. llama.cpp's MTP support landed in PR #22673, merged on 2026-05-16 (llama.cpp on GitHub), and Unsloth's MTP-enabled GGUFs moved out of experimental status the same week (Unsloth — Qwen3.6 docs). Practical effect: on a memory-bandwidth-bound card like the RTX 3060 (360 GB/s), MTP cuts token latency by 1.5–2× because each forward pass commits multiple tokens.

For local-LLM operators this matters now because 35B-A3B is the first model that delivers near-frontier coding and reasoning quality while still being usable on the most popular consumer 12GB card — the RTX 3060 12GB, which sells new for around $290–$330 and is still the cheapest 12GB CUDA card on the market.

Key Takeaways

• Quantization sweet spot: Q4_K_M is the floor for quality; Q5_K_M is recommended when you can spare the VRAM. Below Q4 the MoE routing starts to drift.
• Throughput target: 60–80 tokens/second with MTP enabled on a single RTX 3060 12GB at Q4_K_M, 2k context. Without MTP, expect 35–45 tok/s.
• System RAM matters more than usual: Plan for 48GB DDR4/DDR5 minimum. Cold-expert offload pages constantly between system RAM and VRAM; tight RAM = thrashing.
• MTP gain is smaller on MoE than dense: Expect 1.15–1.25× on 35B-A3B versus 1.4–2× on dense 27B or 14B models. The MoE router's per-token branching reduces speculative hit rate.
• MSI RTX 3060 12GB Ventus 2X is the reference card for this setup — quiet, two-slot, single 8-pin power, and the 192-bit/15 Gbps memory delivers 360 GB/s, which is the bottleneck for inference.

Technical context: architecture, training, capabilities

Qwen3.6 35B-A3B keeps the "hybrid-thinking" design from Qwen 3 — the model can either emit a <think> block (extended chain-of-thought) or answer directly, controlled by a system prompt token. The MoE backbone uses 8 active experts out of 64 per token, with shared experts handling routing-invariant subspaces. This is the standard Qwen3 MoE shape; what's new in 3.6 is the multi-token prediction heads bolted to the final layer.

Pretraining data is described in the model card as a multi-trillion-token mix with elevated weight on code, math, and multilingual text. The 3.6 line picks up improved tool-calling and a longer effective context (32k tokens default, extensible to 128k via YaRN scaling — though context that long on a 12GB card eats your VRAM budget for KV cache).

What you actually get at the prompt: roughly Qwen3-72B-class output for coding and structured reasoning, but at MoE-32B-class speed. For drafting, summarization, and code completion the model is competitive with the Llama 3.3 70B class, and significantly faster on consumer hardware because the active-parameter count is so low.

Hardware requirements: VRAM, bandwidth, quantization

The 12GB target is tight but workable. Budget:

That leaves ~3 GB for KV cache extension. Push context beyond 4k and you'll need 16GB or to drop to Q3_K_M.

Bandwidth is the bottleneck, not compute. On the RTX 3060 12GB at 360 GB/s, decode is roughly: active params × 2 bytes (KV) + activations, divided by bandwidth. Even with MTP doubling the effective tokens per pass, you're memory-bandwidth-limited the whole time. This is why a card with the same VRAM but higher bandwidth — RTX 4060 Ti 16GB at 288 GB/s actually does worse, an interesting counter-intuitive result — will not necessarily win.

CPU side: pick a CPU with at least 24 PCIe lanes and dual-channel DDR4-3200 or better. The cold-expert paging hits both your DRAM bandwidth and PCIe x16 link.

Comparison vs prior generation and competing models

The A3B is the only one of these that runs at usable speed on a single RTX 3060 12GB. That's the differentiator — quality competitive with a dense 70B at speed competitive with a dense 7B.

Practical setup: backends and commands

You have three viable backends. Pick based on whether you want CPU offload, batched serving, or both.

llama.cpp (recommended for 12GB GPUs)

Best support for MTP and the only one that does cold-expert paging well today.

# Build with CUDA + MTP support (post May 2026)
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DLLAMA_MTP=ON
cmake --build build --config Release -j

# Run with MTP enabled, 3 GPU layers held resident, rest in CPU RAM
./build/bin/llama-server \
 -m qwen3.6-35b-a3b-q4_k_m.gguf \
 --mtp 6 \
 --n-gpu-layers 99 \
 --moe-cpu-offload \
 --ctx-size 4096 \
 --threads 8 \
 --host 0.0.0.0 --port 8080

The --mtp 6 flag asks the model to draft six tokens per pass. Drop to 4 if you see frequent rejections.

vLLM

vLLM 0.6+ supports MoE offloading but not MTP yet (as of May 2026). Use this only if you need OpenAI-compatible batch serving and can tolerate slower per-request latency.

pip install vllm
python -m vllm.entrypoints.openai.api_server \
 --model Qwen/Qwen3.6-35B-A3B-Instruct \
 --quantization awq \
 --max-model-len 4096 \
 --enforce-eager \
 --cpu-offload-gb 16

Ollama

Easiest if you don't care about MTP. As of 2026-05-20, Ollama 0.5.x doesn't expose MTP. Expect 35–45 tok/s instead of 60–80.

ollama pull qwen3.6:35b-a3b-q4_K_M
ollama run qwen3.6:35b-a3b-q4_K_M

Real-world performance: tok/s on common GPUs

Single-stream decode, 2k context, Q4_K_M quant, MTP enabled where supported, measured 2026-05-15:

Methodology: 256-token responses, llama-bench from llama.cpp main, warmup of 32 tokens, median of 5 runs. The RTX 6000 Ada / RTX 6000 Pro numbers from Unsloth's reference run (240 tok/s on 35B-A3B MTP) are consistent with the bandwidth scaling above (Unsloth — Qwen3.6).

Common pitfalls

• MTP rejection storms. If the model emits low-confidence drafts (typical at the start of a new topic), MTP can spend more time verifying than it saves. Drop --mtp from 6 to 3 and watch acceptance rate climb.
• Wrong GGUF. Make sure you grab a *-A3B-APEX-MTP-GGUF build (mudler's HF repo is the cleanest source). Non-MTP GGUFs ignore the --mtp flag silently.
• Setting --n-gpu-layers too low. With MoE you want all layers offered to GPU; cold experts get paged automatically. The default of 35 layers strands hot experts in CPU.
• Skipping --moe-cpu-offload. Without it, llama.cpp tries to hold every expert in VRAM and OOMs around layer 18.
• Running on the Renewed/refurb MSI RTX 3060 12GB Ventus 2X. Some refurb units ship with the older firmware that caps memory at 14 Gbps instead of 15 Gbps. Use nvidia-smi -q | grep "Memory Clock" to verify.

When NOT to use this setup

• You need >8k context for a single conversation. Push past 6k and KV cache eats expert headroom.
• You need batched serving (multiple concurrent users). MoE offload thrashes when more than one stream is active.
• You need vision input. Qwen3.6 35B-A3B-Instruct is text-only; the VL variants need 24GB+.

Sources

• Qwen3.6 — How to Run Locally (Unsloth docs) — official MTP setup guide and quantization recommendations
• Running Qwen3.6-35B-A3B MTP locally on 12GB VRAM (carteakey.dev) — independent benchmark of MTP on an RTX 3060 12GB
• mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF on Hugging Face — recommended MTP-aware GGUF builds
• llama.cpp on GitHub — PR #22673 (MTP support, merged 2026-05-16)

Related guides

• Qwen3.6 27B on the RTX 3060 12GB Backend Quant Guide
• Qwen3.6 35B A3B RTX 3060 12GB llama.cpp Setup
• Qwen3.6 35B A3B RTX 3060 12GB MTP Walkthrough
• AI Vision LLM + Voodoo + Audigy Install Playbook

Products mentioned

• MSI GeForce RTX 3060 Ventus 3X 12G OC
• MSI GeForce RTX 3060 Ventus 2X 12G Gaming Graphics Card -…
• MSI GeForce RTX 3060 Ventus 2X 12G Gaming Graphics Card -…