Skip to main content

Qwen3.6 35B-A3B on a 12GB GPU: 95 tok/s on 4070 Super, 34 on 3060

The 80 tok/s MTP figure came from an RTX 4070 Super, not an RTX 3060: cited 12GB tok/s rows, real GGUF sizes and working llama.cpp --n-cpu-moe flags.

As of Sept 2026, Qwen3.6-35B-A3B hits ~95 tok/s with llama.cpp MTP on a 12GB RTX 4070 Super; a 3060 12GB measures 31-39 tok/s with expert offload.

Full model-fit table for 12 GB cards: Best local LLM for a 12GB GPU → lists which models and quants fit in 12 GB, each with its cited tok/s and context.

What fits in 12 GB, for this question

3 rows from the full 12 GB model-fit table on the RTX 3060 12GB, picked for this page's question. A file fits when the card keeps at least 1.5 GB free after the weights (file sizes in decimal GB; the card's 12 GiB is 12.9 GB); speeds are cited community runs, and a row with no cited run says so. See every model →

ModelQuantFits in 12 GB?Cited speed on an RTX 3060 12GB
Qwen3 30B-A3B 18.6 GB fileQ4_K_MDoesn't fit5.7 GB over; runs only with CPU offload— (does not load into VRAM)
Qwen3.5-9B 5.7 GB fileQ4_K_MFits~7.2 GB left for context49.9 tok/s1 cited run, llama.cpp, per TYO Lab blog
gpt-oss-20b 21B-A3.6B · 12.1 GB fileMXFP4Tight fit~0.8 GB left: short contexts in VRAM; longer ones move some layers to the CPU75 tok/s1 cited run, llama.cpp (CUDA), per llama.cpp GitHub (discussion #15396)

Where this fits: the 12 GB tier of the local-LLM GPU picks by VRAM (one card per tier, with cited tok/s). Tiers: 8 GB · 12 GB · 16 GB · 24 GB · 48 GB+ · 128 GB unified.

Qwen3.6 35B-A3B on a 12GB GPU: 95 tok/s on 4070 Super, 34 on 3060

Qwen3.6 35B-A3B on a 12GB GPU with llama.cpp MTP: Direct Answer

As of October 2026, the fastest cited Qwen3.6-35B-A3B result on a 12GB card is 95.6 tok/s, logged on an RTX 4070 Super with Unsloth's UD-Q4_K_XL MTP build and --spec-type draft-mtp in the llama.cpp MTP pull-request thread. It was not an RTX 3060. An RTX 3060 12GB measured 20.5 tok/s without MTP and 31.5–34.5 tok/s with it, and a separate run without MTP reached 38.9 tok/s at UD-Q4_K_M. No 12GB card holds the weights: Unsloth's UD-Q4_K_M file is 22.13 GB, so every 12GB setup keeps expert weights in system RAM with --n-cpu-moe. The offload is what costs speed. InsiderLLM's sweep, updated October 15, 2026 on llama.cpp b10088, measured a 24GB RTX 3090 at 157.66 tok/s with every expert in VRAM and 52.76 tok/s with 24 of 40 expert layers in system RAM, the same -ncmoe 24 setting behind its 38.9 tok/s RTX 3060 run. With MTP at two draft tokens, a separate RTX 3090 study recorded 141.9 tok/s at UD-Q4_K_XL. The comparison beyond 12 GB below has the full table.

Affiliate disclosure: SpecPicks earns commissions on qualifying Amazon purchases.

The full cross-card comparison: Best GPUs for Running Local LLMs in 2026 has the VRAM-tier table from 12 GB through 48 GB, with a source link on every row.

What changed — October 2026

October 16, 2026 update:

October 2026 second pass:

Earlier, in the September 2026 refresh:

  • Corrected: the 80 tok/s headline. It came from an RTX 4070 Super, which logged 95.6 tok/s with MTP, not from an RTX 3060. Cited RTX 3060 runs land between 20.5 tok/s without MTP and 31.5–34.5 tok/s with it, and at 38.9 tok/s at UD-Q4_K_M. The title now states the cited figures.
  • Corrected: MTP flags. The --mtp and --mtp-draft-tokens flags never existed. The setup commands now use the merged --spec-type draft-mtp and --spec-draft-n-max flags, per the llama-server README.
  • Corrected: file size and download. Unsloth's UD-Q4_K_M file is 22.13 GB, not ~19 GB, so no 12GB card holds it and --n-cpu-moe expert offload is now explicit. The non-public Qwen/Qwen3.6-35B-A3B-GGUF repo was replaced with Unsloth's MTP GGUF.
  • Corrected: architecture. The model has 256 experts with 8 routed + 1 shared active per token, not "2 of 8", per the model card.
  • Added: cited 12GB-class rows with active-parameter bandwidth ceilings for the RTX 3060, RTX 4070 Super, RTX 5070 and Arc B580. They include Gemma 4 26B-A4B at ~40–55 tok/s on an RTX 3060 and gpt-oss-20b at 155.87 tok/s on an RTX 5070.
  • Removed: unsourced tables and claims. The first-person context-length table is gone, along with the uncited CPU-utilization, KV-cache and thermal claims. The context table now uses figures from the Qwen3.6-35B 12GB VRAM guide.

What's changed as of September 2026

This article first ran in May 2026, before llama.cpp's MTP support was merged. Six things have changed since then, and several of the original numbers were wrong:

  • MTP is merged, and the flags are different. PR #22673 ("llama + spec: MTP Support") was merged on May 16, 2026. It is enabled with --spec-type draft-mtp and --spec-draft-n-max N, per the llama-server README. The --mtp and --mtp-draft-tokens flags in the earlier version of this article do not exist.
  • The 80 tok/s headline came from a 4070 Super. The earlier version credited ~78 tok/s to a ZOTAC RTX 3060. No public benchmark supports that. The cited 3060 figures above top out near 39 tok/s.
  • The model doesn't fit in 12 GB, and there is no official Qwen GGUF. The Qwen/Qwen3.6-35B-A3B-GGUF repository in the old download command is not a public Hugging Face repo. Community GGUFs are larger than the "~19 GB" claimed: UD-Q4_K_M is 22.13 GB, UD-Q4_K_S is 20.89 GB and UD-IQ4_XS is 17.73 GB. The MTP builds, which include the prediction head, add roughly 0.5 GB (22.66 GB at UD-Q4_K_M). --n-gpu-layers 999 on its own does not work on a 12GB card.
  • MTP helps this MoE model less than it helps dense models. Unsloth's measurements show 1.4–2x for dense models but about 1.15–1.25x for MoE models. Unsloth recommends --spec-draft-n-max 2, since acceptance falls from 83% to 50% at 4 draft tokens. On a 12GB card with heavy expert offload, MTP can be neutral or negative. One RTX 3060 owner running -ncmoe 99 measured generation drop from ~26 to ~24 tok/s with MTP on.
  • The architecture description was wrong. Per the Qwen3.6-35B-A3B model card, the model has 40 layers, 256 experts and 8 routed + 1 shared expert active per token. It does not have "2 of 8" experts.
  • Newer MoE options fit the same setup. Gemma 4 26B-A4B (25.2B total, 3.8B active) and gpt-oss-20b both fit 12GB cards at small quants. Rows are below.

Current 12GB-class rows (September 2026)

Each ceiling is the card's memory bandwidth divided by the bytes streamed per token. For MoE models that is approximated as the active-parameter share of the GGUF file (file size × active ÷ total parameters). Ceilings assume everything is resident in VRAM. With --n-cpu-moe, part of each token's expert reads come from system RAM instead, so the real limit is lower. Every measured figure below sits under its ceiling. Bandwidth figures follow from each card's bus width and memory clock on TechPowerUp: RTX 3060 12GB, 192-bit GDDR6 at 15 Gbps = 360 GB/s; RTX 4070 Super, 192-bit GDDR6X at 21 Gbps = 504 GB/s; RTX 5070, 192-bit GDDR7 at 28 Gbps = 672 GB/s; and Arc B580, 192-bit GDDR6 at 19 Gbps = 456 GB/s.

GPU (12 GB)Model (quant, HF file size)Ceiling (active bytes)Measured decodeSetupSource
RTX 4070 SuperQwen3.6-35B-A3B MTP UD-Q4_K_XL, 22.85 GB~257 tok/s (504 ÷ 1.96 GB)95.6 tok/s (single sample, 98% draft acceptance); 73.2 tok/s on a code benchmark promptupstream MTP branch, --spec-type draft-mtp --spec-draft-n-max 7, -fitt 1024, 131K ctxPR #22673 comment
RTX 4070 SuperQwen3.6-35B-A3B byteshape IQ4_XS, ~18.0 GB~326 tok/s (504 ÷ 1.54 GB)79.8–97.0 tok/s upstream MTP; 110.24 tok/s ik_llama.cppRyzen 7 9700X, 48 GB DDR5-6000, 131K ctx, Q8 KVStartup Fortune write-up of the r/LocalLLaMA post
RTX 4070 SuperQwen3.6-35B-A3B APEX-I-Compact, ~16 GB~368 tok/s (504 ÷ 1.37 GB)64.0 tok/s at 32K ctx; 50.9 tok/s at 258K ctxno MTP, --n-cpu-moe 16–22, i5-14600KF, 32 GB DDR412GB VRAM guide
RTX 4070 (non-Super)Qwen3.6-35B-A3B, lmstudio-community GGUF (quant not stated)n/a (file size not stated)38.76 ± 0.82 tok/s (tg128, average of 3)no MTP, -ngl 99 --cpu-moe (all experts in system RAM)kenimo49 on DEV Community
RTX 3060 12GBQwen3.6-35B-A3B UD-Q4_K_M, 22.13 GB~190 tok/s (360 ÷ 1.90 GB)38.9 tok/s, flat through 8K ctxno MTP, -ngl 99 -ncmoe 24 -fa 1InsiderLLM
RTX 3060 12GBQwen3.6-35B-A3B MTP Q8_0, 37.8 GB~111 tok/s (360 ÷ 3.24 GB)20.0–20.5 tok/s without MTP; 31.5–34.5 tok/s with MTP n-max 3-ngl 5 -ncmoe 32, q8_0 KVPR #22673 comment
RTX 3060 12GBQwen3.6-35B-A3B UD-IQ3_XXS, 13.21 GB~318 tok/s (360 ÷ 1.13 GB)22.9 tok/sno MTP, -ncmoe 25, 64K ctx, Xeon E5-2650 v4, 31 GB DDR4Jean Brito
RTX 3060 12GBGemma 4 26B-A4B UD-IQ2_M, 10.01 GB~238 tok/s (360 ÷ 1.51 GB)~40–55 tok/sfully GPU-resident, 32K ctx, 3-bit "turbo3" KV cache (fork build)julien9679 repo
RTX 5070gpt-oss-20b Q8_0, 12.11 GB~323 tok/s (672 ÷ 2.08 GB)155.87 tok/s (tg128)llama.cpp b7083, VulkanPhoronix
Arc B580gpt-oss-20b Q8_0, 12.11 GB~219 tok/s (456 ÷ 2.08 GB)26.64 tok/s (tg128)llama.cpp b7083, Vulkan (Mesa ANV)Phoronix

Active-parameter counts: Qwen3.6-35B-A3B is 35B total / 3B active, Gemma 4 26B-A4B is 25.2B / 3.8B, and gpt-oss-20b is 21B / 3.6B.

Gaps in the public data. No cited Qwen3.6-35B-A3B decode figure turned up for the RTX 5070 12GB or the Arc B580. The plain RTX 4070 12GB now has one figure, 38.76 tok/s with every expert in system RAM, but no MTP run or partial-offload sweep. The one plain-4070 report in the MTP thread describes a regression on a development build, not a steady-state number. On the RTX 5070 and B580, gpt-oss-20b is the closest measured MoE proxy. The B580's 26.64 tok/s is far below its bandwidth ceiling on the Vulkan build Phoronix used.

Beyond 12 GB: Qwen3.6-35B-A3B on 8–24 GB hardware (October 2026)

The rows above answer the 12GB question. This table shows what changes as memory goes up or down. Each figure was checked against its source page in October 2026. Ceilings use the same active-parameter method as the 12GB table: memory bandwidth ÷ (GGUF file size × 3B ÷ 35B active share). Rows are marked n/a where weights sit in shared or system memory with no single bandwidth figure. File sizes come from Unsloth's GGUF listing unless the source states its own. Memory bandwidth: RTX 3090, 384-bit GDDR6X at 19.5 Gbps ≈ 936 GB/s; RTX 4000 SFF Ada, 160-bit GDDR6 at 280 GB/s; Jetson Orin NX 16GB, 128-bit LPDDR5 at 102.4 GB/s.

Hardware (memory)Quant, file sizeCeiling (active bytes)Measured decodeSetupSource
RTX 3090 (24 GB)UD-Q4_K_XL, 22.36 GB~488 tok/s (936 ÷ 1.92 GB)115.7 tok/s without speculation; 141.9 tok/s with MTP n=2 (+22.7%, 78.4% acceptance)fully GPU-resident, greedy decode, ten fixed promptsthc1006 speculative-decoding repo
RTX 3090 (24 GB)UD-Q4_K_M, 22.13 GB~493 tok/s (936 ÷ 1.90 GB) at -ncmoe 0157.66 tok/s with all experts in VRAM; 52.76 tok/s at -ncmoe 24 (10,550 MiB VRAM peak); 36.11 tok/s at -ncmoe 40no MTP, llama.cpp b10088, headless, llama-bench tg128InsiderLLM, updated Oct 15, 2026
2× RTX 3060 (2 × 12 GB)UD-Q4_K_XL MTPn/a (split across two cards and system RAM)72.6 tok/s decode with MTP; prefill 512 tok/s on a 106,252-token prompt--spec-type draft-mtp --spec-draft-n-max 3, --split-mode layer, 128K ctx, q8_0 KV, llama.cpp b11146, i9-9980XE, 64 GB DDR4; no MTP-off baseline reportedlocal-ai-registry PR #145, Oct 1, 2026
RTX 3090 (24 GB)UD-IQ4_NL_XL, 19.5 GB~560 tok/s (936 ÷ 1.67 GB)139.6 tok/s at 89,600 ctx (CUDA); 122.4 tok/s on Vulkan; 89.1 tok/s at 262,144 ctx with 10 layers' FFNs offloadedno MTP; prefill 3,360 tok/s on CUDAGiles Thomas
RTX 4000 SFF Ada (20 GB)Q3_K_S with MTP layer, 15.55 GB~210 tok/s (280 ÷ 1.33 GB)83.3 tok/s; prefill 1,692.6 tok/sfully GPU-resident, 256K ctx with q4 KV, 70 W boardZimaSpace
RTX 4000 SFF Ada (20 GB)UD-IQ3_XXS, 13.21 GB~247 tok/s (280 ÷ 1.13 GB)74.5 tok/s; 39.2 tok/s at 76K tokens in an agent session; 33.0 tok/s at 101Ksame, 69.8 W measured drawZimaSpace
Arc B390 iGPU (32 GB shared LPDDR5)UD-Q4_K_M MTP, 22 GiBn/a (shared memory)32.75 tok/s Vulkan (33.01 with Level Zero enabled); 30.62 tok/s SYCL; 19.44 tok/s CPU-onlyCore Ultra X7 358H, llama-bench tg128, q4_0 KVgrigio.org
Jetson Orin NX (16 GB unified)IQ2_M, 11.06 GiB (11.88 GB)~101 tok/s (102.4 ÷ 1.02 GB)14.65 tok/s without MTP; 19.20 tok/s with MTP n=1 (+31%)8,129-token prompt, q8_0 KVDavid Noel Ng
Radeon RX 7700S (8 GB, laptop)IQ4_XSn/a (hybrid offload)27 tok/s (the owner's reported ceiling)Vulkan, --n-cpu-moe 26, 64 GB DDR5, 32K ctxllama.cpp discussion #21112

What the table adds to the 12GB picture:

  • 24 GB removes the offload penalty. A 3090 holds the whole 4-bit file, and its 115.7 tok/s baseline is about 3× the best cited RTX 3060 run without MTP (38.9 tok/s). The RTX 3090 also has roughly 2.6× the 3060's bandwidth (936 vs 360 GB/s), so not all of that gap is offload.
  • Offload costs more than the GPU tier. In InsiderLLM's sweep, the same RTX 3090 fell from 157.66 tok/s fully resident to 52.76 tok/s at -ncmoe 24. At that same setting the RTX 3060 measured 38.9 tok/s. Moving 24 expert layers to RAM cut the 3090's speed by about two thirds, while swapping the 3090 for a 3060 at equal offload cost about a quarter.
  • A second 12GB card is the cheapest jump. Two RTX 3060s with MTP reached 72.6 tok/s at 128K context, above every cited single-3060 run, though the source reports no MTP-off figure to isolate the MTP share.
  • MTP pays when the weights are resident. On the 3090 with everything in VRAM, MTP at two draft tokens added 22.7%. On the Jetson, which holds an IQ2_M file in unified memory, it added about 31%. That contrasts with the 12GB rows, where the RTX 3060 with every expert offloaded got slightly slower with MTP. The 3090 repo notes its MTP interval comes from a single invocation, so the true spread is likely wider.
  • A 20 GB, 70 W workstation card beats every cited 12GB card except the 4070 Super. The RTX 4000 SFF Ada's 74.5–83.3 tok/s comes from 3-bit quants that fit entirely in VRAM, despite only 280 GB/s of bandwidth. The source also notes that its Q3_K_S file will not load on a 16 GB card.
  • No discrete GPU at all is roughly RTX 3060 speed. The Arc B390 integrated GPU runs the full 22 GiB Q4_K_M file from shared memory at 32.75 tok/s, inside the 3060's cited 20.5–38.9 tok/s range. On the same laptop, the CPU alone managed 19.44 tok/s.

What MTP is and why the gain is smaller on 12 GB

Multi-Token Prediction (MTP) is a form of speculative decoding. Qwen3.6 ships with an extra prediction head, trained with multi-step MTP per the model card, that drafts the next few tokens. The main model then verifies them in a single forward pass. llama.cpp loads that head from the same GGUF, so no separate draft model is needed (PR #22673). You need an MTP-enabled GGUF, such as Unsloth's MTP build. An ordinary GGUF has no head to draft with.

Two things shrink the gain on a 12GB card:

The practical rule: turn MTP on, then compare --spec-draft-n-max values of 1, 2 and 3 against MTP off on your own card before settling on one.

Best Value: ZOTAC Gaming GeForce RTX 3060 Twin Edge OC 12GB (B08W8DGK3X)

The ZOTAC RTX 3060 Twin Edge OC 12GB is a dual-fan RTX 3060 with a 1777 MHz boost clock and 192-bit GDDR6 (360 GB/s). For Qwen3.6-35B-A3B it is a ~20–39 tok/s card, not an 80 tok/s one. The spread depends on quant, host CPU and whether MTP pays off, per the cited rows above. That is still comfortably interactive for chat. The price case has also changed: the catalog listing on its product page shows $499.99 at the time of this refresh, well above the $290–330 this article quoted in May.

Best alternative: MSI GeForce RTX 3060 Ventus 2X 12G (B08WRVQ4KR)

The MSI RTX 3060 Ventus 2X 12G uses the same GA106 chip and 12 GB of GDDR6, so the same tok/s rows apply. No public Qwen3.6 benchmark separates the two cards. Choose between them on price and cooler size. Its catalog listing on the product page also shows $499.99 at the time of this refresh.

If you are buying new for this workload, compare that price against a 12GB RTX 4070 Super. The 4070 Super is the card behind every 80+ tok/s figure above, with 504 GB/s of memory bandwidth against the 3060's 360 GB/s.

Companion CPU and system RAM: AMD Ryzen 7 5800X (B0815XFSGK)

With --n-cpu-moe, most expert weights sit in system RAM and the CPU computes those layers, so the host matters more here than in a fully GPU-resident setup. The cited 12GB setups ran on a Ryzen 7 9700X with 48 GB DDR5-6000 (the 80–110 tok/s 4070 Super runs), an i5-14600KF with 32 GB DDR4, and a 12-core Xeon E5-2650 v4 with 31 GB DDR4 (22.9 tok/s at IQ3_XXS). An 8-core AMD Ryzen 7 5800X with 32 GB of dual-channel DDR4 is a reasonable AM4 pairing. No public benchmark isolates its effect on this model, so treat CPU gains as varying by workload.

Size system RAM to the file. A 22 GB UD-Q4_K_M with roughly 10–11 GB on the GPU leaves about 11–12 GB of experts in RAM, plus the OS and runtime. That makes 32 GB the practical floor. Both cited 32 GB-class systems ran 64K contexts or more.

llama.cpp setup commands

Bring-up on a Linux host with a 12GB NVIDIA card and a current CUDA toolkit. Every flag below appears in the llama-server README:

bash
# Build llama.cpp with CUDA (MTP support merged in PR #22673, May 2026)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j"$(nproc)"

# Qwen publishes no official GGUF for this model. -hf pulls Unsloth's MTP build;
# UD-Q4_K_M is 22.66 GB, so expert weights must be split between GPU and RAM.
./build/bin/llama-server \
 -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_M \
 --no-mmproj \
 --n-gpu-layers all \
 --n-cpu-moe 24 \
 --ctx-size 32768 \
 --parallel 1 \
 --flash-attn on \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 --spec-type draft-mtp \
 --spec-draft-n-max 2 \
 --port 8080

Key flags explained:

  • --n-gpu-layers all with --n-cpu-moe 24: put every layer on the GPU, then keep the MoE expert weights of the first 24 layers in system RAM. Raise N if you run out of memory and lower it if VRAM is left over. The 12GB guide above found the best N rises with context: 16 at 32K, 17 at 64K, 20 at 128K and 22 at 258K. -ot/--override-tensor does the same job with a regex if you need finer control.
  • Alternatively, omit --n-cpu-moe and let --fit (on by default) size the split automatically. The 95.6 tok/s 4070 Super run used -fitt 1024 (a 1 GiB VRAM margin) instead of a manual N.
  • --parallel 1: a single server slot. The default is auto, and extra slots multiply KV cache.
  • --cache-type-k/v q8_0: halves KV-cache memory compared with the f16 default, leaving room for more experts on the GPU.
  • --spec-type draft-mtp --spec-draft-n-max 2: enable MTP with two draft tokens. The llama-server default is 3; Unsloth recommends 2, and the 3060 owner cited above got the best results at 3. Measure with MTP off as well.

Throughput by context length (RTX 4070 Super 12GB)

Figures from the Qwen3.6-35B 12GB VRAM guide (RTX 4070 Super, i5-14600KF, 32 GB DDR4, ~16 GB APEX-I-Compact quant, no MTP):

ContextKV cache--n-cpu-moeDecode tok/sPrefill
32Kq8_01664.0not reported
64Kq8_01760.7649 tok/s (45K-token input)
128Kq8_02055.3~250 tok/s
258Kq4_02250.9527 tok/s (45K-token input)

The same guide reports that at 258K context, --n-cpu-moe values below 22 cause PCIe thrashing and drop decode to 20–25 tok/s. Offloading too few experts is worse than offloading too many.

When MTP doesn't help: edge cases

  • Heavy expert offload. With nearly all experts in RAM (-ncmoe 99), one 3060 owner measured ~26 tok/s without MTP and ~24 tok/s with it.
  • Too many draft tokens. Unsloth measured acceptance falling from 83% to 50% at 4 draft tokens. The 3060 owner's generation speed also dropped at n-max 4.
  • Separate draft models under offload. On an RTX 3060 at -ncmoe 24, a Qwen 3.5 0.8B drafter took decode from 38.3 to 15.9 tok/s (0.42x). MTP's built-in head avoids loading a second model, but the same VRAM pressure applies.
  • Long prompts. The PR notes that prompt processing slows down with MTP enabled because of device-to-host embedding transfers (PR #22673).
  • Vision input. A slot-corruption bug with MTP plus image input was reported on an RTX 5070 Ti (issue #22867, since closed). If you hit it on an older build, update llama.cpp or run with --no-mmproj.

For other workloads the speedup varies by prompt. Compare MTP on and off on your own prompts.

Common pitfalls to avoid

  1. Assuming Q4 fits in 12 GB. It does not. UD-Q4_K_M is 22.13 GB and UD-Q5_K_M is 26.46 GB. Plan on --n-cpu-moe or --fit at any 4-bit quant. Only quants of about 11–12 GB or less (UD-IQ2_M and below) come close to fitting entirely on the GPU.
  2. Using a non-MTP GGUF with --spec-type draft-mtp. The prediction head has to be in the file. Use an MTP build such as unsloth/Qwen3.6-35B-A3B-MTP-GGUF.
  3. Using the old flag names. --mtp, --mtp-draft-tokens and --draft-max are not valid. The README marks --draft-max as removed in favor of --spec-draft-n-max (llama-server README).
  4. Leaving --parallel on auto. Extra server slots each reserve KV cache and can push a 12GB card into out-of-memory errors at long contexts.
  5. Too little system RAM. With 11+ GB of experts in RAM, 16 GB systems swap. The cited 12GB setups used 31–48 GB.

When NOT to buy a 3060 12GB for local LLM

If you want the 80–110 tok/s figures that circulate for this model, the RTX 3060 is the wrong card. Those came from an RTX 4070 Super with 504 GB/s of bandwidth, versus 360 GB/s on the 3060 (see the TechPowerUp specs linked above). Dense 70B-class models need far more memory than 12 GB at any usable quant. If you only run small dense models, a 12GB card is more than you need.

Citations and sources

  • llama.cpp PR #22673, "llama + spec: MTP Support" (merged May 16, 2026): https://github.com/ggml-org/llama.cpp/pull/22673
  • PR #22673 comment, RTX 4070 Super 12GB, 95.6 tok/s with draft-mtp: https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4443537301
  • PR #22673 comment, RTX 3060 12GB, 20.5 to 31.5–34.5 tok/s with MTP: https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4393961295
  • PR #22673 comment, RTX 3060 12GB with -ncmoe 99, MTP ~26 to ~24 tok/s: https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4409603142
  • PR #22673 comment, RTX 4070 12GB regression report: https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4453841550
  • PR #22673 comment, 8GB RTX 3060 Ti MTP VRAM cost: https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4471492776
  • PR #22673 comment, RTX 3060 laptop 6GB MTP draft sweep: https://github.com/ggml-org/llama.cpp/pull/22673#issuecomment-4375891776
  • thc1006, llama.cpp speculative decoding on one RTX 3090 (115.7 / 141.9 tok/s): https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090
  • Giles Thomas, Benchmarking Qwen 3.6 35B MoE on an RTX 3090 (139.6 tok/s CUDA): https://www.gilesthomas.com/2026/07/benchmarking-qwen-3-6-35b-moe-rtx-3090
  • ZimaSpace, RTX 4000 SFF Ada LLM inference on ZimaOS (74.5 / 83.3 tok/s): https://www.zimaspace.com/docs/hardware/rtx-4000-ada-on-zimaos
  • NVIDIA RTX 4000 SFF Ada datasheet (280 GB/s, 70 W): https://www.nvidia.com/content/dam/en-zz/Solutions/products/workstations/nvidia-rtx-4000-sff-datasheet.pdf
  • grigio.org, llama.cpp backends on Intel Panther Lake / Arc B390 (32.75 tok/s Vulkan): https://grigio.org/benchmarking-llama-cpp-backends-on-intel-panther-lake-vulkan-vs-sycl-vs-openvino-vs-cpu/
  • David Noel Ng, LLMs on a 16GB Jetson Orin NX (14.65 / 19.20 tok/s): https://dnhkng.github.io/posts/jetson-orin-nx-vram-tuning/
  • NVIDIA Jetson Orin specifications (Orin NX 16GB, 102.4 GB/s): https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/
  • llama.cpp discussion #21112, Radeon RX 7700S report (27 tok/s): https://github.com/ggml-org/llama.cpp/discussions/21112
  • itcreations, GeForce RTX 3090 specifications (19.5 Gbps, 384-bit, ~936 GB/s): https://www.itcreations.com/nvidia-gpu/nvidia-geforce-rtx-3090-gpu
  • llama.cpp issue #22867, MTP with vision input: https://github.com/ggml-org/llama.cpp/issues/22867
  • llama-server README (flag reference): https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
  • Startup Fortune, janvitos r/LocalLLaMA RTX 4070 Super benchmark (79.8–97.0 / 110.24 tok/s): https://startupfortune.com/110-toks-on-rtx-4070-super-with-qwen36-35b/
  • InsiderLLM, Qwen3.6 35B on RTX 3060 12GB at 38.9 tok/s and RTX 3090 -ncmoe sweep (157.66 / 52.76 / 36.11 tok/s), updated Oct 15, 2026: https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/
  • local-ai-registry PR #145, Qwen3.6-35B-A3B MTP on 2× RTX 3060 12GB (72.6 tok/s, llama.cpp b11146, Oct 1, 2026): https://github.com/0xSero/local-ai-registry/pull/145
  • kenimo49, Qwen 3.6 vs 3.5 on an RTX 4070 12GB (38.76 tok/s with --cpu-moe), DEV Community: https://dev.to/kenimo49/qwen-36-vs-35-same-37-toks-on-rtx-4070-43-on-frontend-generation-3i0o
  • Jean Brito, Qwen3.6 35B on RTX 3060 via -ncmoe (22.9 tok/s): https://jeanfbrito.github.io/posts/running-qwen3.6-35b-moe-rtx3060-ncmoe/
  • Qwen3.6-35B 12GB VRAM guide (RTX 4070 Super context scaling): https://github.com/shiqikuangsan31/Qwen3.6-35B-12GB-VRAM-Guide
  • Gemma 4 26B-A4B on RTX 3060 12GB: https://github.com/julien9679/Gemma-4-26B-A4B-on-RTX-3060-12GB
  • Phoronix, llama.cpp Vulkan GPU comparison (gpt-oss-20b tg128): https://www.phoronix.com/review/llama-cpp-vulkan-eoy2025/4
  • Unsloth Qwen3.6 guide (MTP speedups, draft-token advice): https://unsloth.ai/docs/models/qwen3.6
  • Qwen3.6-35B-A3B model card: https://huggingface.co/Qwen/Qwen3.6-35B-A3B
  • Unsloth Qwen3.6-35B-A3B GGUF file listing: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
  • Unsloth Qwen3.6-35B-A3B MTP GGUF file listing: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF
  • byteshape Qwen3.6-35B-A3B GGUF file listing: https://huggingface.co/byteshape/Qwen3.6-35B-A3B-GGUF
  • am17an Qwen3.6-35BA3B MTP GGUF (37.8 GB Q8_0): https://huggingface.co/am17an/Qwen3.6-35BA3B-MTP-GGUF
  • Gemma 4 26B-A4B model card: https://huggingface.co/google/gemma-4-26B-A4B-it
  • Unsloth Gemma 4 26B-A4B GGUF file listing: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF
  • gpt-oss-20b model card: https://huggingface.co/openai/gpt-oss-20b
  • Unsloth gpt-oss-20b GGUF file listing: https://huggingface.co/unsloth/gpt-oss-20b-GGUF
  • TechPowerUp RTX 3060 12GB specs: https://www.techpowerup.com/gpu-specs/geforce-rtx-3060-12-gb.c3682
  • TechPowerUp RTX 4070 Super specs: https://www.techpowerup.com/gpu-specs/geforce-rtx-4070-super.c4186
  • TechPowerUp RTX 5070 specs: https://www.techpowerup.com/gpu-specs/geforce-rtx-5070.c4218
  • TechPowerUp Arc B580 specs: https://www.techpowerup.com/gpu-specs/arc-b580.c4244

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.

Products mentioned

Products mentioned in this article

Amazon & eBay listings, plus full specs and alternatives on each product page.

As an Amazon Associate, SpecPicks earns from qualifying purchases; we also earn on qualifying eBay purchases via the eBay Partner Network. Prices shown were last tracked at crawl time and may vary — check the listing for the current price.

Watch a review

MSI Gaming GeForce RTX 3060 12GB: A Deep Dive into Performance and Features — Digital David on YouTube

Frequently asked questions

What does 'A3B' mean in Qwen3.6 35B A3B?
It means about 3 billion active parameters. Per the Qwen model card, the model has 35 billion parameters in total across 40 layers, with 256 experts, and 8 routed plus 1 shared expert run for each token. Memory cost follows the 35B total (a 4-bit GGUF is about 22 GB), while per-token compute and memory reads follow the roughly 3B active share. That is why a 12GB card can run it at interactive speed with expert weights offloaded to system RAM.
What is llama.cpp MTP and how do I enable it?
Multi-Token Prediction is speculative decoding that uses a prediction head built into Qwen3.6: the head drafts a few tokens and the main model verifies them in one pass. llama.cpp merged support in PR #22673 on May 16, 2026. Enable it with --spec-type draft-mtp and --spec-draft-n-max 2 (or 3), and use an MTP GGUF such as unsloth/Qwen3.6-35B-A3B-MTP-GGUF. Flags named --mtp or --mtp-draft-tokens do not exist.
How fast is Qwen3.6 35B-A3B on a 12GB GPU as of September 2026?
It depends on the card. An RTX 4070 Super 12GB has logged 95.6 tok/s with MTP, and 79.8 to 97.0 tok/s on upstream llama.cpp in a separately reported run (110.24 tok/s on the ik_llama.cpp fork). Cited RTX 3060 12GB figures are 38.9 tok/s at UD-Q4_K_M without MTP, and 31.5 to 34.5 tok/s with MTP on a Q8_0 build (up from about 20 tok/s). No cited Qwen3.6 figure exists yet for the RTX 5070 12GB or Arc B580.
How much system RAM do I need alongside the 12GB GPU?
Plan on at least 32 GB. The UD-Q4_K_M file is 22.13 GB, so with --n-cpu-moe roughly 11 to 12 GB of expert weights sit in system RAM, plus the OS and runtime. The cited 12GB setups used 31 to 48 GB of RAM, and a 32 GB DDR4 system ran contexts up to 258K with q4_0 KV cache.
Is the RTX 3060 12GB still worth buying in 2026 for Qwen3.6 35B-A3B?
Only as a budget card that you already own or can find cheaply. The cited figures put it at about 20 to 39 tok/s on this model, which is usable for chat but well short of the 80+ tok/s headline. That headline came from an RTX 4070 Super. The ZOTAC and MSI 3060 12GB listings in the SpecPicks catalog show $499.99 at the time of this refresh, so compare that against a 12GB RTX 4070 Super before buying for this workload.

— Mike Perry · Updated 2026-10-16

Parts this article names

Amazon Associate — prices tracked 2026-11-20, may vary.