Qwen 3.6 27B Quantization Showdown: BF16 vs Q8_0 vs Q4_K_M on Consumer GPUs

Qwen 3.6 27B fits in 24GB of VRAM at Q4_K_M, and the fastest measured single-GPU run at this model size is now an RTX 5090. It ran the same-size Qwen 3.8 27B at 77.8 tok/s in llama.cpp with a 16.5 GB UD-Q4_K_M file, and at 129.7 tok/s with Ollama's built-in MTP draft, per ComputingForGeeks' October 2026 benchmark. On the value pick, a single RTX 3090, the Qwen 3.6 Q4_K_M GGUF runs at 38.3 tok/s in llama.cpp and 61.4 tok/s with multi-token prediction (MTP) switched on, per InsiderLLM's measurement. On a newer llama.cpp build (b10088), a llama-bench run measured 41.77 tok/s decode for Qwen 3.6 27B at UD-Q4_K_XL on the same card (InsiderLLM, August 2026). The weights are 16.8 GB in Unsloth's build and 18.0 GB in bartowski's. A 32k-token FP16 KV cache adds only about 2.1 GB, because just 16 of the model's 64 layers use full attention (Qwen's model config). Q8_0 (28.6–29.1 GB) needs a 32GB card, and full BF16 (53.8–54.7 GB of weights) needs more than 48GB. A 16GB card only fits IQ4_XS or smaller: the RX 9060 XT 16GB, $349 at launch (heise), measured 20.1 tok/s at UD-IQ4_XS (llama.cpp discussion #15021). For most single-GPU owners, Q4_K_M on a 24GB card is still the default.

What changed since the April 2026 version

This guide first ran in April 2026, the month Qwen 3.6 27B shipped. Five things have changed since then, and the tables below have been rebuilt around them.

• MTP landed in mainline llama.cpp. Speculative decoding with the model's own MTP head was merged in llama.cpp PR #22673 on May 16, 2026. On a single RTX 3090 it took Q4_K_M from 38.3 to 61.4 tok/s at a 10k context, a 1.86x wall-clock speedup, and the MTP layer costs about 2.5 GiB of extra VRAM while drafting (InsiderLLM).
• vLLM is faster than llama.cpp on NVIDIA for one user. A 4x RTX 3090 lab measured 70–72 tok/s on narrative prompts for one RTX 3090 running an AutoRound quant in vLLM. It measured 117–118 tok/s with two cards in tensor parallel at 100K context (tfriedel/qwen3.6-rtx3090-lab, re-measured 2026-07-13).
• Two 16GB cards are now a real option. A day-one bake-off ran Q4_K_M on 2x RTX 5060 Ti 16GB (about $400 each). llama.cpp reached a 65,536-token context there, and vLLM with an NVFP4 quant served 345 tok/s of aggregate chat throughput at 64 concurrent requests (LLMKube).
• A successor exists. Qwen 3.8 27B was released on August 14, 2026. It keeps the same decoder size, and its Q4_K_M file is also 16.8 GB. Day-one RTX 3090 decode was about 41 tok/s, under 1% behind 3.6 (RunAIHome). A later llama-bench run on llama.cpp b10088 measured 41.49 tok/s for 3.8 against 41.77 tok/s for 3.6 at UD-Q4_K_XL, 0.67% slower (InsiderLLM). Every VRAM figure here carries over to 3.8.
• Two facts in the first version were wrong. The model's native context is 262,144 tokens, not 128k, and it ships under Apache 2.0, not a custom Qwen license (model card). Its attention is also a hybrid design, described below, rather than plain grouped-query attention.

Added in October 2026:

• Measured RTX 4090, RTX 5090 and L40S figures. The GPU table no longer says "no cited single-user figure" for the 4090 and 5090. On Qwen 3.8 27B at UD-Q4_K_M, llama.cpp single-stream decode measured 49.5 tok/s on an RTX 4090, 77.8 on an RTX 5090 and 39.1 on an L40S 48GB. vLLM with a 19.5 GB INT4 (W4A16) quant measured 54.3, 85.7 and 43.7 tok/s on the same cards (ComputingForGeeks).
• A 16GB Radeon result with MTP. An RX 9070 XT ran a Qwen 3.6 27B IQ3_M build at 25.88 tok/s in llama.cpp on ROCm, and at 46.06 tok/s with MTP at a 62.7% draft acceptance rate, using 14.46 GB of its 16 GB (Lemmy.World LocalLLaMA post).
• The cheapest new card with a measured figure. The RX 9060 XT 16GB ($349 launch price, 320 GB/s per heise) ran Qwen 3.8 27B UD-IQ4_XS (14.25 GB, Unsloth) at 20.1 tok/s decode and 709 tok/s prompt processing on Vulkan (llama.cpp #15021).
• Integrated graphics is not a fit. An Intel Panther Lake Xe3 iGPU (12 EUs) ran Qwen 3.6 27B Q4_K_M at 3.55 tok/s on the SYCL backend (llama.cpp #23313).
• New comparison table. All of the above is collected in the measured-decode table below, with a source on every row.

Why Qwen 3.6 27B is the sweet-spot model for 24GB cards

Two properties explain why this model became the default for 24GB cards.

The first is the architecture. Per the model card, the 64 layers are arranged as 16 repeats of three Gated DeltaNet (linear-attention) blocks followed by one Gated Attention block. The Gated Attention block uses 24 query heads and 4 KV heads.

• Only the 16 full-attention layers keep a KV cache that grows with context.
• The DeltaNet layers hold a fixed-size state.

Long context is therefore cheap. At FP16 the cache grows by about 64 KiB per token, or roughly 66 MB per 1,000 tokens. The native 262,144-token window costs about 17 GB of FP16 cache, or half that with a Q8 cache.

The second is capability per gigabyte. Qwen's published scores for the 27B include:

These are BF16 figures from the model card, not measurements of a quantized build.

Key Takeaways

• VRAM by quant (weights only, Unsloth GGUF file sizes): IQ4_XS 15.4 GB · Q4_K_M 16.8 GB · Q5_K_M 19.5 GB · Q6_K 22.5 GB · Q8_0 28.6 GB · BF16 53.8 GB. Add about 2.1 GB of FP16 KV cache at 32k context. The cache grows linearly with context length.
• Measured single-RTX 3090 speed, Q4_K_M:
• 38.3 tok/s in llama.cpp; 41.77 tok/s (llama-bench tg128, UD-Q4_K_XL) on the newer b10088 build (InsiderLLM, August 2026).
• 61.4 tok/s with MTP (InsiderLLM).
• 70–72 tok/s in vLLM with an AutoRound quant (tfriedel lab).
• Quality: Qwen does not publish per-quant scores. For how far each GGUF format drifts from BF16, use a KL-divergence comparison rather than MMLU deltas; see the quantization formats KLD comparison.
• Long context: IQ4_XS leaves the most KV-cache room on a single 24GB card. At about 66 MB of FP16 cache per 1,000 tokens, 100k tokens adds about 6.6 GB on top of the weights.
• Recommended pick: a used RTX 3090 with MTP enabled for value, and a 32GB card for Q8_0.

How much VRAM does each quantization level require?

The table covers the model weights only, using Unsloth's GGUF file sizes. bartowski's builds run 0.4–1.2 GB larger at the same quant name. The KV cache is extra (formula below).

Headroom assumes roughly 1.5 GB for the runtime and CUDA context, and varies by driver and runtime.

KV cache at FP16 runs about 66 MB per 1,000 tokens, which comes from 16 full-attention layers × 4 KV heads × 256 head dim × K and V × 2 bytes (config.json).

• 32k context is about 2.1 GB.
• 64k is about 4.3 GB.
• 110k is about 7.2 GB.
• Quantizing the cache to Q8 halves each figure.

Which GPU should I pair with Qwen 3.6 27B?

Measured Qwen 3.6 figures exist for the RTX 3090. For the RTX 4090 and RTX 5090 the measured column uses Qwen 3.8 27B, which has the same decoder size and a Q4_K_M file within 0.4 GB of it; the next section has the full table. The ceiling column is bandwidth divided by the 16.8 GB of weights read per token, which is the most a non-speculative decode can reach. Measured llama.cpp speed lands well under the ceiling (38.3 against about 56 on the 3090). MTP can exceed it because each forward pass can accept several drafted tokens.

The RTX 3090 is the value pick for this model. It has the same 24GB ceiling as a 4090, and with MTP enabled the measured speed is above the non-speculative ceiling of either card. A 32GB card is the upgrade that changes what fits: Q8_0 plus a working context, or Q4_K_M plus a very long one. It does not change the tier of model you can run.

Measured Qwen 27B decode beyond the RTX 3090

These are single-stream decode figures for Qwen 3.6 27B and its same-size successor, Qwen 3.8 27B, on cards other than the RTX 3090. Each row comes from a public benchmark linked in the last column. Speculative (MTP) figures are listed separately because they can exceed the bandwidth ceiling.

What the table says about the headline question:

• The RTX 5090 is the only single card here above 75 tok/s without speculation. Its 77.8 tok/s in llama.cpp is about twice the RTX 3090's 38.3 tok/s on Qwen 3.6, and RunAIHome measured Qwen 3.8 within 1% of Qwen 3.6 on the same 3090 (RunAIHome), so the comparison holds across the two releases.
• An RTX 4090 is about 29% faster than a 3090 in llama.cpp (49.5 against 38.3 tok/s), but a 3090 with MTP (61.4 tok/s) is faster than a 4090 without it.
• A 16GB card works only at an IQ4_XS-class quant or below. The RX 9060 XT's 320 GB/s (heise) divided by the 14.25 GB IQ4_XS file gives a ceiling of about 22 tok/s, and the measured 20.1 tok/s sits close to it, so a faster runtime will not add much. The RX 9070 XT needed a 3-bit quant and MTP to reach 46.06 tok/s.
• A data-center L40S is slower than a 4090 here (39.1 against 49.5 tok/s in llama.cpp). Its advantage is 48GB of memory and batch throughput, not single-user speed.
• Integrated graphics is not a practical host. At 3.55 tok/s on the Panther Lake iGPU, a 500-token answer takes over two minutes.

Does IQ4_XS fit 110k context on a single 24GB GPU?

Yes, with a quantized cache. Here is the arithmetic:

That total is on the edge of a 24GB card.

• Quantize the cache to Q8 (-ctk q8_0 -ctv q8_0 in llama.cpp). The cache drops to about 3.6 GB and leaves several gigabytes free.
• Keep one slot (-np 1) so the runtime doesn't reserve cache for parallel requests.
• Budget for MTP if you use it: about 2.5 GiB more while drafting (InsiderLLM). That figure includes KV cache during active speculation, not just the MTP weights. Qwen 3.8 27B's Unsloth file, which includes the MTP block, peaked only 254 MiB higher than Qwen 3.6 27B's with speculation off (InsiderLLM, August 2026). MTP and 110k context together want a Q8 cache.

The full 262,144-token window needs about 8.6 GB of Q8 cache. On one 24GB card that means dropping to a 3-bit weight quant, or moving to a 32GB card or a two-card split.

How much quality does Q4_K_M give up against BF16?

Qwen's published scores (MMLU-Pro 86.2, SWE-bench Verified 77.2) are for the BF16 model (model card). The first version of this guide printed a per-quant MMLU/HumanEval table without a source, and it has been removed.

The reliable way to size the loss is KL divergence (KLD) against the BF16 output distribution. The LLM quantization formats KLD comparison collects the published measurements. For example, it cites Unsloth's GGUF figures on a sibling Qwen model, Qwen3.5-35B-A3B:

Plain Q4_K_M is a reasonable 4-bit default, and it is not the most accurate 4-bit option.

For agentic loops, where one bad token compounds across steps, prefer Q6_K or Q8_0 when the VRAM exists.

What does it cost to run Qwen 3.6 27B at home?

The one cited cost figure comes from the 2x RTX 5060 Ti bake-off at 64 concurrent requests (LLMKube):

• $0.13 per million tokens amortized, including the cards.
• $0.010 per million tokens for electricity alone.

Single-user cost per token is higher, because a lone chat session leaves most of the batch capacity idle. For one user the comparison that matters is up-front card cost against speed, and there a used 24GB card with MTP is the value pick.

Multi-GPU: does splitting Qwen 3.6 27B across two cards help?

It does now. The first version of this guide said to skip multi-GPU for a 27B model, and the 2026 measurements say otherwise:

• Two RTX 3090s in tensor parallel measured 117–118 tok/s at 100K context in vLLM, against 70–72 tok/s for one card (tfriedel lab).
• Two 16GB RTX 5060 Ti cards hold Q4_K_M with a 65,536-token context in llama.cpp (LLMKube). A single 16GB card cannot.

The gain is largest in vLLM's tensor-parallel mode. llama.cpp's default layer split mainly adds capacity (more VRAM for context) rather than single-stream speed. The RTX 3090 supports an NVLink bridge. The 4090 and 5090 do not (NVIDIA RTX 3090 specs), so those cards talk over PCIe.

Verdict matrix

• Get a used RTX 3090 if you want the cheapest 24GB card with cited single-user figures: 38.3 tok/s in llama.cpp, 61.4 with MTP.
• Get a 32GB card (RTX 5090) if you want Q8_0 or a long context at Q4_K_M on a single card.
• Get two RTX 3090s if you want 100K context and more than 100 tok/s, and you are comfortable running vLLM.
• Get two 16GB cards if you already own one. Two RTX 5060 Ti 16GB cards run Q4_K_M with 64k context in llama.cpp.
• Consider Qwen 3.8 27B if you are setting up fresh. It is the same size with the same VRAM math, and it is the newer release.

Bottom line

A used RTX 3090 running Qwen 3.6 27B at Q4_K_M, with MTP enabled in current llama.cpp, is the default single-GPU setup as of September 2026. It is measured at 61.4 tok/s, against 38.3 tok/s without MTP.

• For long context, use IQ4_XS with a Q8 KV cache.
• For near-lossless quality, use Q8_0 on a 32GB card.
• For 100K context at over 100 tok/s, use two 3090s in vLLM.

Recommended quant by VRAM:

Related guides

• Best GPU for Local LLM Inference in 2026
• LLM quantization formats: KLD comparison
• Best local LLM for coding agents on a 24GB GPU
• Ollama Install Guide for Linux, macOS, and Windows

Citations and sources

• https://huggingface.co/Qwen/Qwen3.6-27B: model card (architecture, 262,144-token native context, Apache 2.0, BF16 benchmark scores).
• https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json: layer layout and KV head counts, the basis for the KV-cache figures.
• https://huggingface.co/unsloth/Qwen3.6-27B-GGUF: GGUF weight sizes.
• https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF: GGUF weight sizes.
• https://insiderllm.com/guides/wicked-fast-qwen-3-6-27b-mtp-rtx-3090/: single RTX 3090 Q4_K_M, 38.3 tok/s baseline and 61.4 tok/s with MTP (May 2026, updated August 2026).
• https://insiderllm.com/guides/qwen-3-8-27b-vs-3-6-27b-rtx-3090/: Qwen 3.8 27B vs Qwen 3.6 27B llama-bench on a single RTX 3090, llama.cpp b10088, decode speed and peak VRAM including the MTP block (August 2026).
• https://github.com/ggml-org/llama.cpp/pull/22673: MTP support merged into llama.cpp.
• https://github.com/tfriedel/qwen3.6-rtx3090-lab: vLLM single-3090 and TP=2 measurements (July 2026).
• https://llmkube.com/blog/qwen3-6-27b-bakeoff: 2x RTX 5060 Ti llama.cpp vs vLLM bake-off, context limits and cost per million tokens (April 2026).
• https://runaihome.com/blog/qwen38-27b-consumer-gpu-hardware-guide-2026/: Qwen 3.8 27B release, file size and day-one RTX 3090 speed (August 2026).
• https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622: RTX 3090 memory bandwidth.
• https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889: RTX 4090 memory bandwidth.
• https://www.techpowerup.com/gpu-specs/geforce-rtx-5090.c4216: RTX 5090 memory bandwidth.
• https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090/: RTX 3090 specifications (NVLink).
• https://computingforgeeks.com/ollama-vs-vllm-vs-llama-cpp/: Qwen 3.8 27B single-stream and 32-concurrent decode on RTX 4090, RTX 5090 and L40S in llama.cpp, vLLM and Ollama (October 2026).
• https://lemmy.world/post/48144487: Qwen 3.6 27B IQ3_M on an RX 9070 XT, with and without MTP (June 2026).
• https://github.com/ggml-org/llama.cpp/discussions/15021#discussioncomment-18709095: Qwen 3.8 27B UD-IQ4_XS on an RX 9060 XT 16GB, Vulkan and HIP (October 2026).
• https://github.com/ggml-org/llama.cpp/discussions/23313: Qwen 3.6 27B Q4_K_M on an Intel Panther Lake Xe3 iGPU, SYCL backend (June 2026).
• https://huggingface.co/unsloth/Qwen3.8-27B-GGUF: Qwen 3.8 27B GGUF weight sizes.
• https://www.heise.de/en/news/New-Navi44-chip-Radeon-RX-9060-XT-with-16-GByte-and-RDNA4-from-349-US-dollars-10390507.html: RX 9060 XT 16GB launch price and memory bandwidth.

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.