Qwen3.6 27B on a 12GB GPU: Quantization, Context, and Real-World Tok/s

Yes — you can run Qwen3.6 27B on a 12GB GPU, but only at a 3-bit or low-4-bit quant, and only if you cap context around 16k–32k tokens. As of 2026, q3_K_M leaves ~1.0–1.5GB headroom for KV cache on a 12GB card and generates 22–28 tok/s on an RTX 4070 Super; q4_K_M just barely fits with a 4k–8k window. Expect ~10 tok/s on RX 7800 XT under ROCm 6.4, and ~30 tok/s on the new RTX 5070.

The 12GB-Club moment: why 27B-on-12GB matters

For two years the rule on 12GB consumer cards was "stop at 14B, maybe 22B if you can stomach IQ2." That cracked open in early 2026. Qwen3.6 27B took the open-weights crown under 150B parameters on Artificial Analysis, scoring within striking distance of frontier API models on the Intelligence Index — and the model's KV cache layout, combined with the new q3_K_M quant kernels in llama.cpp, made it the first 27B-class model that genuinely usable on a 12GB card. The r/LocalLLaMA "12GB-Club" thread now has hundreds of benchmark posts pinning down what actually works.

The reason this matters: a 12GB card is the single largest installed base of LLM-capable consumer GPUs. Steam Hardware Survey (2026) puts the RTX 4070 Super, 4070, 3060 12GB, and 5070 collectively above 18% of dedicated GPUs. If you can run a flagship 27B model on what people already own, the gap between "API model" and "local model" closes for hobbyists and small dev teams who can't justify a $1999 RTX 5090. The 24GB tier still wins for fp16, long context, and dual-model workflows — but for most chat, drafting, and code-completion tasks at 4k–16k context, a 12GB card with the right quant is shockingly close.

This guide is the benchmark companion. Real numbers from real cards, real quant trade-offs, and a clear "yes / no / depends" verdict for each common rig.

Key takeaways

• Viable quants on 12GB: q3_K_M (sweet spot), q4_K_M (tight, 4k–8k context only), q2_K (last-resort, noticeable quality drop). q5+ does not fit without offload.
• Generation speed range: 18–30 tok/s on NVIDIA 12GB cards at q3_K_M; 8–12 tok/s on RX 7800 XT under ROCm.
• Context ceiling: ~32k tokens at q3_K_M with full GPU offload; ~16k at q4_K_M; the model's own usable ceiling is ~90k of its claimed 128k window.
• Prefill cost is the gotcha: ingesting 32k tokens takes 6–18 seconds depending on card; for chat use it's invisible, for coding agents it dominates.
• Recommended runtime: llama.cpp 2026.04 builds with -ngl 99 --flash-attn and --cache-type-k q8_0 --cache-type-v q8_0 to cut KV cache memory ~40% versus fp16.

Which quantization fits in 12GB VRAM with usable context?

A 27B model at fp16 is ~54GB. Each quant level cuts that at the cost of perplexity and KL-divergence (KLD) versus fp16. Here's what the GGUF weights actually weigh in at, plus the KV cache budget you have left on a 12GB card after the model loads.

KLD numbers come from the llama.cpp KLD discussion thread on the project's GitHub, calibrated against a 100k-token wikitext sample. The cliff is between q2_K and q3_K_M — q2_K's 0.082 KLD is where you start seeing tangible reasoning regressions on harder prompts. q3_K_M sits at 0.038, which is "you can tell on side-by-side blind tests, but it doesn't break tasks." That's the sweet spot for 12GB.

q4_K_M technically fits the weights, but only with KV cache spilled to system RAM, which murders generation speed. If your card is 12GB, target q3_K_M and stop fighting the math.

How fast is Qwen3.6 27B on an RTX 4070 Super vs RTX 5070 vs RX 7800 XT?

We tested all three cards with q3_K_M, llama.cpp 2026.04 (HEAD as of 2026-04-15), Linux 6.8 (NVIDIA 555.x driver, ROCm 6.4 for AMD), prompt of 512 tokens, generation of 256 tokens, batch=1, full GPU offload (-ngl 99), Flash Attention on.

A few takeaways. The RTX 5070 is the new performance leader at the 12GB tier — GDDR7 bandwidth is a genuine generational step over the 4070 Super, and llama.cpp's CUDA backend already ingests the new SM compute. The RX 7800 XT looks bad on paper here, and the reason is straightforward: ROCm's HIP backend in llama.cpp is still ~40–50% off the theoretical-bandwidth pace versus CUDA. The 7800 XT's 624 GB/s should put it near 5070 territory; in practice it lands closer to a 3060 12GB. AMD has been closing this gap and ROCm 7 (in beta as of April 2026) shows preliminary 30% improvements, but as of the 6.4 release that ships in stable distros today, you pay the tax. The 3060 12GB is still the budget winner — slow but cheap, and at $200–250 used it's the lowest-price ticket into 27B-class local inference.

For a deeper dive on how these cards handle longer-context workloads, see our best 12GB GPU for local LLMs in 2026 buying guide.

What context length can you actually load before OOM?

Theoretical context (the model's 128k claim) and practical context (what your card holds in VRAM) are different problems. KV cache scales linearly with sequence length and quadratically with batch size. On Qwen3.6 27B at q3_K_M, with q8_0 KV cache (the recommended setting), each 1k of context costs ~50–55 MB.

So on a 12GB card with 0.9 GB of headroom after the model:

• 8k context: ~440 MB KV. Comfortable.
• 16k context: ~880 MB. Tight but works.
• 32k context: ~1.76 GB. OOM unless you switch to q4_0 KV (~1.0 GB at the cost of a small quality regression on long-context recall).
• 90k context (the model's true usable ceiling per Artificial Analysis): not reachable on 12GB. You need a 24GB card or higher.

If you absolutely need 32k+ context on 12GB, run q3_K_M with --cache-type-k q4_0 --cache-type-v q4_0. We measured a ~3% regression on needle-in-haystack at 32k versus q8_0 KV, which is acceptable for most use cases. q2_0 KV exists but the recall regressions are sharp; don't bother.

How does prefill latency compare to generation tok/s at 8k / 32k / 90k context?

This is the part of LLM benchmarking that gets ignored in tok/s headlines and bites you the moment you point a coding agent at the model.

Prefill (the first-token-latency dominant phase) processes the entire prompt through the model in parallel before generation begins. At 32k context, you're doing 32,000 forward passes worth of compute up front. Even at 1840 tok/s prefill on a 4070 Super, that's ~17 seconds before the first generated token appears.

OOM at q3_K_M + q8_0 KV; reachable with q4_0 KV but only on 16GB cards. *7800 XT only because it has 16GB; performance is brutal.

For a chat session where prompts are 200–2000 tokens, prefill is invisible. For a coding-agent workflow where the model sees 30k+ of context every turn, prefill dominates wall-clock. If your use case is "agent reads my whole repo and edits a file," budget 15–20 seconds of staring at the screen before any generation, on every turn. That's the actual ergonomics, not the 25 tok/s number.

Where does Qwen3.6 27B degrade — does the reported 90k-on-128k ceiling matter for coding agents?

Artificial Analysis published a long-context evaluation in March 2026 showing Qwen3.6 27B holds coherent task-relevant attention up to ~90k tokens before recall drops below 80% on synthetic needle tests. The model claims 128k context, but the last ~30k is increasingly noisy in practice.

For coding agents this matters differently than for chat. A coding agent typically maintains a rolling context: file contents, recent edits, tool outputs, scratchpad. Once you cross ~60k of accumulated context, you're in the "model still answers but starts losing earlier facts" zone. Symptoms we've seen: misremembering function signatures from files cited 40k tokens earlier, hallucinating import paths, repeating fixes that the agent already tried.

The practical fix for most rigs isn't more context — it's smarter context management. Tools like Aider, Continue, and Cursor's local-mode handle this with periodic summarization. If you're hand-rolling, cap effective context at 32k and let the agent loop with summary-as-context rather than a balloon transcript. You'll get better answers and your prefill stays under 20 seconds.

How does it stack up against Gemma 4 26B-a4b and 35B-a3b on the same 12GB card?

Gemma 4's MoE architecture changes the math. The "26B-a4b" notation means 26B total parameters but only 4B active per token, and "35B-a3b" is 35B total / 3B active. That makes the active-parameter footprint dramatically smaller per inference step, but the full weight set still has to fit in VRAM.

Gemma 4 26B-a4b is the speed champion on a 12GB card — its 4B active params mean ~50% more tok/s than Qwen3.6 27B at comparable quality on broad benchmarks. Where Qwen3.6 still wins: deeper reasoning chains, long-form code generation, and multilingual tasks. Pick by workload, not by tok/s alone.

Gemma 4 35B-a3b is interesting but doesn't fit cleanly on 12GB; it needs partial CPU offload, which collapses tok/s to ~6–8. Wait for a 16GB card or skip it.

Quantization matrix table

(Same as above but with stricter measurements, batch=1, 1024-token generation on RTX 4070 Super.)

Spec + benchmark table: 5 GPUs × quant

*7800 XT has 16GB so context ceiling is higher than the 12GB cards.

Perf-per-dollar and perf-per-watt math

Two metrics matter for sustained workloads: tok/s per dollar of card cost, and tok/s per watt of TGP.

The RTX 5070 wins both metrics among new cards as of 2026 — better bandwidth, better Blackwell-gen efficiency, lower MSRP than the 4070 Super at launch. The RTX 3060 12GB used remains the unlikely value champion for inference if you can find one at $200–250. The RX 7800 XT loses on both axes under current ROCm; revisit after ROCm 7 ships.

Bottom line

If your goal is "run the best open-weights 27B-class model on hardware I already own," and your card is 12GB, the answer is: yes, with q3_K_M, llama.cpp, q8_0 KV cache, and a 16k context cap. Expect 18–31 tok/s depending on which 12GB card you've got. Don't fight q4_K_M on 12GB, don't expect 90k context, and budget for prefill latency if you're building agentic workflows.

If you're shopping today (April 2026) for a card specifically to run Qwen3.6 27B and similar 27B-class models, the RTX 5070 is the new default at $549. The RTX 4070 Super at $599 is fine if you find one discounted. The RTX 3060 12GB used at $200–250 is a bargain entry point. Skip the RX 7800 XT for LLM duty until ROCm 7 lands in stable.

Related guides

• Best 12GB GPU for Local LLMs in 2026
• Best 24GB GPU for Local LLMs in 2026
• Best GPU for an AI Workstation in 2026
• ROCm 2026: AMD Local LLM Status Check

Sources

• Artificial Analysis — Qwen3.6 27B model card (Intelligence Index, hallucination rate, long-context recall): artificialanalysis.ai
• r/LocalLLaMA "12GB-Club" megathread (community quant + tok/s benchmarks)
• llama.cpp KLD discussion thread (KL-divergence numbers per quant): github.com/ggerganov/llama.cpp
• TechPowerUp GPU database — RTX 4070 Super, RTX 5070, RX 7800 XT specifications: techpowerup.com
• Tom's Hardware RTX 5070 review (memory bandwidth, TGP under load): tomshardware.com