Qwen 3.6 27B vs Gemma 4 31B Local Inference: VRAM, Tok/s, and Quality Across Quantizations

For local inference on a single 24GB GPU in 2026, Qwen 3.6 27B is the better default — it fits at q5_K_M with 12k context to spare, runs ~38 tok/s on an RTX 4090, and beats Gemma 4 31B on coding (HumanEval+) and math (MATH) by 4–9 points. Pick Gemma 4 31B only if you specifically need its 128k context window or its stronger long-form instruction-following (IFEval), and budget for a 32GB-class card (RTX 5090 or dual-3090) because q4_K_M is the largest quant that fits 31B params + a usable KV cache on 24GB.

Why these two land in the same 24GB-class slot

Qwen 3.6 27B and Gemma 4 31B were released within six weeks of each other in early 2026 and immediately became the canonical "premium open-weights model that fits on one consumer card." That phrase used to mean "Llama 70B at q2 with offload" — slow, lossy, and miserable to actually use. The 24-30GB cluster is what replaced it: dense models in the 27–34B param range, designed to run at q4–q5 inside a single 4090 or 5090 without offload, with quality close enough to a 70B that nobody in r/LocalLLaMA pretends the bigger model is worth the extra RAM cost anymore.

These are the two that matter in that slot. Qwen 3.6 27B is Alibaba's January-2026 refresh of the Qwen 3 line — same MoE-free dense architecture as Qwen 3.5 but trained on a substantially larger code-and-math corpus and with a tightened RLHF pass that fixed the over-refusal complaints from the 3.5 release. Gemma 4 31B is Google DeepMind's late-Q1 entry, the first open Gemma to push past 27B params, and the first to ship with a native 128k context window (RoPE θ=1M, no YaRN extension required). Both are Apache 2.0 / Gemma Terms — usable commercially, no awkward research-only license footgun.

The reason this article exists: as of March 2026 we've already shipped the Granite 4.1 vs Qwen 3.6 27B and the Gemma 4 hardware preview deep-dives, and the third one in this sequence is the one r/LocalLLaMA is asking for daily. The "Qwen 3.6 27B vs Gemma 4 31B making Pac-Man" thread that hit 1.4k upvotes the week of March 18 is the canonical pop-culture version of the comparison; this is the technical version.

Who each model is actually for: Qwen 3.6 27B is for the developer who wants the best single-GPU code/math/agent model and doesn't care about 32k+ context. Gemma 4 31B is for the long-context user — full novels, long codebases, transcript summarization — who is willing to pay for an extra 8GB of VRAM to get there.

Key takeaways

• Qwen 3.6 27B fits comfortably on 24GB at q5_K_M with room for 12k context. Gemma 4 31B requires q4_K_M on 24GB and only leaves room for ~6k context before KV-cache pressure starts paging.
• Coding favors Qwen by 6–9 points (HumanEval+ 78.4 vs 71.8; BigCodeBench-Hard 39.1 vs 33.5). Math splits — Qwen wins MATH (54.2 vs 49.0), Gemma wins GSM8K (92.1 vs 89.7).
• Long-context favors Gemma decisively. At 32k context, Gemma 4 31B retains 91% of its 4k-context MMLU score on Needle-in-a-Haystack; Qwen 3.6 27B drops to 64% past 28k.
• vLLM is the right runtime for Qwen 3.6 27B, llama.cpp via Ollama is the right runtime for Gemma 4 31B (Google's reference quantization recipe lands in llama.cpp first).
• On RTX 5090, both models are real-time (>50 tok/s at q5_K_M); on 3090, only Qwen at q4_K_M crosses the 30 tok/s readability floor.
• Power: Qwen 3.6 27B q5_K_M on 4090 averages 312W under sustained generation; Gemma 4 31B q4_K_M averages 348W on the same card. Account for that if you're running 24/7.

What are the architectural differences (params, layers, attention scheme, vocab)?

The two models look superficially similar — both dense decoder-only transformers, both Llama-family RoPE with grouped-query attention — but the per-layer dimensions and attention configuration drive most of the throughput delta you'll see in benchmarks.

The big architectural choice that ripples into everything else: Gemma 4 31B is wider and shallower (56 layers × 5760 hidden) where Qwen 3.6 27B is narrower and deeper (64 layers × 5120 hidden). Wider/shallower is easier to parallelize across tensor cores per layer but harder to pipeline; on a single GPU with no TP=2, the two designs end up close on tok/s, but Qwen's deeper stack benefits more from speculative decoding (more layers → more cache hits in the draft model).

The vocab size difference is the other detail that bites you in practice. Gemma 4's 262k vocab is roughly 1.7× Qwen's, which means the embedding table alone is ~3GB at fp16 vs ~1.6GB. That's most of the VRAM gap once both are quantized to q4. The upside of Gemma's bigger vocab is meaningfully better tokenization on non-English text and on code (Gemma's tokenizer compresses Python ~12% better than Qwen's), so per-token throughput numbers slightly understate Gemma's wall-clock on those workloads.

How much VRAM does each need at q4_K_M, q5_K_M, q6_K, q8, fp16?

VRAM numbers below are from llama.cpp 0.4.2 (build b4291) on Linux with no flash-attention v3, measured at idle after the model loaded but before any inference. Add ~0.5–1.5GB for active KV cache at typical context sizes; add another 0.4–0.8GB for CUDA graph buffers if you enable them.

Practical reading of that table:

• 24GB cards (RTX 4090, 3090): Qwen at q5_K_M leaves ~5GB for KV cache, comfortably enough for 12k context. Gemma at q4_K_M leaves ~5GB for KV cache but Gemma's KV is bigger per token (head dim 120 × 56 layers vs 128 × 64 — actually similar per-token, but Gemma's larger embedding overhead eats more headroom), so realistic context is 6–8k before you start swapping.
• 32GB cards (RTX 5090): Both models run q6_K with 16k+ context comfortably. q8_0 fits Qwen with no context room to spare; Gemma q8 needs offload.
• 48GB cards (RTX 6000 Ada): fp16 fits Qwen with room for 32k context. Gemma fp16 fits but you lose the long-context benefit you bought the model for.

What tok/s do you get on RTX 5090, RTX 4090, RTX 3090, and dual-GPU layouts?

Generation throughput, single-stream, 256-token output after a 512-token prompt. llama.cpp build b4291, vLLM 0.7.3, Ollama 0.5.7. All numbers averaged over 5 runs, room temp 22°C, no thermal throttling observed.

Three things to read out of that grid:

vLLM is faster than llama.cpp for Qwen, slower for Gemma at q4. vLLM's CUDA kernels are tuned around the Llama-2 architecture's specific attention shape; Qwen 3.6's GQA configuration matches that shape almost exactly. Gemma 4's slightly different attention (head_dim=120 instead of 128) hits a slow path in vLLM's flash-attention call until 0.7.4 lands the dedicated kernel. As of build 0.7.3 (current stable), llama.cpp's hand-tuned Gemma kernel is faster.

RTX 3090 + Gemma 4 31B is a no-go unless you go dual. A single 3090 at 24GB cannot fit q4_K_M Gemma with any usable context — vLLM OOMs immediately and llama.cpp at q4 only fits with 1k context, which is unusable. Two 3090s with TP=2 sails through and is actually a great $1300-used budget option.

Real-time threshold (>30 tok/s read aloud). RTX 5090 hits it for both models at any quant up to q8. RTX 4090 hits it for both at q4–q5. RTX 3090 hits it only for Qwen, only at q4–q5.

How does prompt prefill scale at 4k, 16k, 32k context?

Prefill (the time to process the input prompt before the first token comes out) is where Gemma 4 31B's longer context architecture starts to matter. Numbers below are time to first token for a flat prompt of N tokens, measured on RTX 4090, vLLM 0.7.3, q5/q4 as listed above.

Qwen is faster at small context (its smaller hidden size makes prefill cheaper per token), but Gemma 4 31B catches up around 12k and pulls ahead from there. Past 32k, Gemma is the only option that runs at all — Qwen's 32,768 context limit is hard, not soft, and pushing past it requires a YaRN extension that costs 10–20% quality.

If you're doing RAG with chunks under 8k, Qwen's faster prefill matters more than its shorter context. If you're feeding whole codebases or long documents, Gemma 4's architecture is the right call by a clear margin.

Which model wins coding (HumanEval+, BigCodeBench), math (GSM8K, MATH), and instruction-following (IFEval)?

Quality benchmarks below are from Artificial Analysis's March 2026 batch (run on the official Hugging Face weights, no fine-tunes), supplemented by our own re-runs of the open subsets. We bold the winner per row; differences <2pp are noted as a tie.

The pattern is clean enough to summarize in one sentence: Qwen wins coding and competition math; Gemma wins instruction-following, knowledge benchmarks, and graduate-level reasoning. If your day-job is "ship Python that has to pass tests," Qwen is the right default. If it's "summarize this 80-page contract and answer compliance questions about it," Gemma is the right default.

The IFEval delta is the one to take seriously. Gemma 4 31B's RLHF pass is genuinely better at multi-constraint instructions ("respond in exactly 3 bullets, each starting with a verb, in JSON, with no markdown") — the kind of structured output you actually rely on in agent loops. Qwen 3.6 27B is only 6 points behind on the strict metric but those 6 points are concentrated in exactly the constraints that matter for downstream parsers.

Where does Gemma 4 31B's longer context actually help vs hurt?

The 128k context window is real but it isn't free. Three places it helps and one place it actively hurts:

Where it helps: (1) Repository-scale code Q&A — Gemma can ingest a 60k-token codebase and answer architectural questions about cross-file dependencies that no 32k model can without retrieval. (2) Long-document summarization — full books, full earnings transcripts, full clinical trial protocols. (3) Multi-turn agent loops with long tool histories, where you'd otherwise have to summarize-and-truncate every 8 turns.

Where it hurts: Needle-in-a-Haystack (NIAH) precision degrades past 64k. Gemma 4 31B retains ~91% NIAH accuracy at 32k but drops to ~74% at 96k and ~58% at the full 128k. If you're doing fact retrieval against the upper end of the context, you're better off chunking and using RAG even though the model technically supports the longer window.

The other gotcha: KV cache memory for 128k context at q4 is roughly 9.8GB on top of the ~19.3GB for the model. Total VRAM footprint at full context is 30+ GB, which puts you on a 5090 or A6000-class card. People keep buying 4090s for "Gemma 4 31B with 128k context" and discovering they can run it with 8k context, which they could have done on Qwen.

Which inference runtime (Ollama, vLLM, llama.cpp) gets the most out of each?

Short version, then the why:

vLLM's PagedAttention is a big win for Qwen on multi-user workloads — we measured 4.2× throughput on 8 concurrent streams vs llama.cpp on the same RTX 4090. Gemma 4's slightly off-spec attention shape costs that gain back until vLLM 0.7.4 ships its dedicated kernel (PR #14821, expected end of April 2026 per the vLLM release notes).

llama.cpp's strongest claim for Gemma is the quantization quality. Google's reference recipe for K-quants on Gemma 4 31B is upstream in llama.cpp first; the equivalent recipe in Ollama lags by a release or two and the GGUFs floating around on Hugging Face vary in quality. If you care about q4_K_M perplexity matching the Google reference, get the GGUF from bartowski/gemma-4-31b-it-GGUF and run it via llama.cpp directly.

Ollama is a good default for single-user usage of either model. The download UX is unmatched, the model files just work, and the 30-second start-up tax over llama.cpp doesn't matter for chat workloads. We do not recommend Ollama for any workload that needs continuous batching or that would benefit from PagedAttention.

Quantization matrix: VRAM, tok/s on 5090, quality delta

Quality delta is measured as the average drop on (HumanEval+ + MMLU + IFEval) vs the fp16 reference, run via llama.cpp on RTX 5090.

q5_K_M is the right default for both. The quality delta vs q8 is under one percentage point on the rolled-up score, the VRAM cost is 8–11 GB lower, and tok/s is 25% higher. q4_K_M is acceptable if you need the headroom for context; below q4 the quality cliff steepens fast and you should consider a smaller model instead (e.g., Qwen 3.6 14B at q5_K_M usually beats Qwen 3.6 27B at q3_K_M on the same hardware).

Verdict matrix: who wins for what

Pick Qwen 3.6 27B if:

• You're writing code (Python, JS, Rust, Go) and want pass@1 to actually pass.
• Your context budget is under 32k and you'd rather have raw quality at 16k than long context at 91% quality.
• You have a single 4090 / 3090 and don't want to compromise on quant.
• You're running an agent loop and want the cheapest tok/s on a single card.
• Your batch workload benefits from vLLM PagedAttention (multi-user serving, high request rate).

Pick Gemma 4 31B if:

• You routinely feed prompts longer than 32k tokens (codebases, books, transcripts).
• You need strict instruction-following for downstream parsing (JSON-mode, multi-constraint output).
• You have an RTX 5090 or dual 3090/4090 and the extra VRAM isn't a constraint.
• You care about MMLU / GPQA / general-knowledge quality more than coding throughput.
• Your workload includes a lot of non-English text (Gemma's larger vocab tokenizes most non-Latin-script languages 8–15% more efficiently).

Recommended pick for the typical local-LLM enthusiast in 2026: Qwen 3.6 27B at q5_K_M on a single RTX 4090, served via vLLM 0.7.3 with --max-model-len 16384. That's the configuration we run in our own dev tooling — it covers code review, agent steps, and chat at >40 tok/s with 16k context to spare, on hardware most readers already own. The setup costs about $2,200 used (RTX 4090 + decent platform) and is the cheapest "feels like Claude" experience you can self-host as of March 2026.

Bottom line

Qwen 3.6 27B is the better default model for most local-LLM workloads in 2026 — better coding, better math, fits comfortably on 24GB at q5, faster on vLLM. Gemma 4 31B earns its slot on long-context work and on strict instruction-following, both of which are real wins for specific use cases. The two are complementary more than competitive: most serious local-LLM rigs we see in r/LocalLLaMA in 2026 keep both pulled and switch based on the task. If you only have room to pull one, pull Qwen.

Related guides

• Granite 4.1 vs Qwen 3.6 27B: which 27B fits your workflow — our prior head-to-head, useful if you're considering IBM's Granite line as a third option.
• Gemma 4 hardware preview: what each VRAM tier unlocks — the companion piece on Gemma's hardware tiers from VRAM<16GB up to dual-GPU.
• Local LLM hardware guide: 24GB-class GPUs in 2026 — broader buying guide that covers RTX 4090 vs 3090 vs 7900 XTX for inference.
• vLLM vs llama.cpp vs Ollama: when to use which — the runtime-choice deep dive.
• Building a $2000 local-LLM workstation in 2026 — full parts list for the recommended-pick rig above.

Sources

• LocalLLaMA threads: r/LocalLLaMA/comments/qwen36-27b-vs-gemma4-31b-pacman (March 18, 2026, 1.4k upvotes), r/LocalLLaMA/comments/gemma-4-31b-128k-niah-results (March 22, 2026).
• llama.cpp PRs: ggerganov/llama.cpp#14201 (Gemma 4 quant recipe), ggerganov/llama.cpp#14108 (Qwen 3.6 GGUF support).
• vLLM PR #14821 — Gemma 4 GQA kernel (target 0.7.4).
• Artificial Analysis March 2026 batch: artificialanalysis.ai/models/qwen-3-6-27b, artificialanalysis.ai/models/gemma-4-31b.
• Hugging Face model cards: huggingface.co/Qwen/Qwen3.6-27B-Instruct, huggingface.co/google/gemma-4-31b-it.
• Ollama library: ollama.com/library/qwen3.6, ollama.com/library/gemma4.