Gemma 4 31B on a 24GB GPU: RTX 3090 vs RTX 4090 vs RX 7900 XTX Quant Table (2026)

This page is for owners of a single 24GB card, and buyers about to get one, who want to run Google's dense Gemma 4 31B rather than the slightly smaller Qwen 3.6 27B. At 24GB this model's 4-bit and 5-bit quants fit entirely in VRAM. That is the line the cheaper tiers cannot cross. The 16GB Gemma 4 31B quant table tops out at Q3_K_M with a quantized KV cache. The RTX 3060 12GB Gemma 4 31B guide is about choosing which layers spill to system RAM. On 24GB the question changes. You are no longer asking whether the model fits, but which quant to run and how much context it leaves you.

Three 24GB cards matter in 2026. The RTX 3090 (Ampere, 2020) is the used-market workhorse. The RTX 4090 (Ada, 2022) is the faster consumer card with the same memory size. The RX 7900 XTX (RDNA3, 2022) is AMD's 24GB option and runs through ROCm or Vulkan rather than CUDA. All three carry 24GB and have memory bandwidth within 8% of each other. For a dense 31B model that bandwidth figure predicts most of the generation speed, so the differences between the cards come from prefill compute, software maturity and power draw.

Every VRAM figure below is either a published file size or calculated from the model's own config.json on Hugging Face, and each one says which. Throughput figures are public third-party measurements. Where no public run exists, the table says so rather than filling the gap.

Key Takeaways

• Q4_K_M is 18.3 GB and fits any 24GB card with room for 32K context. File sizes are from the unsloth/gemma-4-31B-it-GGUF listing. Q5_K_M (21.7 GB) fits at short context. Q6_K (25.2 GB) does not fit.
• The RTX 3090 holds 31–35 tok/s across 4K–32K context: 34.7 tok/s at 4K, 33.5 at 16K and 31.4 at 32K, per Hardware Corner.
• The 32GB RTX 5090 is the ceiling for a single consumer card at 61.1 tok/s (Q4_K, 4K, llama.cpp build 8189, Hardware Corner RTX 5090). That is 1.76× the 3090's generation speed on 1.9× the bandwidth.
• On a comparable dense 32B model the RTX 4090 generates only about 11% faster than the 3090 (38.89 vs 35.1 tok/s), but it prefills about 2.2× faster (2,393 vs 1,087.9 tok/s), per Hardware Corner's RTX 4090 page and its 3090 page.
• Gemma 4's sliding-window design keeps long context cheap. 50 of the 60 layers attend to only 1,024 tokens (config.json), so only 10 global layers grow with context. That works out to about 80 KiB of f16 KV cache per token.

Step 0: Does Gemma 4 31B fit your 24GB card at your context length?

Answer three questions before you change anything.

1. Which quant file? Read its size off the GGUF listing. That is your weight footprint, and it does not change with context.
2. How long a context do you actually use? Chat rarely needs more than 8K. Pasting a codebase or a long PDF pushes you to 32K–128K.
3. What precision is the KV cache? llama.cpp defaults to f16. --cache-type-k q8_0 --cache-type-v q8_0 roughly halves it, and q4_0 cuts it to about a quarter.

The KV-cache math comes straight from the Gemma 4 31B config. The 50 sliding-window layers each have 16 KV heads of dimension 256 and see only a 1,024-token window. llama.cpp sizes that cache to the window plus one micro-batch, so it is a fixed cost of about 1.2 GiB at f16 whatever the context length. The 10 global layers each have 4 KV heads of dimension 512, which comes to about 80 KiB per token at f16 when K and V are stored separately.

Source: calculated from google/gemma-4-31B-it config.json (60 layers, 50 sliding at window 1,024, 10 global). The config's attention_k_eq_v flag means a runtime that shares K and V on the global layers could halve the global column. Treat these figures as the conservative case.

The fit rule used throughout this page is weights + KV + about 0.5 GiB of compute buffer, set against about 23.5 GiB usable on a 24GB card that is not driving a display. That budget is an assumption. If the same card runs your desktop, subtract a few hundred MB more.

Gemma 4 31B VRAM and quantization requirements

File sizes are from the unsloth/gemma-4-31B-it-GGUF listing. Unsloth does not publish a plain Q2_K, so the 2-bit row uses its UD-Q2_K_XL. The VRAM column is calculated with the method above at 8K context and an f16 KV cache. The 16GB column uses about 15.5 GiB usable and the 12GB column about 11.5 GiB, the same budgets as the sibling pages.

Sources: file sizes from unsloth/gemma-4-31B-it-GGUF. VRAM is calculated, not measured. For quality, the quantization formats KLD comparison collects the measured divergence figures. On the closest published 27B-class ladder, akopytko's Qwen3.8-27B GGUF card puts Unsloth's Q4_0 at a mean KLD of 0.0283 and its Q6_K at 0.0029 against BF16. That is a different model, so read it as the shape of the curve, not a Gemma figure.

Loading the vision projector for image input adds roughly another GiB to every row. If you use Gemma 4's multimodal input, Q5_K_M stops being a comfortable fit and Q4_K_M becomes the practical top.

How fast does Gemma 4 31B run on each 24GB card?

Per-card pages: RTX 3090, RTX 4090, RX 7900 XTX, RTX 5090.

The 4090 and 7900 XTX gaps are real. Some sites list single-digit Gemma 4 31B figures for those cards, around 7.8 tok/s. They come either from calculators that assume the Q4_K_M build needs about 36 GB, which contradicts the 18.3 GB file, or from one-off reports where the model spilled into system RAM. Neither is an in-VRAM measurement, so neither is in the table.

The closest same-source proxy for the 4090 is a dense 32B model. Hardware Corner measured Qwen3 32B at Q4_K on the 3090 at 35.1 tok/s generation and 1,087.9 tok/s prefill at 4K. Its 4090 page records a 32B Q4_K_XL (18.64 GiB) at 38.89 tok/s generation and 2,393 tok/s prefill at 4K. The quants differ slightly and the pages were updated at different times, so read the roughly 11% generation gap and 2.2× prefill gap as a direction, not a Gemma result.

Spec-delta table

Sources: TechPowerUp RTX 3090, TechPowerUp RTX 4090, TechPowerUp RX 7900 XTX, TechPowerUp RTX 3060 12 GB. Board-partner cards such as the ASUS ROG Strix RTX 3090 White OC carry factory overclocks and higher power limits, but the same memory configuration.

The table explains the result. The three 24GB cards span only 936 to 1,008 GB/s, a 7.7% range. The 4090's advantage is compute, not memory: it has 16,384 CUDA cores to the 3090's 10,496.

Prefill vs generation: where each card wins

Generating a token on a dense model means streaming every weight through the memory bus once. For an 18.3 GB Q4_K_M file the theoretical ceiling is bandwidth divided by file size:

• RTX 3090: 936 ÷ 18.3 ≈ 51 tok/s ceiling. The measured 34.7 tok/s is about 68% of it.
• RTX 4090: 1,008 ÷ 18.3 ≈ 55 tok/s ceiling.
• RX 7900 XTX: 960 ÷ 18.3 ≈ 52 tok/s ceiling.
• RTX 5090: 1,792 ÷ 18.3 ≈ 98 tok/s ceiling. The measured 61.1 tok/s is about 62% of it.

These are calculations, not measurements. They show why the 4090 cannot pull far ahead in generation: its ceiling is only about 8% above the 3090's.

Prefill, the processing of your prompt before the first token appears, is limited by compute instead. The 5090-to-3090 comparison shows it clearly in the same harness: 2.9× the prefill (3,395 vs 1,155.8 tok/s) but only 1.76× the generation. The 4090 should land between them. On the 32B proxy it prefilled 2.2× faster than the 3090. For a 20,000-token prompt that is the difference between waiting about 25 seconds for the first token on the 3090 at its 16K–32K prefill rates and about 11 on the 4090, if the proxy ratio holds. For short chat turns the gap disappears.

How does context length change the picture?

The 3090 row in Hardware Corner's data shows how little Gemma 4 31B slows down as context grows: generation drops only 9.5% from 4K to 32K (34.7 to 31.4 tok/s), while prefill drops 37% (1,155.8 to 723.7 tok/s). The sliding-window layers are the reason. Most of the attention cost does not grow with context.

Memory is the binding constraint at long context. Combining the KV table with the quant table gives these practical breakpoints on 24GB:

Source: calculated from the config and file sizes above against a 23.5 GiB budget.

Use q8_0 KV first, because its quality cost is generally small. Drop to q4_0 KV only when you need the extra context. If neither gets you there, step down a quant rather than letting llama.cpp offload layers. On a 24GB card a smaller quant that stays entirely in VRAM almost always beats a larger quant that spills.

CUDA vs ROCm for Gemma 4 31B on the RX 7900 XTX

The RX 7900 XTX is a supported ROCm target. AMD's ROCm system requirements list it as RDNA3, LLVM target gfx1100, on Ubuntu 24.04 and 22.04 and RHEL 10 and 9 for the current production release. On Windows, or on any other distro, plan on llama.cpp's Vulkan back end instead.

Vulkan is not a downgrade for generation. In llama.cpp's ROCm performance discussion (#15021), an October 2026 run of Gemma 4 12B on an RDNA4 RX 9060 XT measured 39.0 tok/s generation on Vulkan against 37.6 on HIP/ROCm, with Vulkan also slightly ahead on prefill. Results vary by model and driver, so benchmark both back ends on your own card with llama-bench before settling.

Three caveats specific to the 7900 XTX:

1. No public Gemma 4 31B run exists yet, so its row in the benchmark table stays empty. Its 960 GB/s bandwidth puts its theoretical ceiling level with the 3090's.
2. New model architectures reach CUDA first. Gemma 4's chat template and attention layout needed llama.cpp updates in 2026. Use a current build on either back end.
3. Fewer tools support AMD. vLLM, ExLlamaV3 and most fine-tuning stacks assume CUDA. If you want more than llama.cpp or Ollama inference, factor that in.

What about 12GB? The RTX 3060 offload counter-case

On an MSI Gaming GeForce RTX 3060 12GB, no 3-bit or larger Gemma 4 31B quant fits. Q3_K_M alone is 14.7 GB. Every 4-bit run spills roughly a third of its weights into system RAM, and once layers offload, speed collapses rather than tapering. In llama.cpp issue #26674 a 16GB RTX 5060 Ti running Q6_K with offload measured 0.62 tok/s. On a 12GB card the spill is proportionally larger.

The better 12GB play is a different Gemma 4 model. The 26B-A4B mixture-of-experts variant activates only about 4B parameters per token. On a 12GB RTX 4070 Ti it ran UD-Q4_K_XL at 13.7 tok/s with partial offload, per Alfonso Fortunato's Gemma 4 benchmarks. On a 24GB 3090 the same MoE model reaches 119.4 tok/s at 4K, per Hardware Corner. The best local LLMs for 12GB VRAM guide covers the full 12GB shortlist, and the 3060 Gemma 4 31B guide covers the 2-bit route.

Perf-per-dollar and perf-per-watt

All dollar figures use launch MSRP, if you can find it at that. Used and current street prices move weekly; the live price buttons on each product page carry them.

Sources: throughput from Hardware Corner RTX 3090 and RTX 5090; MSRP and board power from the TechPowerUp pages above. The ceiling column is calculated.

At MSRP, the 7900 XTX has the most bandwidth per dollar by a wide margin. Whether it turns that into tok/s depends on the back end, and no one has published that run yet. Neither the 3090 nor the 4090 is sold new at MSRP in 2026, so compare the live used prices on the RTX 3090 vs RTX 4090 page before deciding.

Gemma 4 31B vs Qwen 3.6 27B on 24GB

Both fit. Qwen 3.6 27B's Q4_K_M is 16.8 GB, per unsloth/Qwen3.6-27B-GGUF, against Gemma's 18.3 GB. That 1.5 GB difference buys Qwen more context, or room for its Q5_K_M (19.5 GB) at longer windows. On speed, Hardware Corner's 3090 page records the previous-generation Qwen3.5 27B at 33.5 tok/s at 4K, close to Gemma 4 31B's 34.7, so the choice is mainly about the model, not tok/s.

Run Gemma 4 31B for image input or for its writing style. Run Qwen 3.6 27B for agentic coding and long-context retrieval. The full comparison is in Qwen 3.6 27B vs Gemma 4 31B for local inference, and the Qwen-side settings are in Qwen 3.6 27B on 24GB: backend and quant settings.

Common pitfalls

• Trusting fit calculators that double-count weights. Several "can I run it" sites claim Gemma 4 31B Q4_K_M needs about 36 GB. The actual file is 18.3 GB. Check the GGUF listing before you buy.
• Leaving the KV cache at f16 and then blaming offload. At 128K context an f16 cache adds about 11 GiB. Set q8_0 KV before you drop a quant.
• Running an old llama.cpp build. Builds from before Gemma 4 support load the weights but produce broken output. Build from ggml-org/llama.cpp main.
• Loading the vision projector by accident. Front ends that auto-attach mmproj add about a GiB. Leave it off for text-only work on Q5_K_M.
• Driving your monitor from the inference card. The desktop compositor can take several hundred MB. Q5_K_M at 8K has only about 1 GiB of headroom.

Verdict matrix

• Get the RTX 3090 if you chat with one model at a time, want Q4_K_M or Q5_K_M fully on the card, and want the cheapest 24GB card with a published Gemma 4 31B result (31–35 tok/s).
• Get the RTX 4090 if you paste long documents, run agents that re-read large contexts, or serve several users. Prefill is where its extra compute shows up, about 2.2× the 3090 on a 32B proxy.
• Get the RX 7900 XTX if you run Linux on a supported ROCm distro or are happy with Vulkan, use llama.cpp or Ollama only, and accept that there is no published Gemma 4 31B run yet.
• Stay on 12GB if you can switch to Gemma 4 26B-A4B or a 12B model. Dense 31B on 12GB is an offload exercise, not a daily driver.

Recommended pick

For most readers the RTX 3090 is the card. It is the only 24GB card here with a published Gemma 4 31B measurement, 34.7 tok/s at 4K and 31.4 at 32K. Its bandwidth sits within 8% of the 4090's, and it fits Q5_K_M at short context. The counter-case is the RTX 4090, for anyone whose workload is prompt-heavy rather than generation-heavy. If you feed 20K-token prompts all day, the faster prefill is worth paying for.

Which GPU fits other models?

If Gemma 4 31B is one of several models you plan to run, start with the best 24GB GPUs for local LLMs. For the card-to-card comparison across model sizes, read RTX 3090 vs RTX 4090 for LLM inference. If you are weighing a used 3090 against a new card, see used RTX 3090 vs new GPU: the Reddit consensus. The Qwen 3.6 27B quantization benchmarks use the same quant-table format for the competing model.

Bottom line

At 24GB, Gemma 4 31B stops being an offload problem. Q4_K_M at 18.3 GB fits all three cards with 32K context at an f16 KV cache, and Q5_K_M fits at 8K. Generation speed tracks memory bandwidth, and the three cards are within 8% of each other on bandwidth. That makes a used RTX 3090 at 34.7 tok/s the value pick, and leaves the RTX 4090's advantage mainly in prompt processing. The RX 7900 XTX has the bandwidth to match the 3090, but it still needs a public Gemma 4 31B run to prove it.

Live price comparison

Check current prices side by side: RTX 3090 vs RTX 4090 and RTX 3090 vs RX 7900 XTX.

Citations and sources

All sources accessed 2026-10-25.

• google/gemma-4-31B-it model card and config (Hugging Face): layer layout, KV heads, head dimensions, sliding window (accessed 2026-10-25)
• unsloth/gemma-4-31B-it-GGUF (Hugging Face): GGUF file sizes (accessed 2026-10-25)
• unsloth/Qwen3.6-27B-GGUF (Hugging Face): Qwen 3.6 27B file sizes (accessed 2026-10-25)
• Hardware Corner: RTX 3090 LLM benchmarks: Gemma 4 31B, Gemma 4 26B, Qwen3 32B and Qwen3.5 27B on the 3090 (accessed 2026-10-25)
• Hardware Corner: RTX 4090 LLM benchmarks: 32B Q4_K_XL proxy run (accessed 2026-10-25)
• Hardware Corner: RTX 5090 LLM benchmarks: Gemma 4 31B on the 5090 (accessed 2026-10-25)
• TechPowerUp: GeForce RTX 3090, GeForce RTX 4090, Radeon RX 7900 XTX, GeForce RTX 3060 12 GB: VRAM, bandwidth, board power, launch MSRP (accessed 2026-10-25)
• AMD ROCm system requirements (Linux): RX 7900 XTX support and gfx1100 target (accessed 2026-10-25)
• llama.cpp ROCm performance discussion #15021: Vulkan vs HIP on Gemma 4 12B (accessed 2026-10-25)
• llama.cpp issue #26674: Gemma 4 31B Q6_K offloaded on a 16GB card (accessed 2026-10-25)
• ggml-org/llama.cpp (GitHub): runtime (accessed 2026-10-25)
• Alfonso Fortunato: Gemma 4 E4B vs 26B local benchmarks: Gemma 4 26B-A4B on a 12GB RTX 4070 Ti (accessed 2026-10-25)
• akopytko/Qwen3.8-27B-NVFP4-GGUF model card (Hugging Face): 27B-class KLD ladder (accessed 2026-10-25)

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported. VRAM totals and throughput ceilings are calculated from the cited model config and file sizes and are labelled as calculations. Where no public run exists, the tables say so.