Skip to main content

Qwen 3.6 27B vs Mistral Medium 3.5 on a 24GB GPU: Only One Fits

Mistral Medium 3.5 is a 128B dense model (74.9 GB at Q4_K_M), not a 27B MoE. Qwen 3.6 27B fits a 24 GB card at 16.8 GB and matches it on SWE-bench Verified.

Short answer: on a 24 GB GPU only one of these models fits, and it is Qwen 3.6 27B. Mistral Medium 3.5 is a 128B dense model, not the 27B mixture-of-experts model this page described in May (Mistral model card). Its smallest GGUF, Unsloth's UD-IQ2_XXS, is 34.9 GB, and the 4-bit Q4_K_M is 74.9 GB (Unsloth). Qwen 3.6 27B's Q4_K_M is 16.8 GB (Unsloth) and runs at 38.3 tok/s on a single RTX 3090 in llama.cpp, or 61.4 tok/s with multi-token prediction (InsiderLLM). On coding the two are close: the model cards report 77.2 (Qwen) and 77.6 (Mistral) on SWE-bench Verified.

Qwen 3.6 27B vs Mistral Medium 3.5 on a 24GB GPU: Only One Fits

Prices were read from SpecPicks' Amazon price tracker on November 12, 2026 and change daily.

What changed — November 2026

  • Corrected Mistral Medium 3.5's size and design. The earlier version listed it as 27.2B total and about 8.5B active parameters in a mixture-of-experts design. Mistral describes it as "a dense 128B model with a 256k context window." The Hugging Face listing counts 127.7B parameters (Mistral, Hugging Face). Every VRAM row and speed row the old page gave for Mistral depended on the wrong size, so they are gone.
  • Removed invented benchmarks. The old page had prefill and generation tables for RTX 5090, 4090 and 3090, a runtime comparison, Elo, MT-Bench and FLORES-200 scores, and power figures. None of them had a source. Every number below links to where it came from.
  • Corrected the license and context claims. Qwen 3.6 27B is Apache 2.0 with a 262,144-token native context, not 32K (Qwen model card). Mistral Medium 3.5 ships under a modified MIT license, which allows commercial use with exceptions for very large companies. It is not the Mistral Research License the old page named.
  • Added measured speeds for 24 GB and 32 GB cards. These cover the RTX 3090, 4090 and 5090, with llama.cpp, MTP and vLLM figures (ComputingForGeeks, InsiderLLM).
  • Updated prices. The old page used April 2026 MSRPs ($1,999 for an RTX 5090, $449 for a used RTX 3090). On November 12, 2026, the lowest tracked Amazon prices were $5,499.99 for an RTX 5090, $4,799.99 for an RTX 4090 and $1,899.99 for a renewed RTX 3090.

Key takeaways

  • Fit on 24 GB: Qwen yes, Mistral no. Qwen's Q4_K_M (16.8 GB) leaves about 5.5 GB for context on a 24 GB card. Mistral's smallest 2-bit file (34.9 GB) needs at least two 24 GB cards.
  • Coding quality is about equal. On SWE-bench Verified, the model cards report 77.2 for Qwen and 77.6 for Mistral. Qwen gets there with about a fifth of the parameters.
  • Speed: Qwen runs 38–78 tok/s on one consumer card. Measured figures: 38.3 tok/s on an RTX 3090 and, for the same-size Qwen 3.8 27B, 49.5 tok/s on an RTX 4090 and 77.8 tok/s on an RTX 5090 in llama.cpp.
  • Mistral on one 24 GB card means CPU offload. With most of the weights in system RAM, the bandwidth math puts generation at about 1.7–3.6 tok/s on a typical dual-channel DDR5 desktop.
  • Verdict: if you own one 24 GB GPU, run Qwen 3.6 27B (or its successor, Qwen 3.8 27B). Run Mistral Medium 3.5 locally only on 96 GB-class hardware, or use Mistral's API.

Spec comparison

Qwen 3.6 27BMistral Medium 3.5
DesignDense, 64 layers: three Gated DeltaNet (linear) blocks per Gated Attention blockDense transformer
Parameters~27.8B, including the vision encoder128B (127.7B listed)
Context262K native, ~1M with YaRN256K
Image inputYesYes
LicenseApache 2.0Modified MIT (commercial use allowed, exceptions for very large companies)
SWE-bench Verified77.277.6
Q4_K_M GGUF16.8 GB74.9 GB
Smallest listed GGUFIQ4_XS 15.4 GB (Unsloth also lists 3-bit and 2-bit builds)UD-IQ2_XXS 34.9 GB

Sources: Qwen model card, Mistral model card, Unsloth Qwen GGUF, Unsloth Mistral GGUF. Qwen also reports 86.2 on MMLU-Pro, 87.8 on GPQA Diamond and 59.3 on Terminal-Bench 2.0. Mistral reports 91.4% on τ³-Telecom. The two vendors publish few benchmarks in common, so SWE-bench Verified is the one direct comparison available.

How much VRAM does each model need?

Weights only, from Unsloth's GGUF file lists. The KV cache comes on top of these figures.

QuantQwen 3.6 27BMistral Medium 3.5Fits one 24 GB card?
~2-bitnot needed on 24 GB34.9 GB (UD-IQ2_XXS)Mistral no
IQ4_XS15.4 GB67.1 GBQwen yes · Mistral no
Q4_K_M16.8 GB74.9 GBQwen yes · Mistral no
Q5_K_M19.5 GB88.3 GBQwen yes · Mistral no
Q6_K22.5 GB103 GBQwen just barely · Mistral no
Q8_028.6 GB133 GBQwen needs a 32 GB card · Mistral no
BF1653.8 GB250 GBNeither

Qwen's long context is cheap because only 16 of its 64 layers use full attention with 4 KV heads. At FP16 the cache costs about 66 MB per 1,000 tokens, or about 2.1 GB at 32K (Qwen config; worked through in our Qwen 3.6 27B quantization guide). So Q4_K_M with a 32K context fits a 24 GB card with room to spare.

How fast is Qwen 3.6 27B on 24 GB and 32 GB cards?

All figures are single-user decode at 4-bit. Each row was checked against a bandwidth ceiling: a dense model can't decode faster than memory bandwidth divided by the bytes read per token, unless speculative decoding such as MTP is on.

GPU (bandwidth)Runtime and buildDecode tok/sCeiling at ~16.5–16.8 GBSource
RTX 3090 24 GB (936 GB/s)llama.cpp, Q4_K_M38.3~56InsiderLLM
RTX 3090 24 GBllama.cpp b10088, UD-Q4_K_XL (llama-bench tg128)41.77~56InsiderLLM, Aug 2026
RTX 3090 24 GBllama.cpp with MTP, Q4_K_M, 10K context61.4MTP can exceed itInsiderLLM
RTX 4090 24 GB (1,008 GB/s)llama.cpp, Qwen 3.8 27B UD-Q4_K_M49.5~61ComputingForGeeks
RTX 4090 24 GBvLLM, INT4 W4A16 (19.5 GB)54.3~52 at 19.5 GB*ComputingForGeeks
RTX 5090 32 GB (1,792 GB/s)llama.cpp, Qwen 3.8 27B UD-Q4_K_M77.8~109ComputingForGeeks
RTX 5090 32 GBvLLM, INT4 W4A1685.7~92ComputingForGeeks

*The 19.5 GB W4A16 checkpoint includes weights that text decoding doesn't read in full every token: the vision tower, and the input embedding table, of which only one row is read per token. The bytes actually read per token are lower than the file size, so a figure slightly above this naive ceiling is plausible.

Qwen 3.8 27B, released August 14, 2026, has the same decoder size and the same 16.8 GB Q4_K_M file as 3.6. On an RTX 3090, llama-bench measured 41.49 tok/s for 3.8 against 41.77 for 3.6, a 0.67% gap (InsiderLLM). The 3.8 figures in the table are therefore a close stand-in for 3.6. For backend flags and KV-cache settings on a 24 GB card, see the Qwen 3.6 27B 24 GB settings guide.

Can you run Mistral Medium 3.5 on a 24 GB GPU at all?

Only by putting most of the model in system RAM. Token speed then depends on how fast the CPU side can read its share of the weights. That makes this a bandwidth calculation, and no published measurement on this exact setup turned up. The arithmetic assumes:

  • about 22 GB of weights on a 24 GB card;
  • dual-channel DDR5-6000 system memory, about 96 GB/s.
Mistral quantIn system RAMApproximate ceiling
Q2_K (46.6 GB)~24.6 GB~3.6 tok/s
Q4_K_M (74.9 GB)~52.9 GB~1.7 tok/s

Those are upper bounds. Real llama.cpp offload usually lands below them. You also need 64–96 GB of system RAM just to hold the offloaded layers.

The published Mistral measurements come from bigger machines:

  • DGX Spark, 128 GB unified memory: 6.9–9.1 tok/s in vLLM with an NVFP4 quant and Mistral's EAGLE draft model, and about 2 tok/s in plain llama.cpp (NVIDIA developer forum).
  • RTX PRO 6000 Blackwell, 96 GB: Mistral Large 123B, the closest dense Mistral model with a published figure, ran at 19.1 tok/s at Q4_K_M (Hardware Corner).

Two 24 GB cards hold the 34.9 GB UD-IQ2_XXS build. With a layer split, each token reads about 17.5 GB from each card in turn, which bounds two RTX 3090s at roughly 27 tok/s. No published run at that quant turned up, and quality at about 2 bits per weight is a real trade-off on a model you'd choose for quality.

Our Mistral Medium 3.5 local inference guide covers 96 GB-class and Mac Studio setups in detail.

What does each setup cost in November 2026?

SetupLowest tracked price, Nov 12, 2026Qwen 3.6 27BMistral Medium 3.5
RTX 3090 24 GB (renewed)$1,899.99Q4_K_M, 38–61 tok/sCPU offload only, ~2–4 tok/s ceiling
RTX 4090 24 GB$4,799.99Q4_K_M, ~50 tok/sCPU offload only
RTX 5090 32 GB$5,499.99Q8_0 fits; ~78 tok/s at 4-bitCPU offload only
2× RTX 3090 (48 GB)~$3,800 at the renewed priceQ8_0 split, or long context2-bit only, ~27 tok/s bound

At these prices a used or renewed RTX 3090 is still the value pick for Qwen. The RTX 4090 costs about 2.5 times as much for about 30% more decode speed. A 5090 makes sense only if you want Q8_0 on one card or plan to run image models alongside the LLM.

Verdict

If you…PickWhy
Own one 24 GB GPUQwen 3.6 27B (or 3.8 27B) at Q4_K_MIt fits with room for 32K+ context and runs 38–50 tok/s
Want the best coding score you can host on one consumer cardQwen 3.6 27B77.2 SWE-bench Verified, within 0.4 points of Mistral's 77.6
Need Apache 2.0 with no revenue exceptionsQwen 3.6 27BMistral's modified MIT license has a carve-out for very large companies
Have a 96 GB card or a 128–256 GB unified-memory boxMistral Medium 3.5 is an option4-bit (74.9 GB) fits; expect roughly 2–19 tok/s depending on the machine
Want Mistral Medium 3.5 on a 24 GB cardUse the APICPU offload bounds it at a few tok/s

Citations and sources

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.

Small-model (7–9B) reference throughput

These figures are for 7–9B models, not the 20–27B models this article covers. The table holds model size fixed so the cards can be compared with each other; for the size in the title, use the article's own figures.

Median generation throughput at 7–9B models (Llama 3.1 8B, Qwen 3 8B), Q4 quantization, from community-reported runs SpecPicks tracks. Each row pools runs from different sources, runtimes and models in that class, so the rows are not a matched head-to-head; where the article compares cards on the same rig, its own figures are the like-for-like result. Street price is the second-lowest listing priced within the last 24 hours at 0.5–2× launch MSRP, so no single listing sets it, and a street figure well above MSRP says so in its cell; where too few listings pass that check the cell shows a dash and the launch MSRP, which is not comparable with the street figures.

GPUVRAM Llama-3-8B class, Q4Street price Sources
NVIDIA GeForce RTX 5090 32 GB 210.8 tok/s5 runs · 4 sources —too few current listings for a street figure · launch MSRP was $1,999 SpecPicks median of 5 runs; sources: llama.cpp GitHub, DatabaseMart, Hardware Corner, nikolasent.github.io (independent…
NVIDIA GeForce RTX 4090 24 GB 130.8 tok/s10 runs · 7 sources —too few current listings for a street figure · launch MSRP was $1,599 SpecPicks median of 10 runs; sources: GPU-Benchmarks-on-LLM-Inference…, Hardware Corner, llama.cpp GitHub (CUDA performanc…, Awesome Agents LLM Leaderboard +3 more
NVIDIA GeForce RTX 3090 24 GB 103.7 tok/s8 runs · 6 sources $2,300street, all listings · 1.5× the $1,499 launch MSRP SpecPicks median of 8 runs; sources: GitHub XiongjieDai/GPU-Benchmarks…, Hardware Corner, kunalganglani.com LLM Benchmarks, llama.cpp GitHub CUDA scoreboard +2 more

Products mentioned in this article

Amazon & eBay listings, plus full specs and alternatives on each product page.

As an Amazon Associate, SpecPicks earns from qualifying purchases; we also earn on qualifying eBay purchases via the eBay Partner Network. Prices shown were last tracked at crawl time and may vary — check the listing for the current price.

Watch a review

I Got the GPU Made With LITERAL GOLD — Linus Tech Tips on YouTube

Frequently asked questions

Does Mistral Medium 3.5 fit on a 24 GB GPU?
No. Mistral Medium 3.5 is a 128B dense model. Its smallest Unsloth GGUF (UD-IQ2_XXS) is 34.9 GB and Q4_K_M is 74.9 GB. On one 24 GB card it only runs with most layers offloaded to system RAM, which bounds generation at roughly 1.7 to 3.6 tok/s on a dual-channel DDR5 desktop.
Is Qwen 3.6 27B as good as Mistral Medium 3.5 for coding?
Close. The model cards report 77.2 on SWE-bench Verified for Qwen 3.6 27B and 77.6 for Mistral Medium 3.5, with Qwen using about a fifth of the parameters.
How fast is Qwen 3.6 27B on an RTX 3090?
InsiderLLM measured 38.3 tok/s at Q4_K_M in llama.cpp and 61.4 tok/s with multi-token prediction. A later llama-bench run on llama.cpp b10088 measured 41.77 tok/s at UD-Q4_K_XL.
How much VRAM does Qwen 3.6 27B need?
Per Unsloth's GGUF files: 15.4 GB at IQ4_XS, 16.8 GB at Q4_K_M, 19.5 GB at Q5_K_M, 22.5 GB at Q6_K and 28.6 GB at Q8_0. A 32K-token FP16 KV cache adds about 2.1 GB.
What hardware runs Mistral Medium 3.5 locally?
At 4-bit (about 75 GB), a 96 GB RTX PRO 6000 Blackwell, a 128 GB DGX Spark or a 256 GB Mac Studio M3 Ultra. A DGX Spark measured 6.9 to 9.1 tok/s with NVFP4 and EAGLE speculative decoding, and about 2 tok/s in plain llama.cpp.

— Mike Perry · Updated 2026-11-12

Parts this article names

Amazon Associate — prices tracked 2026-11-20, may vary.