Short answer: on a 24 GB GPU only one of these models fits, and it is Qwen 3.6 27B. Mistral Medium 3.5 is a 128B dense model, not the 27B mixture-of-experts model this page described in May (Mistral model card). Its smallest GGUF, Unsloth's UD-IQ2_XXS, is 34.9 GB, and the 4-bit Q4_K_M is 74.9 GB (Unsloth). Qwen 3.6 27B's Q4_K_M is 16.8 GB (Unsloth) and runs at 38.3 tok/s on a single RTX 3090 in llama.cpp, or 61.4 tok/s with multi-token prediction (InsiderLLM). On coding the two are close: the model cards report 77.2 (Qwen) and 77.6 (Mistral) on SWE-bench Verified.
Qwen 3.6 27B vs Mistral Medium 3.5 on a 24GB GPU: Only One Fits
Mistral Medium 3.5 is a 128B dense model (74.9 GB at Q4_K_M), not a 27B MoE. Qwen 3.6 27B fits a 24 GB card at 16.8 GB and matches it on SWE-bench Verified.

Prices were read from SpecPicks' Amazon price tracker on November 12, 2026 and change daily.
What changed — November 2026
- Corrected Mistral Medium 3.5's size and design. The earlier version listed it as 27.2B total and about 8.5B active parameters in a mixture-of-experts design. Mistral describes it as "a dense 128B model with a 256k context window." The Hugging Face listing counts 127.7B parameters (Mistral, Hugging Face). Every VRAM row and speed row the old page gave for Mistral depended on the wrong size, so they are gone.
- Removed invented benchmarks. The old page had prefill and generation tables for RTX 5090, 4090 and 3090, a runtime comparison, Elo, MT-Bench and FLORES-200 scores, and power figures. None of them had a source. Every number below links to where it came from.
- Corrected the license and context claims. Qwen 3.6 27B is Apache 2.0 with a 262,144-token native context, not 32K (Qwen model card). Mistral Medium 3.5 ships under a modified MIT license, which allows commercial use with exceptions for very large companies. It is not the Mistral Research License the old page named.
- Added measured speeds for 24 GB and 32 GB cards. These cover the RTX 3090, 4090 and 5090, with llama.cpp, MTP and vLLM figures (ComputingForGeeks, InsiderLLM).
- Updated prices. The old page used April 2026 MSRPs ($1,999 for an RTX 5090, $449 for a used RTX 3090). On November 12, 2026, the lowest tracked Amazon prices were $5,499.99 for an RTX 5090, $4,799.99 for an RTX 4090 and $1,899.99 for a renewed RTX 3090.
Key takeaways
- Fit on 24 GB: Qwen yes, Mistral no. Qwen's Q4_K_M (16.8 GB) leaves about 5.5 GB for context on a 24 GB card. Mistral's smallest 2-bit file (34.9 GB) needs at least two 24 GB cards.
- Coding quality is about equal. On SWE-bench Verified, the model cards report 77.2 for Qwen and 77.6 for Mistral. Qwen gets there with about a fifth of the parameters.
- Speed: Qwen runs 38–78 tok/s on one consumer card. Measured figures: 38.3 tok/s on an RTX 3090 and, for the same-size Qwen 3.8 27B, 49.5 tok/s on an RTX 4090 and 77.8 tok/s on an RTX 5090 in llama.cpp.
- Mistral on one 24 GB card means CPU offload. With most of the weights in system RAM, the bandwidth math puts generation at about 1.7–3.6 tok/s on a typical dual-channel DDR5 desktop.
- Verdict: if you own one 24 GB GPU, run Qwen 3.6 27B (or its successor, Qwen 3.8 27B). Run Mistral Medium 3.5 locally only on 96 GB-class hardware, or use Mistral's API.
Spec comparison
| Qwen 3.6 27B | Mistral Medium 3.5 | |
|---|---|---|
| Design | Dense, 64 layers: three Gated DeltaNet (linear) blocks per Gated Attention block | Dense transformer |
| Parameters | ~27.8B, including the vision encoder | 128B (127.7B listed) |
| Context | 262K native, ~1M with YaRN | 256K |
| Image input | Yes | Yes |
| License | Apache 2.0 | Modified MIT (commercial use allowed, exceptions for very large companies) |
| SWE-bench Verified | 77.2 | 77.6 |
| Q4_K_M GGUF | 16.8 GB | 74.9 GB |
| Smallest listed GGUF | IQ4_XS 15.4 GB (Unsloth also lists 3-bit and 2-bit builds) | UD-IQ2_XXS 34.9 GB |
Sources: Qwen model card, Mistral model card, Unsloth Qwen GGUF, Unsloth Mistral GGUF. Qwen also reports 86.2 on MMLU-Pro, 87.8 on GPQA Diamond and 59.3 on Terminal-Bench 2.0. Mistral reports 91.4% on τ³-Telecom. The two vendors publish few benchmarks in common, so SWE-bench Verified is the one direct comparison available.
How much VRAM does each model need?
Weights only, from Unsloth's GGUF file lists. The KV cache comes on top of these figures.
| Quant | Qwen 3.6 27B | Mistral Medium 3.5 | Fits one 24 GB card? |
|---|---|---|---|
| ~2-bit | not needed on 24 GB | 34.9 GB (UD-IQ2_XXS) | Mistral no |
| IQ4_XS | 15.4 GB | 67.1 GB | Qwen yes · Mistral no |
| Q4_K_M | 16.8 GB | 74.9 GB | Qwen yes · Mistral no |
| Q5_K_M | 19.5 GB | 88.3 GB | Qwen yes · Mistral no |
| Q6_K | 22.5 GB | 103 GB | Qwen just barely · Mistral no |
| Q8_0 | 28.6 GB | 133 GB | Qwen needs a 32 GB card · Mistral no |
| BF16 | 53.8 GB | 250 GB | Neither |
Qwen's long context is cheap because only 16 of its 64 layers use full attention with 4 KV heads. At FP16 the cache costs about 66 MB per 1,000 tokens, or about 2.1 GB at 32K (Qwen config; worked through in our Qwen 3.6 27B quantization guide). So Q4_K_M with a 32K context fits a 24 GB card with room to spare.
How fast is Qwen 3.6 27B on 24 GB and 32 GB cards?
All figures are single-user decode at 4-bit. Each row was checked against a bandwidth ceiling: a dense model can't decode faster than memory bandwidth divided by the bytes read per token, unless speculative decoding such as MTP is on.
| GPU (bandwidth) | Runtime and build | Decode tok/s | Ceiling at ~16.5–16.8 GB | Source |
|---|---|---|---|---|
| RTX 3090 24 GB (936 GB/s) | llama.cpp, Q4_K_M | 38.3 | ~56 | InsiderLLM |
| RTX 3090 24 GB | llama.cpp b10088, UD-Q4_K_XL (llama-bench tg128) | 41.77 | ~56 | InsiderLLM, Aug 2026 |
| RTX 3090 24 GB | llama.cpp with MTP, Q4_K_M, 10K context | 61.4 | MTP can exceed it | InsiderLLM |
| RTX 4090 24 GB (1,008 GB/s) | llama.cpp, Qwen 3.8 27B UD-Q4_K_M | 49.5 | ~61 | ComputingForGeeks |
| RTX 4090 24 GB | vLLM, INT4 W4A16 (19.5 GB) | 54.3 | ~52 at 19.5 GB* | ComputingForGeeks |
| RTX 5090 32 GB (1,792 GB/s) | llama.cpp, Qwen 3.8 27B UD-Q4_K_M | 77.8 | ~109 | ComputingForGeeks |
| RTX 5090 32 GB | vLLM, INT4 W4A16 | 85.7 | ~92 | ComputingForGeeks |
*The 19.5 GB W4A16 checkpoint includes weights that text decoding doesn't read in full every token: the vision tower, and the input embedding table, of which only one row is read per token. The bytes actually read per token are lower than the file size, so a figure slightly above this naive ceiling is plausible.
Qwen 3.8 27B, released August 14, 2026, has the same decoder size and the same 16.8 GB Q4_K_M file as 3.6. On an RTX 3090, llama-bench measured 41.49 tok/s for 3.8 against 41.77 for 3.6, a 0.67% gap (InsiderLLM). The 3.8 figures in the table are therefore a close stand-in for 3.6. For backend flags and KV-cache settings on a 24 GB card, see the Qwen 3.6 27B 24 GB settings guide.
Can you run Mistral Medium 3.5 on a 24 GB GPU at all?
Only by putting most of the model in system RAM. Token speed then depends on how fast the CPU side can read its share of the weights. That makes this a bandwidth calculation, and no published measurement on this exact setup turned up. The arithmetic assumes:
- about 22 GB of weights on a 24 GB card;
- dual-channel DDR5-6000 system memory, about 96 GB/s.
| Mistral quant | In system RAM | Approximate ceiling |
|---|---|---|
| Q2_K (46.6 GB) | ~24.6 GB | ~3.6 tok/s |
| Q4_K_M (74.9 GB) | ~52.9 GB | ~1.7 tok/s |
Those are upper bounds. Real llama.cpp offload usually lands below them. You also need 64–96 GB of system RAM just to hold the offloaded layers.
The published Mistral measurements come from bigger machines:
- DGX Spark, 128 GB unified memory: 6.9–9.1 tok/s in vLLM with an NVFP4 quant and Mistral's EAGLE draft model, and about 2 tok/s in plain llama.cpp (NVIDIA developer forum).
- RTX PRO 6000 Blackwell, 96 GB: Mistral Large 123B, the closest dense Mistral model with a published figure, ran at 19.1 tok/s at Q4_K_M (Hardware Corner).
Two 24 GB cards hold the 34.9 GB UD-IQ2_XXS build. With a layer split, each token reads about 17.5 GB from each card in turn, which bounds two RTX 3090s at roughly 27 tok/s. No published run at that quant turned up, and quality at about 2 bits per weight is a real trade-off on a model you'd choose for quality.
Our Mistral Medium 3.5 local inference guide covers 96 GB-class and Mac Studio setups in detail.
What does each setup cost in November 2026?
| Setup | Lowest tracked price, Nov 12, 2026 | Qwen 3.6 27B | Mistral Medium 3.5 |
|---|---|---|---|
| RTX 3090 24 GB (renewed) | $1,899.99 | Q4_K_M, 38–61 tok/s | CPU offload only, ~2–4 tok/s ceiling |
| RTX 4090 24 GB | $4,799.99 | Q4_K_M, ~50 tok/s | CPU offload only |
| RTX 5090 32 GB | $5,499.99 | Q8_0 fits; ~78 tok/s at 4-bit | CPU offload only |
| 2× RTX 3090 (48 GB) | ~$3,800 at the renewed price | Q8_0 split, or long context | 2-bit only, ~27 tok/s bound |
At these prices a used or renewed RTX 3090 is still the value pick for Qwen. The RTX 4090 costs about 2.5 times as much for about 30% more decode speed. A 5090 makes sense only if you want Q8_0 on one card or plan to run image models alongside the LLM.
Verdict
| If you… | Pick | Why |
|---|---|---|
| Own one 24 GB GPU | Qwen 3.6 27B (or 3.8 27B) at Q4_K_M | It fits with room for 32K+ context and runs 38–50 tok/s |
| Want the best coding score you can host on one consumer card | Qwen 3.6 27B | 77.2 SWE-bench Verified, within 0.4 points of Mistral's 77.6 |
| Need Apache 2.0 with no revenue exceptions | Qwen 3.6 27B | Mistral's modified MIT license has a carve-out for very large companies |
| Have a 96 GB card or a 128–256 GB unified-memory box | Mistral Medium 3.5 is an option | 4-bit (74.9 GB) fits; expect roughly 2–19 tok/s depending on the machine |
| Want Mistral Medium 3.5 on a 24 GB card | Use the API | CPU offload bounds it at a few tok/s |
Related guides
- Qwen 3.6 27B quantization benchmarks: BF16 vs Q8_0 vs Q4_K_M
- Mistral Medium 3.5 (128B) local inference: VRAM, quants and tok/s
- Qwen 3.6 27B on 24 GB: backend and quant settings
- Best 24GB GPU for local LLM inference
- Gemma 4 26B-A4B vs Qwen 3.6 27B: speed vs quality
- llama.cpp vs vLLM vs Ollama
Citations and sources
- Qwen — Qwen3.6-27B model card (architecture, context, license, benchmark scores)
- Qwen — Qwen3.6-27B config.json (layer and KV-head layout)
- Mistral AI — Mistral-Medium-3.5-128B model card (128B dense, 256K, license, SWE-bench Verified)
- Unsloth — Qwen3.6-27B GGUF file sizes
- Unsloth — Mistral-Medium-3.5-128B GGUF file sizes
- InsiderLLM — Qwen 3.6 27B with MTP on an RTX 3090
- InsiderLLM — Qwen 3.8 27B vs 3.6 27B on an RTX 3090
- ComputingForGeeks — Ollama vs vLLM vs llama.cpp on RTX 4090, 5090 and L40S
- NVIDIA developer forum — Mistral Medium 3.5 128B NVFP4 + EAGLE on DGX Spark
- Hardware Corner — RTX PRO 6000 Blackwell LLM benchmarks
- SpecPicks Amazon price tracker, November 12, 2026 (RTX 3090, RTX 4090, RTX 5090)
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
Small-model (7–9B) reference throughput
These figures are for 7–9B models, not the 20–27B models this article covers. The table holds model size fixed so the cards can be compared with each other; for the size in the title, use the article's own figures.
Median generation throughput at 7–9B models (Llama 3.1 8B, Qwen 3 8B), Q4 quantization, from community-reported runs SpecPicks tracks. Each row pools runs from different sources, runtimes and models in that class, so the rows are not a matched head-to-head; where the article compares cards on the same rig, its own figures are the like-for-like result. Street price is the second-lowest listing priced within the last 24 hours at 0.5–2× launch MSRP, so no single listing sets it, and a street figure well above MSRP says so in its cell; where too few listings pass that check the cell shows a dash and the launch MSRP, which is not comparable with the street figures.
| GPU | VRAM | Llama-3-8B class, Q4 | Street price | Sources |
|---|---|---|---|---|
| NVIDIA GeForce RTX 5090 | 32 GB | 210.8 tok/s5 runs · 4 sources | —too few current listings for a street figure · launch MSRP was $1,999 | SpecPicks median of 5 runs; sources: llama.cpp GitHub, DatabaseMart, Hardware Corner, nikolasent.github.io (independent… |
| NVIDIA GeForce RTX 4090 | 24 GB | 130.8 tok/s10 runs · 7 sources | —too few current listings for a street figure · launch MSRP was $1,599 | SpecPicks median of 10 runs; sources: GPU-Benchmarks-on-LLM-Inference…, Hardware Corner, llama.cpp GitHub (CUDA performanc…, Awesome Agents LLM Leaderboard +3 more |
| NVIDIA GeForce RTX 3090 | 24 GB | 103.7 tok/s8 runs · 6 sources | $2,300street, all listings · 1.5× the $1,499 launch MSRP | SpecPicks median of 8 runs; sources: GitHub XiongjieDai/GPU-Benchmarks…, Hardware Corner, kunalganglani.com LLM Benchmarks, llama.cpp GitHub CUDA scoreboard +2 more |
Products mentioned in this article
Amazon & eBay listings, plus full specs and alternatives on each product page.
As an Amazon Associate, SpecPicks earns from qualifying purchases; we also earn on qualifying eBay purchases via the eBay Partner Network. Prices shown were last tracked at crawl time and may vary — check the listing for the current price.
Watch a review
I Got the GPU Made With LITERAL GOLD — Linus Tech Tips on YouTube
Frequently asked questions
Does Mistral Medium 3.5 fit on a 24 GB GPU?
Is Qwen 3.6 27B as good as Mistral Medium 3.5 for coding?
How fast is Qwen 3.6 27B on an RTX 3090?
How much VRAM does Qwen 3.6 27B need?
What hardware runs Mistral Medium 3.5 locally?
Related reading
- Ryzen 7 5800X vs Ryzen 5 5600G for a Budget Local-AI Workstation
- RTX PRO 6000 vs RTX 5090 for Local 70B LLMs: 96GB vs 32GB
- Running a Local Coding Agent on a Small Model: What Actually Breaks (and How to Fix It)
- IBM Granite 4.1 8B vs Qwen 3.6 27B: Which Small Local Model Wins on a 16GB GPU?
- Qwen 3.6-27B on an RTX 5070 Ti (16GB): What Fits at 50K Context and How Fast
- AMD Ryzen AI Max+ 395: The 128GB Mini PC Running 70B LLMs Locally
- Qwen3.8 Max vs Claude Opus 4.8: What the Cost-Per-Task Gap Means for Local Rigs
- Apple Intelligence Ships in China on Alibaba's Qwen: What It Means