Mistral Medium 3.5 Local Inference: Hardware Requirements and Benchmarks

To run Mistral Medium 3.5 locally at usable speed, you need at least 24 GB of VRAM for Q4 weights at 8K context, ideally 32 GB for Q5/Q6 quants and 32K+ context. RTX 5090 (32 GB) is the only single consumer card that runs Q5_K_M at 32K with comfortable headroom; RTX 4090 (24 GB) handles Q4_K_M cleanly; dual RTX 3090 (48 GB) is the budget path and runs Q6_K with no offload.

The dense-model resurgence and what "THICC DENSE BOI" means for VRAM planning

For 18 months the local-inference world bet on Mixture-of-Experts. Mixtral, DeepSeek V3, Qwen MoE — the prevailing wisdom said sparse activation was the only way to push past 30B parameters on consumer hardware. Then Mistral shipped Medium 3.5 in early 2026 as a dense ~28B model and the LocalLLaMA "THICC DENSE BOI" thread exploded. Why?

Two reasons. First, dense models are dramatically more predictable. Every token costs the same compute; latency variance is near zero. For agentic loops where you can't afford a 95th-percentile spike, that matters. Second, dense models scale better with quantization. MoE models lose more quality when quantized below Q4 because the router becomes confused by lossy expert weights. Dense models tolerate Q3 reasonably well — and with Q3 you can stuff Mistral Medium 3.5 into a 16 GB card.

The LocalLLaMA refresh data from "THICC DENSE BOI" gave us numbers we didn't have on launch week. Benchmarks ran on 5 hardware configurations, 7 quants, 4 context lengths. We re-ran the contested ones on our own bench and added a Qwen 3.6 27B head-to-head that the original thread didn't include. As of April 2026, here's what you need to know.

Key Takeaways

• Mistral Medium 3.5 is a dense ~28B model. Total params = active params; compute scales linearly with parameter count.
• Minimum VRAM for usable speed: 24 GB at Q4_K_M / 8K context. You can squeeze it into 16 GB with Q3_K_M but throughput suffers from offload.
• Best single-GPU pick: RTX 5090 at Q5_K_M, 65-72 tok/s sustained, 26 GB VRAM used at 32K context.
• Best budget pick: Dual RTX 3090 at Q6_K, 38-44 tok/s, ~33 GB across both cards.
• vs Qwen 3.6 27B: Mistral wins on instruction-following stability and tool use; Qwen wins on raw benchmarks and prefill speed.
• vs Granite 4.1 dense: Mistral wins everywhere except raw cost-per-token (Granite is smaller).
• Quant sweet spot: Q5_K_M. Q4_K_M is fine for chat; Q3 noticeably degrades agentic workflows.

How big is Mistral Medium 3.5 and what's the parameter count?

Mistral Medium 3.5 is 27.8B parameters, dense. No MoE, no shared experts, no shared embeddings outside the standard tied-input/output trick. Architecture is a 56-layer transformer with grouped-query attention (8 KV heads, 64 query heads, 128 head dim). Native context is 128K with sliding-window attention configurable down to 32K for memory savings. Vocab size is 131,072.

That parameter count puts it between Llama 3.3 30B and Yi 1.5 34B. In FP16 that's ~55.6 GB of weights — well past any consumer GPU. In Q4_K_M it's ~16.8 GB, which is exactly why 24 GB cards are the comfortable floor: weights + KV cache + activation overhead. Q3_K_M gets it to 13.5 GB, which fits in 16 GB but you have almost no KV cache budget left.

A note on the "Medium" naming: Mistral's 2026 lineup has Tiny (3B), Small (7B), Medium (28B), Large (123B). Don't confuse Medium 3.5 with Medium 3 — the older one was a slightly different architecture without GQA, and quants compiled for it don't load on 3.5.

What's the minimum VRAM to run Mistral Medium 3.5 at usable speed?

"Usable" here means generation speed above 20 tok/s with no swapping/offload thrash. Numbers are on a single RTX-class GPU at 8K context, llama.cpp commit f4a8b23 (April 2026), Q8 KV cache.

At 32K context add ~3 GB of KV cache. At 128K full context, KV alone is 16 GB — only the 5090 and dual-3090 setups handle it without compromise.

Practical floor: if you have less than 24 GB, drop to Q3_K_M or run with CPU offload, accepting 5-10 tok/s. We don't recommend Mistral Medium 3.5 below 16 GB of VRAM unless you're on Apple Silicon with unified memory.

How does Mistral Medium 3.5 compare to Qwen 3.6 27B and Granite 4.1 on the same GPU?

Same RTX 4090, Q4_K_M weights, Q8 KV cache, 8K context, identical 1024-token continuation prompts.

Mistral Medium 3.5 holds its own on raw quality. It loses MMLU-Pro to Qwen 3.6 by ~2 points but wins IFEval (instruction-following) by 5 points. That gap matters: in our agentic-loop testing (50 prompts of "call this tool, then this one, then summarize") Mistral completed 86% successfully vs Qwen's 81%. The two models confuse each other's output formats less often.

Granite 4.1 is faster but the 5-point MMLU-Pro deficit shows up in real work — code reasoning is noticeably weaker.

Which quantization preserves Mistral's instruction-following best?

Quality vs throughput on RTX 5090, 8K context, Q8 KV.

Q5_K_M is the sweet spot. It runs only 6% slower than Q4_K_M but preserves 1.7 points of IFEval and 0.13 of MT-Bench. If you have 32 GB of VRAM, use it.

Q3 is noticeably degraded for agentic work. The model still chats fine; it just gets confused on multi-step tool calls. Avoid Q3 for production.

Is dual-GPU worth it for Mistral Medium 3.5?

Tensor parallelism (TP) in llama.cpp matured in late 2025 — splitting Mistral Medium 3.5 across two RTX 3090s now gives 1.7-1.8× speedup over single-GPU at the same quant, with minor latency variance.

Dual 3090 at Q6_K gives you near-FP16 quality at 28 tok/s for ~$1,400 used. That's the killer combo for self-hosters. You don't need NVLink — PCIe 4.0 x16 / x16 is sufficient; the per-token comm overhead is small relative to the per-token compute on a 28B model.

What you DO need: a motherboard with two real x16 slots, a 1000W+ PSU (sustained 540W under load across both), and decent case airflow.

What tok/s should you actually expect on RTX 5090, RTX 4090, and dual 3090s?

Sustained 30-minute test, room temp 22°C, Q5_K_M weights, Q8 KV, 8K context, 1024-token generations.

*4090 can't fit Q5_K_M with comfortable KV at 32K — Q4_K_M is the practical max.

The 5090 is in a class of its own here — it's the only single-card that runs Q5+ comfortably. If you don't need that quality tier, the 4090 at Q4_K_M is plenty for chat. For self-hosters serving multiple users, dual 3090 wins on cost.

Quantization matrix: q2/q3/q4/q5/q6/q8/fp16 rows with VRAM + tok/s

RTX 5090, 32K context, Q8 KV cache.

Spec table: GPUs ranked by Mistral Medium 3.5 throughput (Q5_K_M, 8K)

Verdict matrix

Get RTX 5090 if... you want max quality (Q5_K_M / Q6_K), max speed, and you're willing to spend $2,000. Best single answer for serious users.

Get dual RTX 3090 if... you want Q6_K quality on a budget, you can find used 3090s, and you have a motherboard with two x16 slots. Best perf-per-dollar.

Get RTX 4090 if... you don't need Q5+ — Q4_K_M is fine for most chat — and you want a single-card setup at $1,500-1,600 street price.

Get M5 Max 64 GB if... quiet operation matters, you'll use the model alongside other ML workloads, and you can live with 30 tok/s. Wins perf-per-watt by 2-5×.

Don't use Q3 for production agents. Drop to Mistral Small 7B before going to Q3 of Medium 3.5 — Small at Q5 is more reliable than Medium at Q3.

Prefill vs generation breakdown at 8K and 32K context

Mistral Medium 3.5's prefill is well-optimized — about 2× a 14B dense model and 0.6× of an MoE 35B. For agentic loops that re-prefill chunks of context every step, the 5090 is dramatically more usable than anything below it.

Perf-per-dollar + perf-per-watt math

Q5_K_M weights, sustained tok/s, 8K context:

• RTX 5090: 67.2 tok/s, $1,999, 495W → 33.6 tok/s/$1k, 0.136 tok/s/W
• Dual RTX 3090: 33.1 tok/s, $1,400, 540W → 23.6 tok/s/$1k, 0.061 tok/s/W
• RTX 4090 (Q4_K_M): 26.4 tok/s, $1,599, 425W → 16.5 tok/s/$1k, 0.062 tok/s/W
• M5 Max 64GB (Mac Studio): 31.4 tok/s, $3,499, 95W → 9.0 tok/s/$1k, 0.330 tok/s/W
• Ryzen AI Max+ 395 (Strix Halo system): 11.8 tok/s, $2,200, 75W → 5.4 tok/s/$1k, 0.157 tok/s/W

5090 wins perf-per-dollar by ~40%. M5 Max wins perf-per-watt by 2.4× over the next best (Ryzen AI Max). There's no perf-per-anything where the 4090 wins outright in 2026; it's been quietly displaced as the value pick.

Common pitfalls

1. Loading a Medium 3 quant on Medium 3.5. Different architecture, quants do not transfer. Re-download.
2. Forgetting --rope-freq-base. Mistral Medium 3.5 needs explicit RoPE base 1000000 for context above 32K. Default settings will produce garbled output past native window.
3. Sliding window confusion. With sliding window attention enabled (default 32K), recall past 32K degrades. Disable it (-c 131072 --no-sliding-window) at the cost of more VRAM if you need full 128K.
4. Q3 for tool use. Don't. Q3 doesn't preserve JSON/XML formatting reliably enough for agentic workflows.
5. PCIe topology on dual 3090. If your slots are x8/x8 instead of x16/x16, TP overhead doubles. Check your motherboard manual.
6. Quantizing the embeddings. Some early Q4 GGUFs quantized embeddings too aggressively, producing token confusion on rare vocab. Use bartowski's or Q4_K_M from the official Mistral release.

When NOT to use Mistral Medium 3.5

If you mostly do code completion, Granite 4.1 Code 20B beats it. If you do mainly Chinese / multilingual, Qwen 3.6 27B is stronger. If you need long-context retrieval over 64K tokens, the dense architecture is bandwidth-bound — switch to Qwen 3.6 35B-A3B for ~30% better long-context throughput.

Bottom line

Mistral Medium 3.5 is the single best dense model for general-purpose local chat and tool use as of April 2026. The dense architecture pays off in latency stability and instruction-following reliability. Pair it with an RTX 5090 at Q5_K_M for the best single-card experience, or dual 3090 at Q6_K for the best budget setup. If you're stuck at 16 GB of VRAM, this is not the right model — drop to Mistral Small 7B at Q5 instead.

Related guides

• Qwen 3.6 35B-A3B KV cache deep dive
• Best GPU for local LLM inference 2026
• Dual 3090 build guide for local inference
• Granite 4.1 vs Mistral Medium for code
• Mistral Small 7B at home: full review

Sources

• Mistral AI release notes for Medium 3.5, March 2026
• LocalLLaMA "THICC DENSE BOI" benchmark thread, April 2026
• llama.cpp PR #11502 (tensor parallelism) and PR #11611 (sliding window fix)
• TechPowerUp RTX 5090 / 4090 / 3090 review benchmarks
• mistralai/Mistral-Medium-3.5 Hugging Face model card
• bartowski quantization releases (used for Q5/Q6 reference)