Mistral Medium 3.5 needs about 80 GB of GPU or unified memory to run at 4-bit. Mistral's model card describes it as a "dense 128B" model with a 256k context window, and the Q4_K_M file in bartowski's GGUF release is 78.4 GB. Even the smallest 1-bit quant (IQ1_S, 35.3 GB) is larger than a 32 GB RTX 5090. The realistic single-box options are a 96 GB workstation card, a 128 GB unified-memory machine, or a Mac Studio with M3 Ultra.
Correction (2026-09-26). An earlier version of this page called Mistral Medium 3.5 a dense ~28B model that runs Q4 in 24 GB, and gave tok/s tables for the RTX 5090, RTX 4090 and dual RTX 3090. The model card lists 128B parameters. Those figures described a model that does not exist and have been removed.
Quick answer: file size per quant, and what holds it
| Quant (GGUF) | File size | Smallest memory tier that holds the weights | Source |
|---|---|---|---|
| IQ1_S | 35.3 GB | 48 GB (2× 24 GB cards), little room for context | bartowski GGUF |
| IQ2_XXS | 40.4 GB | 48 GB (2× 24 GB cards), little room for context | bartowski GGUF |
| IQ3_XXS | 52.2 GB | 64 GB (2× 32 GB cards) | bartowski GGUF |
| Q4_K_M | 78.4 GB | 96 GB (RTX PRO 6000, 4× 24 GB cards) | bartowski GGUF |
| Q5_K_M | 91.1 GB | 128 GB unified memory (Ryzen AI Max+ 395 mini PC) | bartowski GGUF |
| Q6_K | 107.8 GB | 128 GB unified memory, tight; 192 GB+ comfortable | bartowski GGUF |
| Q8_0 | 132.9 GB | 192 GB+ (Mac Studio M3 Ultra) | bartowski GGUF |
File sizes are the GGUF parts summed, without the separate vision projector. The weights are not the whole footprint: the KV cache grows with context, so leave headroom above the file size, and more of it for long contexts.
What Mistral Medium 3.5 is
Per the model card, Mistral Medium 3.5 is a dense 128B model, meaning every parameter is used for every token. It has a 256k context window and is released under a modified MIT license with an exception for companies with large revenue. The repository's safetensors metadata totals about 127.7 billion parameters, mostly stored in FP8. Mistral also publishes an EAGLE draft model for speculative decoding with vLLM or SGLang.
The model card also warns that the original Transformers config had an incorrect entry that degraded long-context performance, and that GGUFs built before the fix are affected too. Check the upload date of any GGUF before judging long-context quality.
Which hardware can run it
Single consumer GPUs: no. A 24 GB RTX 4090 or 3090 and a 32 GB RTX 5090 cannot hold even the IQ1_S file. Running it on one of them means offloading most layers to system RAM, which makes generation very slow.
Two or four consumer cards. Two 24 GB cards (48 GB) hold IQ1_S or IQ2_XXS with little room for context. Two 32 GB cards (64 GB) hold IQ3_XXS. Four 24 GB cards (96 GB) hold Q4_K_M. Very low-bit quants of any model lose quality, so the 1- and 2-bit files are a way to try the model rather than a way to use it.
One 96 GB workstation card. NVIDIA lists the RTX PRO 6000 Blackwell with 96 GB of GDDR7 and 1792 GB/s of memory bandwidth. That holds Q4_K_M (78.4 GB) with about 17 GB left for the KV cache.
128 GB unified memory (Ryzen AI Max+ 395). AMD's spec page lists 256-bit LPDDR5x-8000 and 128 GB maximum memory, which works out to 256 GB/s. Mini PCs such as the Minisforum MS-S1 MAX and Bosgame M5 ship with 128 GB. That is enough for Q4_K_M or Q5_K_M once the operating system takes its share, but see the speed section below.
Mac Studio with M3 Ultra. Apple's announcement lists up to 512GB of unified memory and "over 800GB/s" of memory bandwidth. That holds any quant, including Q8_0.
How fast it can be: the bandwidth ceiling
Because the model is dense, generating one token reads roughly all of the weights once. Tokens per second therefore cannot exceed memory bandwidth divided by the file size. Real results land below this ceiling. The ceilings below are that arithmetic, not measurements.
| Hardware | Memory bandwidth (source) | Q4_K_M ceiling (78.4 GB) | Q8_0 ceiling (132.9 GB) |
|---|---|---|---|
| RTX PRO 6000 Blackwell | 1792 GB/s (NVIDIA) | ~23 tok/s | does not fit in 96 GB |
| Mac Studio M3 Ultra | over 800 GB/s (Apple) | ~10 tok/s | ~6 tok/s |
| Ryzen AI Max+ 395 | 256 GB/s (AMD spec) | ~3 tok/s | does not fit in 128 GB |
The ceiling is why a 128 GB mini PC that can load Q5_K_M is still a slow way to use this model. Prompt processing (prefill) is a separate, compute-bound stage, so long prompts add their own wait on top.
Which quant to pick
- Q4_K_M (78.4 GB) is the usual starting point when the memory is there, and the quant most local-inference guides default to.
- Q5_K_M and Q6_K only make sense on 128 GB+ machines, and on bandwidth-limited ones they lower the tok/s ceiling further.
- IQ3 and below are what fit on two consumer cards. Treat output quality there as something to check on your own prompts before you rely on it.
Bottom line
Mistral Medium 3.5 is not a single-GPU model. Plan on about 80 GB for Q4_K_M plus context: a 96 GB workstation card for speed, a Mac Studio M3 Ultra for capacity, or a 128 GB Ryzen AI Max+ 395 box if loading the model matters more than speed. For open models that do fit a 12-32 GB card, see Will it run? and the per-model hardware picker.
Citations and sources
- Mistral AI — Mistral Medium 3.5 128B model card: parameter count, dense architecture, 256k context, license, config warning.
- bartowski — Mistral Medium 3.5 128B GGUF: every file size in the quant table.
- NVIDIA — RTX PRO 6000 Blackwell: 96 GB GDDR7, 1792 GB/s.
- Apple — Mac Studio with M3 Ultra announcement: up to 512GB unified memory, over 800GB/s bandwidth.
- AMD — Ryzen AI Max+ 395 specifications: 256-bit LPDDR5x-8000, 128 GB maximum memory.
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
