Key Takeaways
- DeepSeek-V3 is 671B main-model parameters, 37B activated per token, with a further 14B MTP module — 685B total as published (DeepSeek).
- The weights ship natively in FP8, which is roughly 671 GB before you quantize anything.
- A 4-bit GGUF lands near 400 GB by bits-per-weight arithmetic — still far past any single GPU, including a 32 GB RTX 5090.
- 37B active predicts compute cost per token, not memory footprint. The full expert set must be resident or you pay a stall on every routed token.
- Realistic hosts are a multi-GPU workstation with 8x 24–48 GB cards, a large unified-memory box, or a many-DIMM CPU server — the last of which loads it and runs it slowly.
- A 12 GB card like the RTX 3060 12GB should be running 8–14B models or distills, which is a genuinely different decision rather than a smaller version of this one.
Step 0: are you trying to run the full model, or something that behaves like it?
Answer this before you price a single component, because the two goals have almost nothing in common.
Goal A: run the published weights. You want the actual 671B checkpoint, unmodified, because you are evaluating the model itself, you need reproducibility against the published artifact, or you have a data-residency requirement that forbids the API. This is a hundreds-of-gigabytes problem and it puts you in server hardware. It is a legitimate goal and the rest of this article sizes it honestly.
Goal B: get frontier-ish output on hardware you own. You want strong reasoning and coding output locally. You do not actually care which checkpoint produces it. This is a completely different shopping list — distills, and open-weights models in the 20–35B class, most of which fit a single card.
When Goal A is right: compliance, research reproducibility, or serving enough volume that API spend exceeds hardware amortization. When it isn't: almost everything else. Most people asking about DeepSeek-V3 VRAM requirements are describing Goal A and want Goal B, and the honest advice is to say so rather than sell them a rack.
Why does a 37B-active MoE still need hundreds of gigabytes?
A mixture-of-experts layer replaces one large feed-forward network with many smaller ones plus a router. For each token, the router scores the experts and dispatches the token to a small subset. That is the whole trick: you get the representational capacity of an enormous parameter count while paying the compute of a small one.
What it does not do is reduce how many parameters have to exist somewhere fast. The router's choice is token-dependent and effectively unpredictable. Across a single paragraph of generated output you will touch a large fraction of the expert pool, and there is no useful locality to exploit — the expert selected for token 40 tells you very little about the expert needed for token 41.
So the storage requirement is set by the union of experts that might be needed, which is all of them, while the compute requirement is set by the subset actually used, which is small. Back-ends that try to break this — streaming experts from NVMe, or holding cold experts in system RAM and hot experts in VRAM — do work, and the llama.cpp ecosystem has supported various forms of selective offload for a while. But each of them converts a memory shortfall into a latency penalty, and on a workload where every token can trigger a fetch, that penalty is the dominant term.
The practical rule: active parameters set your tokens-per-second ceiling; total parameters set your bill of materials. Anyone quoting the 37B figure as a hardware requirement has mixed up the two.
VRAM and quantization requirements
The sizes below come from bits-per-weight arithmetic against the published 671B main-model parameter count, using the standard GGUF tier widths documented in llama.cpp. Treat them as close approximations — actual file sizes vary a few percent with how a given quantizer handles attention and embedding tensors — and add KV cache on top of every row.
| Quantization | Bits/weight | Total weight size | Minimum resident memory | Hosting machine class | Quality-loss note |
|---|---|---|---|---|---|
| bf16 | 16 | ~1.34 TB | ~1.4 TB | Multi-node datacenter | Upcast from FP8; no quality gain |
| fp8 (native) | 8 | ~671 GB | ~720 GB | 8x 96 GB datacenter GPUs | Reference; this is how it ships |
| q8_0 | ~8.5 | ~713 GB | ~760 GB | 8x 96 GB datacenter GPUs | Lossless in practice; larger than native FP8 |
| q6_K | ~6.6 | ~553 GB | ~600 GB | 8x 80 GB, or a big RAM server | Effectively indistinguishable from q8 |
| q5_K_M | ~5.7 | ~478 GB | ~520 GB | 8x 64 GB, or 512 GB RAM server | Very minor degradation |
| q4_K_M | ~4.8 | ~403 GB | ~450 GB | 8x 48–64 GB, or 512 GB RAM server | The standard tier; small measurable loss |
| q3_K_M | ~3.9 | ~327 GB | ~370 GB | 8x 48 GB, or 384 GB RAM server | Noticeable degradation on reasoning |
| dynamic ~1.6-bit | ~1.6 | ~134 GB | ~180 GB | 2x 80 GB, or 192 GB RAM server | Substantial degradation; feasibility demo |
The dynamic sub-2-bit row deserves a caveat rather than enthusiasm. Aggressive mixed-precision quantization schemes have made "run the 671B model on one workstation" technically true, and the demonstrations are genuinely impressive engineering. What they do not deliver is the model people think they are getting. At that compression the reasoning behavior that makes the checkpoint worth wanting is measurably degraded, and you are frequently better served by a well-quantized 32B model that fits properly.
What hardware actually hosts it?
Three brackets, in descending order of "actually usable".
Multi-GPU workstation or server (8x 48–96 GB). This is the reference configuration and the only one where generation speed is interactive. Eight datacenter-class accelerators with tensor parallelism across NVLink or a fast fabric gives you 384–768 GB of pooled HBM at multiple terabytes per second of aggregate bandwidth. Nothing in the consumer channel touches this. It is also, at current pricing, comfortably more expensive than a car, which is the honest framing that most "run it locally" content omits.
Unified-memory box. A single machine with several hundred gigabytes of shared CPU/GPU memory sidesteps the pooling problem entirely — the whole model is addressable without a fabric. Bandwidth is far below HBM, so generation is slower, but the model loads and runs without exotic sharding. This bracket has become the most practical single-box answer for large MoE models specifically, because MoE's low compute-per-token profile tolerates lower bandwidth better than a dense model of equivalent size would.
CPU plus system RAM server. A used dual-socket server with 512 GB or more of registered DDR4 will load a q4 quantization for a fraction of the GPU cost. It will also produce tokens at a rate that makes interactive chat unpleasant, because CPU inference is bounded by memory bandwidth and even eight-channel DDR4 is an order of magnitude behind HBM. This is a reasonable batch-processing rig — overnight document runs, offline evaluation — and a poor assistant. Be clear with yourself about which one you are building.
For per-GPU numbers on the consumer cards that show up in these builds, our benchmark pages carry the per-SKU bandwidth and throughput data.
What can a 12 GB GPU run instead?
If you arrived here from a desktop with a 12 GB card, this is the section that matters. The realistic substitutes:
| Option | Params | Q4_K_M size | Fits 12 GB? | Context headroom | What you give up |
|---|---|---|---|---|---|
| 7–8B dense | 8B | ~4.8 GB | Yes, easily | Very large | Breadth of world knowledge |
| 13–14B dense | 14B | ~8.5 GB | Yes | Moderate | Some multi-step reasoning depth |
| Reasoning distill, 14B class | 14B | ~8.5 GB | Yes | Moderate | Frontier-level breadth, not chain-of-thought style |
| 20–35B-total sparse MoE | 30B | ~18 GB | No, needs offload | Poor | Fits 24 GB comfortably; 12 GB struggles |
| 32B dense | 32B | ~19 GB | No | None | Requires 24 GB+ |
The 12 GB tier's sweet spot is 8B at q6_K with a generous context window, or 14B at q4_K_M with a tighter one. Reasoning distills in that size range are the practical stand-in for a frontier MoE on this hardware — you lose breadth of world knowledge rather than usability, and everything stays resident in VRAM, which is worth more in daily use than a model that half-fits.
Both the ZOTAC Gaming GeForce RTX 3060 Twin Edge 12GB and the MSI GeForce RTX 3060 Ventus 2X 12G carry 12 GB of GDDR6 at 360 GB/s on a 170 W TGP (TechPowerUp), and either one hosts that tier without complaint. The full model-fit map is in our RTX 3060 12GB local LLM guide, and the sparse-MoE-specific case is in the Qwen3 30B-A3B GPU guide.
How much system RAM and CPU do you need for offload?
If you are attempting a partial-offload configuration — some experts in VRAM, the rest in system memory — the sizing rule is blunt: system RAM must be at least the size of the quantized weights you are not holding in VRAM, plus headroom for the OS and KV cache. For a q4 quantization of this model that puts you past 384 GB, which is registered-DIMM server territory. Desktop platforms cap out at 128–256 GB, and that is not enough for the full weights at any quantization above roughly 2 bits.
The real limiter on the CPU side is memory bandwidth, not core count. Offloaded layers stream weights from DRAM on every token, so the number of populated memory channels sets your ceiling. A dual-channel desktop platform gives you a fraction of what an eight-channel server board provides, and no CPU upgrade within a platform fixes that.
Core count still matters up to a point — enough threads to saturate the channels you have. The AMD Ryzen 7 5800X is a sensible desktop host at eight cores and sixteen threads on a 105 W TDP (AMD), which is comfortably past the knee of the curve for dual-channel DDR4. Adding cores beyond that returns very little for inference specifically, though it helps if the box also runs other services. Offload hosts are compared in the CPU-offload host piece.
Where CPU offload stops being usable, stated plainly: once a meaningful fraction of the routed experts live in system RAM, per-token latency is dominated by DRAM reads and the interactive experience collapses. It remains fine for batch jobs where you do not watch the tokens appear.
Where do you store several hundred gigabytes of weights?
Two constraints: raw capacity, and the fact that you will want more than one quantization on hand.
A q4 checkpoint of this model is around 400 GB. A q6 is around 550 GB. Keeping both, plus a couple of smaller models for comparison, is a multi-terabyte library — and re-downloading a 400 GB checkpoint because you deleted it to make room is a genuinely bad afternoon.
Drive speed affects load time, not tokens per second, because weights are read once and then live in memory. That makes cheap bulk capacity the right buy. The Crucial BX500 1TB rates up to 540 MB/s sequential read and the Kingston A400 960GB up to 500 MB/s. At 500 MB/s, a 400 GB checkpoint takes on the order of thirteen minutes to read cold — slow, but a once-per-boot cost, and NVMe reduces it to a few minutes rather than eliminating it. Buy capacity over speed, and see the NVMe vs SATA model-library comparison if you are weighing the upgrade.
What does context length cost on top of the weights?
The KV cache is the second budget, and it grows linearly with prompt length independent of the weights.
The arithmetic follows the transformer architecture the back-ends implement (llama.cpp): bytes per token equals 2 x layers x KV heads x head dimension x bytes per element. Deep models with many layers pay proportionally more, and a model of this scale has a lot of layers.
| Context | Rough KV cache, fp16, deep large model | On top of a ~400 GB q4 checkpoint |
|---|---|---|
| 8K | Single-digit GB | Negligible against the weights |
| 32K | Tens of GB | Noticeable; plan for it |
| 128K | Well over 100 GB | A second machine's worth of memory |
Two mitigations apply. Quantizing the KV cache to 8-bit halves it at a small quality cost and is a flag in most back-ends. Architectures using compressed latent attention cut it much further, which is one reason large MoE releases have gravitated toward those schemes. Neither changes the shape: long context is a separate, linear, easily-forgotten line item, and on a machine sized exactly to the weights it is what pushes you over.
Cost per token: self-hosted vs API
The uncomfortable arithmetic. A machine that hosts this model at a usable quantization and interactive speed is a five-to-six-figure capital purchase plus continuous power draw plus your time maintaining it. Hosted inference for models in this class is priced per million tokens and requires none of that.
Self-hosting pencils out when at least one of these holds:
- Volume. You are generating enough tokens per month that API spend crosses hardware amortization. That is a real threshold and some workloads clear it, but it is much higher than most people estimate.
- Data residency. Your inputs legally cannot leave your infrastructure. This is the strongest reason and it does not care about the math.
- Reproducibility. You need the exact published weights pinned indefinitely, immune to a provider deprecating an endpoint.
- Research access. You need to inspect activations, modify routing, or fine-tune — none of which an API exposes.
If none of those describe you, self-hosting the full 671B checkpoint is a hobby expense rather than an economic decision. That is a perfectly good reason to do it, but it should be an honest one.
Verdict matrix
Self-host the full weights if… you have a compliance requirement, a reproducibility requirement, or genuine research need, and you have access to a multi-GPU server or a large unified-memory box. Budget q4_K_M as your working tier — it is the point where quality loss is small and the hardware requirement is merely enormous rather than absurd.
Run a distill or a smaller open-weights model locally if… you want strong reasoning and coding output on hardware you can actually buy. A 14B distill on a 12 GB card, or a 32B model on a 32 GB card, delivers most of the daily utility at a thousandth of the capital cost. This is the right answer for the large majority of readers.
Use the API if… your volume is moderate, your data is not restricted, and you want the published model's behavior without owning the infrastructure. Nothing about running weights on your own metal makes the output better — it makes it yours, which is a different property and only sometimes the one you need.
Bottom line
DeepSeek-V3's 37B active-parameter figure is the most successfully misread number in local inference. It tells you the model computes cheaply. It tells you nothing about the roughly 400 GB of 4-bit weights that have to sit somewhere the router can reach in microseconds. Size for total parameters, budget the KV cache separately, and if the resulting bill of materials does not fit your situation, run a 14B distill on the card you already own — you will get more done.
Related guides
- RTX 3060 12GB Local LLM Guide: Which Models Actually Fit
- Best GPU for Qwen3 30B-A3B: What the 3B Active Path Really Needs
- i7-9700K vs Ryzen 7 5800X as a CPU-Offload Host
- Ollama vs vLLM vs llama.cpp on a 12GB GPU
- NVMe vs SATA SSD for Local LLM Model Libraries
Citations and sources
- DeepSeek — DeepSeek-V3 model card on Hugging Face (accessed 2026-09-09)
- llama.cpp — GGUF quantization tiers, KV cache and expert-offload implementation (accessed 2026-09-09)
- TechPowerUp — GeForce RTX 3060 12 GB specifications (accessed 2026-09-09)
- AMD — Ryzen 7 5800X product page (accessed 2026-09-09)
- NVIDIA — GeForce RTX 5090 (accessed 2026-09-09)
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported. Quantized file sizes are arithmetic estimates from the published parameter count and standard bits-per-weight tiers; verify against the specific quantization you download.
