Running large language models on your own hardware instead of a cloud API comes down to three resources: GPU VRAM (or system RAM if you're going CPU-only), memory bandwidth, and storage fast enough to load multi-gigabyte model weights without stalling. The rest — which brand of silicon, which motherboard, which case — is secondary to getting those three numbers right for the model sizes you actually want to run.
This synthesis walks through the current GPU landscape across AMD, NVIDIA, and Intel, translates model size into concrete RAM/VRAM requirements, and covers CPU-only and storage considerations that most local LLM guides skip.
The short answer: what actually matters
| Priority | Why it matters | Rule of thumb |
|---|---|---|
| GPU VRAM | Determines the largest model you can load without swapping to system RAM | Match VRAM to your target model's quantized size |
| Memory bandwidth | Determines tokens-per-second at a given model size | Higher-bandwidth memory (HBM > GDDR > DDR) inference faster |
| System RAM | Needed for CPU offload, larger context windows, and multi-model setups | At least equal to GPU VRAM, more for CPU-only inference |
| Storage | Model weights for 30B+ models run 15-40GB+ per checkpoint | A fast SSD avoids multi-minute load times |
For deeper hardware-specific comparisons, see this catalog's AMD Instinct vs Radeon Pro local LLM benchmarks and the Intel Arc B580 write-up for a budget-tier alternative.
GPU choice: AMD, NVIDIA, and Intel for local inference
NVIDIA remains the default recommendation for most people starting out, mainly because CUDA has the deepest and most mature support across inference frameworks — vLLM, llama.cpp, ExLlama, and most quantization tooling target CUDA first and add other backends afterward. Consumer cards like the RTX 4090 (24GB) are a common reference point for what a single high-end gaming GPU can handle.
AMD's position has shifted meaningfully. On the data-center side, the Instinct MI300 series ships with substantially more onboard memory than NVIDIA's H100 (per AMD's published specifications), which matters directly for how large a model fits on a single card without splitting it across multiple GPUs — a bigger practical win for local hosting than raw throughput numbers. On the workstation side, AMD's Radeon Pro W7900 targets the same VRAM-per-dollar niche that professional NVIDIA RTX cards occupy, and llama.cpp's ROCm backend has closed much of the earlier compatibility gap for GGUF-format models. The catalog's dedicated AMD Instinct vs Radeon Pro benchmarks piece goes deeper on where ROCm still lags CUDA in practice.
Intel's Arc line, meanwhile, has become a legitimate budget entry point specifically because of VRAM-per-dollar rather than raw speed — the Arc B580 at 12GB for under $300 undercuts equivalent NVIDIA and AMD VRAM tiers, at the cost of a less mature software stack (OpenVINO and IPEX-LLM rather than CUDA or ROCm).
| GPU family | VRAM tier (representative) | Best fit | Software maturity |
|---|---|---|---|
| NVIDIA RTX 4090 / workstation RTX | 24GB (consumer) | Broadest framework support, fastest to get running | Highest — CUDA-first tooling |
| AMD Instinct MI300-series | High-capacity HBM | Large models on a single accelerator | Improving via ROCm |
| AMD Radeon Pro W7900 | High VRAM workstation tier | VRAM-per-dollar for mid-to-large models | ROCm/llama.cpp, gap narrowing |
| Intel Arc B580 | 12GB | Budget entry, small-to-mid quantized models | Newest, least mature |
None of these choices is universally "best" — it depends on whether you're optimizing for framework compatibility (NVIDIA), VRAM capacity per dollar (AMD), or absolute entry price (Intel).
VRAM and system RAM: how much do you actually need?
Model size in parameters translates fairly predictably into memory footprint once you know the precision. At FP16 (the default "full precision" most models ship in), each parameter costs roughly 2 bytes. Quantized to 4-bit GGUF — the format most local runners default to — that drops to roughly 0.5-0.6 bytes per parameter including overhead, per llama.cpp's quantization documentation and Hugging Face's inference guidance.
| Model size | FP16 footprint | 4-bit (GGUF Q4) footprint | Realistic GPU tier |
|---|---|---|---|
| 7-8B | ~14-16GB | ~4-5GB | Any 8GB+ consumer GPU |
| 13B | ~26GB | ~7-8GB | 12GB+ consumer GPU |
| 30-34B | ~60-68GB | ~17-20GB | 24GB consumer GPU, tight |
| 70B | ~140GB | ~35-40GB | Dual 24GB GPUs or one 48GB workstation card |
System RAM should generally match or exceed your GPU VRAM if you plan on any CPU offload (common when a model slightly exceeds what fits on the GPU alone). For CPU-only inference, RAM is your only ceiling — the same quantized-footprint numbers above apply directly, just against system memory instead of VRAM.
CPU-only and hybrid CPU/GPU inference
A GPU isn't strictly required. llama.cpp and Ollama both run entirely on CPU, splitting model layers across available RAM. Throughput is the tradeoff: CPU-only inference is dramatically slower than GPU inference for anything beyond small quantized models, and the gap widens as model size grows. Modern high-core-count CPUs with fast memory channels narrow that gap somewhat but don't close it — this remains the reason most serious local LLM builds still center on GPU VRAM rather than CPU horsepower.
Hybrid setups — partial GPU offload with the rest of the model in system RAM — are a common middle ground for running models slightly larger than available VRAM, at a throughput cost proportional to how much sits on the CPU side. Runner choice affects how well this is handled in practice; the catalog's Ollama vs LM Studio vs vLLM comparison and its vLLM vs llama.cpp on an RTX 3060 piece both cover how offload behavior differs by tool.
Storage: don't overlook load times
A 30B-class model in 4-bit GGUF format runs 17-20GB per checkpoint; a 70B model can be 35-40GB. Loading that from a slow spinning disk adds real, noticeable delay every time you swap models or restart a session. A fast external SSD — something like a WD 2TB My Passport SSD — is a reasonable way to keep a rotating library of quantized checkpoints without eating into your system drive, while a higher-capacity spinning drive like a Seagate 4TB One Touch works fine as bulk cold storage for models you use less often. Price and specs for both are current catalog listings, not benchmarked by this piece.
Budget tiers: from experimentation to multi-GPU workstations
At the low end, small quantized models can run on surprisingly modest hardware — the catalog's Raspberry Pi for local LLMs guide covers just how far a Pi 4 or 5 can be pushed for lightweight, low-throughput experimentation.
At the high end, the calculus shifts toward whether local hosting is even the right call versus cloud API access for frontier-scale models — a tradeoff explored in Anthropic's 2GW AMD compute deal and what it means for local vs. cloud. For workloads that blend LLM inference with other GPU-heavy tasks like 3D rendering or diffusion models, the AI rigging and 3D model hardware guide covers overlapping hardware considerations.
Most home builders land somewhere in the middle: a single 24GB-class consumer or workstation GPU, 64GB of system RAM, and enough fast storage to keep a handful of quantized models on hand — comfortably covering 7B through 30B-class models with room for the occasional heavily-quantized 70B experiment.
Choosing a runner to match your hardware
Hardware choice and software runner choice aren't independent decisions. llama.cpp's GGUF format and broad backend support (CUDA, ROCm, Metal, CPU) make it the most hardware-agnostic option, which matters if you're on AMD or mixed hardware. vLLM targets higher-throughput serving scenarios and leans more heavily on CUDA maturity. Ollama wraps llama.cpp with a simpler interface aimed at getting a model running quickly rather than squeezing out maximum throughput. The Ollama vs LM Studio vs vLLM comparison breaks down which fits which use case, and the vLLM vs llama.cpp on a budget RTX 3060 piece is a useful reference point for VRAM-constrained builds specifically.
Citations and sources
- llama.cpp (GitHub) — quantization formats and backend support
- Hugging Face: efficient inference on a single GPU — precision and memory footprint guidance
- AMD Instinct MI300 series product page — memory specifications
- NVIDIA H100 product page — memory specifications
- AMD Radeon Pro W7900 product page — VRAM and workstation positioning
- r/LocalLLaMA — community hardware and runner discussion
- Ollama model library — model size and quantization reference
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
