Building or buying a local AI server to run large language models comes down to four questions: how much memory does the target model need, which GPU tier fits the budget, is storage and network fast enough to not become the bottleneck, and can the system sustain that load 24/7 without thermal throttling. This synthesis walks through each, grounded in public specs and community-published sizing guides rather than invented benchmark numbers.
How Much RAM and VRAM a Local AI Server Actually Needs
Memory sizing for local LLMs is one of the most frequently discussed topics on r/LocalLLaMA and in the llama.cpp and Ollama documentation, and the community has converged on a simple heuristic: memory footprint scales with parameter count and quantization level, not with any single "minimum spec."
| Model size | Approx. size at Q4 quantization | Approx. size at fp16 |
|---|---|---|
| 7B | ~4-5GB | ~13-14GB |
| 13B | ~7-8GB | ~24-26GB |
| 34B | ~19-20GB | ~65-68GB |
| 70B | ~38-40GB | ~130-140GB |
These figures come from the widely cited quantization math published alongside the llama.cpp project and repeated across community sizing threads — treat them as approximate, since exact footprint also depends on context length and KV cache settings. The practical takeaway: a 13B model comfortably fits on a 24GB consumer GPU at Q4, while running the same model at full fp16 precision pushes into data-center territory. For a deeper walkthrough of matching RAM, VRAM, and NVMe to a 13B+ build, see Local AI Server Build 2026: RAM, VRAM and NVMe for 13B+ Models.
System RAM matters separately from VRAM when any part of the model spills to CPU offload (common with llama.cpp's --n-gpu-layers partial offload). Running short on system RAM doesn't degrade gracefully — it stalls or crashes the process — so sizing RAM above the model's total footprint, not just matching it, is the safer default.
GPU Tiers for a Local LLM Server
GPU choice for local inference splits into three practical tiers: consumer cards repurposed for AI work, prosumer/workstation cards with more VRAM per dollar than their consumer siblings, and data-center accelerators built for multi-tenant clusters.
Consumer tier covers cards like the RTX 4090, RTX 3060 12GB, and AMD's RX 7900 XTX. These are the entry point for most home-lab builds because they're available at retail and supported by mainstream inference stacks (llama.cpp, Ollama, vLLM). VRAM capacity — not just raw compute — is usually the limiting factor here, which is why a 12GB card like the RTX 3060 remains a popular budget pick for 7B-class models; see Intel Arc B580 vs RTX 3060 12GB for Local LLMs in 2026 and Local LLM PC Build 2026: What 12GB of VRAM Really Runs for what that VRAM ceiling actually supports.
Prosumer/workstation tier includes cards like AMD's Radeon PRO W-series and Intel's Arc Pro B60, which trade gaming clocks for higher VRAM capacity at a given price point — useful when a model's footprint sits just above what consumer cards offer. Intel Arc Pro B60 AI: 24GB VRAM for Local LLMs in 2026 covers one example of this tier.
Data-center tier — AMD's Instinct MI300X, NVIDIA's H100/H200 — targets multi-tenant inference clusters running dozens of concurrent sessions, not single-user home servers. These parts use HBM3-class memory (the MI300X ships with roughly 192GB, per AMD's own product page) specifically to hold very large models or serve many requests in parallel; procurement and power requirements put them outside the scope of a home or small-office build. If a workload genuinely needs that scale, it's worth budgeting for enterprise leasing rather than a one-off purchase.
Exact throughput comparisons between AMD and NVIDIA cards shift frequently as ROCm and CUDA software stacks update, so treat any specific tokens-per-second multiplier as a snapshot of one benchmark run on one software version rather than a durable rule — check current numbers on TechPowerUp or r/LocalLLaMA before making a purchase decision based on raw throughput.
New Model Releases Change the Math
Local server sizing isn't static — new open-weight releases regularly shift what fits on a given GPU. When Google shipped a stealth update to Gemma addressing tool-calling behavior, the VRAM math for running it locally stayed the same, but the practical reliability of local agentic workflows changed; see Gemma 4 Stealth Update Fixes Tool Calling: What Changes Locally. Similarly, when a new large open model like Kimi K3 lands, the first question a home-lab server operator asks is whether it fits in a given VRAM budget at all — see Can You Run Kimi K3 Locally? VRAM Math for the New Open Model for that specific breakdown.
Storage and Networking: The Parts People Forget
Model loading time is dominated by storage read speed, which is why NVMe SSDs are the default recommendation over SATA drives for any server that swaps between multiple models — a large model loading from a slow drive can add tens of seconds to cold-start latency compared to a fast NVMe drive, though the exact gap depends on model size and the specific drive's sustained read performance.
Networking is the piece most home-lab builders under-provision. A local LLM server accessed only from the machine it runs on doesn't need much beyond a stable link, but the moment a second device, a remote client, or a multi-node setup joins, wired Ethernet becomes worth the extra cable run. A dual-band Wi-Fi 6 router with solid wired backhaul, like the TP-Link Archer AX55, handles mixed wired/wireless access reasonably well, and short, well-shielded Cat6 patch cables — the 1ft, 3ft, and 5ft 5-packs from Jadaol cover typical rack and desk-to-switch runs — keep a multi-machine setup from bottlenecking on the network instead of the GPU.
Cooling and Power for Sustained LLM Workloads
Inference workloads differ from gaming in one important way: sustained utilization. A gaming session has natural load variance as scenes change; a batch inference job can pin a GPU at high utilization for the full duration of a long generation or a batch job. That sustained load pattern makes case airflow and cooler quality more important than peak boost clocks. Actual operating temperatures depend heavily on case design, ambient room temperature, fan curve tuning, and the specific cooler on a given card — without a source-backed number for a specific configuration, treat any thermal claim as workload- and case-dependent rather than a fixed spec that transfers across builds.
Choosing a Practical Build Tier
| Use case | Typical model range | GPU tier | Notes |
|---|---|---|---|
| Personal coding assistant / chat | 7B-13B at Q4 | Single consumer GPU (12-24GB) | See Local LLM in VSCode: The 2026 Setup Guide for IDE integration |
| Small-office shared server | 13B-34B at Q4-Q8 | Single 24GB+ or prosumer card | Prioritize VRAM headroom for concurrent sessions |
| Compact/low-power home server | 7B-13B at Q4 | Mini PC with integrated or discrete GPU | See Best Mini PC for Local LLMs in 2026 |
| Multi-tenant / large-model serving | 34B+ or multiple concurrent models | Data-center accelerator or multi-GPU cluster | Typically enterprise procurement, not home-lab budget |
For most self-hosted use cases — a personal assistant, a small team's shared inference endpoint, or an always-on coding helper — the consumer and prosumer tiers cover the realistic model range. Data-center accelerators only make sense once the workload genuinely requires serving many concurrent large-model sessions.
Citations and sources
- https://github.com/ggerganov/llama.cpp
- https://www.reddit.com/r/LocalLLaMA/
- https://ollama.com/
- https://www.techpowerup.com/
- https://www.amd.com/en/products/accelerators/instinct/mi300x.html
- https://www.tomshardware.com/
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
