This tutorial walks you through running Llama 3.1 70B on an NVIDIA GeForce RTX 5080. Exact commands, expected tokens-per-second, and the tradeoffs you should know before starting.
Does it fit?
NVIDIA GeForce RTX 5080 has 16 GB of GDDR7. Llama 3.1 70B at q4_K_M wants ~42 GB of it for weights alone.
Verdict: ⚠️ Tight fit. You'll need heavy CPU offload (llama.cpp with only ~13 of 80 layers on the GPU) or drop to q3_K_S. Expect 1-3 tok/s with offload.
Install Ollama (the easy path)
Ollama handles GPU detection automatically on NVIDIA (via CUDA), AMD (via ROCm on Linux), and Apple Silicon (via Metal). No manual configuration needed.
Install llama.cpp (more control)
llama.cpp gives you flag-level control over quantization, context length, and layer offload. Build from source:
With 16GB VRAM, offload ~13 layers; the rest stays on CPU. Expect roughly half the all-GPU speed but much cleaner than running at smaller quants.
Expected performance
- Community reports from LocalLLaMA suggest ~1-3 tok/s for Llama 3.1 70B q4_K_M on a 16GB card with heavy CPU offload. All-GPU 70B inference requires ≥48GB VRAM.
If you need more speed, bigger VRAM is the answer. Consider upgrading to a card with more headroom →.
Common issues
"out of memory" on the first prompt: reduce context length (-c 2048) or quantization (q4_K_S instead of q4_K_M).
Slow first token but fast generation: that's prompt processing ("prefill"). Normal — blame the KV cache building. Subsequent messages in the same session will be snappy.
Frequent swapping / system hangs: VRAM is full AND system RAM is full. Close Chrome. Add more DDR5.
Related
NVIDIA GeForce RTX 5080 specs: 16GB memory, 360W TDP, 2025 launch. MSRP $999.
Does it fit? Full quantization matrix
Weight-only VRAM for Llama 3.1 70B at every common quant, plus the KV-cache overhead for a 4K-token context. KV cache scales linearly with context — see the context-length table further down.
| Quant | Weights | + KV @ 4K ctx | Total | Fits on this GPU? | Quality loss |
|---|---|---|---|---|---|
| q2_K_S | 21.0 GB | 5.6 GB | 26.6 GB | ❌ | Severe (15-25%) |
| q3_K_M | 31.5 GB | 5.6 GB | 37.1 GB | ❌ | Noticeable (5-8%) |
| q4_K_M | 42.0 GB | 5.6 GB | 47.6 GB | ❌ | Minimal (1-3%) — community default |
| q5_K_M | 49.0 GB | 5.6 GB | 54.6 GB | ❌ | <1% |
| q6_K | 56.0 GB | 5.6 GB | 61.6 GB | ❌ | Effectively lossless |
| q8_0 | 77.0 GB | 5.6 GB | 82.6 GB | ❌ | Inference-lossless |
| fp16 | 140.0 GB | 5.6 GB | 145.6 GB | ❌ | Baseline (original precision) |
Values are approximate — actual footprint depends on batch size, whether the KV cache is quantized (-ctk q8_0 -ctv q8_0 in llama.cpp halves it), and whether you reserve VRAM for a display. Rule of thumb: budget 5-10% headroom on top of the table.
How public benchmarks show and compared
Every tok/s, FPS, and synthetic score in this article is pulled live from the SpecPicks benchmark catalog (hardware_specs, ai_benchmarks, synthetic_benchmarks). We cite the source_name on each row — the vast majority are community-reported numbers from r/LocalLLaMA and llama.cpp GitHub Discussions, with synthetic scores from PassMark, Phoronix, and Tom's Hardware's GPU hierarchy.
Where DB rows exist for a specific model+quant+GPU combination, we quote the number exactly. Where they don't, we fall back to published spec-sheet values (VRAM capacity, TDP, memory bandwidth) plus the closest community-verified ballpark — clearly flagged as a ballpark, not a measurement. We prefer "we don't know" over a fabricated number.
SpecPicks does not run paid hardware review cycles; we aggregate. If you see a number you can improve on, pull-request the row.
Measured tok/s on this GPU
Live data from ai_benchmarks for NVIDIA GeForce RTX 5080, filtered to the Llama 3.1 70B family where available:
| Model | Quant | Runtime | Gen tok/s | VRAM used | Source |
|---|---|---|---|---|---|
| No direct matches in the DB yet — see community thread below |
For the full tok/s matrix on this card across every model we've logged, see the NVIDIA GeForce RTX 5080 benchmark page.
Context length and VRAM — the hidden cost
KV cache grows linearly with context. Here's the approximate overhead on top of 42.0 GB of q4_K_M weights for Llama 3.1 70B:
| Context | KV cache | Total VRAM |
|---|---|---|
| 2K tokens | ~0.65 GB | ~42.7 GB |
| 4K tokens | ~1.3 GB | ~43.3 GB |
| 8K tokens | ~2.6 GB | ~44.6 GB |
| 32K tokens | ~10 GB | ~52 GB |
| 128K tokens | ~40 GB | ~82 GB |
(Llama 3.1 70B uses GQA with 8 KV heads across 80 layers — much smaller per-token KV cost than non-GQA models.)
For long-context workloads (≥32K tokens) on consumer hardware, use llama.cpp's KV-cache quantization — -ctk q8_0 -ctv q8_0 roughly halves cache footprint with sub-1% quality loss. This is the single biggest VRAM-saving flag for long context.
Which runtime wins on this hardware?
Three mainstream runtimes target NVIDIA GeForce RTX 5080; the right one depends on your workload:
- Ollama — easiest. Auto-detects CUDA, handles model downloads, exposes an OpenAI-compatible API out of the box. Wraps llama.cpp; you give up fine-grained control for zero setup.
- llama.cpp — direct flag-level control over quant, context, KV-cache precision, batch size, split layers across GPUs. Where the LocalLLaMA community benchmarks its numbers (see the Apple-Silicon megathread #4167 for reference tok/s across M-series chips).
- vLLM — built for production serving. Tensor parallelism, PagedAttention, continuous batching. Linux + NVIDIA CUDA primary target. If you're not serving multiple concurrent users, the overhead isn't worth it.
For head-to-head numbers and install commands across all three, see our Ollama vs llama.cpp vs vLLM guide.
Troubleshooting — three failure modes and fixes
1. First token takes 5-30 seconds, then generation is fast. That's normal prefill: the model is processing your prompt before it can start generating. On a long prompt (4K+ tokens) prefill dominates the first-token latency. If it's unexpectedly slow, check that you actually offloaded layers to the GPU — nvidia-smi or powermetrics (macOS) should show near-100% utilisation during prefill. If utilisation is flat, your inference is running on CPU.
2. "Out of memory" halfway through a long chat. The KV cache grew past what the card can hold. Drop to a smaller quant (q4_K_M → q3_K_M), cut -c context length, or enable KV-cache quantization (-ctk q8_0 -ctv q8_0 in llama.cpp). On Ollama set num_ctx smaller in your Modelfile.
3. Tok/s is ~30% of what LocalLLaMA threads report. Three usual suspects: (a) power/thermal throttling — check sustained clocks during a long prompt; (b) PCIe x8 or x4 link when you expected x16 — nvidia-smi --query-gpu=pcie.link.width.current --format=csv; (c) running the wrong binary, e.g. a non-CUDA llama.cpp build on an NVIDIA card. Rebuild with GGML_CUDA=1 and confirm.
Frequently asked questions
Can I run Llama 3.1 70B on NVIDIA GeForce RTX 5080 without offloading to CPU?
No. q4_K_M weights alone are ~42 GB, well beyond the RTX 5080's 16 GB. You need CPU offload (heavy — most layers on CPU) or a card with ≥48 GB VRAM (e.g. RTX 6000 Ada/Pro, or dual-GPU setups).
What quantization should I use on NVIDIA GeForce RTX 5080?
q4_K_M is the community default — 1-3% quality loss vs fp16 with less than half the memory. Drop to q3_K_M only when VRAM is tight. Go to q6_K or q8_0 when you have headroom and want to eliminate quant damage as a variable.
Is NVIDIA GeForce RTX 5080 bottlenecked by memory or compute for this model?
Dense-weight inference is memory-bandwidth-bound on almost every consumer card. The RTX 5080's ~960 GB/s of GDDR7 bandwidth sets a ceiling: sustained tok/s ≈ memory bandwidth ÷ weight bytes per token — i.e. all-GPU q4_K_M on a 70B model tops out around 20-23 tok/s on this card if the weights fit, which they don't. The compute units are rarely the limit for single-user inference; they matter more for batched serving.
Does multi-GPU help for this model?
For a 70B model, usually no. If the model already fits in one card, a second card mainly helps batch throughput (vLLM) not single-user latency. Tensor parallelism adds inter-GPU traffic that often nets negative for interactive chat. Multi-GPU pays off on 70B+ models where you need to stack VRAM across cards.
Where can I report or compare my own tok/s numbers?
The r/LocalLLaMA community benchmark threads are the canonical place. llama.cpp also maintains a GitHub Discussions thread for Apple Silicon and per-platform performance. SpecPicks imports numbers from both into ai_benchmarks; if you want a figure added, pull-request the row.
Sources
- r/LocalLLaMA (community tok/s threads)
- llama.cpp GitHub Discussions #4167 — Apple Silicon benchmark thread
- Tom's Hardware — RTX 5080 Founders Edition review
- Phoronix — RTX 5080/5090 Linux performance review
Related guides
- NVIDIA GeForce RTX 5080 full benchmark page
- Best GPU for Llama 3.1 70B
- Ollama vs llama.cpp vs vLLM
- What VRAM do you need for local LLMs?
