Skip to main content

How to run Llama 3.1 70B on NVIDIA GeForce RTX 5090

Exact commands, expected tok/s, VRAM math for this specific combination.

By , Founder & Editor-in-Chief · Published 2026-04-21 · Updated 2026-10-17 · 2 min read

Requires CPU offload — step-by-step Ollama and llama.cpp setup plus real tok/s numbers for Llama 3.1 70B on NVIDIA GeForce RTX 5090.

How to run Llama 3.1 70B on NVIDIA GeForce RTX 5090

Which models fit on an RTX 5090?

Llama 3.1 70B's Q4_K_M file is 42.5 GB (per its Hugging Face listing, bartowski/Meta-Llama-3.1-70B-Instruct-GGUF); on the RTX 5090, the weights do not fit, so the remaining layers run on the CPU from system RAM and system-RAM bandwidth sets the speed. SpecPicks tracks 3 community runs of that size on this card, median 17.8 tok/s. RTX 5090 carries 32 GB of VRAM. The weights column is the range of real Q4_K_M files on Hugging Face for the models in each size class (about 0.6 GB per billion parameters), and the runtime plus a usable context window wants about 2 GB on top; every tokens-per-second figure is a median over community-reported Q4 runs SpecPicks tracks for this card, with the run count and the source beside it.

Showing the model sizes this article covers and the band either side. Every size from 3B to 70B+, for every card SpecPicks tracks, is in the local-LLM GPU table.

Model size Weights at Q4 Fits in 32 GB? Measured Left for context Source
30-35B (Qwen 3 32B, QwQ 32B)The step change. A 24 GB card holds this entirely in VRAM; below that it is CPU offload. about 19.9 GB Fitsweights and a usable context window 55.6 tok/s8 runs · 2 sources ~12 GBfor runtime and KV cache Hardware Corner
70B+ (Llama 3.3 70B, Qwen 2.5 72B)One 48 GB card or two 24 GB cards. A 32 GB card runs it only with layers in system RAM. about 42.5 GB Nospills to system RAM — system-RAM bandwidth sets the speed 17.8 tok/s3 runs · 3 sources none LocalLLaMA

Every RTX 5090 benchmark run, with its source → GPU picks for local LLM — the same table across every card we track How we source these numbers

This tutorial walks you through running Llama 3.1 70B on an NVIDIA GeForce RTX 5090. Exact commands, expected tokens-per-second, and the tradeoffs you should know before starting.

Does it fit?

NVIDIA GeForce RTX 5090 has 32 GB of GDDR7. Llama 3.1 70B at q4_K_M wants ~42 GB of it for weights alone.

Verdict: ⚠️ Doesn't fit at q4_K_M. The 32 GB card is ~10 GB short, so you'll need real CPU offload (llama.cpp with ~20–25 layers on CPU) or drop to q3_K_M / q3_K_S. Expect 6-12 tok/s with offload.

Install Ollama (the easy path)

bash
# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:70b
ollama run llama3.1:70b

Ollama handles GPU detection automatically on NVIDIA (via CUDA), AMD (via ROCm on Linux), and Apple Silicon (via Metal). No manual configuration needed.

Install llama.cpp (more control)

llama.cpp gives you flag-level control over quantization, context length, and layer offload. Build from source:

bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

# Download a quantized GGUF (community favorite: bartowski or TheBloke on HuggingFace)
./llama-cli -m ~/models/llama-3-1-70b-q4_k_m.gguf \
 -n 512 -c 4096 \
 -ngl 26 \
 -p "Write a haiku about GPUs"

With 32GB VRAM you can typically place ~55–60 of Llama 3.1 70B's 80 layers on the GPU (-ngl 55 to start, then tune up while watching VRAM headroom); the remainder stays on CPU. Expect roughly half the all-GPU speed but much cleaner output than running at smaller quants.

Expected performance

  • Community reports from LocalLLaMA suggest ~10-25 tok/s on this class of hardware.

If you need more speed, bigger VRAM is the answer. Consider upgrading to a card with more headroom →.

Common issues

"out of memory" on the first prompt: reduce context length (-c 2048) or quantization (q4_K_S instead of q4_K_M).

Slow first token but fast generation: that's prompt processing ("prefill"). Normal — blame the KV cache building. Subsequent messages in the same session will be snappy.

Frequent swapping / system hangs: VRAM is full AND system RAM is full. Close Chrome. Add more DDR5.

NVIDIA GeForce RTX 5090 specs: 32GB memory, 575W TDP, 2025 launch. MSRP $1,999.

Does it fit? Full quantization matrix

Weight-only VRAM for Llama 3.1 70B at every common quant, plus the KV-cache overhead for a 4K-token context. KV cache scales linearly with context — see the context-length table further down.

QuantWeights+ KV @ 4K ctxTotalFits on this GPU?Quality loss
q2_K_S21.0 GB5.6 GB26.6 GB✅Severe (15-25%)
q3_K_M31.5 GB5.6 GB37.1 GB❌Noticeable (5-8%)
q4_K_M42.0 GB5.6 GB47.6 GB❌Minimal (1-3%) — community default
q5_K_M49.0 GB5.6 GB54.6 GB❌<1%
q6_K56.0 GB5.6 GB61.6 GB❌Effectively lossless
q8_077.0 GB5.6 GB82.6 GB❌Inference-lossless
fp16140.0 GB5.6 GB145.6 GB❌Baseline (original precision)

Values are approximate — actual footprint depends on batch size, whether the KV cache is quantized (-ctk q8_0 -ctv q8_0 in llama.cpp halves it), and whether you reserve VRAM for a display. Rule of thumb: budget 5-10% headroom on top of the table.

How public benchmarks show and compared

Every tok/s, FPS, and synthetic score in this article is pulled live from the SpecPicks benchmark catalog (hardware_specs, ai_benchmarks, synthetic_benchmarks). We cite the source_name on each row — the vast majority are community-reported numbers from r/LocalLLaMA and llama.cpp GitHub Discussions, with synthetic scores from PassMark, Phoronix, and Tom's Hardware's GPU hierarchy.

Where DB rows exist for a specific model+quant+GPU combination, we quote the number exactly. Where they don't, we fall back to published spec-sheet values (VRAM capacity, TDP, memory bandwidth) plus the closest community-verified ballpark — clearly flagged as a ballpark, not a measurement. We prefer "we don't know" over a fabricated number.

SpecPicks does not run paid hardware review cycles; we aggregate. If you see a number you can improve on, pull-request the row.

Measured tok/s on this GPU

Live data from ai_benchmarks for NVIDIA GeForce RTX 5090, filtered to the Llama 3.1 70B family where available:

ModelQuantRuntimeGen tok/sVRAM usedSource
No direct matches in the DB yet — see community thread below

For the full tok/s matrix on this card across every model we've logged, see the NVIDIA GeForce RTX 5090 benchmark page.

Context length and VRAM — the hidden cost

KV cache grows linearly with context. Here's the approximate overhead on top of 42.0 GB of q4_K_M weights for Llama 3.1 70B:

ContextKV cacheTotal VRAM
2K tokens~2.8 GB~44.8 GB
4K tokens~5.6 GB~47.6 GB
8K tokens~11.2 GB~53.2 GB
32K tokens~44.8 GB~86.8 GB
128K tokens~179.2 GB~221.2 GB

For long-context workloads (32K and above) on consumer hardware, use llama.cpp's KV-cache quantization — -ctk q8_0 -ctv q8_0 roughly halves cache footprint with sub-1% quality loss. This is the single biggest VRAM-saving flag for long context.

Which runtime wins on this hardware?

Three mainstream runtimes target NVIDIA GeForce RTX 5090; the right one depends on your workload:

  • Ollama — easiest. Auto-detects CUDA, handles model downloads, exposes an OpenAI-compatible API out of the box. Wraps llama.cpp; you give up fine-grained control for zero setup.
  • llama.cpp — direct flag-level control over quant, context, KV-cache precision, batch size, split layers across GPUs. Where the LocalLLaMA community benchmarks its numbers (see the Apple-Silicon megathread #4167 for reference tok/s across M-series chips).
  • vLLM — built for production serving. Tensor parallelism, PagedAttention, continuous batching. Linux + NVIDIA CUDA primary target. If you're not serving multiple concurrent users, the overhead isn't worth it.

For head-to-head numbers and install commands across all three, see our Ollama vs llama.cpp vs vLLM guide.

Troubleshooting — three failure modes and fixes

1. First token takes 5-30 seconds, then generation is fast. That's normal prefill: the model is processing your prompt before it can start generating. On a long prompt (4K+ tokens) prefill dominates the first-token latency. If it's unexpectedly slow, check that you actually offloaded layers to the GPU — nvidia-smi or powermetrics (macOS) should show near-100% utilisation during prefill. If utilisation is flat, your inference is running on CPU.

2. "Out of memory" halfway through a long chat. The KV cache grew past what the card can hold. Drop to a smaller quant (q4_K_M → q3_K_M), cut -c context length, or enable KV-cache quantization (-ctk q8_0 -ctv q8_0 in llama.cpp). On Ollama set num_ctx smaller in your Modelfile.

3. Tok/s is ~30% of what LocalLLaMA threads report. Three usual suspects: (a) power/thermal throttling — check sustained clocks during a long prompt; (b) PCIe x8 or x4 link when you expected x16 — nvidia-smi --query-gpu=pcie.link.width.current --format=csv; (c) running the wrong binary, e.g. a non-CUDA llama.cpp build on an NVIDIA card. Rebuild with GGML_CUDA=1 and confirm.

Sources

  1. r/LocalLLaMA (community tok/s threads)
  2. llama.cpp GitHub Discussions #4167 — Apple Silicon benchmark thread
  3. Tom's Hardware — RTX 5090 Founders Edition review
  4. Phoronix — RTX 5080/5090 Linux performance review
  5. PassMark — GeForce RTX 5090 video card benchmark

Products mentioned

Small-model (7–9B) reference throughput

These figures are for 7–9B models, not the 70B models this article covers. The table holds model size fixed so the cards can be compared with each other; for the size in the title, use the article's own figures.

Median generation throughput at 7–9B models (Llama 3.1 8B, Qwen 3 8B), Q4 quantization, from community-reported runs SpecPicks tracks. Each row pools runs from different sources, runtimes and models in that class, so the rows are not a matched head-to-head; where the article compares cards on the same rig, its own figures are the like-for-like result. Street price is the second-lowest listing priced within the last 24 hours at 0.5–2× launch MSRP, so no single listing sets it, and a street figure well above MSRP says so in its cell; where too few listings pass that check the row shows launch MSRP instead. Rows marked for comparison are not the cards this article is about — they are included so the throughput column has something to be read against.

GPUVRAM Llama-3-8B class, Q4Street price Sources
NVIDIA GeForce RTX 5090 32 GB 224.8 tok/s4 runs · 3 sources $1,999MSRP SpecPicks median of 4 runs; sources: llama.cpp GitHub, DatabaseMart, Hardware Corner
Radeon RX 7900 XTXfor comparison 24 GB 138.6 tok/s9 runs · 3 sources $999MSRP SpecPicks median of 9 runs; sources: GitHub / 1337hero, llama.cpp GitHub issue #20934, Markaicode / LocalScore
NVIDIA RTX A5000 24GBfor comparison 24 GB 133 tok/s4 runs · 3 sources $1,999MSRP SpecPicks median of 4 runs; sources: llama.cpp GitHub (CUDA scoreboard), knightli.com (llama.cpp CUDA…, MyAIHardware

Products mentioned in this article

Amazon & eBay listings, plus full specs and alternatives on each product page.

As an Amazon Associate, SpecPicks earns from qualifying purchases; we also earn on qualifying eBay purchases via the eBay Partner Network. Prices shown were last tracked at crawl time and may vary — check the listing for the current price.

Watch a review

Gigabyte Aorus Master RTX 5090 Review — Hardware Unboxed on YouTube

Frequently asked questions

What are the key differences between Ollama and llama.cpp for running Llama 3.1 70B?
Ollama simplifies setup with automatic GPU detection and model downloads, making it ideal for users seeking ease of use. In contrast, llama.cpp offers granular control over quantization, context length, and layer offloading, catering to advanced users who need fine-tuned performance adjustments. Ollama is better for quick deployment, while llama.cpp is suited for benchmarking and optimization.
Why is the first token generation slower than subsequent tokens?
The slower first token generation is due to the prefill phase, where the model processes the input prompt and builds the KV cache. This step is computationally intensive, especially for long prompts. Subsequent tokens benefit from the precomputed cache, resulting in faster generation speeds.
How can I optimize VRAM usage for long-context workloads?
For long-context workloads, enable KV-cache quantization in llama.cpp using flags like `-ctk q8_0 -ctv q8_0`. This reduces the KV cache footprint by approximately 50% with minimal quality loss. Additionally, consider lowering the context length or using a smaller quantization level, such as q3_K_M, to fit within VRAM limits.
What should I do if I encounter 'out of memory' errors during inference?
To address 'out of memory' errors, reduce the context length (e.g., `-c 2048`), switch to a lower quantization level (e.g., q4_K_M to q3_K_M), or enable KV-cache quantization. Closing background applications to free up system RAM can also help if VRAM and RAM are both fully utilized.
Is the NVIDIA GeForce RTX 5090 suitable for running Llama 3.1 70B at full precision?
No, the NVIDIA GeForce RTX 5090's 32 GB of VRAM is insufficient for running Llama 3.1 70B at full precision (fp16), which requires approximately 145.6 GB of memory. Quantized formats like q4_K_M or q3_K_M are necessary to fit the model within the available VRAM.

Related reading

Browse all SpecPicks reviews →

— Mike Perry · Updated 2026-10-17