How to run Llama 3.1 70B on NVIDIA GeForce RTX 5070

Quick answer. Running Llama 3.1 70B on a 12 GB NVIDIA GeForce RTX 5070 is a heavy-CPU-offload play: only about 25 % of the model's 80 layers fit on the GPU even at q4_K_M, so most of the work runs through your DDR5 memory bus. Realistic throughput is 6–10 tokens/sec at 4K context, 4–6 tok/sec at 16K. It's usable — far better than the 0.6 tok/sec of a pure-CPU build — but if you need 70B-class quality at conversational speeds, the 3090 24 GB or 5090 32 GB are the right tools.

The constraint up front

Llama 3.1 70B is Meta's 70-billion-parameter dense transformer with 80 hidden layers, a 128K-token context window, and Grouped-Query Attention (GQA) that keeps the KV cache compact. As GGUF quants:

A 12 GB RTX 5070 can't hold any of those in their entirety. The realistic option is q4_K_M with --n-gpu-layers 18–22 and the remaining 58–62 layers running on the CPU side via llama.cpp's GGML backend. q3_K_M loses about 4–7 % on the standard reasoning eval suite but lets you push 24–28 layers onto the GPU; that's the speed/quality knob.

VRAM math for an RTX 5070

Each of Llama 3.1 70B's 80 layers at q4_K_M is approximately 485 MB. Add a 1.5 GB embedding + output head, CUDA/activation overhead, and the KV cache. Here's the realistic 4K-context budget:

That's already over budget. In practice with a 250 W card sharing its slot with a display, you'll find -ngl 18 is the sweet spot: ~8.7 GB of weights + 0.6 GB of cache and overhead, comfortably below the 12 GB ceiling. Push to 20 layers and a long prompt will OOM mid-prefill.

The remaining 60+ layers run on CPU. A Ryzen 7 7700X with DDR5-6000 sustains ~83 GB/s memory bandwidth. The RTX 5070 sustains ~672 GB/s. That's an 8× ratio, and it's why CPU offload caps your speed: the layers on the GPU might generate at 60+ tok/sec, but the CPU portion limits you to 7–9 tok/sec overall.

Install — Ollama or llama.cpp

Ollama path (easy)

curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
ollama pull llama3.1:70b-instruct-q4_K_M # ~42 GB download
ollama run llama3.1:70b-instruct-q4_K_M

Ollama defaults the 5070 to n_gpu_layers ≈ 18 and num_ctx 2048. For 70B you almost certainly want a longer context; create a Modelfile:

FROM llama3.1:70b-instruct-q4_K_M
PARAMETER num_ctx 4096
PARAMETER num_predict 1024
PARAMETER temperature 0.6

Then ollama create my-llama70 -f Modelfile.

llama.cpp path (control)

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_F16=ON
cmake --build build --config Release -j
# Download a 70B q4_K_M GGUF from Hugging Face
./build/bin/llama-server \
 -m models/llama-3.1-70b-instruct-q4_K_M.gguf \
 -ngl 18 -c 4096 -t 8 -fa \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 --port 8080

Flags worth a deeper look:

• -ngl 18: Llama 3.1 70B has 80 layers; this offloads layers 0–17 to GPU. Tune ±2 with nvidia-smi watching VRAM.
• -c 4096: Llama 3.1's KV cache is much smaller than Llama 2 thanks to GQA (8 KV heads vs 64 attention heads), so 4K-8K context is cheap. 16K is also fine; 128K (the model's max) is not.
• -fa: Flash-Attention 2. Required for stable 8K+ context with offload — without it, attention activations spike.
• --cache-type-k q8_0 --cache-type-v q8_0: KV cache at 8-bit. Negligible quality cost, ~50 % memory savings.
• -t 8: CPU threads. Match this to physical core count, not SMT thread count. Hyperthreaded threads thrash the memory bus.

Real-world numbers

Benchmark rig: Ryzen 9 7900X (12 cores), 64 GB DDR5-6000, RTX 5070 12 GB, Ubuntu 24.04, CUDA 12.6, llama.cpp build of 2026-04-29:

Compare to the same model on bigger cards:

For a deeper card-vs-card walk-through see our running Llama 3.1 70B locally hardware requirements guide.

Common pitfalls — five we see repeatedly

1. Wrong CPU memory-channel layout. Llama 3.1 70B at 42 GB lives almost entirely in DDR5. If you bought a 32 GB single-stick kit, the memory controller runs in single-channel mode at ~40 GB/s instead of 80 GB/s and your tok/sec collapses by half. Always 2×16 or 2×32, never 1×32.

2. Half the model gets paged out to swap. Default Linux on a 32 GB system will start swapping during prefill. Either jump to 64 GB+ of RAM or set vm.overcommit_memory=1 and put the GGUF on a fast NVMe.

3. Speculative decoding doesn't help here. Llama 3.1 70B has a draft model (Llama 3.2 1B) that would speed up generation on a GPU-resident setup. On CPU-offload it tanks throughput because the draft model has to be loaded on the GPU, eating into the layers budget. Disable with --draft-max 0 if your launcher tries to enable it.

4. The Q4_0 format from older GGUF builds is 8 % slower than q4_K_M. Always grab the K-quants (with the _K_M or _K_S suffix) — they're both smaller and faster on modern llama.cpp.

5. Output is correct but "robotic." Some downloadable q4_K_M GGUFs were quantized from instruction-tuned variants where the rope_scaling metadata wasn't preserved. The model still produces text but ignores the system prompt. Re-download from bartowski or unsloth accounts on Hugging Face which preserve metadata.

When NOT to use this combo

• Production chat. Even at 9 tok/sec, a 600-token answer takes ~67 seconds. Acceptable for personal use, brutal for end users. Hosted inference at $0.40/M tokens is cheaper for low volume.
• Code completion. Tab-complete needs <500 ms first-token; 70B on offload averages 4–5 seconds to first token. Use Qwen 3 14B or Llama 3.1 8B on the same card instead.
• Agent workflows. Multi-step agents fan out 5–20 LLM calls per task. Throughput-multiplied that's 5+ minutes per task at 8 tok/sec, which kills the iteration loop.
• Heavy fine-tuning. Even LoRA at 70B needs 60+ GB across optimizer state and gradients. Rent an H100 hour.

If you need 70B-class quality and your hardware budget is the 5070, consider running a smaller model with similar evals: Qwen 3 32B (covered separately at how to run Qwen 3 32B on RTX 5070) or DeepSeek-R1 32B-distill gets ~85 % of Llama 70B's quality at twice the speed on the same hardware.

Worked example — summarise a long PDF

The realistic 70B use-case on a 5070 is offline summarisation. Take a 25-page PDF (~10K tokens), ask Llama 3.1 70B-Instruct for a 500-token executive summary:

• Prefill 10K input tokens at 88 PP-tok/s → 114 seconds to first token
• Generate 500 output tokens at 8 TG-tok/s → 62 seconds
• Total: ~3 minutes per document

For a batch overnight job that's fine — you can run 240 documents in 12 hours. For an interactive "summarize this for me right now" feature, it's too slow. The 70B model on a 5090 finishes the same job in <60 seconds.

Hardware shortcut — pair the 5070 with more RAM

If you're committed to the 5070 chassis, the cheapest speedup is CPU memory, not GPU. Going from 32 GB DDR5-5200 to 64 GB DDR5-6400 (matched 2×32 kit) gets you 25–30 % more tok/sec on offloaded layers — and 70B at q4_K_M won't even load into 32 GB of system RAM once the model is mapped, the Linux kernel, and your browser tabs are accounted for.

If you can also move the display to integrated graphics (free up the 5070 entirely), you'll claw back another ~400 MB of VRAM and can run -ngl 20 instead of -ngl 18 — about a 10 % throughput improvement.

Tuning recipe by use case

Overnight batch summarisation (highest throughput, fixed context):

-ngl 18 -c 4096 -fa \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 --temp 0.3 --top-p 0.9 \
 -t 8 --no-mmap

--no-mmap forces the GGUF into RAM rather than memory-mapping it from disk; on a 64 GB system that gets you ~10 % more steady-state tok/s by avoiding page-fault stalls.

Interactive chat (lowest first-token latency, medium output):

-ngl 18 -c 2048 -fa \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 --temp 0.7 --top-p 0.9 \
 -t 8 --predict 512

~9 tok/s but with first-token-latency around 4 seconds instead of 8 — the shorter context means less prefill work and a smaller KV cache.

Research analysis (long context, accuracy over speed):

-ngl 14 -c 32768 -fa \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 --rope-scaling linear --rope-freq-scale 0.5 \
 --temp 0.4 --top-p 0.95 -t 8

~5 tok/s but you can stuff a small book into the prompt. The rope-scaling settings extend Llama 3.1's 8K native to 32K with degraded but still useful accuracy.

Benchmark methodology

All measurements above used the same protocol:

./build/bin/llama-bench -m models/llama-3.1-70b-q4_K_M.gguf \
 -p 512 -n 128 -ngl 18 -c 4096 \
 --cache-type-k q8_0 --cache-type-v q8_0 \
 -t 8 -r 30

Reviewers ran 5 warmup iterations and 30 measured iterations. Median values are reported. The host system used a fixed CPU governor (performance), no other foreground processes, and a fresh model load on each invocation to remove disk-cache hot-start effects.

For the prefill numbers (PP), we vary -p from 256 to 4096 and pick the value closest to the use case. Generation (TG) is measured at 128 output tokens, which is short enough to avoid context-shift effects but long enough for the runtime to settle into steady-state.

Second worked example — a research workflow

If you're using the 5070 to research a topic with 70B, the realistic flow is "load a context, ask one question, write notes, repeat":

• Load 8K tokens of source material into context — prefill 90 s
• First question (200 tokens, get a 400-token answer): 50 s
• Write notes for 2–3 minutes (model is idle, but the KV cache is preserved)
• Second question on same context (200 tokens, 400-token answer): 50 s
• Third question: 50 s

A 30-minute research session of one focused topic might involve 5–8 model interactions and 4–8 minutes of waiting. That's fast enough to keep flow if you have something else to do (note-taking, web reading) while it generates. For pure "type, wait, read" workflows, 70B on a 5070 is too slow.

See also

• Best GPUs for running local LLMs in 2026
• RTX 5090 vs RTX A6000 for Local LLMs
• How to run Llama 3.1 70B on RTX 5080 — same model, more VRAM
• How to run Llama 3.1 70B on RTX 5090 — the comfortable-fit version
• Mac Studio M3 Ultra vs RTX 5090 for AI Inference in 2026

Cited sources

• llama.cpp KV-cache and offload behaviour docs (GitHub discussion 4167)
• Ollama install script + Modelfile reference (ollama.com/install.sh)
• vLLM paged-attention reference (GitHub — vllm-project/vllm)
• Community tok/s reports on the 5070 + 70B combo (reddit.com/r/LocalLLaMA)
• 2026 GPU price/perf table (Tom's Hardware GPU hierarchy)

As of May 2026 — Meta's 70B-class release cadence is annual; if Llama 4 70B ships with MoE routing, these tok/sec numbers will roughly double on the same card.