A second GPU does not simply double llama.cpp's tokens-per-second. Per the project's own multi-GPU documentation and community measurements shared on r/LocalLLaMA, the realistic benefit of a dual-GPU llama.cpp setup is usually VRAM pooling first, throughput scaling second — a second card lets a larger quantization or longer context fit in memory, while generation speed gains depend heavily on split mode, interconnect bandwidth, and whether the workload is compute- or memory-bandwidth-bound. This guide synthesizes how llama.cpp's multi-GPU code actually behaves, how to configure it on both NVIDIA/CUDA and AMD/ROCm hardware, and what public benchmark trackers are worth checking before spending on a second card.
Understanding Dual GPU Acceleration for Llama.cpp
llama.cpp exposes multi-GPU behavior through two independent controls documented in the project's multi-GPU support notes: a split mode (-sm / --split-mode) and a tensor split ratio (-ts / --tensor-split).
- Layer split (
-sm layer, the default) assigns contiguous blocks of transformer layers to each GPU. Inference still runs mostly sequentially — GPU 0 finishes its layers, hands the activations to GPU 1 over PCIe, and so on. This tolerates slower interconnects but caps per-token latency at roughly the sum of each GPU's per-layer time plus the hand-off cost. - Row split (
-sm row) splits individual weight matrices across GPUs so both cards work on the same layer simultaneously. This can raise throughput on well-matched cards but is far more sensitive to PCIe bandwidth and driver-level peer-to-peer support, which is why community reports in r/LocalLLaMA multi-GPU threads frequently note row split helping more on NVLink-adjacent NVIDIA pairs than on consumer PCIe-only rigs.
Quantized KV cache flags (-ctk, -ctv, e.g. q8_0 or q4_0 instead of the default f16) matter more in multi-GPU setups than single-GPU ones: the cache is replicated or split alongside the model, so shrinking it frees headroom for either a bigger model or a longer context window on the same pair of cards.
AMD's MI300X uses a multi-chip-module (MCM) design that stacks compute and memory dies on one package — a single-card trait, not a multi-GPU one — but it's relevant here because it changes the math on whether you need two cards at all. A single MI300X's large HBM3 pool can hold models that would otherwise require splitting across two smaller cards, which is a different value proposition than "dual GPU for speed."
Net takeaway from the current state of the code: expect sub-linear throughput scaling from adding a second GPU under the default layer-split mode, and treat 'bigger model now fits' as the primary, more reliable win. Anyone quoting a flat multiplier (e.g. '4x faster with two GPUs') without naming the model size, quantization, split mode, and interconnect isn't describing a reproducible result — treat those numbers skeptically and check current figures on public trackers before buying hardware.
How to Set Up Dual GPU for Llama.cpp (Step-by-Step)
1. Confirm PCIe lane allocation before buying a second card
Consumer motherboards frequently drop the second x16 physical slot to x4 or x8 electrical lanes once populated. For layer-split workloads this is usually tolerable; for row-split it becomes the bottleneck. Check your motherboard's block diagram, not just the slot's physical size — an x16-length slot wired for x4 electrical is common on mid-range boards.
2. Pick the matching backend
- NVIDIA/CUDA: build llama.cpp with
-DGGML_CUDA=ON; both cards are detected automatically vianvidia-smi-visible devices. - AMD/ROCm: build with the HIP backend (
-DGGML_HIP=ON) against a ROCm version that lists both GPUs'gfxtargets as supported in AMD's ROCm compatibility docs. Multi-GPU ROCm support has historically trailed CUDA in maturity for llama.cpp-style split-mode workloads — verify both cards show up inrocm-smibefore assuming the build will use them.
3. Set the tensor split ratio to match VRAM, not card count
--tensor-split takes a ratio, not a percentage — --tensor-split 1,1 is only correct for two identically-sized cards. Pairing a 48GB Radeon Pro W7900 with a 16GB RX 6800 XT (an unusual but sometimes-seen budget pairing) needs something closer to --tensor-split 3,1 so the larger card carries proportionally more layers.
4. Launch with layer split first, then experiment
Start with -sm layer -ngl 999 (offload all layers) and confirm both GPUs show utilization in nvtop (NVIDIA) or radeontop (AMD). Only try -sm row after layer split is stable — it's the mode most likely to fail silently or crash on unsupported peer-to-peer configurations.
5. Quantize the KV cache if context length matters more than raw speed
Add -ctk q8_0 -ctv q8_0 to shrink the cache footprint on both cards, freeing memory for longer context windows — useful when the pooled VRAM from two cards is what's actually enabling a bigger model in the first place.
AMD GPU Benchmark Comparison: Dual vs Single GPU
Public spec sheets make the VRAM-pooling case clear even before running a single benchmark:
| Card | VRAM | Class | Typical multi-GPU role |
|---|---|---|---|
| AMD Instinct MI300X | 192GB HBM3 | Datacenter accelerator | Often runs large models on a single card; pairing two is about fitting frontier-scale models, not consumer-style speedups |
| Radeon Pro W7900 | 48GB GDDR6 | Workstation | Common dual-GPU pick for 70B-class models at moderate quantization |
| Radeon RX 6800 XT | 16GB GDDR6 | Consumer/gaming | Pairing two extends usable context/model size for 13B–34B-class models |
| Radeon RX 6600 XT | 8GB GDDR6 | Consumer/entry | VRAM-constrained even paired; best suited to small models or as a secondary card offloading a handful of layers |
(Specs per AMD's product pages for the MI300X, the W7900, and the RX 6000 series.)
Rather than repeat a specific speedup multiplier here — reported numbers vary widely by model, quantization, split mode, and llama.cpp build version, and shift with nearly every release — check a current run before deciding. The most reliable places to see reproducible, dated numbers are the llama.cpp GitHub discussions (search 'multi-gpu' or 'tensor-split') and active r/LocalLLaMA benchmark megathreads, both of which typically include the exact command line and hardware used — the minimum needed to trust a number at all.
The RX 6600 XT is worth flagging separately: at 8GB, even a pair (16GB pooled) is tight for anything past small/quantized models once KV cache and system overhead are accounted for, making it a poor second-card candidate for anyone chasing larger context windows rather than a specific speed target.
Optimization Techniques for Dual GPU Systems
Quantized vs full-precision KV cache. Default f16 cache is safest for output quality but doubles memory use versus q8_0 and roughly quadruples it versus q4_0. In a dual-GPU setup where VRAM is the actual scarce resource, dropping to q8_0 is usually the better trade than buying a third card, at a small, generally acceptable quality cost.
PCIe bandwidth strategy. If your board only offers x8/x8 or x8/x4 once both slots are populated, default to layer split — it crosses the PCIe bus once per layer boundary rather than continuously. Reserve row split for boards that keep both GPUs on genuine x16 lanes or PCIe 5.0, where the extra bandwidth headroom absorbs row split's chattier communication pattern.
Thermal and power headroom. Dual high-TDP cards (workstation or datacenter-class) in one case need airflow and PSU headroom sized for sustained load, not just peak spec — llama.cpp's inference workload keeps both cards near their power limit for the duration of generation, unlike bursty gaming loads. Power-limiting via nvidia-smi -pl or the ROCm equivalent is a low-cost way to trade a small amount of throughput for meaningfully lower sustained heat in a cramped dual-GPU case.
Storage for model files. Multi-GPU rigs tend to accumulate several large GGUF quantizations for A/B testing split modes — a dedicated external drive for model storage, such as a SanDisk 256GB Ultra Dual Drive with USB-C, keeps the OS drive from filling up, and a 4-port powered USB hub helps once a dual-GPU build has already claimed most of the case's front-panel USB headers.
Related reading
For builders comparing dual-GPU paths more broadly, these SpecPicks pieces cover adjacent ground:
- Dual RTX 3090 Setup Guide 2026: PSU, NVLink, Cooling
- Dual RTX 3090 LLM Training: 2026 Benchmarks & Build Guide
- Dual RTX 3090 vs RTX 5090: Gaming vs AI Training
- Dual RTX 3090 PC Builds: PSU, Cooling, and Cost in 2025
- llama.cpp vs Ollama for Single-User Chat on an RTX 3060 (2026)
- vLLM vs llama.cpp on an RTX 3060 12GB for Local Chat
- Qwen MTP on LLaMA.cpp + TurboQuant: What's Actually Verified
FAQs
Does a second GPU double llama.cpp's tokens per second?
No. Per llama.cpp's own multi-GPU documentation, the default layer-split mode runs largely sequentially across cards, so throughput gains are typically well below a 2x multiplier; the more consistent benefit is pooled VRAM for a larger model or longer context.
Should I use tensor-split by layer or by row for multi-GPU?
Start with layer split (-sm layer), the default and the more broadly compatible mode. Only try row split if both cards share genuine x16 or PCIe 5.0 bandwidth, since it's far more sensitive to interconnect speed.
Is ROCm as fast as CUDA for multi-GPU llama.cpp?
ROCm multi-GPU support has historically matured more slowly than CUDA's for llama.cpp-style workloads. Verify both cards' gfx targets are listed as supported in AMD's current ROCm compatibility docs, and confirm rocm-smi sees both devices before assuming a build will use them.
Do I need NVLink for dual-GPU llama.cpp?
No — most consumer and workstation dual-GPU llama.cpp setups run over plain PCIe without NVLink. Fast interconnects mainly matter if you're experimenting with row-split mode.
What's the benefit of a quantized KV cache in multi-GPU setups?
Flags like -ctk q8_0 -ctv q8_0 shrink the KV cache's memory footprint on both cards, freeing VRAM that a dual-GPU setup can then spend on a larger model or a longer context window instead.
Can I mix GPU brands or models in one llama.cpp setup?
llama.cpp's CUDA and HIP backends are generally card-agnostic within a vendor family, but mixing an AMD and NVIDIA card in the same process isn't a supported configuration; mismatched VRAM sizes within the same vendor work fine as long as --tensor-split is set proportionally.
Citations and sources
- https://github.com/ggml-org/llama.cpp
- https://rocm.docs.amd.com/
- https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html
- https://www.amd.com/en/products/graphics/workstations/radeon-pro/w7900.html
- https://www.amd.com/en/products/graphics/desktops/radeon/6000-series/amd-radeon-rx-6800-xt.html
- https://www.reddit.com/r/LocalLLaMA/
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
