NVIDIA H200 SXM 141GB — Benchmarks & Specs
Short answer: the NVIDIA H200 SXM 141GB generates 139 tokens/sec on qwen3.6:35b-a3b (q8_0), a mixture-of-experts model with ~3B active parameters (dense models of similar size run far slower), under llama.cpp in a single cited run, per r/LocalLLaMA; there are too few comparable runs on file for a median. All 27 models in the Will-It-Run matrix fit in its 141 GB of VRAM at their listed quantization with an ~8K context, the largest in the sizing matrix being DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run); the matrix does not include larger models, so this is not the card's ceiling.
- Single cited run 139 tok/s qwen3.6:35b-a3b q8_0, per r/LocalLLaMA
- Fits in 141 GB all 27 matrix models → at ~8K context; the largest in the sizing matrix is DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run), and the Will-It-Run matrix has no larger model
Who should buy this instead: the NVIDIA H200 SXM 141GB's 141 GB puts it in the 48 GB+ tier of the local-LLM GPU picks by VRAM (one card per tier, each with cited tok/s). If the models you run fit in 24 GB, compare the 24 GB pick. If they need more than 141 GB, start at the 128 GB unified pick.
Bottom line: how fast is the NVIDIA H200 SXM 141GB?
For local LLM inference it generates 139.3 tokens/sec running qwen3.6:35b-a3b (a mixture-of-experts model with ~3B active parameters, so dense models of similar size run far slower) at q8_0 under llama.cpp, per r/LocalLLaMA. In MLPerf Inference v5.0 — Llama 2 70B Server Scenario it scores 33,000 tokens/s, per CoreWeave MLPerf v5.0 Blog. It has 141 GB of VRAM, and the largest model on file at 4-bit quantization on this card is command-r+:104b (104B parameters, q4_K_M), generating 9.8 tokens/sec under ollama, per Ollama Benchmarks. Larger models have no sourced run on file. That run used a 2,048-token context; the Will-It-Run matrix sizes each model at its listed quantization with an ~8K context, and by that measure DeepSeek-R1-Distill-Llama 70B at Q4_K_M is the largest that fits in 141 GB.
Every figure above is a row in the tables below, and each row links out to the review or public benchmark database the number was taken from. SpecPicks aggregates published measurements; it does not report first-party benchmark runs.
Check what fits in 141 GB: by file size at each listed quantization, 27 of the 27 open-weight model files in the SpecPicks Will-It-Run matrix load entirely into the NVIDIA H200 SXM 141GB's 141 GB while keeping at least 1.5 GB reserved for an ~8K context; the largest, DeepSeek-R1-Distill-Llama 70B (a 42.5 GB Q4_K_M file per its Hugging Face listing), leaves roughly 108.9 GB of the card's 151.4 GB (141 GiB) free. Check a model in the Will-It-Run matrix →
What fits in the NVIDIA H200 SXM 141GB's 141 GB, and what doesn't
7 of these 7 model files load whole into 141 GB with room for an ~8K context. A file fits when the card keeps at least 1.5 GB free after the weights; file sizes are the GGUF downloads on Hugging Face in decimal GB, and the card's 141 GB (binary GiB) is 151.4 GB in the same units. Speeds are this card's single-user generation speed from cited community runs at the file listed; a row with no cited run says so rather than estimating one.
| Model | Quant | Fits in 141 GB? | Cited speed on this card |
|---|---|---|---|
| Llama 3.1 8B 4.9 GB file | Q4_K_M | Fits~146.5 GB left for context | No public measurement on file |
| Qwen3 14B 9 GB file | Q4_K_M | Fits~142.4 GB left for context | No public measurement on file |
| gpt-oss-20b 21B-A3.6B · 12.1 GB file | MXFP4 | Fits~139.3 GB left for context | No public measurement on file |
| Qwen3 30B-A3B 18.6 GB file | Q4_K_M | Fits~132.8 GB left for context | No public measurement on file |
| Qwen3 32B 19.8 GB file | Q4_K_M | Fits~131.6 GB left for context | No public measurement on file |
| Gemma 3 27B 28.7 GB file | Q8_0 | Fits~122.7 GB left for context | No public measurement on file |
| Llama 3.3 70B 42.5 GB file | Q4_K_M | Fits~108.9 GB left for context | No public measurement on file |
The NVIDIA H200 SXM 141GB is a datacenter accelerator from the Hopper family released in 2024 from NVIDIA. Key on-paper specs include 141 GB of HBM3e memory, 4800 GB/s of memory bandwidth, 700W TDP. Data on this page draws on 4 synthetic benchmark results, 15 community AI inference reports (top 12 shown), 2 measured game frame-rate results, compiled from 3DMark, Koyeb GPU LLM Benchmarks, NVIDIA Technical Blog, NVIDIA TensorRT-LLM Blog, Ollama Benchmarks, r/LocalLLaMA, CoreWeave MLPerf v5.0 Blog, Hardware Unboxed, Millstone AI, Tom's Hardware; each table row links to its source. Read this page when shopping the NVIDIA H200 SXM 141GB, comparing it against other datacenter accelerators for your server or cluster, or sizing it for a specific workload (LLM inference and serving, fine-tuning, or HPC compute).
Gaming Performance (measured FPS)
Average and 1% low frame rates by game, resolution, and quality preset. Bars are scaled against the fastest result on this page.
| Game | Resolution | Settings | Relative | Avg FPS | 1% low | Source |
|---|---|---|---|---|---|---|
| Call of Duty: Modern Warfare 2 | 1440p | Ultra FSR Quality | 120 fps | 95 fps | Hardware Unboxed 2023-10-22 | |
| Cyberpunk 2077 | 4K | Ultra RT on DLSS Ultra | 60 fps | 45 fps | Tom's Hardware 2023-11-15 |
AI Inference Performance
Tokens per second under each model + quantization. Higher = faster generation. Bars compare runs across the same model.
| Model | Quantization | Relative | Tokens/sec | VRAM used | Source |
|---|---|---|---|---|---|
| qwen2.5:7b-instruct | — vllm | 182.2 tok/s | — | Koyeb GPU LLM Benchmarks 2026-05-27 | |
| llama3.3:70b (speculative decoding, Llama 3.2 1B draft) | FP8 TensorRT-LLM 0.15.0 | 181.7 tok/s | — | NVIDIA Technical Blog 2024-12-17 | |
| llama3.1:8b-instruct | — vllm | 168.0 tok/s | — | Koyeb GPU LLM Benchmarks 2026-05-27 | |
| qwen3-coder:30b-a3b | FP8 vllm | 164.9 tok/s | — | Millstone AI 2026-02-03 | |
| qwen3.6:35b-a3b | q8_0 llama.cpp | 139.3 tok/s | — | r/LocalLLaMA (u/Bird476Shed comment, llama-bench text table) 2026-04-30 | |
| command-r+:104b | q4_K_M ollama | 9.8 tok/s | 135.2 GB | Ollama Benchmarks 2023-10-12 |
Batched serving throughput
| Model | Quantization | Relative | Tokens/sec (aggregate) | VRAM used | Source |
|---|---|---|---|---|---|
| gpt-oss:20b | FP8 TensorRT-LLM batched | 13890.0 tok/s aggregate | — | NVIDIA TensorRT-LLM Performance Overview 2026-11-03 | |
| llama2:13b | FP8 tensorrt-llm batched | 11819.0 tok/s aggregate | — | NVIDIA TensorRT-LLM Blog 2023-11-13 | |
| gpt-oss:120b | FP8 TensorRT-LLM batched | 6798.0 tok/s aggregate | — | NVIDIA TensorRT-LLM Performance Overview 2026-11-03 | |
| llama2:70b | FP8 tensorrt-llm batched | 3803.0 tok/s aggregate | — | NVIDIA Technical Blog 2023-12-04 | |
| llama2:70b | FP8 tensorrt-llm batched | 3014.0 tok/s aggregate | — | NVIDIA TensorRT-LLM Blog 2023-11-13 | |
| llama2:70b | FP8 TensorRT-LLM 0.5 batched | 341.0 tok/s aggregate | — | NVIDIA TensorRT-LLM GitHub (H200 launch blog) 2023-11-13 |
Synthetic Benchmarks
Higher is better. Bars are scaled within each benchmark family (multi-thread, single-thread, etc.) so you can compare like-with-like at a glance.
| Benchmark | Relative | Score | Source |
|---|---|---|---|
| MLPerf Inference v5.0 — Llama 2 70B Server Scenario | 33,000 tokens/s | CoreWeave MLPerf v5.0 Blog 2025-04-02 | |
| MLPerf Inference v4.0 — Llama 2 70B Offline Scenario | 31,712 tokens/s | NVIDIA Developer — AI Inference Performance 2024-03 | |
| 3DMark Time Spy | 15,200 points | 3DMark 2023-07-15 | |
| Port Royal (RT) | 11,800 points | 3DMark 2023-07-15 |
Full Specifications
| TDP | 700 W |
|---|---|
| VRAM | 141 GB |
| Memory type | HBM3e |
| CUDA cores | 16896 |
| Fp8 tflops | 3958 |
| Process | 4 nm |
| Form factor | SXM5 |
| Nvlink | 900 GB/s |
| Architecture | Hopper GH100 |
| Tensor cores | 528 |
| Memory bandwidth | 4800 GB/s |
NVIDIA H200 SXM 141GB — Frequently Asked Questions
What is the NVIDIA H200 SXM 141GB best used for?
When was the NVIDIA H200 SXM 141GB released?
Where do the NVIDIA H200 SXM 141GB benchmark numbers come from?
Can the NVIDIA H200 SXM 141GB run local LLMs?
Where can I buy the NVIDIA H200 SXM 141GB?
Buying guides that rank the NVIDIA H200 SXM 141GB's class
This page is the raw performance data. The guides below turn it into a ranked pick for a specific build.
Editorial guides covering the NVIDIA H200 SXM 141GB
In-depth SpecPicks reviews, build guides, and head-to-heads referencing this datacenter accelerator.
- NVIDIA H200 vs B200 for LLM Inference: What Changes in 2026
- Used RTX 3090 vs New GPU for Local LLMs: 2026 Reddit Consensus
- Tesla P40 vs RTX 3090: Used 24GB GPUs for Local LLMs in 2026
- RTX 3060 12GB Local LLM: Reddit Consensus on What Runs (2026)
- gpt-oss on a 12GB GPU: 2026 Community Consensus (RTX 3060)
- Best GPU for Local LLMs by VRAM: 2026 Reddit Consensus
- RTX PRO 6000 vs RTX 5090 for Local 70B LLMs: 96GB vs 32GB
- RTX 5090 vs RTX Spark for Local LLMs: 32 GB GDDR7 vs 128 GB Unified (2026)
- Browse all SpecPicks reviews →
More guides & deep dives from the SpecPicks archive
Browse all articles & guides →- Best 1440p Gaming GPUs in 2026
- The Complete Voodoo5 5500 AGP Driver Guide (2026 Edition)
- Best Retro Handhelds in 2026: Pocket, Mid-Tier and FPGA Picks
- Best Budget Gaming PC Build 2026 — ~$1,000 Parts List
- Emulation Hardware in 2026: FPGA, Software, and Cart-Reader Ecosystems
- How to Build a Windows 98 Retro PC in 2026
- RTX 4070 Super vs RX 7800 XT — Which to Buy in 2026
More reviews from the SpecPicks archive
Browse all reviews →- IBM Granite 4.1 (3B / 8B / 30B): Local Inference Benchmarks and Hardware Picks
- Ryzen 5 5600G vs Core i7-9700K for 1080p Gaming
- Can a Raspberry Pi 4 Run a Local LLM? Ollama tok/s Tested
- Intel Arc LLM Support: What Works in 2026
- Best Budget SSD for Gaming and PC Upgrades in 2026
- Hackaday Europe 2026: Building a Retro PC From Scratch
- Building a Local AI-Agent Eval Rig After AISI's Benchmark Warning
- GPT-5.6 Sol at One-Third the Cost: When Local Inference Still Wins
- Logitech G29 vs HORI Force Feedback: Best Beginner Sim-Racing Wheel in 2026
- Best Storage for a Raspberry Pi 4 Home Server: microSD vs SATA SSD vs NVMe
- Dump and Archive Your Retro PC's IDE Drives in 2026: A USB Adapter Workflow
- KOORUI 27" 4K vs Samsung Odyssey 4K for Console + PC Gaming
- Ryzen AI Halo vs a DIY RTX 3060 Box for Local LLMs in 2026
- $100 CPU Shootout: Ryzen 5 5500 vs Core i3-14100F vs i3-12100F
- Black Friday Monitor Deals 2026: 5 Picks for Gaming and AI Desks
- Blue Yeti vs HyperX QuadCast 2 S: Best USB Mic for New Streamers
- Kimi K3 vs Qwen 3.8 for Local Coding Agents in 2026
- Noctua NH-U12S vs Cooler Master ML240L for Ryzen 5800X
- ASUS TUF VG27AQ vs Dell G3223Q: 1440p HDR or 4K 32-Inch in 2026
- Best Mechanical Keyboard for Office and Hybrid Coding in 2026
- Best Internal SATA SSDs for PC Upgrades (2026)
- Qwen3 8B: Raspberry Pi 4 8GB vs RTX 3060 12GB (2026)
- Irtysh 32-Core CPU: Is the 30+ FPS Witcher 3 Claim Real?
- Best Gaming Setup for Kids and Teens in 2026: 5 Picks
More buying guides from SpecPicks
Browse all buying guides →- Best NVMe External Enclosures for 2026
- Best Graphics Cards for Gaming in 2026
- Best Tools for Building and Repairing Retro PCs in 2026
- Best Retro Gaming Consoles & Handhelds for 2026
- Best DDR5 RAM for Gaming PCs in 2026
- Best AM5 Motherboards for 2026
- Best 4K Monitors for Content Creators in 2026
- Best NVMe SSDs for Gaming in 2026
- Best Mechanical Keyboards for Gaming in 2026
- Best CPUs for Content Creators in 2026
- Best GPUs for Running Local LLMs in 2026
- Best Gaming Mice for 2026
- Best GPUs for 4K Gaming in 2026
- Best Controllers for PC Gaming in 2026
- Best External SSDs for Content Creators in 2026
- Best Gaming Monitors for 2026
- Best PC Cases for Building in 2026
- Best CPU Coolers for 2026
- Best 1440p 240Hz Gaming Monitors in 2026
- Best CPUs for Gaming in 2026
- Best GPU for Running 27B-32B Local LLMs in 2026
Hardware benchmark data on SpecPicks
All benchmarks →- NVIDIA GB300 Grace Blackwell Ultra Desktop (DGX Station) — benchmarks & specs
- Apple M3 Ultra 32 Core — benchmarks & specs
- AMD Radeon Pro W7800 32GB — benchmarks & specs
- NVIDIA Jetson AGX Orin 64GB — benchmarks & specs
- Quadro RTX 6000 — benchmarks & specs
- Ryzen 5 2400G — benchmarks & specs
- NVIDIA GeForce RTX 5070 Ti — benchmarks & specs
- NVIDIA GeForce GTX 1650 SUPER — benchmarks & specs
- GRID RTX6000-8Q — benchmarks & specs
- Ryzen 7 3700X — benchmarks & specs
- Ryzen 5 5600X — benchmarks & specs
- Ryzen 5 3550H — benchmarks & specs
- NVIDIA GeForce RTX 3090 — benchmarks & specs
- Ryzen 7 5700G — benchmarks & specs
- NVIDIA GeForce RTX 3070 Ti — benchmarks & specs
- Radeon RX 6800 XT — benchmarks & specs
- Radeon 9500 Pro — benchmarks & specs
- Radeon RX 5500 XT — benchmarks & specs
- NVIDIA RTX Spark (N1X) 18-Core — benchmarks & specs
- AMD Ryzen 5 7400 — benchmarks & specs
- AMD Ryzen Threadripper PRO 9985WX — benchmarks & specs
- Pentium 4 2.4B (Northwood) — benchmarks & specs
- Radeon RX 9070 XT — benchmarks & specs
- Apple M1 Max 10 Core 3200 MHz — benchmarks & specs