NVIDIA L40 48GB — Benchmarks & Specs
Short answer: the NVIDIA L40 48GB generates a median 152 tokens/sec on 7-9B dense models at 4-bit (Q4) under llama.cpp, across 3 cited runs from llama.cpp GitHub perf thread and GPU-Benchmarks-on-LLM-Inference. All 27 models in the Will-It-Run matrix fit in its 48 GB of VRAM at their listed quantization with an ~8K context, the largest in the sizing matrix being DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run); the matrix does not include larger models, so this is not the card's ceiling.
- Median speed 152 tok/s 3 cited runs, 7-9B at Q4, llama.cpp
- Fits in 48 GB all 27 matrix models → at ~8K context; the largest in the sizing matrix is DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run), and the Will-It-Run matrix has no larger model
Who should buy this instead: the NVIDIA L40 48GB's 48 GB puts it in the 48 GB+ tier of the local-LLM GPU picks by VRAM (one card per tier, each with cited tok/s). If the models you run fit in 24 GB, compare the 24 GB pick. If they need more than 48 GB, start at the 128 GB unified pick.
NVIDIA L40 48GB: verdict by use case
- For everyday local models (7-9B dense): a median 152 tok/s at 4-bit (Q4) under llama.cpp, across 3 cited runs (llama.cpp GitHub perf thread, GPU-Benchmarks-on-LLM-Inference).
- For large models in 48 GB: all 27 models in the Will-It-Run matrix fit with an ~8K context, the largest in the sizing matrix being DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run); the matrix does not include larger models, so 48 GB has headroom beyond it that is not sized here.
Each verdict quotes a figure from a linked source or a row in the tables below. Where no cited figure exists for a use case, there is no verdict line for it.
Bottom line: how fast is the NVIDIA L40 48GB?
SpecPicks median: 152 tokens/sec across 3 cited single-user runs of 7-9B dense models at 4-bit (Q4) under llama.cpp (range 114–154 tok/s), from llama.cpp GitHub perf thread (2 runs), GPU-Benchmarks-on-LLM-Inference. For local LLM inference it generates 15.3 tokens/sec running llama3.1:70b at Q4_K_M under llama.cpp, per GPU-Benchmarks-on-LLM-Inference. In Geekbench 5 OpenCL it scores 292,357–333,904 points, across two published results (technical.city GPU benchmark listing, CpuTronic GPU benchmark listing). Its 48 GB of VRAM is the binding constraint for local inference: the llama3.1:70b run above (70B parameters) is the largest model on file at 4-bit quantization on this card.
Every figure above is a row in the tables below, and each row links out to the review or public benchmark database the number was taken from; the SpecPicks median is computed here from the runs it names, each a row in the AI table. SpecPicks aggregates published measurements; it does not report first-party benchmark runs.
Check what fits in 48 GB: by file size at each listed quantization, 27 of the 27 open-weight model files in the SpecPicks Will-It-Run matrix load entirely into the NVIDIA L40 48GB's 48 GB while keeping at least 1.5 GB reserved for an ~8K context; the largest, DeepSeek-R1-Distill-Llama 70B (a 42.5 GB Q4_K_M file per its Hugging Face listing), leaves roughly 9 GB of the card's 51.5 GB (48 GiB) free. Check a model in the Will-It-Run matrix →
What fits in the NVIDIA L40 48GB's 48 GB, and what doesn't
7 of these 7 model files load whole into 48 GB with room for an ~8K context. A file fits when the card keeps at least 1.5 GB free after the weights; file sizes are the GGUF downloads on Hugging Face in decimal GB, and the card's 48 GB (binary GiB) is 51.5 GB in the same units. Speeds are this card's single-user generation speed from cited community runs at the file listed; a row with no cited run says so rather than estimating one.
| Model | Quant | Fits in 48 GB? | Cited speed on this card |
|---|---|---|---|
| Llama 3.1 8B 4.9 GB file | Q4_K_M | Fits~46.6 GB left for context | 45.1 tok/s1 cited run, llamafile, per LocalScore (Mozilla Builders) |
| Qwen3 14B 9 GB file | Q4_K_M | Fits~42.5 GB left for context | No public measurement on file |
| gpt-oss-20b 21B-A3.6B · 12.1 GB file | MXFP4 | Fits~39.4 GB left for context | No public measurement on file |
| Qwen3 30B-A3B 18.6 GB file | Q4_K_M | Fits~33 GB left for context | No public measurement on file |
| Qwen3 32B 19.8 GB file | Q4_K_M | Fits~31.8 GB left for context | No public measurement on file |
| Gemma 3 27B 28.7 GB file | Q8_0 | Fits~22.8 GB left for context | No public measurement on file |
| Llama 3.3 70B 42.5 GB file | Q4_K_M | Fits~9 GB left for context | No public measurement on file |
The NVIDIA L40 48GB is a graphics card from the Ada Lovelace Pro family released in 2023 from NVIDIA. Key on-paper specs include 48 GB of GDDR6 VRAM, 300W TDP. Data on this page draws on 7 synthetic benchmark results, 17 community AI inference reports (top 13 shown), compiled from GitHub: MarkoPekas/runpod-ollama-speedtest-guide, CpuTronic, GPU-Benchmarks-on-LLM-Inference, llama.cpp GitHub perf thread, technical.city, Crusoe AI Blog, Koyeb GPU Benchmarks, LocalScore, PassMark Software, vLLM GitHub Issues #5007, VMware Cloud Foundation Blog; each table row links to its source. Read this page when shopping the NVIDIA L40 48GB, comparing it against other graphics cards in your build, or sizing it for a specific workload (synthetic benchmark scores or local LLM inference).
AI Inference Performance
Tokens per second under each model + quantization. Higher = faster generation. Bars compare runs across the same model.
| Model | Quantization | Relative | Tokens/sec | VRAM used | Source |
|---|---|---|---|---|---|
| llama2:7b | Q4_0 llama.cpp | 153.8 tok/s | — | llama.cpp GitHub perf thread (Performance of llama.cpp on Nvidia CUDA) 2025-10-18 | |
| llama2:7b | q4_0 llama.cpp | 152.0 tok/s | — | llama.cpp GitHub Discussion #15013 2024-01 | |
| llama3:8b | q4_K_M llama.cpp | 113.6 tok/s | — | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) 2024-05 | |
| mixtral:8x7b-instruct-v0.1-q2_K | q2_K ollama | 65.5 tok/s | — | GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11 | |
| mixtral:8x7b | q4_0 ollama | 56.9 tok/s | — | GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11 | |
| mixtral:8x7b-instruct-v0.1-q3_K_S | q3_K_S ollama | 56.7 tok/s | — | GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11 | |
| llama3:8b | FP16 vllm | 54.1 tok/s | — | VMware Cloud Foundation Blog 2024-09-25 | |
| mixtral:8x7b-instruct-v0.1-q5_1 | q5_1 ollama | 46.8 tok/s | — | GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11 | |
| llama-3.1:8b | — vllm | 46.0 tok/s | — | Koyeb GPU Benchmarks 2024-06 | |
| llama3.1:8b | q4_K_M llamafile | 45.1 tok/s | — | LocalScore (Mozilla Builders) retrieved 2026-07-09 | |
| llama3.1:70b | Q4_K_M llama.cpp | 15.3 tok/s | — | GPU-Benchmarks-on-LLM-Inference (GitHub) 2024-07 |
Batched serving throughput
| Model | Quantization | Relative | Tokens/sec (aggregate) | VRAM used | Source |
|---|---|---|---|---|---|
| llama3:8b | W4A8KV4 QServe 128 concurrent requests | 3568.8 tok/s aggregate | — | Crusoe AI Blog 2024-08-02 | |
| llama-2:7b | FP16 vllm batched | 3115.3 tok/s aggregate | — | vLLM GitHub Issues #5007 2024-05-23 |
Synthetic Benchmarks
Higher is better. Bars are scaled within each benchmark family (multi-thread, single-thread, etc.) so you can compare like-with-like at a glance.
| Benchmark | Relative | Score | Source |
|---|---|---|---|
| Geekbench 5 OpenCL | 333,904 points | technical.city (Geekbench Browser community) retrieved 2026-04-29 | |
| Geekbench 5 OpenCL | 292,357 points | CpuTronic retrieved 2026-09-08 | |
| Geekbench 5 Vulkan | 249,130 points | CpuTronic retrieved 2026-09-08 | |
| Geekbench 5 Vulkan | 237,295 points | technical.city (Geekbench Browser community) retrieved 2026-04-29 | |
| PassMark G3D Mark | 27,355 points | PassMark Software retrieved 2026-06-09 | |
| PassMark GPU Compute | 23,067 points | PassMark VideoCardBenchmark retrieved 2026-04-29 | |
| Blender Benchmark | 4,336 points | CpuTronic retrieved 2026-09-08 |
Full Specifications
| TDP | 300 W |
|---|---|
| VRAM | 48 GB |
| CUDA cores | 18176 |
| Memory type | GDDR6 |
NVIDIA L40 48GB — Frequently Asked Questions
What is the NVIDIA L40 48GB best used for?
When was the NVIDIA L40 48GB released?
Where do the NVIDIA L40 48GB benchmark numbers come from?
Can the NVIDIA L40 48GB run local LLMs?
Where can I buy the NVIDIA L40 48GB?
Buying guides that rank the NVIDIA L40 48GB's class
This page is the raw performance data. The guides below turn it into a ranked pick for a specific build.
Editorial guides covering the NVIDIA L40 48GB
In-depth SpecPicks reviews, build guides, and head-to-heads referencing this graphics card.
- RTX 3060 12GB Local LLM: Reddit Consensus on What Runs (2026)
- RTX PRO 6000 vs RTX 5090 for Local 70B LLMs: 96GB vs 32GB
- Tesla P40 vs RTX 3090: Used 24GB GPUs for Local LLMs in 2026
- Best GPU for Local LLMs by VRAM: 2026 Reddit Consensus
- Used RTX 3090 vs New GPU for Local LLMs: 2026 Reddit Consensus
- gpt-oss on a 12GB GPU: 2026 Community Consensus (RTX 3060)
- FLUX.2 Klein 4B for Halloween Art: RTX 3060 12GB vs RTX 5060 8GB
- gpt-oss 120B on a 128GB Box: DGX Spark (ASUS Ascent GX10) vs Ryzen AI Max+ 395 tok/s Table
- Browse all SpecPicks reviews →
More guides & deep dives from the SpecPicks archive
Browse all articles & guides →- Best Retro Handhelds in 2026: Pocket, Mid-Tier and FPGA Picks
- How to Build a Windows 98 Retro PC in 2026
- The Complete Voodoo5 5500 AGP Driver Guide (2026 Edition)
- Best 1440p Gaming GPUs in 2026
- RTX 4070 Super vs RX 7800 XT — Which to Buy in 2026
- Emulation Hardware in 2026: FPGA, Software, and Cart-Reader Ecosystems
- Best Budget Gaming PC Build 2026 — ~$1,000 Parts List
More reviews from the SpecPicks archive
Browse all reviews →- Raspberry Pi 5 Price Guide: What You'll Actually Pay
- AMD Ryzen 9 9950X Hitting 98°C While Streaming?
- 4K 60Hz or 1440p 240Hz? Picking a Monitor for an RTX 3060-Class GPU
- MSI MPG 322UR QD-OLED X24: 4K 240Hz Performance Analyzed
- Should You Overprovision vCPU Per VM in a Homelab?
- Raspberry Pi Dual Monitor Setup: Driving Two 4K Displays in 2026
- 8BitDo Pro 2 vs GameSir G7 SE vs DualSense: Best Controller for Emulation
- Samsung 870 EVO vs WD Blue SN550: Best Storage for Forza Horizon 6 Fast Loading
- Raspberry Pi Zero 2W: Privacy-Preserving Ring Alternative
- Best CPU Cooler for a Ryzen 7 5800X (AM4) in 2026
- RX 9070 XT vs RTX 5080 for a 9800X3D Build (2026)
- Inkling 975B Open Weights: What a Frontier Speech Model Means for Local Rigs
- RTX 3060 12GB Showdown: ZOTAC Twin Edge vs MSI Ventus 2X (2026)
- Build a Raspberry Pi 4 Cyberdeck Music Workstation in 2026
- Best Mid-Range CPU for Streaming and Gaming in 2026: Ryzen 7 5800X vs 5700X vs 5600G
- Building a Period-Correct 1999 Quake III Arena LAN Rig: Voodoo3, Pentium III, and Win98 SE
- Best Gaming Controllers for PC in 2026: 5 Picks Tested Across Genres
- Best CPU Cooler for a Ryzen Build in 2026 (Air & AIO)
- Best Wireless Gaming Controllers for PC and Console in 2026
- Best Budget Gaming CPUs for 1080p in 2026
- How to Backup a NAS on a Budget in 2026
- IBM Granite 4.1 (3B / 8B / 30B): Local Inference Benchmarks and Hardware Picks
- Can a Raspberry Pi 4 8GB Run Local LLMs in 2026? Ollama tok/s + SSD-Boot Setup
- Best NVMe SSD for Local AI Model Storage in 2026
More buying guides from SpecPicks
Browse all buying guides →- Best DDR5 RAM for Gaming PCs in 2026
- Best Graphics Cards for Gaming in 2026
- Best 4K Monitors for Content Creators in 2026
- Best Tools for Building and Repairing Retro PCs in 2026
- Best CPUs for Content Creators in 2026
- Best Mechanical Keyboards for Gaming in 2026
- Best NVMe External Enclosures for 2026
- Best Gaming Monitors for 2026
- Best GPUs for 4K Gaming in 2026
- Best CPU Coolers for 2026
- Best AM5 Motherboards for 2026
- Best GPUs for Running Local LLMs in 2026
- Best CPUs for Gaming in 2026
- Best GPU for Running 27B-32B Local LLMs in 2026
- Best PC Cases for Building in 2026
- Best Retro Gaming Consoles & Handhelds for 2026
- Best Gaming Mice for 2026
- Best Controllers for PC Gaming in 2026
- Best 1440p 240Hz Gaming Monitors in 2026
- Best External SSDs for Content Creators in 2026
- Best NVMe SSDs for Gaming in 2026
Hardware benchmark data on SpecPicks
All benchmarks →- NVIDIA GeForce RTX 3060 Ti — benchmarks & specs
- AMD Ryzen 5 7640S — benchmarks & specs
- Intel Core i7-14700K — benchmarks & specs
- AMD Ryzen 9 7945HX3D — benchmarks & specs
- NVIDIA GeForce RTX 5090 — benchmarks & specs
- Apple M2 Ultra 24 Core — benchmarks & specs
- AMD Ryzen 7 7840U — benchmarks & specs
- NVIDIA GeForce GTX 1660 — benchmarks & specs
- GeForce 3 Ti 500 — benchmarks & specs
- Radeon RX 9070 GRE — benchmarks & specs
- Apple M3 8 Core — benchmarks & specs
- Quadro RTX 5000 (Mobile) — benchmarks & specs
- Ryzen 7 5700X3D — benchmarks & specs
- Radeon RX 5600 XT — benchmarks & specs
- AMD Ryzen 7 7735H — benchmarks & specs
- NVIDIA GeForce GTX 1660 Ti — benchmarks & specs
- AMD Ryzen 9 5980HX — benchmarks & specs
- AMD Ryzen 7 PRO 5850U — benchmarks & specs
- Radeon RX 6650M — benchmarks & specs
- AMD Ryzen 9 9955HX3D — benchmarks & specs
- Radeon RX 9060 XT 8GB — benchmarks & specs
- Apple M1 Pro 8 Core 3200 MHz — benchmarks & specs
- Radeon RX 7800 XT — benchmarks & specs
- NVIDIA GeForce RTX 5070 Ti — benchmarks & specs