NVIDIA L40S 48GB — Benchmarks & Specs
Short answer: the NVIDIA L40S 48GB generates 15.3 tokens/sec on llama3:70b (q4_K_M) under llama.cpp in a single cited run, per GPU-Benchmarks-on-LLM-Inference; there are too few comparable runs on file for a median. All 27 models in the Will-It-Run matrix fit in its 48 GB of VRAM at their listed quantization with an ~8K context, the largest in the sizing matrix being DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run); the matrix does not include larger models, so this is not the card's ceiling.
- Single cited run 15.3 tok/s llama3:70b q4_K_M, per GPU-Benchmarks-on-LLM-Inference
- Fits in 48 GB all 27 matrix models → at ~8K context; the largest in the sizing matrix is DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run), and the Will-It-Run matrix has no larger model
Who should buy this instead: the NVIDIA L40S 48GB's 48 GB puts it in the 48 GB+ tier of the local-LLM GPU picks by VRAM (one card per tier, each with cited tok/s). If the models you run fit in 24 GB, compare the 24 GB pick. If they need more than 48 GB, start at the 128 GB unified pick.
Bottom line: how fast is the NVIDIA L40S 48GB?
For local LLM inference it generates 15.3 tokens/sec running llama3:70b at q4_K_M under llama.cpp, per GPU-Benchmarks-on-LLM-Inference. In Geekbench 6 OpenCL it scores 352,507 points, per Tom's Hardware. Its 48 GB of VRAM is the binding constraint for local inference: the llama3:70b run above (70B parameters) is the largest model on file at 4-bit quantization on this card.
Every figure above is a row in the tables below, and each row links out to the review or public benchmark database the number was taken from. SpecPicks aggregates published measurements; it does not report first-party benchmark runs.
Check what fits in 48 GB: by file size at each listed quantization, 27 of the 27 open-weight model files in the SpecPicks Will-It-Run matrix load entirely into the NVIDIA L40S 48GB's 48 GB while keeping at least 1.5 GB reserved for an ~8K context; the largest, DeepSeek-R1-Distill-Llama 70B (a 42.5 GB Q4_K_M file per its Hugging Face listing), leaves roughly 9 GB of the card's 51.5 GB (48 GiB) free. Check a model in the Will-It-Run matrix →
What fits in the NVIDIA L40S 48GB's 48 GB, and what doesn't
7 of these 7 model files load whole into 48 GB with room for an ~8K context. A file fits when the card keeps at least 1.5 GB free after the weights; file sizes are the GGUF downloads on Hugging Face in decimal GB, and the card's 48 GB (binary GiB) is 51.5 GB in the same units. Speeds are this card's single-user generation speed from cited community runs at the file listed; a row with no cited run says so rather than estimating one.
| Model | Quant | Fits in 48 GB? | Cited speed on this card |
|---|---|---|---|
| Llama 3.1 8B 4.9 GB file | Q4_K_M | Fits~46.6 GB left for context | 45.9 tok/s1 cited run, llama.cpp, per LocalScore (Mozilla Builders) |
| Qwen3 14B 9 GB file | Q4_K_M | Fits~42.5 GB left for context | No public measurement on file |
| gpt-oss-20b 21B-A3.6B · 12.1 GB file | MXFP4 | Fits~39.4 GB left for context | No public measurement on file |
| Qwen3 30B-A3B 18.6 GB file | Q4_K_M | Fits~33 GB left for context | No public measurement on file |
| Qwen3 32B 19.8 GB file | Q4_K_M | Fits~31.8 GB left for context | No public measurement on file |
| Gemma 3 27B 28.7 GB file | Q8_0 | Fits~22.8 GB left for context | No public measurement on file |
| Llama 3.3 70B 42.5 GB file | Q4_K_M | Fits~9 GB left for context | No public measurement on file |
The NVIDIA L40S 48GB is a graphics card from the Ada Lovelace Pro family released in 2023 from NVIDIA. Key on-paper specs include 48 GB of GDDR6 VRAM, 350W TDP. Data on this page draws on 6 synthetic benchmark results, 28 community AI inference reports (top 14 shown), compiled from GPU-Benchmarks-on-LLM-Inference, Koyeb, PassMark Software, Red Hat, technical.city, Crusoe AI / MIT Han Lab, FPSBench, LocalScore, lucataco GPU Benchmark Gist, NVIDIA NIM LLMs Benchmarking, Spheron Network blog, Thunder Compute, Tom's Hardware; each table row links to its source. Read this page when shopping the NVIDIA L40S 48GB, comparing it against other graphics cards in your build, or sizing it for a specific workload (synthetic benchmark scores or local LLM inference).
AI Inference Performance
Tokens per second under each model + quantization. Higher = faster generation. Bars compare runs across the same model.
| Model | Quantization | Relative | Tokens/sec | VRAM used | Source |
|---|---|---|---|---|---|
| Llama 3 8B | Q4_K_M llama.cpp | 113.6 tok/s | — | GPU-Benchmarks-on-LLM-Inference (GitHub) 2024-05 | |
| llama3.1:8b | FP8 vllm | 72.6 tok/s | — | NVIDIA NIM LLMs Benchmarking 2025-03 | |
| llama3.1:8b | FP8 vllm | 70.0 tok/s | — | Thunder Compute 2024-08 | |
| llama2:13b | FP16 other | 58.7 tok/s | — | lucataco GPU Benchmark Gist (GitHub) 2023-11 | |
| deepseek-r1-distill-llama-8b | — vllm | 46.0 tok/s | — | Koyeb GPU Benchmarks 2025-05 | |
| llama3.1:8b | FP16 vllm | 46.0 tok/s | — | Koyeb GPU Benchmarks 2025-05 | |
| llama3.1:8b | FP16 vllm | 46.0 tok/s | — | Spheron Network blog 2026-03-15 | |
| llama3.1:8b | q4_K_M llama.cpp | 45.9 tok/s | — | LocalScore (Mozilla Builders) retrieved 2026-08-08 | |
| Llama 3 70B | Q4_K_M llama.cpp | 15.3 tok/s | — | GPU-Benchmarks-on-LLM-Inference (GitHub) 2024-05 | |
| llama3:70b | q4_K_M llama.cpp | 15.3 tok/s | — | XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) |
Batched serving throughput
| Model | Quantization | Relative | Tokens/sec (aggregate) | VRAM used | Source |
|---|---|---|---|---|---|
| llama3:8b | W4A8KV4 QServe 128 concurrent requests | 3568.8 tok/s aggregate | — | Crusoe AI / MIT Han Lab 2024-07 | |
| Llama 3.1 8B Instruct | FP8 vLLM (MLPerf v5.1 Offline) batched | 1642.0 tok/s aggregate | — | Red Hat (MLPerf Inference v5.1) 2025-09-10 | |
| Llama 3.1 8B Instruct | FP8 vLLM (MLPerf v5.1 Server) batched | 1207.0 tok/s aggregate | — | Red Hat (MLPerf Inference v5.1) 2025-09-10 | |
| llama3.1:8b | FP16 vllm 8 concurrent requests | 336.0 tok/s aggregate | — | Koyeb 2024-10 |
Synthetic Benchmarks
Higher is better. Bars are scaled within each benchmark family (multi-thread, single-thread, etc.) so you can compare like-with-like at a glance.
| Benchmark | Relative | Score | Source |
|---|---|---|---|
| Geekbench 6 OpenCL | 352,507 points | Tom's Hardware 2024-06-25 | |
| Geekbench 5 OpenCL | 336,576 points | technical.city retrieved 2026-07-09 | |
| Geekbench OpenCL | 330,727 points | FPSBench (aggregated from Geekbench Browser) 2024-03 | |
| Geekbench 5 Vulkan | 260,799 points | technical.city retrieved 2026-07-09Same result reported by FPSBench (aggregated from Geekbench Browser) | |
| PassMark G3D Mark | 20,023 points | PassMark Software retrieved 2026-06-09 | |
| PassMark GPU Compute | 18,866 Ops/Sec | PassMark Software retrieved 2026-07-09 |
Full Specifications
| TDP | 350 W |
|---|---|
| VRAM | 48 GB |
| CUDA cores | 18176 |
| Memory type | GDDR6 |
NVIDIA L40S 48GB — Frequently Asked Questions
What is the NVIDIA L40S 48GB best used for?
When was the NVIDIA L40S 48GB released?
Where do the NVIDIA L40S 48GB benchmark numbers come from?
Can the NVIDIA L40S 48GB run local LLMs?
Where can I buy the NVIDIA L40S 48GB?
Buying guides that rank the NVIDIA L40S 48GB's class
This page is the raw performance data. The guides below turn it into a ranked pick for a specific build.
Editorial guides covering the NVIDIA L40S 48GB
In-depth SpecPicks reviews, build guides, and head-to-heads referencing this graphics card.
- Running Llama 3.1 70B Locally: Hardware Requirements & Performance Benchmarks
- NVIDIA RTX A6000 48GB Review: The Workstation Card That Still Owns Local 70B Inference (2026)
- NVIDIA RTX PRO 6000 Blackwell vs RTX A6000: Is 96 GB Worth $4,000 More?
- Tesla P40 vs RTX 3090: Used 24GB GPUs for Local LLMs in 2026
- gpt-oss on a 12GB GPU: 2026 Community Consensus (RTX 3060)
- Best GPU for Local LLMs by VRAM: 2026 Reddit Consensus
- Used RTX 3090 vs New GPU for Local LLMs: 2026 Reddit Consensus
- RTX PRO 6000 vs RTX 5090 for Local 70B LLMs: 96GB vs 32GB
- Browse all SpecPicks reviews →
More guides & deep dives from the SpecPicks archive
Browse all articles & guides →- How to Build a Windows 98 Retro PC in 2026
- The Complete Voodoo5 5500 AGP Driver Guide (2026 Edition)
- Emulation Hardware in 2026: FPGA, Software, and Cart-Reader Ecosystems
- RTX 4070 Super vs RX 7800 XT — Which to Buy in 2026
- Best 1440p Gaming GPUs in 2026
- Best Budget Gaming PC Build 2026 — ~$1,000 Parts List
- Best Retro Handhelds in 2026: Pocket, Mid-Tier and FPGA Picks
More reviews from the SpecPicks archive
Browse all reviews →- Build a Raspberry Pi 4 AirPlay Receiver in 2026: Full Walkthrough
- RTX 3060 12GB vs Ryzen 5 5600G iGPU for Entry Local LLM Inference
- Retro PC storage in 2026: CompactFlash-to-IDE vs a SATA SSD for Win98/XP
- Strix Halo Gaming Benchmarks: What AMD's APU Can Really Do
- Imaging a Vintage IDE Drive to CompactFlash for a Silent Win98 Boot
- Best Wired Controller for Fighting Games on PC (2026)
- Solar-Powered Raspberry Pi Bird Identifier Shows the Pi AI Camera's Reach
- Modern CRT Monitor for Retro Gaming: The 2025 Reality Check
- Shared ChatGPT and Claude Chat Links Are Spreading Malware (And Local LLMs Fix It)
- Best GPU for AI Workstations in 2026
- Halloween Talking Prop: Qwen3 4B on Jetson Orin Nano Super vs Raspberry Pi 5 16GB
- How to Run a SATA SSD on a Windows XP Retro Gaming PC (AHCI + Adapters)
- HyperX QuadCast 2 vs Blue Yeti for Streaming in an Untreated Room
- OpenAI Tripled Revenue to $5.7B but Burned $3.7B: What It Means for Local AI
- This Open-Source KVM Runs a Pentium 4 Instead of a Raspberry Pi
- Best Storage Upgrades for Gaming and Retro PCs in 2026
- Qwen3.7-Plus vs Gemma 4 12B for Local Agents on a 12GB GPU
- How to run Llama 3.1 70B on Arc B580
- Ryzen AI Max+ 395 Specs: Full Breakdown and Comparison
- RTX 3060 12GB vs Ryzen 9 3900X for Qwen2.5 14B: GPU or CPU in 2026?
- Claude Fable 5: Anthropic Admits 'Wrong Tradeoff' on Throttling
- ComfyUI on an RTX 3060 12GB: Flux and SDXL Speeds in 2026
- UT99 OldUnreal 469 Patch in 2026: Migrate Configs, Rejoin Modern Servers, and Tune Mouse Precision
- AMD Ryzen Buying Guide 2026: Every Tier Compared
More buying guides from SpecPicks
Browse all buying guides →- Best GPUs for Running Local LLMs in 2026
- Best PC Cases for Building in 2026
- Best Gaming Mice for 2026
- Best Tools for Building and Repairing Retro PCs in 2026
- Best 4K Monitors for Content Creators in 2026
- Best NVMe SSDs for Gaming in 2026
- Best Retro Gaming Consoles & Handhelds for 2026
- Best GPU for Running 27B-32B Local LLMs in 2026
- Best Mechanical Keyboards for Gaming in 2026
- Best Graphics Cards for Gaming in 2026
- Best CPUs for Content Creators in 2026
- Best Gaming Monitors for 2026
- Best DDR5 RAM for Gaming PCs in 2026
- Best GPUs for 4K Gaming in 2026
- Best Controllers for PC Gaming in 2026
- Best CPU Coolers for 2026
- Best 1440p 240Hz Gaming Monitors in 2026
- Best CPUs for Gaming in 2026
- Best NVMe External Enclosures for 2026
- Best AM5 Motherboards for 2026
- Best External SSDs for Content Creators in 2026
Hardware benchmark data on SpecPicks
All benchmarks →- Apple M4 Max — benchmarks & specs
- Radeon RX 6800S — benchmarks & specs
- Radeon RX 6650M — benchmarks & specs
- AMD Ryzen 9 5900 — benchmarks & specs
- AMD Ryzen 9 5900H — benchmarks & specs
- AMD Ryzen 9 7945HX — benchmarks & specs
- GeForce RTX 5050 9 GB — benchmarks & specs
- NVIDIA GeForce GTX 1650 — benchmarks & specs
- Ryzen 5 4500U — benchmarks & specs
- AMD Ryzen 7 PRO 5750G — benchmarks & specs
- Apple M4 8 Core — benchmarks & specs
- Radeon RX 7600S — benchmarks & specs
- NVIDIA Jetson Orin Nano 8GB — benchmarks & specs
- NVIDIA GeForce GTX 1050 Ti — benchmarks & specs
- AMD Ryzen 9 7845HX — benchmarks & specs
- AMD Ryzen 5 5600X3D — benchmarks & specs
- Ryzen 7 1700 — benchmarks & specs
- AMD Ryzen Threadripper PRO 7995WX — benchmarks & specs
- AMD Ryzen 9 PRO 7945 — benchmarks & specs
- Ryzen 7 5700U — benchmarks & specs
- AMD Ryzen 5 PRO 5650G — benchmarks & specs
- RTX6000-Ada-6Q — benchmarks & specs
- GeForce RTX 4060 Ti — benchmarks & specs
- Intel Core i5-11400F — benchmarks & specs