Skip to main content
NVIDIA L40 48GB
NVIDIA · GPU · Ada Lovelace Pro

NVIDIA L40 48GB — Benchmarks & Specs

48 GB VRAM300W TDP2023

Short answer: the NVIDIA L40 48GB generates a median 152 tokens/sec on 7-9B dense models at 4-bit (Q4) under llama.cpp, across 3 cited runs from llama.cpp GitHub perf thread and GPU-Benchmarks-on-LLM-Inference. All 27 models in the Will-It-Run matrix fit in its 48 GB of VRAM at their listed quantization with an ~8K context, the largest in the sizing matrix being DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run); the matrix does not include larger models, so this is not the card's ceiling.

  • Median speed 152 tok/s 3 cited runs, 7-9B at Q4, llama.cpp
  • Fits in 48 GB all 27 matrix models → at ~8K context; the largest in the sizing matrix is DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run), and the Will-It-Run matrix has no larger model

Who should buy this instead: the NVIDIA L40 48GB's 48 GB puts it in the 48 GB+ tier of the local-LLM GPU picks by VRAM (one card per tier, each with cited tok/s). If the models you run fit in 24 GB, compare the 24 GB pick. If they need more than 48 GB, start at the 128 GB unified pick.

NVIDIA L40 48GB: verdict by use case

  • For everyday local models (7-9B dense): a median 152 tok/s at 4-bit (Q4) under llama.cpp, across 3 cited runs (llama.cpp GitHub perf thread, GPU-Benchmarks-on-LLM-Inference).
  • For large models in 48 GB: all 27 models in the Will-It-Run matrix fit with an ~8K context, the largest in the sizing matrix being DeepSeek-R1-Distill-Llama 70B at Q4_K_M (estimated to fit, not a measured run); the matrix does not include larger models, so 48 GB has headroom beyond it that is not sized here.

Each verdict quotes a figure from a linked source or a row in the tables below. Where no cited figure exists for a use case, there is no verdict line for it.

Bottom line: how fast is the NVIDIA L40 48GB?

SpecPicks median: 152 tokens/sec across 3 cited single-user runs of 7-9B dense models at 4-bit (Q4) under llama.cpp (range 114–154 tok/s), from llama.cpp GitHub perf thread (2 runs), GPU-Benchmarks-on-LLM-Inference. For local LLM inference it generates 15.3 tokens/sec running llama3.1:70b at Q4_K_M under llama.cpp, per GPU-Benchmarks-on-LLM-Inference. In Geekbench 5 OpenCL it scores 292,357–333,904 points, across two published results (technical.city GPU benchmark listing, CpuTronic GPU benchmark listing). Its 48 GB of VRAM is the binding constraint for local inference: the llama3.1:70b run above (70B parameters) is the largest model on file at 4-bit quantization on this card.

Every figure above is a row in the tables below, and each row links out to the review or public benchmark database the number was taken from; the SpecPicks median is computed here from the runs it names, each a row in the AI table. SpecPicks aggregates published measurements; it does not report first-party benchmark runs.

Check what fits in 48 GB: by file size at each listed quantization, 27 of the 27 open-weight model files in the SpecPicks Will-It-Run matrix load entirely into the NVIDIA L40 48GB's 48 GB while keeping at least 1.5 GB reserved for an ~8K context; the largest, DeepSeek-R1-Distill-Llama 70B (a 42.5 GB Q4_K_M file per its Hugging Face listing), leaves roughly 9 GB of the card's 51.5 GB (48 GiB) free. Check a model in the Will-It-Run matrix →

What fits in the NVIDIA L40 48GB's 48 GB, and what doesn't

7 of these 7 model files load whole into 48 GB with room for an ~8K context. A file fits when the card keeps at least 1.5 GB free after the weights; file sizes are the GGUF downloads on Hugging Face in decimal GB, and the card's 48 GB (binary GiB) is 51.5 GB in the same units. Speeds are this card's single-user generation speed from cited community runs at the file listed; a row with no cited run says so rather than estimating one.

ModelQuantFits in 48 GB?Cited speed on this card
Llama 3.1 8B 4.9 GB fileQ4_K_MFits~46.6 GB left for context45.1 tok/s1 cited run, llamafile, per LocalScore (Mozilla Builders)
Qwen3 14B 9 GB fileQ4_K_MFits~42.5 GB left for contextNo public measurement on file
gpt-oss-20b 21B-A3.6B · 12.1 GB fileMXFP4Fits~39.4 GB left for contextNo public measurement on file
Qwen3 30B-A3B 18.6 GB fileQ4_K_MFits~33 GB left for contextNo public measurement on file
Qwen3 32B 19.8 GB fileQ4_K_MFits~31.8 GB left for contextNo public measurement on file
Gemma 3 27B 28.7 GB fileQ8_0Fits~22.8 GB left for contextNo public measurement on file
Llama 3.3 70B 42.5 GB fileQ4_K_MFits~9 GB left for contextNo public measurement on file

The NVIDIA L40 48GB is a graphics card from the Ada Lovelace Pro family released in 2023 from NVIDIA. Key on-paper specs include 48 GB of GDDR6 VRAM, 300W TDP. Data on this page draws on 7 synthetic benchmark results, 17 community AI inference reports (top 13 shown), compiled from GitHub: MarkoPekas/runpod-ollama-speedtest-guide, CpuTronic, GPU-Benchmarks-on-LLM-Inference, llama.cpp GitHub perf thread, technical.city, Crusoe AI Blog, Koyeb GPU Benchmarks, LocalScore, PassMark Software, vLLM GitHub Issues #5007, VMware Cloud Foundation Blog; each table row links to its source. Read this page when shopping the NVIDIA L40 48GB, comparing it against other graphics cards in your build, or sizing it for a specific workload (synthetic benchmark scores or local LLM inference).

AI Inference Performance

Tokens per second under each model + quantization. Higher = faster generation. Bars compare runs across the same model.

Local LLM inference throughput on the NVIDIA L40 48GB, in generated tokens per second for a single request (one user's generation speed). Higher is better; each row links to the community report or benchmark database it came from. Batched serving runs are listed separately below.
Model Quantization Relative Tokens/sec VRAM used Source
llama2:7b Q4_0 llama.cpp 153.8 tok/s — llama.cpp GitHub perf thread (Performance of llama.cpp on Nvidia CUDA) 2025-10-18
llama2:7b q4_0 llama.cpp 152.0 tok/s — llama.cpp GitHub Discussion #15013 2024-01
llama3:8b q4_K_M llama.cpp 113.6 tok/s — XiongjieDai/GPU-Benchmarks-on-LLM-Inference (GitHub) 2024-05
mixtral:8x7b-instruct-v0.1-q2_K q2_K ollama 65.5 tok/s — GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11
mixtral:8x7b q4_0 ollama 56.9 tok/s — GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11
mixtral:8x7b-instruct-v0.1-q3_K_S q3_K_S ollama 56.7 tok/s — GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11
llama3:8b FP16 vllm 54.1 tok/s — VMware Cloud Foundation Blog 2024-09-25
mixtral:8x7b-instruct-v0.1-q5_1 q5_1 ollama 46.8 tok/s — GitHub (MarkoPekas/runpod-ollama-speedtest-guide) 2024-01-11
llama-3.1:8b — vllm 46.0 tok/s — Koyeb GPU Benchmarks 2024-06
llama3.1:8b q4_K_M llamafile 45.1 tok/s — LocalScore (Mozilla Builders) retrieved 2026-07-09
llama3.1:70b Q4_K_M llama.cpp 15.3 tok/s — GPU-Benchmarks-on-LLM-Inference (GitHub) 2024-07

Batched serving throughput

Aggregate output tokens per second on the NVIDIA L40 48GB summed across 128 concurrent requests (serving runtimes such as vLLM). This is total server throughput, not one user's generation speed — each request generates far slower — so these rows are not comparable with the single-stream table above.
Model Quantization Relative Tokens/sec (aggregate) VRAM used Source
llama3:8b W4A8KV4 QServe 128 concurrent requests 3568.8 tok/s aggregate — Crusoe AI Blog 2024-08-02
llama-2:7b FP16 vllm batched 3115.3 tok/s aggregate — vLLM GitHub Issues #5007 2024-05-23

Synthetic Benchmarks

Higher is better. Bars are scaled within each benchmark family (multi-thread, single-thread, etc.) so you can compare like-with-like at a glance.

Synthetic benchmark scores for the NVIDIA L40 48GB — higher is better. Each row links to the public database the number was taken from. A benchmark listed more than once is separate published results, shown as published rather than averaged.
Benchmark Relative Score Source
Geekbench 5 OpenCL 333,904 points technical.city (Geekbench Browser community) retrieved 2026-04-29
Geekbench 5 OpenCL 292,357 points CpuTronic retrieved 2026-09-08
Geekbench 5 Vulkan 249,130 points CpuTronic retrieved 2026-09-08
Geekbench 5 Vulkan 237,295 points technical.city (Geekbench Browser community) retrieved 2026-04-29
PassMark G3D Mark 27,355 points PassMark Software retrieved 2026-06-09
PassMark GPU Compute 23,067 points PassMark VideoCardBenchmark retrieved 2026-04-29
Blender Benchmark 4,336 points CpuTronic retrieved 2026-09-08

Full Specifications

TDP300 W
VRAM48 GB
CUDA cores18176
Memory typeGDDR6

NVIDIA L40 48GB — Frequently Asked Questions

What is the NVIDIA L40 48GB best used for?
NVIDIA L40 48GB is a 48 GB VRAM Ada Lovelace Pro-family graphics card. High-end 4K gaming and local LLM inference are the headline use cases; the benchmark tables on this page place it within its generation.
When was the NVIDIA L40 48GB released?
NVIDIA L40 48GB was released in 2023, an Ada Lovelace Pro-generation part.
Where do the NVIDIA L40 48GB benchmark numbers come from?
Each benchmark row on this page names its source (public benchmark databases such as TechPowerUp, PassMark, Geekbench and Cinebench, and community reports such as r/LocalLLaMA threads) and links to it, so every number can be checked against the original.
Can the NVIDIA L40 48GB run local LLMs?
Yes — AI inference results for the NVIDIA L40 48GB are on file; the AI Inference Performance section on this page lists model, quantization and tokens-per-second for each run. With 48 GB VRAM, 32B-parameter open-weight models fit at Q4 quantization.
Where can I buy the NVIDIA L40 48GB?
SpecPicks has no verified listing of the NVIDIA L40 48GB on file yet; the Amazon search link at the top of this page shows its current offers. SpecPicks earns a small affiliate commission on qualifying purchases.

Buying guides that rank the NVIDIA L40 48GB's class

This page is the raw performance data. The guides below turn it into a ranked pick for a specific build.

Editorial guides covering the NVIDIA L40 48GB

In-depth SpecPicks reviews, build guides, and head-to-heads referencing this graphics card.

More guides & deep dives from the SpecPicks archive

Browse all articles & guides →

More reviews from the SpecPicks archive

Browse all reviews →

More buying guides from SpecPicks

Browse all buying guides →

Hardware benchmark data on SpecPicks

All benchmarks →