Skip to main content
NVIDIA A100 PCIe 40GB
NVIDIA · GPU · Ampere

NVIDIA A100 PCIe 40GB — Benchmarks & Specs

40 GB VRAM250W TDP$12,000 MSRP2020

Short answer: the NVIDIA A100 PCIe 40GB generates 42.0 tokens/sec on qwen:32b (q4_0) under ollama in a single cited run, per DatabaseMart; there are too few comparable runs on file for a median. Its 40 GB of VRAM holds up to Gemma 3 27B at Q8_0 with an ~8K context, by the Will-It-Run matrix's sizing (24 of 27 models fit). A used eBay listing for it is $4,000.00, checked (listings can end at any time).

View used on eBay — $4,000.00 → Search Amazon for the NVIDIA A100 PCIe 40GB →

Who should buy this instead: the NVIDIA A100 PCIe 40GB's 40 GB puts it in the 24 GB tier of the local-LLM GPU picks by VRAM (one card per tier, each with cited tok/s). If the models you run fit in 16 GB, compare the 16 GB pick. If they need more than 40 GB, start at the 48 GB+ pick.

Bottom line: how fast is the NVIDIA A100 PCIe 40GB?

For local LLM inference it generates 42.0 tokens/sec running qwen:32b at q4_0 under ollama, per DatabaseMart. In PassMark G3D Mark it scores 13,326 points, per PassMark VideoCardBenchmark. Its 40 GB of VRAM is the binding constraint for local inference: the largest model on file at 4-bit quantization on this card is llama3:70b (70B parameters, q4_0), generating 24.1 tokens/sec under ollama, per DatabaseMart. That run's source does not report a context length; the Will-It-Run matrix sizes each model at its listed quantization with an ~8K context, and by that measure Gemma 3 27B at Q8_0 is the largest that fits in 40 GB.

Every figure above is a row in the tables below, and each row links out to the review or public benchmark database the number was taken from. SpecPicks aggregates published measurements; it does not report first-party benchmark runs.

eBay listing (Used): $4,000.00, checked . eBay listings are third-party and can end at any time; as an eBay Partner Network member, SpecPicks may earn a commission.

Check what fits in 40 GB: by file size at each listed quantization, 24 of the 27 open-weight model files in the SpecPicks Will-It-Run matrix load entirely into the NVIDIA A100 PCIe 40GB's 40 GB while keeping at least 1.5 GB reserved for an ~8K context; the largest that fits, Gemma 3 27B (a 28.7 GB Q8_0 file per its Hugging Face listing), leaves roughly 11.3 GB free. Check a model in the Will-It-Run matrix →

What fits in the NVIDIA A100 PCIe 40GB's 40 GB, and what doesn't

6 of these 7 model files load whole into 40 GB with room for an ~8K context. A file fits when the card keeps at least 1.5 GB free after the weights; file sizes are the GGUF downloads on Hugging Face. Speeds are this card's single-user generation speed from cited community runs at the Q4-class file listed; a row with no cited run says so rather than estimating one.

ModelQuantFits in 40 GB?Cited speed on this card
Llama 3.1 8B 4.9 GB fileQ4_K_MFits~35.1 GB left for context107 tok/s1 cited run, ollama, per DatabaseMart
Qwen3 14B 9 GB fileQ4_K_MFits~31 GB left for contextNo public measurement on file
gpt-oss-20b 21B-A3.6B · 12.1 GB fileMXFP4Fits~27.9 GB left for contextNo public measurement on file
Qwen3 30B-A3B 18.6 GB fileQ4_K_MFits~21.4 GB left for contextNo public measurement on file
Qwen3 32B 19.8 GB fileQ4_K_MFits~20.2 GB left for context42.1 tok/s1 cited run, ollama, per DatabaseMart
Gemma 3 27B 28.7 GB fileQ8_0Fits~11.3 GB left for contextNo public measurement on file
Llama 3.3 70B 42.5 GB fileQ4_K_MDoesn't fit2.5 GB over; runs only with CPU offload— (does not load into VRAM)

The NVIDIA A100 PCIe 40GB is a datacenter accelerator from the Ampere family released in 2020 from NVIDIA. Key on-paper specs include 40 GB of HBM2 memory, 250W TDP. It launched with a $12,000 MSRP, though street prices typically diverge meaningfully from launch pricing. Data on this page draws on 5 synthetic benchmark results, 39 community AI inference reports (top 14 shown), compiled from DatabaseMart, llama.cpp GitHub Discussion #3359, Bizon Tech GPU Benchmarks, PassMark VideoCardBenchmark, Puget Systems Labs, vLLM GitHub Discussions; each table row links to its source. Read this page when shopping the NVIDIA A100 PCIe 40GB, comparing it against other datacenter accelerators for your server or cluster, or sizing it for a specific workload (LLM inference and serving, fine-tuning, or HPC compute).

AI Inference Performance

Tokens per second under each model + quantization. Higher = faster generation. Bars compare runs across the same model.

Local LLM inference throughput on the NVIDIA A100 PCIe 40GB, in generated tokens per second for a single request (one user's generation speed). Higher is better; each row links to the community report or benchmark database it came from. Batched serving runs are listed separately below.
Model Quantization Relative Tokens/sec VRAM used Source
llama2:7b q4_K_M llama.cpp 127.2 tok/s — llama.cpp GitHub Discussion #3359 2023-09-27
DeepSeek-R1:8B q4_K_M ollama 108.2 tok/s — DatabaseMart 2025-01
llama3.1:8b q4_K_M ollama 106.7 tok/s — DatabaseMart 2025-01
llama2:7b q8_0 llama.cpp 101.7 tok/s — llama.cpp GitHub Discussion #3359 2023-09-27
llama2:13b q4_K_M llama.cpp 81.9 tok/s — llama.cpp GitHub Discussion #3359 2023-09-27
llama2:7b FP16 llama.cpp 72.1 tok/s — llama.cpp GitHub Discussion #3359 2023-09-27
qwen2.5:14b q4_K_M ollama 65.0 tok/s — DatabaseMart 2025-01
gemma3:12b q4_K_M ollama 61.6 tok/s — DatabaseMart 2025-03
qwen:32b q4_0 ollama 42.0 tok/s — DatabaseMart 2025-02
llama3:70b q4_K_M ollama 24.1 tok/s — DatabaseMart 2025-01

Batched serving throughput

Aggregate output tokens per second on the NVIDIA A100 PCIe 40GB summed across 50 concurrent requests (serving runtimes such as vLLM). This is total server throughput, not one user's generation speed — each request generates far slower — so these rows are not comparable with the single-stream table above.
Model Quantization Relative Tokens/sec (aggregate) VRAM used Source
gemma-3-4b-it FP16 vllm 50 concurrent requests 3385.7 tok/s aggregate 8.1 GB DatabaseMart 2025-03
llama2:7b FP16 vllm batched 2246.0 tok/s aggregate — vLLM GitHub Discussions 2023-06-27
llama-7b — vllm batched 2246.0 tok/s aggregate — vllm GitHub Discussion #275 2023-06-27
DeepSeek-R1-Distill-Llama-8B FP16 vllm 50 concurrent requests 2225.3 tok/s aggregate 15.0 GB DatabaseMart 2025-03

Synthetic Benchmarks

Higher is better. Bars are scaled within each benchmark family (multi-thread, single-thread, etc.) so you can compare like-with-like at a glance.

Synthetic benchmark scores for the NVIDIA A100 PCIe 40GB — higher is better. Each row links to the public database the number was taken from.
Benchmark Relative Score Source
PassMark G3D Mark 13,326 points PassMark VideoCardBenchmark retrieved 2026-05-01
HPL (High Performance LINPACK) FP64 10,940 Gflops Puget Systems Labs 2021-05-21
Blender (Bizon composite) 3,788 points Bizon Tech GPU Benchmarks 2024-01
V-Ray GPU 1,555 points Bizon Tech GPU Benchmarks 2024-01
OctaneBench 498 points Bizon Tech GPU Benchmarks 2024-01

Full Specifications

TDP250 W
VRAM40 GB
CUDA cores6912
Memory typeHBM2

NVIDIA A100 PCIe 40GB — Frequently Asked Questions

What is the NVIDIA A100 PCIe 40GB best used for?
NVIDIA A100 PCIe 40GB is a 40 GB VRAM Ampere-family graphics card. High-end 4K gaming and local LLM inference are the headline use cases; the benchmark tables on this page place it within its generation.
When was the NVIDIA A100 PCIe 40GB released, and what was its launch MSRP?
NVIDIA A100 PCIe 40GB launched in 2020 at a $12,000 MSRP. Street prices drift from launch pricing over a product's life, so check the current listing before buying.
Where do the NVIDIA A100 PCIe 40GB benchmark numbers come from?
Each benchmark row on this page names its source (public benchmark databases such as TechPowerUp, PassMark, Geekbench and Cinebench, and community reports such as r/LocalLLaMA threads) and links to it, so every number can be checked against the original.
Can the NVIDIA A100 PCIe 40GB run local LLMs?
Yes — AI inference results for the NVIDIA A100 PCIe 40GB are on file; the AI Inference Performance section on this page lists model, quantization and tokens-per-second for each run. With 40 GB VRAM, 32B-parameter open-weight models fit at Q4 quantization.
Where can I buy the NVIDIA A100 PCIe 40GB?
SpecPicks has no verified listing of the NVIDIA A100 PCIe 40GB on file yet; the Amazon search link at the top of this page shows its current offers. SpecPicks earns a small affiliate commission on qualifying purchases.

Buying guides that rank the NVIDIA A100 PCIe 40GB's class

This page is the raw performance data. The guides below turn it into a ranked pick for a specific build.

Related GPUs

Editorial guides covering the NVIDIA A100 PCIe 40GB

In-depth SpecPicks reviews, build guides, and head-to-heads referencing this datacenter accelerator.

More guides & deep dives from the SpecPicks archive

Browse all articles & guides →

More reviews from the SpecPicks archive

Browse all reviews →

More buying guides from SpecPicks

Browse all buying guides →

Hardware benchmark data on SpecPicks

All benchmarks →