NVIDIA RTX A6000 48GB — Benchmarks & Specs

Bottom line: how fast is the NVIDIA RTX A6000 48GB?

SpecPicks median: 102 tokens/sec across 3 cited single-user runs of 7-9B dense models at 4-bit (Q4) under llama.cpp (range 102–145 tok/s), from llama.cpp GitHub Discussion #15013, GPU-Benchmarks-on-LLM-Inference, OpenLLMBenchmarks. For larger models, the SpecPicks median is 14.6 tokens/sec across 4 cited single-user runs of 65-73B dense models at 4-bit (Q4) under llama.cpp (range 14.1–14.6 tok/s), from OpenLLMBenchmarks (2 runs), GPU-Benchmarks-on-LLM-Inference, LocalScore. For local LLM inference it generates 14.5 tokens/sec running qwen2.5:72b at q4_K_M under ollama, per DatabaseMart. In 3DMark Fire Strike Graphics it scores 27,511 points, per technical.city. Its 48 GB of VRAM is the binding constraint for local inference: the largest models on file at 4-bit quantization on this card are 72B-class: the qwen2.5:72b run above and qwen:72b (q4_0), which generates 14.5 tokens/sec under ollama, per DatabaseMart. That run's source does not report a context length; the Will-It-Run matrix sizes each model at its listed quantization with an ~8K context, and by that measure DeepSeek-R1-Distill-Llama 70B at Q4_K_M is the largest that fits in 48 GB.

Every figure above is a row in the tables below, and each row links out to the review or public benchmark database the number was taken from; the SpecPicks median is computed here from the runs it names, each a row in the AI table. SpecPicks aggregates published measurements; it does not report first-party benchmark runs.