How to run Llama 3.1 8B on Apple M3 Ultra

Llama 3.1 8B runs comfortably on every Apple M3 Ultra Mac Studio configuration — the 96 GB base SKU has 12× the memory the model needs, and the 819 GB/s memory bandwidth pushes generation to 75–110 tok/s at q4_K_M as of 2026. The interesting question isn't whether it works; it's why you'd buy an M3 Ultra to run a model this small, and the honest answer is: you wouldn't, unless you're using the headroom for parallel sessions, long context, or running multiple models concurrently.

What "Llama 3.1 8B" is, and why it's still the default 2026 small model

Meta-Llama-3.1-8B-Instruct shipped in July 2024 as part of Meta's Llama 3.1 release. It's an 8.03B-parameter dense transformer with 128K native context, trained on ~15T tokens. Despite being eighteen months old at this point, it's still the community baseline for "small model that gets work done" — it punches above its weight on coding (HumanEval ~62), summarization, and tool-use, and the ecosystem of fine-tunes around it (Hermes, Nous, Dolphin) is the deepest in the open-source LLM world.

For Mac users, 8B is the sweet spot because:

• It fits everywhere — even a 16 GB MacBook Air runs it at acceptable speed
• The q4_K_M GGUF is ~4.7 GB, leaving room for long context
• It generates at human-reading-speed-or-faster on M2/M3/M4 Macs
• Apple's MLX team and Ollama both ship optimized builds

If you're new to local LLMs and want to feel the workflow without buying a $7,000 Mac Studio, this is the model you start with.

Apple M3 Ultra: what makes it actually different

The M3 Ultra launched in the Mac Studio in March 2025 and remains Apple's flagship for local LLM work as of mid-2026. It's two M3 Max dies stitched via UltraFusion, yielding:

The thing that matters for LLM inference is 819 GB/s memory bandwidth. That's roughly 1.5× the 16-core M4 Max (546 GB/s) and ~50% higher than the M3 Max it's stitched from. For an 8B model — which is small enough that you'll never run out of memory — this bandwidth translates directly into tokens per second. The 80-core GPU variant doesn't help much for an 8B model; prompt processing is fast either way and generation is bandwidth-bound, not compute-bound.

The Apple page is your reference for the official specs.

VRAM math (and why it's almost irrelevant here)

Llama 3.1 8B at q4_K_M:

Total at 128K context: ~22 GB. The Mac Studio base 96 GB SKU has 70+ GB of headroom even with the full 128K context loaded. That's the entire point — you can spawn multiple instances, run a long-context retrieval task, and keep room for an embedding model and a transcription model on the side without touching swap.

Install with Ollama

curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3.1:8b
ollama run llama3.1:8b

Ollama maps llama3.1:8b to the Meta Llama-3.1-8B-Instruct q4_K_M build by default. Tags like llama3.1:8b-instruct-q8_0 or llama3.1:8b-instruct-fp16 get you higher-precision variants — feel free to pick one, the M3 Ultra won't break a sweat.

To push the context window up before launch:

OLLAMA_NUM_CTX=32768 ollama run llama3.1:8b

For 128K context, raise both OLLAMA_NUM_CTX and num_gpu and expect ~16 GB of KV cache. Worth it for long-doc summarization.

Install with llama.cpp

llama.cpp gives you reproducible benchmarks via llama-bench and full control over batch size and KV-cache precision.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build -j

huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
 Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf --local-dir ./models

./build/bin/llama-cli \
 -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
 -c 32768 -ngl 99 \
 -p "Summarize Hamlet in 200 words."

-ngl 99 puts every layer on the GPU. On the M3 Ultra, anything else leaves performance on the table — there's no PCIe penalty because the GPU and CPU share the same memory pool.

For reproducible numbers:

./build/bin/llama-bench \
 -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
 -ngl 99 -p 512 -n 128 -r 3

Expected tok/s

Numbers below from the llama.cpp M-series benchmark thread and our runs on an M3 Ultra 96 GB. Generation is single-stream at 4K context, q4_K_M.

Generation tok/s converges across the 60-core and 80-core M3 Ultra because the model is too small to exhaust the GPU. Prompt eval still benefits from extra GPU cores. If you're doing RAG over long documents, prefer the 80-core; if you're doing chat, the 60-core is fine.

For perspective: 8B at 90 tok/s is faster than most people can read. The bottleneck shifts from the model to your reading speed.

MLX path for maximum speed

Apple's MLX framework is roughly 15–25% faster than llama.cpp for the 8B size on M-series silicon, because it uses Apple's optimized GEMM kernels directly rather than going through GGML's portable implementations.

pip install mlx-lm

python -m mlx_lm.generate \
 --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
 --prompt "Write a Python function to compute longest common subsequence." \
 --max-tokens 400

On an 80-core M3 Ultra 96 GB, the MLX 4-bit Llama 3.1 8B runs at ~115 tok/s generation. That's faster than humans can read and approaches the network round-trip floor for a hosted API.

Quantization choices: just use q4_K_M, but here's the ladder

The 8B model is small enough that you can push to q8_0 on any M3 Ultra without breaking a sweat. The 0.5% MMLU lift from q4 to q8 is below most users' noise floor, but if you're using it for math or code where edge cases matter, q8_0 is worth the extra few GB of memory and ~10% generation-speed hit.

Common pitfalls

1. Treating 8B as a knowledge model. It's a small dense model — its world knowledge is 2024-ish and shallow on anything niche. Use it for transformation tasks (summarize, format, classify) and routing, not direct factual recall.
2. Not using a system prompt. Out of the box, Llama 3.1 8B is verbose and apologetic. A 3-line system prompt setting tone and brevity changes the outputs more than you'd expect.
3. Comparing against ChatGPT. Llama 3.1 8B is 1/100th the parameter count of GPT-4-class models. It's a different tool. Use it for what it's good at.
4. Running on an M3 Ultra for the wrong reason. If you're only ever running an 8B model, an M4 Max 36 GB at $2,500 is 80% of the M3 Ultra's tok/s for 30% of the price. The M3 Ultra makes sense when you have larger models in the rotation (see Llama 3.1 70B, DeepSeek-R1 32B).
5. Forgetting to set temperature=0 for code generation. The default temperature of 0.7–1.0 makes code generation flaky.

When NOT to use Llama 3.1 8B on M3 Ultra

• Cost-sensitive personal use. A $7,000 Mac Studio is not the right machine for a model that runs fine on a $1,200 Mac mini M4 Pro.
• Production serving. For >10 RPS, host the model on an L40S or H100 instance — much higher throughput, much more predictable latency.
• Anything requiring world knowledge past mid-2024. Combine with retrieval or use a larger model.
• Reasoning tasks where the chain matters. Use DeepSeek-R1 32B or Qwen 3 32B instead.

Worked example: classify-and-route service

A common 8B pattern: classify an incoming request and route to a larger model only when needed.

import requests

CLASSIFIER_PROMPT = '''You classify user prompts into one of:
SIMPLE - chat, formatting, lookup
REASONING - multi-step problem
CODE - generate or debug code
Respond with ONE word.

User: {user}
Class:'''

def classify(user_msg: str) -> str:
 r = requests.post("http://localhost:11434/api/generate", json={{
 "model": "llama3.1:8b",
 "prompt": CLASSIFIER_PROMPT.format(user=user_msg),
 "options": {{"temperature": 0, "num_predict": 8}},
 "stream": False,
 }})
 return r.json()["response"].strip().split()[0]

# Route accordingly
cls = classify(user_msg)
target_model = {{
 "SIMPLE": "llama3.1:8b",
 "REASONING": "deepseek-r1:32b",
 "CODE": "qwen3:32b",
}}[cls]

The 8B model classifies in ~80 ms, then you only pay the 32B model's latency on the 30% of requests that need it. Cuts overall throughput cost 4–5× for a typical mixed workload.

Power, thermals, and what "Mac Studio idle" actually means

Running Llama 3.1 8B at q4_K_M on an M3 Ultra Mac Studio:

For an 8B model the Mac Studio never breaks a sweat — sustained generation for an hour barely raises the chassis temperature. That's the practical difference vs a desktop GPU box: you can leave the Mac Studio in a shared workspace and never hear it. Compare to an RTX 4090 tower at full load (350+ W and 40+ dBA fan noise), and the productivity difference of not having a leaf-blower next to your desk is real.

For laptop users on the M3 Max or M4 Max equivalents, Llama 3.1 8B pulls 18–35 W under generation. Battery life under sustained inference: roughly 5–6 hours on a 14" MBP, 7–9 hours on a 16" MBP. Plenty for coding-on-a-flight workflows where you don't want to commit to an API roundtrip.

Reproducible benchmarking with llama-bench

If you want to validate the numbers in this article (or any other Apple Silicon LLM article), llama-bench is the standard tool. It measures prompt-processing throughput (pp) and token-generation throughput (tg) separately, runs multiple repetitions, and reports standard deviation.

./build/bin/llama-bench \
 -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
 -ngl 99 \
 -p 128,512,2048 \
 -n 64,128,512 \
 -r 5

The -p 128,512,2048 flag tests prompt processing at three lengths; -n 64,128,512 tests generation at three lengths; -r 5 runs each combination five times. Output gives you a clean table with tokens/sec and ±sigma. Use that for comparison shopping between quantizations or chips — it's far more reliable than eyeballing ollama run latency.

For an apples-to-apples comparison against published numbers, also pin the q8 KV cache (-ctk q8_0 -ctv q8_0), disable the macOS GPU thermal throttle (impossible on Apple Silicon — you just have to wait until the chip is cold), and run on AC power. Battery-attached benchmarks see roughly 15–25% lower throughput because the chip enters a power-conservation profile.

TL;DR

• Any M3 Ultra Mac Studio runs Llama 3.1 8B at 80–110 tok/s — faster than you can read.
• The M3 Ultra is overkill for 8B alone; the bandwidth+memory headroom only pays off when you're also running 32B–70B models or many parallel sessions.
• Start with Ollama, move to MLX for the last 20% speed boost.
• Use q4_K_M as your default; consider q8_0 on math/code if you have memory to spare.
• The Mac Studio is silent under any 8B workload — leaving it always-on next to your desk is a non-issue.