How to run Llama 3.1 70B on Apple M4 Max

How to run Llama 3.1 70B on Apple M4 Max

Llama 3.1 70B runs natively on an Apple M4 Max with no CPU offloading — provided you have at least 48 GB of unified memory. At Q4_K_M the model weighs ~40.5 GB and at Q5_K_M about 50 GB, so the 64 GB, 96 GB, and 128 GB SKUs handle it easily and the 48 GB SKU works at Q4_K_M with a tight budget. Expect 15–22 tokens per second on the 40-core GPU variant using Ollama's Metal backend. The whole setup is brew install ollama && ollama run llama3.1:70b — most of this guide is about choosing the right quant, configuring context, and knowing where the M4 Max wins versus a discrete GPU.

What "running 70B" actually takes

Meta's Llama 3.1 70B has 70.6 billion parameters across 80 hidden layers with a native 128k context. It's the open-weights flagship as of 2026 — comparable to GPT-3.5-class proprietary models on most benchmarks, with significantly stronger code and reasoning than its 8B and 13B siblings. The challenge is purely memory: at FP16 it weighs ~140 GB, far above any consumer GPU; even Q8_0 lands around 75 GB.

That's where the M4 Max is interesting. Apple's late-2024 Max chip ships with 36, 48, 64, 96, or 128 GB of unified memory — the latter two configurations can hold the model entirely on one chip with room left for KV cache and macOS. The bandwidth is 546 GB/s on the 40-core GPU and 410 GB/s on the 32-core part, which at small batch sizes is the binding constraint for inference. The architecture pays off: no PCIe transfers, no VRAM/RAM split, no offloading bookkeeping.

Memory budget per quant

Llama 3.1 70B at common quants:

Sizing by Mac:

• 36 GB M4 Max — IQ3_XXS only. Quality is OK for chat, weak for code and reasoning. Don't run 70B on this SKU as your primary workflow.
• 48 GB M4 Max — Q4_K_M with 8k context. Tight; close other heavy apps. Q5_K_M doesn't reliably fit alongside macOS.
• 64 GB M4 Max — Q4_K_M with 32k context, or Q5_K_M with 8k context. Comfortable.
• 96 GB / 128 GB M4 Max — Q5_K_M or Q6_K with 32k–131k context. Q8_0 fits on 128 GB. You're memory-rich.

For 70B specifically, quality drops noticeably below Q4_K_M; Q5_K_M is the sweet spot when memory allows. The KV cache scales linearly with context — at 131k context (Llama 3.1's native ceiling) the cache alone is ~40 GB at FP16, which is why anything beyond casual long-context work on 70B wants quantized KV cache via --kv-cache-type-k q8_0 --kv-cache-type-v q8_0.

Step 1 — Install Ollama and pull a quant

Ollama ships Metal support on macOS:

brew install ollama
brew services start ollama

# Pull Llama 3.1 70B at Q4_K_M (the community default):
ollama pull llama3.1:70b
# Or, explicitly Q5_K_M for slightly better quality:
ollama pull llama3.1:70b-instruct-q5_K_M

The Q4_K_M download is ~40 GB; budget 5–10 minutes on a fast connection. Once it's cached, run it:

ollama run llama3.1:70b
>>> Walk me through how vector clocks resolve concurrent edits in a CRDT.

First-prompt latency is ~15–25 s on the cold cache (Metal kernel compile + first weight stream into the GPU page table). Subsequent prompts are streaming-fast.

Step 2 — llama.cpp for direct control

For prompt-cache reuse, custom sampling, or building your own daemon, drive llama.cpp directly:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

huggingface-cli download bartowski/Meta-Llama-3.1-70B-Instruct-GGUF \
 Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf --local-dir ./models

./build/bin/llama-cli \
 -m models/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf \
 --gpu-layers 999 \
 --ctx-size 8192 \
 --kv-cache-type-k q8_0 \
 --kv-cache-type-v q8_0 \
 --threads 8 \
 --temp 0.6 --top-p 0.9 \
 -p "Explain Raft leader election in plain English."

--gpu-layers 999 keeps all 80 layers on the GPU; the value is clamped internally. The quantized KV-cache flags shave ~50% off cache memory at negligible quality cost — a near-free win on 70B.

Step 3 — Use Llama 3.1's long context wisely

Llama 3.1 70B has a 128k native context. At Q4_K_M with an unquantized KV cache, a 32k context allocates ~10 GB just for the cache; 131k allocates ~40 GB. For 64 GB and 96 GB SKUs that means:

# Sensible default for a coding agent on 64 GB M4 Max:
./build/bin/llama-cli \
 -m models/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf \
 --ctx-size 32768 \
 --kv-cache-type-k q8_0 --kv-cache-type-v q8_0 \
 --gpu-layers 999 -p "..."

For RAG over a large corpus, switch to a 96/128 GB SKU and you can keep a 64k or 131k context resident without paging. The bigger lever, though, is prompt caching — see the Pro Tip section below.

Real-world numbers

Measurements from an M4 Max 16-core CPU / 40-core GPU / 64 GB unified memory, macOS 15.3, Ollama 0.4.6, Q4_K_M, 4096-token context:

The 32-core GPU M4 Max trails by ~25%, landing around 13–15 tok/s at Q4_K_M. The M3 Max 40-core posts ~16–17 tok/s on the same workload. The M2 Ultra Mac Studio (800 GB/s bandwidth, 192 GB max unified) lands at ~28 tok/s — the bandwidth advantage shows.

Compared to discrete GPUs: an RTX 3090 24 GB requires CPU offload for 70B Q4_K_M (the weights are 40 GB, the card is 24) and lands around 13 tok/s. An RTX 4090 24 GB hits ~17 tok/s with offload. A dual-3090 rig with NVLink reaches ~28 tok/s by sharding the model across two cards. An H100 80 GB SXM at FP8 posts ~75 tok/s. The M4 Max isn't fastest, but it's the only consumer machine that fits 70B at Q5_K_M without thinking about layer offload, and the 64–128 GB SKUs are widely available off the shelf.

Common pitfalls

• Running on the 36 GB SKU. Even IQ3_XXS pushes memory pressure into yellow under normal macOS. Plan for at least 48 GB if 70B is your primary model.
• Q5_K_M on 48 GB. Resident memory is ~52 GB. macOS will swap heavily; tok/s drops by 40% and you'll see beachballs. Stick to Q4_K_M on this SKU.
• Unquantized KV cache at long context. 32k context at FP16 KV cache is ~10 GB; 131k is ~40 GB. Always set --kv-cache-type-k q8_0 --kv-cache-type-v q8_0 for 70B unless you have a specific quality reason not to.
• Forgetting to plug in. On battery, GPU clocks throttle 25–40%. Tok/s drops from 18 to ~12.
• Too many threads. M4 Max has 4 efficiency + 12 performance cores. --threads 16 includes the e-cores and slows the run. Use --threads 8 or 0 (auto).
• Mixing fine-tunes. Community fine-tunes (Hermes, Dolphin, etc.) often ship as different GGUFs; Ollama and llama.cpp template detection can drift. If outputs look weirdly verbose or skip the assistant turn, run ollama show <tag> --modelfile and confirm the TEMPLATE matches Llama-3-Instruct's <|begin_of_text|><|start_header_id|>system<|end_header_id|> format.

When not to do this

If you need 70B at peak throughput for a multi-user service, an M4 Max is the wrong tool — the architecture serves one user at a time well, but doesn't multiplex. A pair of used RTX 3090s on a $1500 PC + tensor parallelism via vLLM beats it for shared workloads. For a single-user coding assistant, the M4 Max 64 GB or 96 GB is the most ergonomic option on the market today.

If you'd rather get peak quality and don't care about local: the OpenAI o1 and Anthropic Claude 4.7 Opus APIs run circles around any local 70B for $10–$30 per million tokens. Local 70B wins on privacy, offline, and steady-state cost (no per-token API spend); it loses on raw quality and on long-context cost.

For smaller models on the same hardware, see Llama 3.1 8B on M4 Max, Qwen 3 14B on M4 Max, or Qwen 3 32B on M4 Max. For the same 70B workload on a $250 GPU, see Llama 3.1 70B on Arc B580 — slower, cheaper, requires heavy CPU offload.

Power, heat, and the laptop-vs-Studio decision

A sustained Llama 3.1 70B Q4_K_M session on a 16" MacBook Pro M4 Max pulls 70–90 W from the wall, stabilizes at package temperatures of 95–98 °C, and runs the fan at a clearly-audible level after 3–4 minutes of load. On the 14" form factor expect about 5–10% lower steady-state tok/s due to throttling and a more frantic fan curve.

If 70B is your primary model and you'll run it for hours each day, the Mac Studio M4 Max is the better long-term host. The chassis cools the same silicon meaningfully better, the fan is large enough to stay nearly inaudible, and there's no battery to worry about. For a static workstation use, the Studio's price-per-tok/s is also better — the 64 GB Studio M4 Max comes in around $2,500, where a 64 GB 16" MacBook Pro is $3,500.

Across M-series generations on 70B Q4_K_M tok/s: M2 Ultra 76-core (~28) > M4 Max 40-core (~18) ≈ M3 Max 40-core (~17) > M4 Max 32-core (~14) — and that's it; the Pro chips and below aren't realistic 70B hosts due to memory caps. The Ultra's 800 GB/s bandwidth and 192 GB max unified memory put it in a different tier; if your budget reaches that ceiling, an M2 or M3 Ultra Mac Studio is the consumer-tier ceiling for 70B local.

Use-case fit at the 70B mark

70B is where local open-weights models start to seriously compete with proprietary APIs on reasoning and code. Llama 3.1 70B specifically posts MMLU and GSM8K numbers within a few points of GPT-4o, with HumanEval near 80%. In practical terms:

Strong fits:

• Multi-file code refactoring with the full 128k context and prompt caching.
• Long-form writing where tone control and consistency matter (technical docs, marketing copy, fiction).
• Multi-step agentic workflows with 10+ tools and complex routing.
• Bilingual or multilingual document work.
• Privacy-critical workloads where API calls aren't an option.

Marginal fits (consider an API):

• Sub-second first-token latency for a chat UX. Even with prompt caching, 70B's per-token speed is the binding constraint at ~18 tok/s.
• Multi-user serving on a single Mac — possible up to 2 concurrent users on 128 GB SKUs, but tok/s halves per user.

Not a fit:

• Real-time voice or TTS pipelines where target latency is under 200 ms.
• Cost-sensitive workloads — at $0.50/day for an equivalent API call rate, the API is cheaper than amortized hardware for most single-user cases.

Concurrent model loadout on 96/128 GB

The killer use of a 96 GB or 128 GB M4 Max is keeping multiple models hot. A practical loadout for a serious local-AI workstation:

ollama pull llama3.1:70b # Q4_K_M, ~40 GB
ollama pull qwen3:14b # Q5_K_M, ~10 GB — fast assistant
ollama pull nomic-embed-text # ~270 MB — embeddings for RAG
# Optional 4th model:
ollama pull qwen3:14b-instruct-q5_K_M # specialized chat

Total working set is around 55 GB with all three loaded, leaving 70+ GB on a 128 GB SKU for OS, browsers, IDE, and headroom. The Ollama daemon serves all three concurrently; route requests to the right model via the model field in the API. Set OLLAMA_KEEP_ALIVE=24h to keep them all resident across the working day.

This is the "private personal AI" platform's strongest argument: a single machine, no API contracts, no cloud round trips, and you can ask a fast model first then escalate to 70B when you need depth. Nothing in the consumer GPU world delivers this experience as smoothly.

Pro tip: prompt caching is non-negotiable at 70B

70B's prefill cost is steep — a 2k-token prompt eats ~11 s before the first output token, and a 16k-token RAG prompt eats ~90 s. If your workflow reuses a long prefix (system prompt, codebase index, document set), llama.cpp's --prompt-cache flag saves the prefill KV cache to disk so subsequent calls with the same prefix skip almost all prefill:

./build/bin/llama-cli \
 -m models/Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf \
 --prompt-cache ./caches/codebase-index.bin \
 --prompt-cache-all \
 --gpu-layers 999 \
 --ctx-size 32768 \
 --kv-cache-type-k q8_0 --kv-cache-type-v q8_0 \
 -f codebase-system-prompt.txt \
 -p "Find every place we mutate auth state outside the reducer."

The first invocation builds the cache; subsequent invocations re-read it in well under a second of effective prefill. Ollama's daemon-side prefix caching (0.4.5+) does the same thing transparently — confirm by running OLLAMA_DEBUG=1 ollama serve and watching for prefix cache hit log lines after the second prompt with the same prefix.

For agent workflows where the system prompt rarely changes but the task does, this is the difference between a 12 s "thinking" pause and a 1 s pause — and at 70B it matters a lot more than at 8B.

Sources

• Meta — Llama 3.1 model card (parameters, context, benchmarks)
• Apple — M4 Max product specs (memory bandwidth, GPU cores)
• Ollama and the Ollama install script
• llama.cpp — Metal backend
• llama.cpp KV-cache quantization discussion — and its follow-up benchmark
• vLLM — for tensor-parallel comparison with discrete-GPU rigs
• Community benchmarks: 21 LLMs on MacBook Air M5 and general threads at r/LocalLLaMA