Training an LLM in Swift, Part 1: Matrix Mult Speed

Matrix multiplication is the single most expensive operation in transformer training — the bulk of an LLM's forward and backward pass is, mechanically, a long sequence of GEMM (general matrix multiply) calls. Before any Swift-based training loop can be worth building, that core kernel has to move off a naive loop and onto hardware-appropriate vectorized code. This first installment stays narrowly focused on that jump: what a naive Swift matrix multiply actually costs, where the time goes, and which concrete changes move throughput from single-digit Gflop/s into Tflop/s territory.

Why Swift Is a Serious Option for the Numerical Core

Swift is not a typical choice for ML research code, and Python's ecosystem — PyTorch, JAX, Hugging Face — is not going anywhere. But Swift has one underrated advantage for exactly this kernel-level work: it compiles to native code with no interpreter overhead, and on Apple platforms it has first-class access to the same vendor-tuned linear algebra libraries that Python libraries eventually call into anyway.

Three pieces of the Swift/Apple stack matter for matrix multiplication specifically:

• Accelerate — Apple's system framework exposing BLAS and LAPACK routines, vectorized for Apple silicon (Apple Developer documentation).
• Metal / Metal Performance Shaders Graph — the GPU compute path, for routing large GEMM operations onto the on-chip GPU (Apple Metal documentation).
• MLX and mlx-swift — Apple's array-programming framework for machine learning, with an official Swift package built around Apple silicon's unified memory model (MLX on GitHub, mlx-swift on GitHub).

Swift also has swift-numerics, Apple's open-source package for numeric protocols and SIMD-friendly types, which is the lower-level building block underneath hand-rolled vectorized code before reaching for Accelerate (swift-numerics on GitHub).

For internal comparison points on the GPU side of an LLM training rig — the piece that ultimately consumes whatever matrix-multiply throughput a training loop can generate — see Best GPU for LLM Training and Inference in 2026 and Dual RTX 3090 vs RTX 5090: Gaming vs AI Training.

From Naive Loops to Gflop/s: What a First Implementation Looks Like

A first-pass matrix multiply in any language, Swift included, is usually a triple-nested loop over row, column, and inner-product index. It is easy to write and easy to reason about — and it is almost always memory-bound rather than compute-bound. As matrix dimensions grow past what fits in L1/L2 cache, the naive access pattern re-reads rows and columns from progressively slower memory tiers, and the CPU's arithmetic units sit idle waiting on data far more often than they're actually multiplying and accumulating.

This is the standard story behind why raw Gflop/s figures for naive matrix multiply implementations, in any compiled language, tend to sit well below a chip's theoretical peak — the bottleneck is data movement, not arithmetic throughput. The fix is not a faster loop; it's a different memory-access pattern.

Profiling With Instruments

Before optimizing, the standard step is profiling. Xcode's Instruments tooling (part of the standard Xcode toolchain) attaches to a running Swift binary and reports where cycles actually go — cache misses, memory stalls, and time spent per function — rather than relying on guesswork about which loop is slow. For matrix multiplication specifically, this typically confirms the memory-bound diagnosis directly: a profile of a naive Swift GEMM implementation on any modern chip generally shows the CPU's actual multiply-accumulate units running well under capacity while memory-subsystem stalls dominate the timeline.

Memory Hierarchy: Why Tiling and Cache-Aware Layout Matter

The standard fix for a memory-bound matrix multiply is blocking (also called tiling): instead of iterating over entire rows and columns, the algorithm processes small sub-blocks of the input matrices that fit entirely within a cache level, computes a partial result, and moves to the next block. This keeps data resident in fast cache for longer, sharply reducing the number of round-trips to main memory for the same amount of arithmetic work.

This is precisely the technique vendor BLAS libraries implement internally, tuned per-architecture for the specific cache sizes and associativity of the target chip — which is the practical reason hand-written Swift loops rarely beat a well-tuned Accelerate call: the blocking parameters in Accelerate are already matched to the hardware it's running on.

A useful mental table for what each optimization stage is actually addressing:

For readers building a CPU-heavy workstation for this kind of kernel-level experimentation, Troubleshooting Ryzen 7 5800X Memory Training, Boot Loops, and Cold-Starts is a relevant companion piece — memory subsystem stability matters as much as raw clock speed once workloads become this cache-sensitive.

Vectorization: SIMD, Accelerate, and Metal

Once data movement is under control, the next lever is keeping arithmetic units fed with wide, single-instruction-multiple-data (SIMD) operations rather than scalar multiply-adds. Swift exposes this in two ways:

1. Direct SIMD types — Swift's standard library and swift-numerics package expose fixed-width SIMD vector types that map onto hardware vector registers, letting a developer write explicitly vectorized inner loops.
2. Accelerate's BLAS entry points — calling cblas_sgemm (single-precision) or cblas_dgemm (double-precision) hands the entire operation to Apple's pre-vectorized, pre-blocked implementation, which is the practical default for most Swift numerical code rather than re-deriving blocking and vectorization by hand.

For the largest matrices — the ones that actually resemble transformer weight matrices during training — the more direct throughput path is the GPU rather than the CPU. Metal Performance Shaders Graph exposes GEMM-equivalent operations that execute on the on-chip GPU, and MLX's Swift bindings build directly on top of this, giving Swift a GPU-backed tensor path without hand-written Metal shader code (mlx-swift).

From Gflop/s to Tflop/s: What Realistic Targets Look Like

The jump from a naive-loop implementation to a properly blocked, vectorized, BLAS-backed one is consistently the largest single step in this progression — moving from a memory-bound, mostly-idle execution unit to one that's actually saturated with useful arithmetic. The further jump from CPU-side Gflop/s into sustained Tflop/s generally requires moving the computation to the GPU, since GPU compute throughput is an order of magnitude higher than CPU throughput on the same chip package for this class of workload.

Apple's own published hardware specifications illustrate the scale of that gap: Apple stated the M2 Ultra's 76-core GPU configuration delivers up to 27.2 teraflops of GPU compute performance at launch, versus a CPU peak that is a small fraction of that figure (Apple Newsroom, M2 Ultra introduction). That gap is the practical argument for routing large-matrix GEMM operations through Metal rather than treating Accelerate's CPU-side BLAS as the finish line — Accelerate gets a naive implementation to a respectable, well-tuned Gflop/s baseline; the GPU path is what makes Tflop/s achievable.

Exact achievable throughput for any specific matrix shape, precision, and chip varies by workload and is worth benchmarking directly with Instruments rather than assumed from a spec sheet — theoretical peak and sustained real-world GEMM throughput are consistently different numbers.

Hardware Considerations for Swift-Based Training

Swift's tightest integration is with Apple silicon, where Accelerate, Metal, and MLX form a coherent stack sharing the same unified memory pool — a genuine architectural advantage for training workloads, since it avoids explicit host-to-device data copies between CPU and GPU memory that discrete-GPU systems require.

That said, most production LLM training still happens on discrete GPUs with a mature CUDA or ROCm ecosystem, and readers evaluating a dedicated training rig outside the Swift/Apple stack should weigh those options directly: Dual RTX 3090 LLM Training: 2026 Benchmarks & Build Guide covers a common budget-conscious multi-GPU configuration, while Intel Arc AI Training in 2026: What Workloads Actually Work covers a non-Nvidia alternative with a different maturity profile. For smaller-scale model training rather than full LLM pretraining, Best Budget GPU for CNN and Image-Model Training in 2026 and Best GPU for Training CNNs at Home in 2026 cover a lower-cost entry point relevant to the same matrix-multiply-bound workloads discussed here, just at smaller model scale.

For Swift-and-Metal-specific development, the practical reference implementation many projects build against or borrow techniques from is llama.cpp's Metal backend, which demonstrates GPU-accelerated inference (a closely related matrix-multiply-bound workload to training) on Apple silicon in production use (llama.cpp on GitHub).

Where Part 1 Leaves Off

The throughline here is that Swift's matrix-multiply performance ceiling is not set by the language — it's set by how directly a given implementation routes work through Accelerate's pre-tuned BLAS and Metal's GPU compute path versus a hand-written loop. Getting the kernel-level primitive right is a prerequisite, not the finish line: a fast GEMM call by itself doesn't train a model. The natural next step is wiring that primitive into an actual forward/backward training loop, where kernel-level Tflop/s gains either do or don't show up as reduced wall-clock training time — the subject for Part 2.

Citations and sources

• Apple Developer — Accelerate framework documentation
• Apple — Accelerate overview
• MLX — Apple's array framework for machine learning (GitHub)
• mlx-swift — Swift API for MLX (GitHub)
• swift-numerics (GitHub)
• Apple Developer — Metal
• Apple Newsroom — Apple introduces M2 Ultra
• llama.cpp (GitHub)

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.