loom
Zero-dependency deep-learning compiler in C++23 that JITs GPT-2 to AArch64 machine code
SIM · real instruction words, random operands
- 114.8 GFLOP/s measured result SGEMM, 1 thread 768×768×3072 fp32 on an M3 Pro, best of 5 runs: ~89% of the P-core’s theoretical FMA peak, 55× a naive -O3 C++ loop
- ~200 tok/s measured result GPT-2 decode, 4 threads fp32, batch 1, greedy, 4 threads on an Apple M3 Pro; rounded from single runs that vary with load (README: ~190–220 tok/s)
- 2.9 KB measured result machine code per decode step, 9 kernels 9 deduplicated kernels: the 12 identical layers reuse one copy of each kernel
- 6.4× measured result buffer reuse Decode program: 0.20 MB arena vs 1.31 MB raw
Problem
loom asks what it takes to run a real transformer with no BLAS, no libtorch and no assembler. Every layer (tensor IR, fusion passes, memory planner, kernels, instruction encoder, executable memory, tokenizer and model loader) has to be built and tested in-repo, and the result has to run GPT-2 124M.
Approach
GPT-2 is expressed in an 11-op, transformer-altitude tensor IR; each prompt length gets its own shape-specialized Program, compiled in under a millisecond and cached, and every decode step reuses the T = 1 Program. Compiler passes fuse the residual add into LayerNorm, fold bias into the matmul epilogue, lower GELU to one separate fused elementwise pass, choose GEMV for T = 1 and SGEMM otherwise, and deduplicate kernels by descriptor. A liveness-based planner sorts buffers by size and places each at the lowest 64-byte-aligned arena offset that doesn’t collide with a buffer live at the same time. Kernel emitters (12×8 SGEMM microkernel, packed GEMV, LayerNorm, causal softmax, attention scores/mix, and a fused elementwise SSA emitter with an inlined polynomial exp that tanh and GELU build on) drive loom’s own AArch64 encoder, and each Program’s kernels are copied into one MAP_JIT block with the W^X toggle and i-cache invalidation that Apple Silicon requires. Threads split work only on output-disjoint ranges, so each element’s reduction order is fixed regardless of thread count, and a scalar fp64 interpreter runs the same Program as the correctness referee.
Result
On an Apple M3 Pro, GPT-2 124M decodes at ~200 tokens/s (fp32, batch 1, greedy, 4 threads), and each decode step runs 9 deduplicated kernels totaling 2.9 KB of machine code. Single-thread SGEMM reaches 114.8 GFLOP/s on 768×768×3072 (~89% of the P-core’s theoretical fp32 FMA peak), and the GEMV kernel streams packed weights at 90.7 GB/s on the 768×3072 decode shape. On real GPT-2 weights, 20 greedy tokens from the JIT match the fp64 interpreter exactly, and CI runs the test suite under ASan+UBSan and the threaded runtime under TSan.
Highlights
- Own AArch64 instruction encoder, built from the Arm ARM field diagrams: 82 instructions cross-checked word-for-word against system clang, 0 mismatches
- 12×8 NEON SGEMM microkernel (24-register accumulator tile): 114.8 GFLOP/s single-thread, ~89% of an M3 Pro P-core’s theoretical FMA peak, 55× naive -O3 C++
- Whole GPT-2 decode step compiles to 9 deduplicated kernels, 2.9 KB of machine code; the liveness planner packs 1.31 MB of buffers into a 0.20 MB arena
- 20 greedy GPT-2 tokens from the JIT match an fp64 interpreter twin exactly; JIT prefill (SGEMM) and token-by-token decode (GEMV) logits agree to 6e-5
- Work splits only on output-disjoint ranges (tests: bitwise-equal logits at 1 vs 4 threads); a thread-pool use-after-free was fixed, now guarded under TSan
Lab note
Folding GELU into the GEMV weight-streaming loop measured 4% slower at decode than a separate fused pass, so the compiler keeps it separate.
- Model I/O
- mmap safetensors loader, own JSON parser, byte-level BPE tokenizer; GPT-2 124M weights repacked once at load into the blocked layout GEMV and SGEMM share
- to Tensor IR — weights + token ids
- Tensor IR
- 11 transformer-altitude ops; one Program per prompt length, compiled once and cached (XLA-style, sub-ms compile); decode reuses the T=1 Program
- to Compiler passes — graph
- Compiler passes
- residual-Add+LayerNorm fusion · bias in matmul epilogue, GELU as one separate fused pass · GEMV (T=1) vs SGEMM · kernel dedup by descriptor
- to Liveness memory planner — step schedule
- to Kernel emitters — op descriptors
- Liveness memory planner
- live intervals over the step schedule → largest buffer first, each at the lowest free 64-B-aligned arena offset (6.4× reuse on decode)
- to JIT executor — arena offsets
- to fp64 interpreter twin — same schedule + arena
- Kernel emitters
- 12×8 SGEMM microkernel, packed GEMV, lm_head GEMV, LayerNorm, causal softmax, attention scores/mix, fused elementwise SSA (polynomial exp; tanh and GELU built on it)
- to A64 encoder → MAP_JIT — encoder calls
- A64 encoder → MAP_JIT
- raw 32-bit words, cross-checked vs clang; one JitBlock per Program with W^X flip + i-cache flush; decode step = 9 kernels / 2.9 KB
- to JIT executor — machine code
- JIT executor
- thread pool splits work on output-disjoint ranges (bitwise-equal logits at 1 vs 4 threads in tests); KV-cached decode, seeded fp64 sampler
- fp64 interpreter twin
- runs the same Program with scalar fp64 reference math: the correctness referee
- to JIT executor — differential check: 20 greedy GPT-2 tokens identical
| Run | tok/s |
|---|---|
| 1 thread | 143 |
| 4 threads | 193 |
| 8 threads | 165 |
| Run | GFLOP/s |
|---|---|
| loom | 114.8 |
| naive -O3 C++ | 2.1 |
Limits
fp32, batch 1, GPT-2 124M, AArch64.
Stack
- C++23 (no third-party libraries)
- AArch64 + NEON (self-emitted machine code)
- MAP_JIT / W^X on Apple Silicon
- CMake + Ninja
- ASan + UBSan, TSan
- GitHub Actions (macOS arm64 + Linux)
Numbers traced to loom @ 58c07ce.
Sources
- 114.8 GFLOP/s
SGEMM 768×768×3072 | 114.8 GFLOP/s | 2.1 GFLOP/s | 55×
loom / README.md @ 58c07ce - ~200 tok/s
decode at 4 threads varies ~190–220 tok/s run to run with ambient load
loom / README.md @ 58c07ce - 2.9 KB
The entire GPT-2 decode step compiles to 9 deduplicated kernels — 2.9 KB of machine code
loom / README.md @ 58c07ce - 6.4×
6.4× reuse on the decode program (0.20 MB arena vs 1.31 MB raw)
loom / docs/DESIGN.md @ 58c07ce - Lab note
was implemented, measured 4% slower at decode (constant reloads and a long dependency chain inside the weight-streaming loop), and removed
loom / docs/DESIGN.md @ 58c07ce