Engineering notes

Recent writing

1. 01

   Part 1 of 20

   Transformers from the Weights Up, Part 1: The 30,000-Foot View

   Text in, token out, and what a 50 GB weights file really is

   1 Nov 2026

2. 02

   Part 2 of 20

   Transformers from the Weights Up, Part 2: Dot products, the neuron, and shapes

   x·w as similarity and projection, and every layer as tensors flowing through matmuls.

   1 Nov 2026

3. 03

   Part 3 of 20

   Transformers from the Weights Up, Part 3: Activation functions and nonlinearity

   Why a stack of linear maps collapses, and how sigmoid, ReLU, GELU, and SwiGLU keep depth.

   2 Nov 2026

4. 04

   Part 4 of 20

   Transformers from the Weights Up, Part 4: Layers as matrix multiplies: FLOPs, bytes, and arithmetic intensity

   A layer is a handful of matrix multiplies. What decides speed is whether the chip is doing those FLOPs or waiting on the weights.

   3 Nov 2026

5. 05

   Part 5 of 20

   Transformers from the Weights Up, Part 5: MLPs and what the FFN stores

   Each FFN hidden unit is a memory slot: gate and up rows are keys, and a down_proj column is the value written back.

   4 Nov 2026