Engineering notes

Recent writing

1. 01

   Part 1 of 20

   Transformers from the Weights Up, Part 1: The 30,000-Foot View

   Text in, token out, and what a 50 GB weights file really is

   1 Nov 2026

2. 02

   Part 2 of 20

   Transformers from the Weights Up, Part 2: Dot products, the neuron, and shapes

   x·w as similarity and projection, and every layer as tensors flowing through matmuls.

   1 Nov 2026

3. 03

   Part 3 of 20

   Transformers from the Weights Up, Part 3: Activation functions and nonlinearity

   Why a stack of linear maps collapses, and how sigmoid, ReLU, GELU, and SwiGLU keep depth.

   2 Nov 2026

4. 04

   Part 4 of 20

   Transformers from the Weights Up, Part 4: Layers as matrix multiplies: FLOPs, bytes, and arithmetic intensity

   A layer is a handful of matrix multiplies. What decides speed is whether the chip is doing those FLOPs or waiting on the weights.

   3 Nov 2026

5. 05

   Part 5 of 20

   Transformers from the Weights Up, Part 5: MLPs and what the FFN stores

   Each FFN hidden unit is a memory slot: gate and up rows are keys, and a down_proj column is the value written back.

   4 Nov 2026

6. 06

   Part 6 of 20

   Transformers from the Weights Up, Part 6: Training: cross-entropy, backprop, AdamW

   Training is forward, cross-entropy on the next token, one backward sweep, and an AdamW step. That is where the 6N FLOPs and the 16 bytes per parameter come from.

   4 Nov 2026

7. 07

   Part 7 of 20

   Transformers from the Weights Up, Part 7: Tokenization (BPE) and embeddings

   A tokenizer turns text into integers, and embed_tokens turns each integer into a vector by row lookup. The vocabulary quietly sets sequence length, a slice of the parameter budget, and some of a model's blind spots.

   5 Nov 2026