Skip to content
ginsberg.si

Transformers from the Weights Up/Part 4 of 20

Transformers from the Weights Up, Part 4: Layers as matrix multiplies: FLOPs, bytes, and arithmetic intensity

A layer is a handful of matrix multiplies. What decides speed is whether the chip is doing those FLOPs or waiting on the weights.

3 Nov 2026

Image 4 is σ(Wx + b) for one input. Stack M inputs (tokens, from any mix of batch and sequence, as Lesson 2 showed) as the rows of X and the same weights serve all of them in one matrix-matrix product. The bias add and the activation touch M · N elements, a rounding error next to the 2MNK of the matmul, and Llama drops biases entirely.

IMAGE 04x1 0.30x2 −0.60x3 0.80x4 0.50x5 0.90unit 1unit 2unit 3unit 4unit 5unit 6unit 7unit 8unit 9unit 10unit 11unit 12unit 13unit 14unit 15unit 16W is 16×5y = σ(Wx + b)
Image 4. Five inputs fully connected to 16 units: y = σ(Wx + b), with W a 16×5 matrix.
(r) One matmul, one multiply-add at a time
A [3, 4]B [4, 5]C = A·B [3, 5]1-20321-10-102120-1131-120-2031-21-120102··············FLOPs so far2 / 120Distinct elements touched3·4 + 4·5 + 3·5 = 47Intensity if each is moved once (bf16)120 / (47·2 B) = 1.28NowC[0,0] += A[0,0]·B[0,0]

C = A·B with A [M, 4] and B [4, 5]; small integers so you can check by hand. Every output is a K-long dot product, K multiplies and K adds, so the whole thing is 2·M·N·K FLOPs. Each A element is reused N times and each B element M times; that reuse is what arithmetic intensity measures. Switch to M = 1 (one token, the decode case) and B's elements are used only once each.

Counting FLOPs: 2N forward, 6N training

Each weight wᵢⱼ appears in exactly one product per token, wᵢⱼ · xⱼ, and that product is added to an accumulator. One multiply plus one add is 2 FLOPs (an FMA counts as 2, which is also how NVIDIA quotes peak numbers). Summed over every matmul weight, the forward pass costs 2 FLOPs per parameter per token, plus attention's context-dependent term from Lesson 2.

Forward

Llama-3-8B: 2 × (6.98B layer weights + 0.53B lm_head) ≈ 15.0 GFLOP. The 0.53B embedding is a row lookup, not a matmul.

Backward

Two matmuls per layer: one for the gradient with respect to the input (to keep backpropagating), one for the gradient with respect to the weights. Each costs as much as the forward one.

Training total

D is the number of training tokens. Meta reports Llama 3 was pretrained on over 15T tokens, so the 8B model is about 6 × 8.03 × 10⁹ × 1.5 × 10¹³ ≈ 7.2 × 10²³ FLOPs.

Bytes, not FLOPs: arithmetic intensity

A chip has two budgets: peak compute π (FLOP/s) and memory bandwidth β (bytes/s). An operation's arithmetic intensity I is FLOPs done per byte moved between memory and the compute units. The roofline model (Williams, Waterman and Patterson, 2009) says the best you can do is the smaller of the two limits.

b is bytes per element (2 in bf16). The ridge point π/β is the intensity a kernel needs to keep the math units busy: 989 × 10¹² ÷ (3.35 × 10¹²) ≈ 295 FLOP/byte on an H100 SXM, and 312 ÷ 2.039 ≈ 153 on an A100. Now plug in the two shapes that matter.

(s) The roofline: attainable FLOP/s vs arithmetic intensity
11010010001101001000arithmetic intensity (FLOP per byte moved)TFLOP/smemory-boundcompute-boundridge ≈ 295 FLOP/Bdecode, 1 tokenprefill, 2048 tokens
Intensity at this M
1.00 FLOP/B ≈ M when M ≪ 4096
Attainable throughput
3.35 TFLOP/s (0.3% of peak)
Regime
memory-bound: more tokens per weight read are nearly free

Log-log axes. The slanted roof is bandwidth × intensity, the flat roof is peak dense bf16 tensor throughput (H100 SXM 989 TFLOPS and 3.35 TB/s; A100 80GB SXM 312 TFLOPS and 2,039 GB/s, from NVIDIA's spec sheets). The dot is Llama-3-8B's gate_proj (4096 → 14336) run on M tokens at once, with weights, inputs and outputs each moved once in bf16. Real kernels reach maybe 80–85% of the flat roof.

A real layer on the roofline

Here are Llama-3-8B's seven per-layer weights with the simplest roofline time estimate. At M = 1 every matmul is memory-bound and the layer costs the time to read 436.2 MB. Raise M to 128 and the time barely moves, because the same bytes now serve 128 tokens: that is why serving engines batch many users' decode steps together. Past the ridge, time grows linearly with M and you are buying FLOPs.

(t) One Llama-3-8B layer's seven matmuls on a roofline budget
q_proj[4096, 4096]AI 1.0010.0 µsk_proj[1024, 4096]AI 1.002.5 µsv_proj[1024, 4096]AI 1.002.5 µso_proj[4096, 4096]AI 1.0010.0 µsgate_proj[14336, 4096]AI 1.0035.1 µsup_proj[14336, 4096]AI 1.0035.1 µsdown_proj[4096, 14336]AI 1.0035.1 µs
Layer FLOPs
436.2 MFLOP
Layer bytes moved
436.4 MB (weights alone 436.2 MB)
Layer time, × 32 layers
130.3 µs × 32 = 4.17 ms
Time per token
4.17 ms (240 tokens/s through the layers)

Time per matmul = max(FLOPs ÷ peak, bytes ÷ bandwidth), the simplest roofline estimate, with weights, inputs and outputs each moved once in bf16. Brass bars are memory-bound, blue bars are compute-bound. Attention scores, norms and SiLU are left out; Lesson 13 adds the KV-cache reads that decode also pays.

RegimeTypical MBound byWhat helps
Single-user decode1Memory bandwidth (I ≈ 1)Fewer bytes per weight (quantization), faster memory, speculative decoding
Batched serving decodetens to hundredsBandwidth until M nears the ridgeContinuous batching; KV-cache memory becomes the limit (Lesson 13)
Prefill / trainingthousandsCompute (I in the hundreds to thousands)Tensor cores, lower-precision math (fp8, Lesson 15), better kernels

Why decode is memory-bound, and the tokens/s ceiling

Generating one token at batch 1 means one matrix-vector product against every weight matrix. Each weight is read from memory, used for 2 FLOPs, and not needed again until the next token. So the time per token is at least the bytes of the model divided by bandwidth, no matter how many TFLOPS the chip advertises.

The 15.01 GB is everything streamed per step: 32 layers × 436.2 MB plus lm_head's 1.05 GB (the embedding contributes one 8 KB row). Compute for the same token is 15.0 GFLOP, or 15.2 µs at the H100's peak, against 4.48 ms to read the weights. That 0.3% utilization is the single most important number for understanding local inference.

(u) Decode: every token streams the whole model through the chip
HBM / unified memory: weights at restlm_headcompute units: 2 FLOPs per weight, then that weight is not needed again this token
Bytes read per token
7.505B weights × 16 bits ÷ 8 = 15.01 GB
Ceiling, batch 1
3,350 GB/s ÷ 15.01 GB = 223 tokens/s (4.5 ms/token)
Tokens generated (slowed)
0

Each block is one of Llama-3-8B's 32 layers (436.2 MB each in bf16); the last is lm_head (1.05 GB in bf16). The embedding is a single-row lookup per token, so it is not streamed. Ceiling = bandwidth ÷ bytes read per token, at batch 1, ignoring KV-cache reads and kernel overhead, so real numbers are lower. Bandwidths: H100 SXM 3.35 TB/s, A100 80GB SXM 2,039 GB/s, RTX 4090 1,008 GB/s, M2 Ultra 800 GB/s. Q8_0 is exactly 8.5 bits/weight; Q4_K_M varies by model and is shown at an assumed 4.85. Animation slowed 100×.

Lesson 4: key numbers

2MNK
FLOPs of an M by K times K by N productone multiply plus one add per term
2N / 6N
FLOPs per tokenforward, and training (forward plus backward)
≈ 1
FLOP/byte, bf16 decodeeach weight: 2 bytes, 2 FLOPs
≈ M
intensity with M tokenswhile M is much smaller than K and N
295
H100 SXM ridge (FLOP/byte)989 TFLOPS ÷ 3.35 TB/s
153
A100 80GB ridge (FLOP/byte)312 TFLOPS ÷ 2,039 GB/s
15.01 GB
Llama-3-8B bytes per decode stepbf16, layers plus lm_head
≈ 223 tok/s
H100 batch-1 ceiling, bf16real numbers land below this

Lesson 4: self-check

1. How many FLOPs is Llama-3-8B's gate_proj applied to a 2048-token prompt?

Answer. 2 × 2048 × 4096 × 14336 = 240,518,168,576 ≈ 240.5 GFLOP, for one matrix in one layer.

2. Why is a bf16 matrix-vector product's arithmetic intensity about 1, independent of matrix size?

Answer. Every weight is 2 bytes, is read once, and takes part in exactly one multiply-add (2 FLOPs). The input and output vectors are tiny next to the matrix, so FLOPs divided by bytes is about 2NK divided by 2NK, which is 1.

3. What is the batch-1 tokens per second ceiling for Llama-3-8B in Q8_0 on an A100 80GB?

Answer. 7.505B streamed weights × 8.5 bits ÷ 8 ≈ 7.97 GB per token. 2,039 GB/s ÷ 7.97 GB ≈ 256 tokens/s, ignoring KV-cache reads and overhead.

4. An H100 serves 64 users' decode steps in one batch. Roughly how do time per step and total throughput change versus one user?

Answer. Intensity is about 64, still well below the ridge of about 295, so the step stays memory-bound and takes about as long as one user's (weights dominate the bytes). Throughput rises nearly 64 times. In practice each user's KV cache adds its own reads, which Lesson 13 accounts for.

5. Training uses about 6N FLOPs per token. Where do the extra 4N come from?

Answer. Backward needs two matmuls per forward matmul: the gradient with respect to X is the gradient with respect to Y times W (to pass gradients down), and the gradient with respect to W is the transpose of the gradient with respect to Y times X (to update the weight). Each costs the same 2MNK as the forward product.

6. Fusing SiLU of the gate, times up, into the matmul saves no FLOPs. What does it save, and why does that matter in the decode regime?

Answer. Memory traffic: gate and up (M by 14336 each) are never written out and read back. When the chip is waiting on bytes, removing bytes is the only optimization that speeds things up.

Lesson 4: resources