Skip to content
ginsberg.si

Transformers from the Weights Up/Part 7 of 20

Transformers from the Weights Up, Part 7: Tokenization (BPE) and embeddings

A tokenizer turns text into integers, and embed_tokens turns each integer into a vector by row lookup. The vocabulary quietly sets sequence length, a slice of the parameter budget, and some of a model's blind spots.

5 Nov 2026

  1. 1

    Pre-tokenize

    A regex splits text into chunks (words with their leading space, runs of up to 3 digits, punctuation runs, whitespace). Merges never cross chunk boundaries.

  2. 2

    Bytes

    Each chunk becomes UTF-8 bytes. With 256 byte tokens as the base vocabulary, any string at all can be encoded: there is no unknown token.

  3. 3

    Merge

    Apply the learned merge list in rank order until no learned pair remains. Llama 3 has 128,000 regular tokens, built from its byte base by merges.

  4. 4

    Ids → vectors

    Add special tokens such as <|begin_of_text|> (id 128000), then gather rows of embed_tokens: [T] ints → [T, 4096] bf16. That tensor is the residual stream's starting point (image 7).

IMAGE 07xhyRMSNormAttention+residual addRMSNormFFN+residual addh = x + Attn(RMSNorm(x)), y = h + FFN(RMSNorm(h))
Image 7. Pre-norm residuals: RMSNorm and attention add back, then RMSNorm and the FFN add back.

Byte-pair encoding: learning the merges

BPE began as a 1994 compression trick and was adapted to neural machine translation by Sennrich, Haddow and Birch (2015). Training is greedy and purely statistical: start with a base alphabet, count every adjacent pair of symbols in the corpus (weighted by how often each pre-tokenized chunk occurs), merge the most frequent pair into a new symbol, record that merge, and repeat. The output is two artifacts: a vocabulary and an ordered merge list. The order matters, because encoding replays it.

(ae) Training BPE: count adjacent pairs, merge the most frequent, repeat
corpus segmentation after 0 merges×5low_×2lower_×6newest_×3widest_tokens in corpus: 95 · vocab: 11 base + 0 merges = 11pair counts (frequency-weighted)e + s9s + t9t + _9w + e8l + o7o + w7n + e6next: merge "e" + "s" → "es" (9 occurrences)merge list (rank = id order)

Toy corpus from Sennrich et al. (2015): 'low' ×5, 'lower' ×2, 'newest' ×6, 'widest' ×3, with _ marking the end of a word. Every count and merge is computed live; ties go to the pair seen first. Grey chips are base symbols; each colored chip is a token created by one merge, and its color matches its row in the merge list. Llama 3's tokenizer is the same loop run over bytes of a huge corpus until 128,000 tokens exist.

Step it to the end and look at what happened. The suffix "est_" was built first, because it is shared by two frequent words; "low" came next; "newest" became a single token, as did "low_". The rare word "widest" stays in pieces. That is the general shape at scale: common words and common subwords get their own ids, and rare words are spelled out of reusable fragments. Ten merges took the toy corpus from 95 symbol occurrences to 28.

Encoding: applying merges in rank order

Encoding a new string does not search for the "best" segmentation. Within each pre-tokenized chunk it finds the adjacent pair with the lowest merge rank, merges it everywhere in the chunk, and repeats until no learned pair is left. It is deterministic and fast (tiktoken and Hugging Face tokenizers do it in Rust), and it means a word's tokens depend on merge history, not on meaning.

(af) Encoding new text: apply the learned merges in rank order
using the first 10 of 10 mergeslowestlowest_ids [31, 29]newernewer_ids [33, 4, 17, 26]widerwider_ids [36, 3, 4, 17, 26]slowslow_ids [18, 35]
Tokens
13
Characters (incl. word ends)
24
Characters per token
1.85

Uses the merges learned in panel (ae). For each word, repeatedly find the adjacent pair with the lowest merge rank and merge it, until no learned pair remains. Words never seen in training ('slow', 'wider') still encode, as pieces. Toy alphabet: lowercase letters plus _ (ids 0–26); merge r gets id 27 + r. Other characters are dropped here; byte-level BPE never has to drop anything, because its 256 base tokens cover every byte.

Llama 3's pre-tokenizer is the regex below, from Meta's reference tokenizer.py. It attaches a word's leading space to the word (so " the" and "the" are different tokens), keeps English contractions as their own chunks, and caps digit runs at three, so "2026" is always split as "202" + "6" before any merge can see it.

(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+
Behavior you have seenMechanism in the tokenizer
Counting letters in a word is surprisingly hardThe model receives one id for a whole word or a few chunks; the spelling is never in its input and must be learned indirectly.
Arithmetic on long numbers is brittleDigits arrive in chunks of up to 3, and the chunk boundaries come from the left, so the same place value lands in different positions within its token.
Leading spaces and capitalization change outputs" Paris", "Paris" and " paris" are different ids with separately learned embeddings.
Non-English text costs more per wordFewer merges were learned for it, so the same content takes more tokens: more compute, more KV cache, less fits in context. Llama 3's extra 28K tokens target exactly this.
Glitch tokensA token that won a merge in the tokenizer's corpus but barely appears in the model's training data keeps a near-random embedding (next section). Land and Bartolo (2024) built detectors for these under-trained tokens across many open models.

The embedding table is a [V, d] lookup

embed_tokens.weight in Llama-3-8B is a [128256, 4096] bf16 tensor: 525,336,576 parameters, 1.05 GB. Row i is token i's learned vector. Conceptually the input is a one-hot vector eᵢ ∈ ℝ^V multiplied by the table, but nobody does that matmul: it selects one row, so the implementation is a gather.

(ag) The embedding table: a row gather forward, a sparse scatter-add backward
embed_tokens E [V, d]0 <bos>1 the2 cat3 sat4 on5 mat6 dog7 ran8 a9 rug10 moon11 ideaoutput X [T = 7, d = 8]t=0 id 0t=1 id 1t=2 id 2t=3 id 3t=4 id 4t=5 id 1t=6 id 5X[6] = E[5] = one-hot(5) · E ("mat")

Toy table, V = 12 tokens × d = 8, with made-up values (color = value, from the diverging palette). Forward: each id selects one row, which is exactly one-hot(id) · E. Backward into the embedding: only rows whose tokens appeared in the batch get a gradient, and 'the', used twice, gets the sum of both positions' gradients. Backward into an untied lm_head: p − y is dense, so every row gets a gradient on every step. Llama-3-8B's tables are [128256, 4096].

The original Transformer multiplied embeddings by √d_model; Llama does not scale them, and every block reads an RMSNorm-ed copy of the stream anyway (Lesson 11). Token embeddings carry no information about position: "the" at position 1 and position 400 start as the identical vector. Position enters inside attention, through RoPE (Lesson 10).

Vocab size trade-offs, tied and untied

A bigger vocabulary compresses text into fewer tokens. Every downstream cost that scales with sequence length (attention's T² work, KV-cache bytes, the context window, and the number of decode steps for a given answer) gets cheaper. The price is paid in the two [V, d] tables, in the lm_head matmul and softmax on every position, and in rarer tokens getting fewer gradient updates each.

TokenizerVEnglish chars per tokenTokens for 10,000 chars
Llama 2 (SentencePiece BPE)32,0003.17≈ 3,155
Llama 3 (tiktoken-based BPE)128,2563.94≈ 2,538 (19.5% fewer)

Compression figures are from the Llama 3 paper, measured on "a sample of English data"; the 10,000-character column is just arithmetic on them.

(ah) What vocabulary size costs: parameters, bytes, and FLOPs
params6.98Bembed_tokenslm_headeverything else (blocks, norms)
One [V, d] table
128,256 × 4096 = 525,336,576 params, 1.05 GB
Vocabulary share of all params
13.1% of 8.03B
lm_head FLOPs per token
2·V·d = 1.051 GFLOP, 6.5% of the ≈ 2N forward
fp32 logits per position
4·V = 513 KB

Preset configurations from each model's published config.json (GPT-2 small also has a 1024 × 768 learned position table, counted in 'everything else'). Moving the V slider changes only the vocabulary tables and keeps every other tensor fixed, which is a what-if, not a real model. GB = 10⁹ bytes, bf16. lm_head FLOPs are 2·V·d per token; the forward pass is about 2 FLOPs per parameter per token (Lesson 4). The tied toggle shows what sharing one table would save; at batch-1 decode the embedding side only reads one row per token, so most of the decode cost of a big vocabulary is the lm_head read and matmul.

Tying means using one matrix for both ends: logits = h Eᵀ. Press and Wolf (2016) showed that sharing the input and output embeddings lowers perplexity in language models while removing a whole table, and GPT-2 does it. The trade is pure budget: in GPT-2 small, the shared 50,257 × 768 table is 38.6M of 124.4M parameters (31%). Llama-3.2-1B ties its 128,256 × 2048 table (262.7M of 1.24B, 21.3%); with a separate lm_head it would carry another 262.7M. Llama-3-8B unties: its two tables total 1,050,673,152 parameters, 13.1% of the model, and the output side gets to learn "what predicts token i" separately from "what token i means as input."

Lesson 7: key numbers

128,256
Llama 3 vocabulary128,000 BPE tokens + 256 reserved special ids
525.3M
params per [128256, 4096] table1.05 GB in bf16
13.1%
of Llama-3-8B is vocabulary tablesembed_tokens + untied lm_head = 1,050,673,152
8 KB
embedding bytes read per tokenone bf16 row of 4096
3.17 → 3.94
English chars per token, Llama 2 → 3≈ 19.5% fewer tokens for the same text
50,257
GPT-2 vocabulary256 bytes + 50,000 merges + 1 special
1.05 GFLOP
lm_head per token, Llama-3-8B2·V·d, ≈ 6.5% of the forward pass
21.3%
Llama-3.2-1B's tied table share262.7M of 1.24B params

Lesson 7: self-check

1. In BPE training, a pair (a, b) has the highest count. What exactly gets added to the tokenizer, and why does the order of additions matter?

Answer. A new vocabulary entry ab and a new row at the end of the merge list. Encoding replays merges by rank, lowest first, so the same string can segment differently depending on which merges were learned earlier. Reorder the list and you have a different tokenizer, even with the same vocabulary.

2. Why can byte-level BPE encode any string, and what is the cost?

Answer. Its base vocabulary is all 256 byte values, so any UTF-8 (or arbitrary binary) input decomposes into known tokens with no unknown token. The cost is that rare scripts and odd strings fall back toward one token per byte, so they use many more tokens per character.

3. Llama-3-8B processes the id 9906 at one position. What tensor slice does that touch, and how many bytes does it read at bf16?

Answer. model.embed_tokens.weight[9906, :], a [4096] row: 4096 × 2 = 8,192 bytes. No matmul happens; it is a gather.

4. A batch contains the token ' the' 300 times. What gradient does its embedding row receive?

Answer. The sum of the 300 per-position gradients ∂L/∂xₜ arriving at the bottom of the network for those positions. Rows for tokens absent from the batch get zero gradient from the loss (weight decay, if applied to embeddings, still shrinks them).

5. Why does Lesson 4's streamed-parameter count for decode exclude embed_tokens but include lm_head?

Answer. The embedding is a lookup of one row per token, so almost none of its 1.05 GB is read. lm_head computes a logit for all 128,256 tokens, which is a full [128256, 4096] matvec, so every byte of it is read every step. 8,030,261,248 − 525,336,576 = 7,504,924,672.

6. You double the vocabulary of a Llama-3-8B-shaped model to 256,512 and keep everything else. What grows, and roughly by how much?

Answer. Each table grows by another 525.3M params, so about +1.05B params (+2.1 GB bf16) total; lm_head FLOPs per token double to ≈ 2.1 GFLOP; fp32 logits per position double to ≈ 1 MB. What you buy is fewer tokens per document, which shrinks attention, KV-cache, and decode-step costs; how much depends on the data and is not known without measuring.

7. Why might a model produce garbage when it sees a particular rare token?

Answer. If the token was created by merges on the tokenizer's corpus but almost never appears in the model's training data, its embedding row received few or no gradient updates and stays close to its random initialization. The model has never learned what that vector means.

Lesson 7: resources