Guus Bouwens
← Writing & notes

FROM THE NOTEBOOK

LLMs explained: mathematically and in plain english

Understanding LLM’s Architecture from First Principles

Modern Large Language Models like GPT-4, and their contemporaries have revolutionized how we interact with AI. But beneath the surface of these conversational marvels lies an elegant mathematical architecture called the decoder-only transformer. This article will take you on a journey through the complete mathematical machinery that powers these models, explaining each step in both precise mathematical notation and intuitive plain English. Think of this as peeling back the layers of a sophisticated system, revealing the beautiful simplicity at its core.

The Big Picture: What Problem Are We Solving?

Before diving into matrices and equations, let's establish what we're actually building. At its heart, an LLM is a next-token predictor, a mathematical function that takes a sequence of words (or more precisely, tokens) and outputs a probability distribution over all possible next words. When you type "The cat sat on the", the model computes probabilities like: "mat" has 80% probability, "floor" has 10%, "chair" has 5%, and so on for every word it knows.

This seemingly simple task turns out to be very powerful. By training on trillions of words and learning these prediction patterns, the model implicitly learns grammar, facts, reasoning patterns, and even some forms of world knowledge. However, the nature of this "understanding" is still a subject of ongoing research and philosophical discussion. Crucially, everything the model becomes is a direct reflection of this training data; its capabilities, knowledge, and even its inherent biases are all learned from the statistical patterns within those trillions of words.

Consider the profound implications of this through a powerful analogy: imagine you're reading a detective novel, and the author carefully plants clues throughout the story: a suspicious character's nervous tic, an inconsistency in an alibi, a hidden motive lurking beneath the surface. The mystery unfolds page after page, building tension and misdirection. Then, at the climax, the detective dramatically reveals the killer, and in that moment of revelation, all the scattered clues suddenly cohere into a coherent picture. An LLM trained on thousands of such detective novels learns to recognize these patterns so deeply that when you describe the plot, the clues, and the setup of the mystery, it can predict who the killer will be before the revelation, not through explicit instruction, but through having internalized the statistical patterns of how mystery narratives work. This isn't mere pattern matching; it's evidence of a deeper understanding. The model hasn't memorized the books; it's learned the underlying logic of mystery construction, character motivation, and narrative structure. When it successfully predicts the killer, it demonstrates that somewhere within its billions of parameters, it has captured something akin to reasoning about human psychology, causality, and logical deduction. This example reveals something remarkable: by optimizing for the simple task of predicting the next word across vast amounts of human-generated text, the model has implicitly developed reasoning capabilities that allow it to make sophisticated inferences about complex scenarios it has never explicitly seen.

The decoder-only transformer architecture is the mathematical framework that makes this learning and prediction possible at massive scale.

Chapter 1: From Words to Numbers: Embeddings and Tokenization

Plain English: Computers don't understand words; they only understand numbers. The first step in any language model is converting text into a format that can be processed mathematically: vectors of real numbers.

Tokenization: Breaking Text Into Pieces

Let's start with a concrete example. When you input the text "The cat sat on the mat", the model doesn't process it as six separate English words. Instead, it breaks it down into tokens: units that could be whole words, parts of words, or even individual characters.

For our example, using a simple tokenizer, we might get: "The" becomes token ID 150, "cat" becomes token ID 2847, "sat" becomes token ID 5284, "on" becomes token ID 319, another "the" (in lowercase) becomes token ID 150 again, and "mat" becomes token ID 4521.

The tokenization process is fundamentally a mapping function that transforms a continuous string of text into a discrete sequence of integers. Given a vocabulary of size V (typically 50,000 to 100,000 tokens), tokenization creates a sequence of integer indices where each index uniquely identifies a particular token in the vocabulary.

Word Embeddings: Giving Tokens Meaning

Once we have token IDs, we need to convert them into embeddings: dense vector representations that capture semantic meaning. The key insight is that words with similar meanings should have similar vector representations. For instance, "cat" and "dog" should have vectors that point in similar directions, while "cat" and "democracy" should be far apart in this vector space.

Plain English: An embedding is like a GPS coordinate for words in a high-dimensional "meaning space". Just as latitude and longitude specify a location on Earth, an embedding vector specifies a location in meaning-space.

We maintain an embedding matrix that we'll call E with dimensions of V by dmodel, where V is the vocabulary size and dmodel is the embedding dimension (typically 768, 1024, or larger). For a token with ID i, we look up its embedding by selecting the i-th row of this matrix. This is essentially a table lookup operation where we retrieve a pre-computed vector associated with each token.

eᵢ = E[i, :] ∈ ℝ^(dmodel)

If our embedding dimension is 4, which is much smaller than reality but good for illustration, the token "cat" might be represented as a vector containing the values 0.82, negative 0.43, 1.21, and 0.15. These numbers are learned during training through backpropagation (explained later), starting from random values and gradually adjusting to capture semantic relationships between different words.

Positional Encoding: Where Are We in the Sequence?

Here's a critical problem: the attention mechanism we'll discuss later treats input as a set, not a sequence. That means "The cat sat on the mat" and "The mat sat on the cat" would look identical to the model! Clearly, word order matters enormously in language.

Plain English: We need to inject information about each word's position into its representation. We do this by adding a position-specific vector to each word embedding, creating a unique "fingerprint" for each position in the sequence.

The positional encoding uses sinusoidal functions to create unique patterns for each position. For a position we call pos and a dimension we call dim, the positional encoding is defined as: for even dimensions (2dim), we compute sine of (pos divided by 10000 raised to the power 2dim divided by dmodel). For odd dimensions (2dim plus 1), we compute the cosine of (pos divided by 10000 raised to the power 2dim divided by dmodel).

PE(pos, 2dim) = sin(pos / 10000^(2dim/dmodel)

PE(pos, 2dim+1) = cos(pos / 10000^(2dim/dmodel)

Plain English: The sinusoidal functions create wavelike patterns with different frequencies across dimensions. Lower dimensions oscillate quickly, capturing local position information. Meanwhile, higher dimensions oscillate slowly, capturing broader position context. This allows the model to learn both absolute positions and relative positions.

The final input representation for token i at position pos is computed by taking the embedding for that token and adding the positional encoding for that position. This combined vector now contains both the semantic meaning from the embedding and positional information from the encoding.

xᵢ = eᵢ + PE(pos)

Chapter 2: The Heart of the Transformer: Self-Attention

Now we arrive at the most crucial innovation in modern language models: the self-attention mechanism. This is where the magic happens, where each word in a sentence gets to "look at" every other word and decide which ones are relevant for understanding its meaning.

Plain English: Imagine you're reading a sentence and trying to understand each word. For the word "it" in "The cat drank the milk because it was thirsty", you need to look back at "cat" (not "milk") to understand what "it" refers to. Self-attention is the mathematical mechanism that allows the model to do exactly this, to dynamically focus on relevant context.

The Query-Key-Value Framework

Self-attention operates through three learned transformations of the input, called Queries (Q), Keys (K), and Values (V). Think of these as three different "views" of each token:

Given input embeddings arranged in a matrix X with dimensions n by dmodel (where n is sequence length), we compute three transformed versions. We multiply X by a learned weight matrix W_Q to get the query matrix Q. We multiply X by a learned weight matrix W_K to get the key matrix K. We multiply X by a learned weight matrix W_V to get the value matrix V. Each of these weight matrices has dimensions dmodel by d_k, where d_k is the dimension of queries and keys.

Q = X W_Q K = X W_K V = X W_V

Plain English: Each row of X is one token's embedding. When we multiply X by W_Q, we're transforming each token's embedding into its query representation. The weight matrices are learned parameters—numbers that the model adjusts during training to capture useful query, key, and value representations.

Computing Attention Scores

Now comes the core operation: determining which tokens should pay attention to which other tokens.

Step 1: Similarity Scores

We compute a similarity score between each query and every key using the dot product. The scores matrix equals Q multiplied by K transposed.

Scores = Q Kᵀ

Plain English: The dot product of query vector q_i and key vector k_j measures how aligned two vectors are. A large positive value means the query and key are similar (token i finds token j relevant); a value near zero means they're unrelated.

Step 2: Scaling

We divide by the square root of d_k. This scaling factor is computed as the square root of the dimension of the keys.

Scaled_Scores = (Q Kᵀ) / √d_k**

Why divide by square root? As the dimension d_k grows, dot products tend to grow in magnitude. Large values pushed into softmax cause extremely peaked distributions with tiny gradients, making training difficult. The 1/√d_k factor keeps the variance of scores stable regardless of dimension.

Step 3: Causal Masking (Critical for Decoder-Only Models)

This is where decoder-only transformers differ from encoders. When predicting the next token, we must not allow the model to "peek" at future tokens. We apply a causal mask that sets attention scores to negative infinity for all future positions.

The masking rule is: if the current position i is greater than or equal to the position j we're attending to, we keep the score. If i is less than j (meaning we're trying to look into the future), we set the score to negative infinity.

Scoresᵢⱼ = Scaled_Scoresᵢⱼ if i ≥ j, else -∞

Plain English: Token at position i can only attend to tokens at positions 0, 1, through i (past and present), never to positions i+1, i+2, and beyond (future). This ensures the model learns to predict based only on preceding context, making it suitable for sequential generation.

Step 4: Softmax Normalization

We convert scores into probabilities using softmax. For each row of the scores matrix, we compute the softmax: the exponential of each element divided by the sum of exponentials of all elements in that row.

softmax(z)ᵢ = e*ᶻⁱ / Σⱼ eᶻʲ*

Plain English: Softmax takes any set of numbers and converts them to probabilities that sum to 1. The exponential function emphasizes larger scores. If token j has a much higher score than others, it gets a probability close to 1. The negative infinity masked values become exponential of negative infinity, which equals zero, effectively removing future tokens from consideration.

Step 5: Weighted Sum of Values

Finally, we compute a weighted combination of the value vectors. We take the attention weights matrix and multiply it by the values matrix V.

Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V

Plain English: For a token at position i, we compare its query against all keys at positions 0 through i, getting attention scores. We apply softmax to get attention weights that sum to 1. We take a weighted average of value vectors using these weights. The result is a new representation of token i that incorporates information from relevant context tokens.

Multi-Head Attention: Multiple Perspectives

A single attention mechanism captures one type of relationship. Multi-head attention runs multiple attention operations in parallel, each learning different aspects of relationships between tokens.

With h heads, we have h separate sets of weight matrices. Each head computes its own attention using its own query, key, and value projections. These outputs are concatenated together and then projected through a final weight matrix W_O.

MultiHead(X) = Concat(head₁, ..., head_h) W_O

Plain English: Different heads might specialize in different relationships: one might focus on syntax (subject-verb agreement), another on semantic similarity, another on long-range dependencies. The concatenation and projection combine these diverse perspectives into a unified representation.

Chapter 3: Building the Decoder Block

A complete decoder block combines several components in a specific architectural pattern. Let's build it piece by piece.

Layer Normalization

Before and after major operations, we apply layer normalization to stabilize training.

For a vector x with d dimensions, layer normalization works as follows. First, compute the mean (μ) by summing all elements and dividing by d. Then, compute the variance (σ²) by summing the squared differences from the mean and dividing by d. Next, take each element, subtract the mean, divide by the square root of the variance plus a small epsilon constant (like 10 to the negative 5th power) for numerical stability. Finally, multiply by a learned scaling parameter γ and add a learned shifting parameter β.

LayerNorm(x) = γ ⊙ (x - μ) / √(σ² + ε) + β

Plain English: Layer normalization ensures that activations in each layer have mean 0 and variance 1, then allows the model to learn optimal scale and shift. This prevents activations from growing too large or too small as they flow through many layers, which would cause training instability.

Feed-Forward Network

After attention, each token passes through a position-wise feed-forward network (FFN): a simple two-layer neural network applied identically to each position.

The feed-forward network computation proceeds as follows. Take the input x and multiply by a weight matrix W_1, adding bias b_1. This expands the dimension (typically multiplying dmodel by 4). Apply the ReLU activation function, which outputs z if z is positive and 0 if z is negative. Multiply the result by another weight matrix W_2, adding bias b_2, projecting back to the original dmodel dimension.

FFN(x) = ReLU(x W_1 + b_1) W_2 + b_2**

Plain English: This is a two-layer neural network with a "bottleneck" expansion. First, we expand from dimension 768 (typical) to 3072 (4 times larger), apply ReLU to introduce nonlinearity, then project back to 768. This happens independently for each token: the FFN doesn't look at other tokens, it just processes each representation individually.

Why ReLU? The Rectified Linear Unit creates a simple nonlinear transformation: negative values become zero, positive values pass through unchanged. This nonlinearity is crucial, without it, stacking many linear transformations would still be just one big linear transformation, severely limiting the model's expressive power.

Residual Connections: The Training Stabilizer

Perhaps the most important architectural detail is residual connections (also called skip connections).

The standard approach would compute output equals F of x, where F is some function. Instead, with residual connections, we compute output equals x plus F of x.

y = x + F(x)

Plain English: We add the input directly to the output of each sublayer. This creates "highway" paths that allow gradients to flow backward through the network during training without getting diminished.

Why does this matter? During backpropagation, gradients must flow backward from the output through all layers to update early-layer parameters. In deep networks, gradients can become exponentially small (vanishing gradients). The residual connection provides a direct path: when computing the derivative of output with respect to x, we get the identity matrix plus the derivative of F with respect to x. The identity matrix ensures gradients always have at least a magnitude of 1, preventing vanishing.

Complete Decoder Block Architecture

Putting it all together, here's the full computation for one decoder block. The input x flows through the following sequence of operations: apply layer normalization to x. Compute multi-head self-attention on the normalized input. Add the attention output back to x (residual connection). Apply layer normalization to this result. Compute the feed-forward network on the normalized input. Add the feed-forward output back (residual connection) to get the final output.

Plain English: The input flows through layer norm, then attention (with causal masking), then gets added back to itself. This result flows through another layer norm, then the feed-forward network, then gets added to itself again. Modern models like GPT-3 stack 96 of these blocks sequentially!

Chapter 4: From Decoder Blocks to Predictions

After the input flows through all decoder blocks, we need to convert the final representations into actual predictions: probabilities over the vocabulary for the next token.

Output Projection

The final layer's output (h_final) is in dmodel dimensions. We project this to vocabulary size by multiplying by a weight matrix W_out with dimensions dmodel by V (vocabulary size) and adding a bias b_out with dimensions equal to vocabulary size. This produces logits (z), one score for each token in the vocabulary.

z = h_final W_out + b_out

Plain English: Each dimension of the logits vector corresponds to one token in the vocabulary. A high value means the model thinks that token is likely next; a low value means unlikely.

Softmax to Probabilities

We apply softmax to convert logits to probabilities. The probability of token i given the context equals **exponential of logit_**i divided by the sum over all vocabulary tokens of exponential of that token's logit.

P(token_i*| context) = e*ᶻⁱ / Σⱼ eᶻʲ

Plain English: This ensures all probabilities are positive and sum to exactly 1, creating a valid probability distribution over all possible next tokens.

Token Selection Strategies

During generation, how do we pick the next token? There are several approaches:

Chapter 5: Training the Model: How Learning Happens

Now we understand the forward pass, how a trained model makes predictions. But how does it learn in the first place?

The Training Objective: Next-Token Prediction

Training data consists of long sequences of text. For each sequence, we create multiple training examples by using prefixes. For instance, take the sequence "The cat sat on the mat". We can create training examples by saying: given the prefix "The", predict "cat". Given the prefix "The cat", predict "sat". Given the prefix "The cat sat", predict "on". And so on through the sequence.

Plain English: Every prefix becomes a training example where we try to predict the next token. A single sequence yields multiple training opportunities.

Cross-Entropy Loss

For each training example, the model outputs a probability distribution over all tokens. We represent the true next token as a one-hot vector, a vector that has 1 in the position corresponding to the correct token and 0 everywhere else.

The loss function (L) measures how far off the prediction is. We compute -1 multiplied by the sum from i equals 1 to V of (y_i times logarithm of predicted probability i). Since y_i equals 1 only for the true token and 0 elsewhere, this simplifies to the negative logarithm of the predicted probability of the true token.

L = -Σ yᵢ log(p̂ᵢ) = -log(p̂_true)

Plain English: If the model assigns high probability (close to 1) to the correct next token, negative logarithm of that probability is close to 0 (low loss). If it assigns low probability (say 0.001), the loss is very high (negative logarithm of 0.001 is approximately 6.9). We want to minimize this loss.

Backpropagation and Gradient Descent

Once we compute the loss, we need to adjust the model's parameters (all those weight matrices) to reduce it. This happens through backpropagation.

We compute gradients of the loss with respect to each parameter. These gradients tell us how much and in which direction to change each parameter to reduce the loss.

***∇***W L = ∂L/∂W

Plain English: Imagine you're hiking in a foggy mountain and want to reach the valley (minimum loss). You can't see far ahead, but you can feel the slope under your feet. Gradients are like the direction of steepest descent, they point downhill. We take a small step in that direction, recompute the gradient, and repeat.

The parameter update rule is: new weight (W_new) equals old weight (W_old) minus learning rate times the gradient of the loss (***∇***W L) with respect to that weight. The learning rate (often called eta (η)) is the step size, how far we move in the direction of the gradient.

W**_new = W_old - η ∇W L

In practice: We don't compute gradients over the entire dataset (too slow). Instead, we use mini-batch gradient descent: compute loss and gradients on a small batch (say 32 sequences), update parameters, then move to the next batch. This is called stochastic gradient descent because each batch gives a noisy estimate of the true gradient, but it's much faster.

The Chain Rule: How Gradients Flow Backward

The magic of backpropagation is the chain rule from calculus. For composed functions where f depends on g and g depends on x, the derivative of f with respect to x equals the derivative of f with respect to g multiplied by the derivative of g with respect to x.

∂f/∂x = (∂f/∂g) · (∂g/∂x)

Plain English: If the output depends on some intermediate value, which depends on the input, we can decompose the gradient into stages. The gradient "flows backward" through each layer, multiplying by local gradients at each step.

In a transformer, this means the following sequence of operations: First, compute gradient of loss with respect to output logits. Second, backpropagate through output projection. Third, backpropagate through each decoder block, starting from the last and working backward. Fourth, backpropagate through embeddings. Fifth, update all parameters using their computed gradients.

Why are residual connections crucial? Remember that residual connections add x plus F of x. When backpropagating, gradients flow through both paths: the direct path (through x) and the function path (through F of x). Even if gradients through F diminish, the direct path ensures some gradient always flows backward, enabling training of very deep networks (96+ layers in GPT-3).

Chapter 6: Putting It All Together: The Complete Architecture

Let's trace a complete forward pass through a decoder-only transformer, step by step, with both math and plain English.

Input: Text sequence "The cat sat on"

Step 1: Tokenization: The text is broken into tokens: "The" becomes 150, "cat" becomes 2847, "sat" becomes 5284, "on" becomes 319.

Step 2: Embedding Lookup: For each token ID i, we look up embedding eᵢ in the embedding matrix. Assuming dmodel equals 768, each embedding is a vector of 768 real numbers. We arrange these into a matrix where each row is one token's embedding.

Step 3: Add Positional Encoding: We compute positional encodings for each position (0, 1, 2, 3) and add them to the embedding matrix. The first token gets positional encoding for position 0, the second for position 1, and so on.

Step 4: Pass Through Decoder Blocks: For each of L decoder blocks (typically 12, 24, or more): Apply layer normalization, compute multi-head self-attention with causal masking, add attention output back (residual), apply layer normalization again, compute feed-forward network, add feed-forward output back (residual). The output of one block becomes input to the next.

Step 5: Output Projection: We take the last token's final representation and multiply it by W_out to get a vector of vocabulary size. This vector is in log-probability space (the logits).

Step 6: Softmax to Probabilities: We apply softmax to convert logits to a proper probability distribution over all 50,257 tokens in the vocabulary.

Step 7: Sample Next Token: Using top-p (nucleus) sampling with p equals 0.9: Sort tokens by probability, find the smallest set of highest-probability tokens with cumulative probability exceeding 0.9, renormalize these probabilities, sample one token from this distribution. Suppose we sample token 150, which corresponds to "the".

Step 8: Autoregressive Generation: This step reveals the generative power of the model. The process is "autoregressive," meaning it uses its own previous outputs as input for future predictions.

The newly sampled token ("the") is appended to the end of the current input sequence. The resulting, longer sequence ("The cat sat on the") now becomes the complete input for the next forward pass. The model re-processes this entire sequence, starting from Step 4, to generate a new probability distribution and sample the next token (likely "mat"). This cycle of predict, append, and re-process continues, with the input sequence growing by one token at each step, until a maximum length is met or a designated stop token is produced.

Plain English Summary: The text is converted to token IDs, looked up in an embedding table, combined with positional encodings, passed through many decoder blocks (each applying attention and feed-forward transformations with residual connections), projected to vocabulary size, converted to probabilities, and sampled to produce the next token. This process repeats autoregressively, using previous outputs as inputs, to generate entire sequences.

Conclusion: The Beauty of Simplicity at Scale

What makes decoder-only transformers remarkable is not complexity, but elegant simplicity repeated at massive scale. The core operations are mathematically straightforward. The innovation lies in how they're composed:

  1. Self-attention allows dynamic, context-dependent routing of information between tokens. Each token can selectively focus on other tokens that are relevant to understanding its meaning.
  2. Causal masking enforces sequential prediction, enabling autoregressive generation. Tokens can only see past and present context, never future tokens.
  3. Residual connections stabilize training of very deep networks. They provide direct paths for gradients to flow backward during training.
  4. Layer normalization keeps activations in a healthy range. It prevents numerical instability and helps the model learn more robustly.
  5. Multi-head attention captures diverse types of relationships. Different heads can specialize in different linguistic phenomena simultaneously.

When you scale this architecture to billions of parameters and train on trillions of tokens, something extraordinary emerges: a model that can engage in dialogue, write code, reason through problems, and even exhibit some forms of common sense. All of this emerges from the deceptively simple objective of predicting the next token.

The mathematics we've explored here forms the foundation of modern AI language capabilities. Understanding these mechanisms demystifies how systems like ChatGPT work, transforming them from mysterious "black boxes" into comprehensible (if impressively large) mathematical machines.

Every time you interact with an LLM, this entire mathematical ballet executes billions of times: tokens are embedded, attention is computed, gradients flow backward, parameters update. The next token emerges, one probability distribution at a time, guided by the elegant mathematics of the decoder-only transformer.

The journey from raw text to meaningful generation happens entirely through these mathematical transformations. There's no magic, only powerful mathematics that scales to produce intelligence.

______________________________________________________________________________

Tokens: Tokenization vs Embedding - How are they Different? | Airbyte

Query/Key/Value: What are Query, Key, and Value in the Transformer Architecture and Why Are They Used? | Towards Data Science

Decoder: Vaswani, A., et al. "Attention is all you need." Advances in neural information processing systems 30 (2017)

All writing