← Fine Tuning From First Principles

Backpropagation Through a Transformer

Trace a small causal Transformer, derive its vector-Jacobian products, and check selected parameter gradients independently with finite differences.

A training script can call backward() successfully while updating the wrong subset of parameters. It can also produce a zero gradient for a perfectly legitimate reason. The practical task is to distinguish a correct gradient path from a disconnected tensor, a frozen parameter, an unselected target or an unfortunate initialisation. Printing one final loss cannot make those distinctions.

We will trace the batch from Chapter 07 through a small causal Transformer and back from Chapter 08’s scalar loss. Every major operation is visible. This is a randomly initialised educational model; the work checks derivatives and information flow, not the quality of a pretrained language model.

Begin with a chain small enough to differentiate

Take a scalar prediction $u=wx$, residual $r=u-y$ and squared loss $L=r^2$. The chain rule gives

$$ \frac{\partial L}{\partial w} =\frac{\partial L}{\partial r} \frac{\partial r}{\partial u} \frac{\partial u}{\partial w} =2r\cdot1\cdot x. $$
At $w=1$, $x=2$ and $y=4$, the residual is $-2$ and the derivative is $-8$. The negative sign says that increasing $w$ slightly decreases this loss locally. It does not prescribe how large an update should be or guarantee improvement after a large update. Choosing and applying that update is the next chapter's subject.

A Transformer contains many vector-valued intermediate functions instead of this short scalar chain. For $\mathbf y=f(\mathbf x)$, write the upstream loss derivative as $\mathbf g=\partial L/\partial\mathbf y$. Backpropagation needs a vector-Jacobian product, $J_f(\mathbf x)^\top\mathbf g$, rather than the complete Jacobian stored as a dense matrix. The upstream derivative tells the local operation which combination of its output sensitivities matters to the current scalar loss.

Autograd records the operations and the tensors required for those local products. Its graph is a record of the executed computation, not a proof that we supplied the intended labels or attention mask. PyTorch’s autograd notes describe that mechanism; the examples here execute against the pinned 2.6.0 implementation.

Follow the forward path before following its derivative

The educational model consists of token and position embeddings, one explicit causal block, a final layer normalisation and an untied output projection. The block uses pre-normalisation: normalise before the attention sublayer, add a residual, normalise before the MLP, and add a second residual. This choice makes its code concise and keeps the identity paths visible.

The reference derivative probe has width 8, two heads of width 4, MLP width 16 and context capacity 16. It has 4,936 parameters. The later training configuration uses wider dimensions but the same implementation. We do not replace the model between the derivative chapter and the training loop with an unrelated framework model.

The Transformer family was introduced in Attention Is All You Need. Our bounded implementation omits an encoder, cross-attention, tokenisation-specific architecture details and production optimisations. Its purpose is to expose the causal decoder computations we need to differentiate.

For input shape $B\times T$, embedding lookup produces $B\times T\times d$ values. A position embedding of the same width is added at each position. Inside the block, one linear projection produces queries, keys and values. They are reshaped into $h$ heads, each with width $d_k=d/h$.

For one head,

$$ \mathbf S=\frac{\mathbf Q\mathbf K^\top}{\sqrt{d_k}},\qquad \mathbf P_{\mathrm{attn}}=\operatorname{softmax}(\mathbf S+\mathbf M),\qquad \mathbf C=\mathbf P_{\mathrm{attn}}\mathbf V. $$
The mask $M$ is zero at allowed positions and negative infinity at disallowed ones. The allowed keys must belong to the real sequence and must not be in the future. A non-empty right-padded row always begins with a real token, so each query has at least one permitted key. An all-padding row is rejected rather than passed to a softmax of entirely negative-infinite values.

The heads are concatenated and projected back to width $d$. After its residual addition, the MLP expands to width $d_{ff}$, applies GELU, and projects back. The final projection returns $B\times T\times V$ logits. Chapter 08’s loss then selects the shifted target positions. No generation sampling appears in this backward calculation.

Backward through the small Transformer

The left column follows the forward chain downward with derivatives returning upward; the right column collects what each backward step needs, including the frozen-parameter and zero-initial-path cases that look like failures but are not. The strip records the independent finite-difference agreement on selected parameters.

Differentiate a linear map and an embedding lookup

For a matrix product $Y=XW$, with upstream derivative $G$, differential notation gives

$$ dY=dX\,W+X\,dW. $$
Collecting the coefficients of $dX$ and $dW$ in the scalar differential of the loss yields
$$ \frac{\partial L}{\partial X}=GW^\top, \qquad \frac{\partial L}{\partial W}=X^\top G. $$
A bias derivative sums the upstream contributions over the rows that used that bias. PyTorch's `Linear` stores its weight in output-by-input order, so its implementation writes the forward product with a transposed stored weight. Keep the storage convention explicit when comparing formulae with tensor shapes.

An embedding lookup is a row selection. Its backward operation adds each selected position’s upstream derivative into the corresponding embedding row. If ID 1 occurs twice, its gradient receives both contributions. Our small independent test sums embeddings for IDs [1, 1, 2]; row 1 receives [2, 2], row 2 receives [1, 1], and unselected rows receive zero. Repeated-token contributions are accumulated, not overwritten.

That example also clarifies why an unused embedding can have a zero gradient while other embeddings train normally. Zero alone is not evidence of a broken backward pass. Ask whether that parameter participated in a scored prediction’s computation.

Pass through softmax and masked attention

For a softmax row $\mathbf p$ and upstream derivative $\mathbf g$, its vector-Jacobian product is

$$ \frac{\partial L}{\partial s_i} =p_i\left(g_i-\sum_j p_jg_j\right). $$
This avoids explicitly storing the matrix with entries $p_i(\delta_{ij}-p_j)$. A change to one score alters the normalising sum and hence affects all allowed probabilities in that row. The derivative therefore cannot be obtained by treating those probabilities as independent sigmoid outputs.

For attention output $\mathbf C=\mathbf P_{\mathrm{attn}}\mathbf V$, the upstream derivative $\mathbf G_C$ gives

$$ \mathbf G_V=\mathbf P_{\mathrm{attn}}^\top\mathbf G_C,\qquad \mathbf G_P=\mathbf G_C\mathbf V^\top. $$
Apply the row-wise softmax product to $\mathbf G_P$ to obtain $\mathbf G_S$. Then the score product gives
$$ \mathbf G_Q=\frac{\mathbf G_S\mathbf K}{\sqrt{d_k}},\qquad \mathbf G_K=\frac{\mathbf G_S^\top\mathbf Q}{\sqrt{d_k}}. $$
The implementation compares these manually assembled products with autograd on a two-position causal example. In the reference float64 run, the largest query/key/value derivative discrepancy was approximately $8.7\times10^{-19}$.

A forbidden future score is replaced by a constant before softmax. Its probability is zero and the derivative back through that masked score is zero. This does not require the derivative with respect to the probability tensor itself to be zero: the masking and softmax Jacobian determine which upstream contributions reach the scores.

The forward causality test changes a future input token and verifies that earlier logits remain unchanged. This tests the information restriction directly, independently of a gradient being small. Both checks are useful: an incorrectly broadcast mask may still produce finite gradients while allowing the model to read the answer it should predict.

Residual additions and layer normalisation

For a residual computation $Y=X+F(X)$, the incoming derivative has an identity contribution and a sublayer contribution:

$$ G_X=G_Y+J_F(X)^\top G_Y. $$
The addition gives a direct derivative path. It does not guarantee that gradients cannot explode, vanish or cancel. The sublayer Jacobian and the sequence of later operations still matter. A trace is more informative than assuming every residual architecture must have healthy gradients.

Layer normalisation couples the features within a position. For feature width $d$, define

$$ \mu=\frac1d\sum_i x_i,\quad \sigma^2=\frac1d\sum_i(x_i-\mu)^2,\quad \hat x_i=\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}}. $$
The affine output is $y_i=\gamma_i\hat x_i+\beta_i$. Let $g_i$ include the multiplication by $\gamma_i$ from the upstream derivative. The derivative with respect to the input is
$$ \frac{\partial L}{\partial x_i} =\frac{1}{\sqrt{\sigma^2+\epsilon}} \left[g_i-\operatorname{mean}(g) -\hat x_i\operatorname{mean}(g\hat x)\right]. $$
The learned scale and bias receive sums of the upstream derivative times $\hat x$ and the upstream derivative itself, respectively, across the positions sharing those parameters. The explicit comparison tests the input formula with unit scale and zero bias. Its observed maximum error against autograd was approximately $1.7\times10^{-16}$.

The stabilising $\epsilon$ remains inside both forward and derivative calculations. Omitting it in one calculation changes the function being checked, especially when the variance is small. A finite-difference disagreement can therefore come from comparing two different normalisation definitions rather than from an autograd failure.

Follow the MLP and the scalar objective

The MLP is a composition of two linear maps and a nonlinearity. For the exact GELU definition used here,

$$ \operatorname{GELU}(x)=x\Phi(x),\qquad \operatorname{GELU}'(x)=\Phi(x)+x\phi(x), $$
where $\Phi$ and $\phi$ are the standard normal cumulative distribution and density. The code independently constructs this derivative using `erf` and an exponential. It agrees with the tested GELU backward calculation to approximately $2.3\times10^{-16}$ in float64. An approximate GELU variant would be a different function and needs its own corresponding derivative.

At the output, a single selected cross-entropy term has derivative $p_v-\mathbb{1}[v=y]$ with respect to its logit. Under a token mean, divide that contribution by the total selected-target count. Ignored target positions contribute zero. From there, linear products, normalisation, residual additions and attention carry the derivative backwards.

Our trace retains intermediate gradients for embeddings, queries, keys, values, scores, attention probabilities, aggregated context, residuals and MLP states. A retained intermediate gradient costs memory; it is a teaching instrument rather than the default training configuration. Chapter 11 will measure the different memory categories instead of assuming that the visible parameter arrays describe the whole computation.

Check selected parameters without trusting the backward implementation

For a scalar parameter $w$, a central difference estimates

$$ \frac{\partial L}{\partial w} \approx\frac{L(w+\varepsilon)-L(w-\varepsilon)}{2\varepsilon}. $$
The comparison uses the same forward function, labels and mask for both evaluations. It restores the original weight afterwards, including when the objective raises an exception. We turn dropout off and use CPU float64 so random masks and FP32 rounding do not dominate the small perturbation.

With $\varepsilon=10^{-6}$, the reference probe observed:

Selected parameter Autograd derivative Central difference Absolute error
Output weight, ID 79 and feature 0 0.02349374836 0.02349374828 $7.9\times10^{-11}$
Query/key/value projection entry 0,0 -0.003095272024 -0.003095272039 $1.5\times10^{-11}$
Embedding entry for H, feature 0 0.00004130864482 0.00004130917830 $5.4\times10^{-10}$

These selected checks are not an exhaustive proof of every derivative. They are an independent way to detect several wrong paths or shape conventions. Very small $\varepsilon$ increases cancellation; a large value measures curvature over too wide an interval. Near a non-differentiable point, the meaning of a numerical check needs further care. Our selected smooth computation avoids that complication.

The bounded CUDA demonstration separately compares FP32 logits and parameter gradients with a CPU model loaded from the same state. It uses absolute tolerance $10^{-5}$ and relative tolerance $10^{-4}$, and the observed comparisons passed. We do not demand binary identity across different devices and kernels, or call a GPU float32 comparison an independent float64 finite-difference check.

Distinguish accumulation, freezing and an initially zero path

Calling backward adds to existing parameter gradients. In a small regression we call it twice on equivalent forward computations and observe twice the first gradient. Clearing gradients with zero_grad(set_to_none=True) then removes that accumulated buffer. An optimiser step does not, by itself, define when those gradients should be cleared.

Freezing a parameter sets its requires_grad flag to false. The model still uses its numerical values in the forward pass. Other trainable weights can receive derivatives through computations containing frozen weights. Our frozen token and position embeddings have no parameter gradient, while the output head still receives a non-zero gradient. Evaluation mode changes operations such as dropout; it does not freeze parameters.

Low-rank initialisation supplies another instructive zero. Write a linear weight as $\mathbf W_0+\mathbf B\mathbf A$, with $W_0$ frozen. If $G$ is the derivative with respect to that effective weight, then

$$ \mathbf G_B=\mathbf G\mathbf A^\top,\qquad \mathbf G_A=\mathbf B^\top\mathbf G. $$
With a non-zero $A$ and zero $B$, the initial derivative of $A$ is zero, while the derivative of $B$ can be non-zero. The miniature probe observed norms zero and approximately 3, respectively. That is the expected consequence of the factorisation, not a failed adapter. Initialising both factors to zero would remove both first-step paths in this simple construction. Chapter 22 develops the full method and its design choices.

The output of this chapter is a checked derivative path, not an improvement claim. We have tested causality, selected analytical products, central differences, frozen parameters and gradient buffers using one shared model. Chapter 10 now asks how an optimiser turns those gradients into controlled changes to the stored weights.

Further Reading

  1. Ashish Vaswani et al., “Attention Is All You Need” (2017). Read the source. The architecture paper whose attention, residual and normalisation blocks the chapter differentiates by hand.
  2. Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton, “Layer Normalization” (2016). Read the source. The normalisation paper behind the chapter’s lengthiest hand derivation, explaining why the backward pass couples features within a position.
  3. PyTorch contributors, “Autograd mechanics” (PyTorch 2.6 notes). Read the source. The mechanism reference justifying the warning that autograd records what ran but cannot certify the intended labels or frozen set.