LoRA From Linear Algebra
Derive low-rank updates, factor gradients, initialisation, memory counts and merge behaviour, then verify them against PEFT.
A dense linear layer contains millions of weights, but a particular adaptation may not need an independent change in every direction. Can we restrict the update while preserving the original transformation?
LoRA answers by learning two factors whose product forms an additive weight update. The base matrix remains frozen. The proposal originates in Hu and colleagues’ LoRA paper. Here we derive the layer and its gradients directly, then compare an executable reference with the selected PEFT implementation.
The restriction is an inductive choice, not a theorem that every useful update has low rank. It reduces trainable state and constrains the adaptation space. Whether that constraint helps a task needs a controlled experiment.
Fix the storage and batch convention first
Let a linear layer accept input vectors of width d_in and produce outputs of width d_out. We store its matrix as
Introduce a rank parameter r and factors
Some libraries store a projection transposed, and some attention modules fuse several projections. Their layouts require an explicit translation. A matching module name does not establish a matching matrix convention.
| Object | Shape | Role |
|---|---|---|
W (base) |
d_out by d_in |
Frozen; computes the XWᵀ path |
A (factor) |
r by d_in |
Trainable; maps input to r coordinates, H = XAᵀ is n by r |
B (factor) |
d_out by r |
Trainable; maps r coordinates to output space |
s = α/r (scale) |
scalar | Multiplies the update and both factor gradients |
X (row batch) |
n by d_in |
Shared input to both paths |
G = ∂L/∂Y |
n by d_out |
Loss gradient at the outputs |
∂L/∂B |
d_out by r |
Equals sGᵀXAᵀ; same shape as B |
∂L/∂A |
r by d_in |
Equals sBᵀGᵀX; same shape as A |
∂L/∂X |
n by d_in |
Equals GW + sGBA; crosses the frozen base |
Interpret the rank restriction geometrically
A maps an input into r coordinates. B maps those coordinates into output space. The correction therefore lies in the output subspace spanned by B’s columns:
If r is small, the factors may express a useful restricted family of changes. If the task requires important directions outside that family, more optimisation cannot recover an arbitrary dense update. Increasing rank expands the available parameterisation but does not guarantee better held-out performance.
The figure shows the bottleneck, the frozen base path, and the d = 4096, r = 8 counts. Notice that the update lives in the column space of B while W itself can stay full rank — the restriction bounds the change, not the model.
The construction resembles a matrix factorisation, but training LoRA is not generally the same operation as taking the best rank-r truncated singular-value decomposition of a known desired update. In adaptation we usually do not know that desired matrix. We optimise a data-dependent network loss through the factors.
Factorisations are also non-unique. For an invertible r by r matrix Q, the factors BQ and Q^-1 A produce the same BA. Equivalent forward maps can nevertheless have different factor gradients and optimiser trajectories. Parameterisation therefore matters beyond the effective weight matrix.
Count the savings in the chosen units
A dense matrix contains d_out d_in parameters. The adapter contains
This ratio describes trainable parameters, not total model memory. The base must still be stored and used. Activations, logits, kernels, temporary buffers and adapter state remain.
If the adapter parameters and gradients are FP32, each contributes four bytes per parameter. Two FP32 Adam moment arrays contribute another eight bytes. For this example the moments require 524,288 bytes, compared with 134,217,728 bytes for two FP32 moments over every element of the dense matrix. Scalar step state and implementation metadata add smaller terms.
Storage choices can differ. A separate FP32 master copy is an additional term, not something to add automatically whenever the word “mixed precision” appears. An embedding or output head made trainable also belongs in the ledger. Ignoring such extra modules can make a small claimed adapter unexpectedly large.
The reference now takes an explicit adapter dtype. When the base is FP16 or BF16,
the default adapter dtype is FP32, matching PEFT’s autocast_adapter_dtype
behaviour. The adapter path casts its input to the adapter dtype, computes the
two factor products there and casts the result back to the base output dtype. In
PEFT 0.18.0 the same sequence is visible in Linear.forward: _cast_input_dtype
uses the adapter weight dtype, then result.to(torch_result_dtype) restores the
base output dtype. Measured FP16-base/FP32-adapter parity is recorded with its
tolerance rather than asserted as exact equality. Merge forms the dense delta in
the adapter dtype and casts it to the base weight dtype before addition; unmerge
restores the stored base bytes exactly. That is a floating merge, not a
quantised-kernel merge.
Work through a complete numerical layer
Take
This tiny layer has five adapter parameters and six base parameters, so it is
not a persuasive memory-saving demonstration. That is intentional: its purpose
is to make every element inspectable. The larger parameter-count calculation
above establishes the algebraic scaling separately. The implementation also
enforces that the rank does not exceed min(d_in, d_out); a larger rank is no
longer a low-rank restriction and raises LORA_RANK. Wrapping a LoRALinear in
another adapter, or attaching to a *.base path inside one, raises LORA_BASE:
double adapters change the topology that the trainable-set check is intended to
bind.
Derive the factor gradients
Let G be the derivative of the scalar loss with respect to Y, shaped n by d_out. For the adapter contribution, write H=XA^T. Then Y_adapter=sHB^T.
Using the matrix differential and collecting terms gives
The gradient passed to the layer input is
For the worked example, choose the loss sum(Y squared), which equals 44. Then G=2Y, and the factor gradients are
The absence of a stored W gradient is a software freezing property. It does not mean the mathematical derivative of the loss with respect to an unfrozen W would be zero.
Why setting both factors to zero stops learning
A convenient initial condition preserves the original model’s output: Delta W must initially be zero. Setting both factors to zero achieves that, but it creates a stationary adapter parameterisation:
Instead initialise A nonzero and B zero. The forward update remains zero, but
The deterministic experiment records factor-gradient norms of [0,0] for the both-zero case. With one nonzero factor, the first A gradient is zero while the first B gradient norm is about 19.14. After one SGD step, the A gradient norm is about 14.95. These numbers describe the deliberately small squared-loss fixture, not a typical language-model gradient scale.
A zero first gradient therefore does not prove that A was frozen or disconnected. We must inspect the expected dependency and later steps. Conversely, a requires_grad flag alone does not prove useful gradient flow.
Scaling and dropout change the experiment
The conventional scale alpha/r multiplies the effective update and both factor gradients. Holding alpha constant while increasing rank changes that scale. Holding alpha/r constant avoids that particular confound, but does not make different ranks identical optimisation problems.
The number and distribution of initial factor entries, optimiser state and available update directions still change. Other scaling conventions, including rank-stabilised choices, are different interventions and need separate identities. A result obtained with one cannot silently validate another.
Adapter dropout applies to the adapter input in our implementation. During training its sampled mask changes the correction. During evaluation dropout is disabled. A fixed merged matrix reproduces the deterministic evaluation map; it does not reproduce every possible training-time dropout mask.
This is why merging is refused while the reference layer is in training mode. The training loop must unmerge before updating factors. Treating a merged layer as if it still used the factor path would silently stop adapter gradients.
Compare the reference with PEFT
Our selected library check uses the LoRA Linear layer from PEFT 0.18.0, whose versioned reference documents the relevant configuration surface.
The test copies identical base weights and factor tensors into the reference and PEFT layers, fixes rank and alpha, disables dropout, and uses identical FP64 inputs. It compares forward outputs, A gradients, B gradients and evaluation merge behaviour. The recorded forward and factor-gradient differences are zero for the tested layer, within a 1e-12 acceptance tolerance.
That is meaningful implementation parity, but its scope is small. It does not verify automatic target selection in a Qwen model, a fused projection layout, a quantised backend, adapter distribution format or a complete training run. Those need their own checks when the corresponding checkpoint is admitted.
The generic SFT loop from Chapter 21 uses the inspectable reference layer. Library parity helps explain its meaning without turning a library call into the source of the mathematics.
Merge only what has been verified
For a floating-point evaluation layer, merging stores W+sBA in the base matrix and disables the separate adapter contribution. Adding the contribution both to W and again in forward would double the update.
In exact arithmetic, merged and unmerged evaluation outputs agree. Floating-point association and rounding can produce small differences. Our tests use explicit tolerances and record the actual error rather than asserting universal bitwise equality.
The reference retains a copy of the original base when merging and restores that copy on unmerge. This avoids repeated subtraction drift in the demonstration, but consumes an extra full matrix. It is not a memory-free deployment strategy. Merging twice, unmerging an unmerged layer and training while merged are explicit errors.
A quantised base requires a different decision. Dequantising, adding an update and requantising introduces a new approximation, while preserving a dequantised merged base changes storage. The reference refuses to merge a quantised layer under the floating-point operation’s name. Chapter 24 examines that boundary.
What the experiments justify
The CPU tests establish the chosen factor convention, analytical gradients, initialisation behaviour, parameter counts, freezing and save/reload integration. The selected PEFT layer agrees under the tested conditions. A bounded CUDA tensor check also verifies the small adapter path on the reference device.
They do not establish an optimal rank, a useful target-module set, reduced end-to-end latency, real-model memory fit or improved task performance. No pretrained weights are involved.
The next experiment must therefore vary a stated choice under controls, preserve the resulting negative or positive evidence and avoid choosing a configuration through protected evaluation results. That is an experimental-design problem, rather than another matrix identity.
Further Reading
- Edward J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021). Read the source. Proposes frozen-base low-rank updates, sourcing the chapter’s storage convention and rank bound; also recommended in Chapter 21 for adapter identity.
- Hugging Face contributors, “LoRA” (PEFT documentation). Read the source. Documents zero-B initialisation, scaling and merge utilities against which the chapter checks forward and gradient parity.
- Damjan Kalajdzievski, “A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA” (2023). Read the source. Argues for scaling by the square root of rank, clarifying why rank comparisons need separate identities.