← Fine Tuning From First Principles

Adapter Composition and Model Merging

Derive what adding, merging and sequencing rank-1 adapter deltas does to a weight matrix, measure which capabilities survive composition, show a conflicting delta erasing a retained behaviour, and check that merged and unmerged FP64 probabilities agree exactly in this tiny model.

The compatible system that passed every challenge in Chapter 39 was not a single adapter. It was two adapters added to one base model. Adapter A was constructed to refuse a certain erase, and adapter B to send uncertain or unknown requests to review. Neither alone could do the whole task. Together they scored 24/24.

That outcome invites a general strategy: adapt separately for each capability, then combine. It is attractive in practice. Separate adapters can be trained, reviewed and replaced independently, and the base model is stored once. This chapter asks whether two adaptations can be combined, and what compatibility actually means. The answer begins with linear algebra, because for LoRA adapters on a linear layer the combination is exact arithmetic on weight matrices. It ends with the limits of that arithmetic. Adding two weight deltas is always possible. Whether the result keeps both behaviours depends on whether the deltas interfere.

The integrated record is FT-P7-INTEGRATED-01-40-s37, at data seed 37 only. The adapters are constructed. This chapter demonstrates algebra, compatibility checks and interference, not the success of multi-task fine-tuning.

Four ways to use two adapters

Given a frozen base layer with weight W₀ and two LoRA adapters, there are several distinct operations that are easily confused.

  • Switching. Exactly one adapter is active at a time, chosen per request or per deployment. Nothing is combined. Each behaviour exists only while its adapter is loaded.
  • Weighted delta composition. Both adapters are active, and the layer computes with W₀ + w_A·ΔW_A + w_B·ΔW_B. The adapters can stay as separate modules while doing this.
  • Merge and unload. The composed weight is computed once and written into a single dense matrix, and the adapter modules are discarded. The result is an ordinary layer with no adapter structure.
  • Output combination. Two separately adapted models each produce probabilities, and the probabilities or decisions are combined, for example by averaging. This is an ensemble, not a weight-space operation.

A fifth idea, sequential application, appears whenever adapted layers are stacked. One transformation is applied, then another. That is function composition, and its order matters.

These operations are not equivalent in general. The rest of the chapter derives when two of them coincide and when they do not.

The deltas, written out

Chapter 37 introduced the constructed adapters. A rank-r LoRA adapter on a linear layer with weight W ∈ ℝ^{out×in} has factors A ∈ ℝ^{r×in} and B ∈ ℝ^{out×r}. It contributes

$$ \Delta W = \frac{\alpha}{r}\, B A , $$
following the convention of Hu et al. (2021) as used throughout this book. With r = 1 and α = 1, the delta is an outer product. For adapter A, with A = [0, 0, 1, 0] and B = [−2, 2, 0]ᵀ,
$$ \Delta W_A = \begin{bmatrix} -2 \\ 2 \\ 0 \end{bmatrix} \begin{bmatrix} 0 & 0 & 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 0 & -2 & 0 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix}. $$
Only the erase column is non-zero. The adapter moves 2 units of score from `complete` to `refuse` whenever the erase feature is on, and does nothing otherwise. For adapter B, with A = [0, 0, 0, 1] and B = [−3, 0, 3]ᵀ,
$$ \Delta W_B = \begin{bmatrix} 0 & 0 & 0 & -3 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 3 \end{bmatrix}. $$
Only the uncertainty column is non-zero. The cross-check rebuilt both deltas from the stated factors and found them equal, element by element, to the deltas stored in the experiment record.

The composed weight with unit weights is

$$ W = W_0 + \Delta W_A + \Delta W_B = \begin{bmatrix} 1 & 1 & -1 & -2 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & 3 \end{bmatrix}. $$
The two deltas touch different input columns. They do not overlap, so neither can disturb what the other does on inputs where only its own feature is active. On an uncertain erase, x = [0, 0, 1, 1], both are active. The first row gives −1 − 2 = −3, the second gives 2 and the third gives 3, so the scores are (−3, 2, 3) and review wins by one unit. Composition works on this input because B's push towards review (3) is larger than A's push towards refusal (2).

Delta addition is not factor addition

It is tempting to combine two LoRA adapters of the same rank by adding their factors, forming B = B_A + B_B and A = A_A + A_B, and storing one adapter. That computes a different thing:

$$ (B_A + B_B)(A_A + A_B) = B_A A_A + B_B A_B + B_A A_B + B_B A_A . $$
The first two terms are the intended deltas. The last two are **cross terms** that neither adapter contains. For these adapters, B_A·A_B puts −2 and +2 into the uncertainty column, so A's refusal push would now fire on uncertain requests. B_B·A_A puts −3 and +3 into the erase column, so B's review push would fire on every erase. Factor addition would turn two clean, non-overlapping behaviours into an entangled one. Low-rank adapters compose by adding **products**, ΔW_A + ΔW_B, not by adding factors. The sum generally has rank up to r_A + r_B and cannot be stored as one rank-r adapter.

Delta addition versus factor addition with interference consequence

The figure contrasts the correct product sum (disjoint columns, both capabilities kept) with factor addition (cross terms entangle the columns), alongside the measured capability result: the conflicting delta keeps Task A at 12/12 while erasing Task B to 0/12. It is derived from the constructed weights and the recorded 12 + 12 capability counts.

Merged and unmerged are the same function

The unmerged implementation keeps three modules: the frozen base and one LoRA layer per adapter, each LoRA layer containing its own copy of the base. It computes

$$ s(x) = W_0 x + \sum_i w_i\,\big[(W_0 + \Delta W_i)\,x - W_0 x\big] . $$
The merged implementation computes s(x) = (W₀ + Σᵢ wᵢ ΔWᵢ) x with one matrix. In exact arithmetic the two are identical, because matrix multiplication distributes over addition. This is why merging a LoRA adapter adds no inference cost compared with a dense fine-tuned layer (Hu et al., 2021).

In floating point they need not be identical. The two computations add terms in different orders, and floating-point addition is not associative. The experiment compared the softmax probabilities of the merged and unmerged composed systems on all 24 test cases in FP64. The largest absolute difference was 0 on every case. The cross-check obtained the same zero from the recorded probabilities. It also obtained zero independently, by recomputing both forms in plain Python floating point from the factors.

That exact zero has a specific explanation, and it should not be generalised. Every weight is a small integer, and every feature is 0 or 1. Every intermediate score is therefore a small integer, and small integers are represented exactly in FP64. No rounding occurs in these score calculations, in either order. With non-integer weights, lower precision (FP16, BF16 or a quantised base), larger matrices with long accumulations, or kernels that reorder reductions, merged and unmerged outputs typically differ by small amounts. A merge must then be checked against a stated tolerance on the actual model, precision and hardware. A merge into a quantised base is a further approximation, because the merged weight must be requantised.

Sequential application does not commute

Delta composition on one linear layer is commutative: ΔW_A + ΔW_B = ΔW_B + ΔW_A. Stacked layers are not. If one layer applies a map M₁ and the next applies M₂, the result is M₂M₁x, and in general M₂M₁ ≠ M₁M₂. The experiment records a two-dimensional example with separate activation maps. It is not a case of nested LoRA on the same target. With x = (1, 2),

$$ M_a = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix}, \quad M_b = \begin{bmatrix} 1 & 0 \\ 0 & 2 \end{bmatrix}, \qquad M_b M_a x = (3, 4), \quad M_a M_b x = (5, 4). $$
Order changed the first coordinate from 3 to 5. Add a nonlinearity between the layers and even the commutativity of individual deltas stops being enough to predict the composite. A language model contains dozens of such layers. Composing adapters on different layers of a Transformer is therefore not guaranteed to equal applying them one after another, and neither is guaranteed to equal averaging their outputs. Each operation has to be evaluated as itself.

Compatibility before arithmetic

Weight addition is only meaningful if the deltas were built for the same coordinates. The implementation refuses to compose unless every adapter declares:

  • the same base identity, a digest of the frozen base’s state;
  • the same target module, with matching input and output dimensions;
  • the same rank profile and scaling convention;
  • the same policy as the active system, unless a stale diagnostic is explicitly requested (Chapter 39 used that override once, on purpose);
  • the same precision (FP64 here);
  • finite, bounded factors, with at most four adapters and composition weights bounded in magnitude by 4.

These checks establish that the arithmetic is well defined. They say nothing about whether the result is any good. A delta trained against a different base model can be shaped correctly and still be meaningless when added to this one. Base-identity checking is the only defence against that, and it is a check of provenance, not of behaviour.

Interference

Compatibility does not prevent interference. Consider a conflicting adapter that uses the uncertainty feature like B but pushes the other way, with B = [3, 0, −3]ᵀ. Combined with adapter A, it gives

$$ W_{\text{conflict}} = W_0 + \Delta W_A + \Delta W_{\text{conflict}} = \begin{bmatrix} 1 & 1 & -1 & 4 \\ 0 & 0 & 2 & 0 \\ 0 & 0 & 0 & -3 \end{bmatrix}. $$
Every request whose uncertainty feature is on now has its `complete` score raised by 3 and its `review` score lowered by 3. An uncertain read scores (5, 0, −3). An uncertain erase scores (3, 2, −3). An unknown operation scores (4, 0, −3). All of them are completed, although every one of them required review. The clear requests, whose uncertainty feature is off, are untouched.

The experiment measures this on two capability subsets of the 24 test cases. Task A is the 12 clear requests: certain reads, copies and erases. Task B is the 12 requests that require review. Task B is the capability that must be retained when anything is added to a system that already has it.

System Task A: clear requests Task B: review required
Base 8/12 0/12
Adapter A only 12/12 0/12
Adapter B only 8/12 12/12
A + B, unmerged 12/12 12/12
A + B, merged 12/12 12/12
A + conflicting delta 12/12 0/12

Each adapter alone carries one capability. Adapter A gives 12/12 on clear requests and nothing on review. Adapter B gives 12/12 on review but still completes the 4 certain erases. Composition keeps both, and merging changes nothing. The conflicting delta keeps task A at 12/12 and destroys task B: 0/12. Overall task success for that system is 12/24. That is the same total as adapter A alone, reached by a different route, and a single aggregate would not distinguish the two.

The conflict is easy to see here because the deltas are rank 1 and each touches one named feature. In a real model, two adapters trained on different data change thousands of shared directions. Their interference is spread across many small conflicts in sign and magnitude, not concentrated in one column. That is why merging methods designed for real models do more than add.

TIES and DARE

Two published merging methods address interference directly. Neither was implemented or run in this experiment. Their status in the engineering record is LITERATURE_ONLY_NOT_IMPLEMENTED, and weighted delta summation must not be relabelled as either.

Task arithmetic (Ilharco et al., 2023) treats the difference between fine-tuned and pre-trained weights as a task vector. It reports that adding task vectors can produce a model that does well on several tasks, and that negating one can reduce performance on its task. The composition in this chapter is the LoRA-delta special case of that addition.

TIES-Merging (Yadav et al., 2023) identifies two sources of interference when task vectors are added: redundant small changes and sign disagreements between models. It applies three steps. It trims each task vector by resetting entries that changed only a small amount. It elects a sign for each parameter by resolving conflicts across models. Then it performs a disjoint merge, averaging only the entries that agree with the elected sign. Applied to this chapter’s conflicting pair, the uncertainty column has opposing signs, and sign election would keep one adapter’s direction and discard the other’s. Which one survives would depend on magnitudes. No such merge was performed here.

DARE (Yu et al., 2024) randomly drops entries of each delta with probability p and rescales the survivors by 1/(1 − p), so that each entry keeps its expected value. The authors report that supervised fine-tuning deltas are highly redundant, so many entries can be dropped, and they use DARE to sparsify deltas before merging to reduce interference. That redundancy claim concerns large fine-tuned language models. The tiny constructed delta does not establish the redundancy reported for those large models. Removing an entry can alter behaviour, depending on which entry is removed; it need not delete a capability.

Parameter accounting

Counting parameters for composed adapters requires saying what is being counted.

Quantity Elements
Frozen base weight (3 × 4) 12
One rank-1 adapter: A (1 × 4) plus B (3 × 1) 7
Logical unmerged total: base plus two adapters 26
Logical merged total: one dense weight 12
Runtime cached nn.Parameter elements, unmerged implementation 50
Additionally cached composed-weight elements 12

The logical counts describe the mathematics. The runtime count describes this readable implementation, in which each LoRA layer holds its own copy of the base weight: 12 + 2 × (12 + 7) = 50. Neither is a memory measurement. Chapter 41 shows why the process-level memory figures on this machine cannot separate a 26-element model from everything else the interpreter loads.

What the experiment shows and does not show

It shows that weighted LoRA delta composition on one linear layer is exact arithmetic, and that merged and unmerged forms agreed exactly in FP64 for this model. It shows that composition keeps capabilities that occupy separate directions and that a conflicting delta can erase a retained capability while leaving the others intact. And it shows that compatibility checks are necessary and not sufficient.

It does not show that independently trained adapters compose well. Nothing was trained. It does not show that merging is numerically exact for real models, precisions or kernels. It says nothing about the effectiveness of TIES, DARE or any other merging method, none of which was run. And the rule baseline still solves the whole task without composing anything.

Practical lessons

  • Decide which operation you mean: switching, delta composition, merging, output combination or sequential application. Then evaluate that operation itself.
  • Compose adapters by adding their deltas, never their factors.
  • Check base identity, target, shapes, rank, policy and precision before any arithmetic, and treat those checks as preconditions, not as quality evidence.
  • Measure every capability separately before and after composition. Retention of old behaviour needs its own denominator.
  • Verify a merge numerically against the unmerged form on real inputs, at the deployment precision, with a stated tolerance.
  • Report logical parameter counts and memory measurements separately.

Limitations

The model is one 3 × 4 linear layer with two constructed rank-1 adapters, so its exact merge equivalence follows from integer arithmetic. The capability matrix uses 12 + 12 cases at a single data seed. The noncommutativity example uses separate 2 × 2 maps, not stacked LoRA layers. TIES and DARE are described from their papers and were not implemented. The record carries AUTHENTICATED_EXECUTION = NOT_ESTABLISHED.

FineTuneLab

ftlab.delivery.composition builds the base and the constructed adapters. It provides delta, combined and sequential, and its System class predicts in merged or unmerged form, computing model and adapter identities from content digests. Adapter validation enforces base identity, shape, rank, policy and precision. The experiment is ch40 in ftlab.delivery.experiments.

Where this leads

A correct composed weight is still only a tensor in memory. To be used it must be written to an artifact, loaded again elsewhere and placed behind an interface. The loaded copy must be shown to be the same model, and the interface must behave like the library it wraps. Chapter 41, Exporting, Serving and Measuring the Result, follows the composed system out of the library and into a local service.

Further Reading

  1. Gabriel Ilharco et al., “Editing Models with Task Arithmetic” (2023). Read the source. Defines task vectors and their arithmetic, the real-model generalisation of the chapter’s delta addition.
  2. Prateek Yadav et al., “TIES-Merging: Resolving Interference When Merging Models” (2023). Read the source. Trim, elect-sign and disjoint merge as the published remedy for the interference the chapter demonstrates; described as literature only, not implemented in the book.
  3. Le Yu et al., “Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch” (2024). Read the source. Introduces DARE sparsification before merging; its redundancy claims must not be transferred to the tiny constructed model.