Multilingual and Multi-Task Adaptation
Write a multi-task, multi-language objective as a weighted sum of slice losses, then show with executed fixtures how a pooled score can rise or stay flat while one slice gets worse, why the comparator must always be named, and why synthetic language renderings say nothing about multilingual competence.
Part VII ended with a release that could be rolled back and an audit trail that recorded every step. All of that machinery assumed one task, one population and one number that said whether the candidate was better. Real adaptation rarely looks like that. A support system answers in several languages. A single adapter is asked to handle two policy revisions, or a classification head and a generation head at once. The moment there is more than one slice, the summary number stops being a single fact and becomes a choice about how to weigh one group’s success against another’s.
This chapter asks one question: how can an aggregate improve, or stay exactly flat, while a task or language slice gets worse? The answer is arithmetic, not mystery, and once the arithmetic is written down it becomes clear that the choice of aggregate is a decision about whom the system serves. A pooled score that rises can still justify rejecting a candidate.
Slices, weights and the objective
Multi-task learning, training one model on several related tasks at once, has a long history in deep learning; Ruder’s overview is a good map of it. This chapter needs only the part of that story that concerns measurement.
Index a declared slice by a task t and a language ℓ, and write s = (t, ℓ). Each slice has its own loss over its own examples. A multi-task, multilingual objective is a weighted sum of slice losses:
Because differentiation is linear, the gradient of the weighted objective is the weighted sum of slice gradients:
How one slice’s step affects another
Take two slices, a and b, with gradients g_a and g_b. Suppose a training step follows only g_a with learning rate η, so that θ′ = θ − η g_a. A first-order Taylor expansion of slice b’s loss around θ gives:
The diagnostic has limits that are easy to forget. It is local, valid only for a small step at the current parameters. It depends on the parameterisation, so the same model written differently can show a different angle. It measures training loss, not held-out task success. And it does not say how large a useful step is. A negative dot product is a reason to look, not a verdict.
FineTuneLab’s record FT-P8-INTEGRATED-02-P8-43A-s37 contains one worked instance
of the arithmetic. It is labelled MATHEMATICAL_CONSTRUCTED_NOT_TRAINED_GRADIENTS
in the raw record, and it should be read that way: these are supplied vectors,
not gradients measured during training. With g₁ = (1, 0) and g₂ = (−1, 1), the
dot product is 1 × (−1) + 0 × 1 = −1, so the two directions conflict. With slice
weights 0.75 and 0.25 and slice losses 2 and 4, the weighted loss is
0.75 × 2 + 0.25 × 4 = 2.5. No PCGrad projection was implemented and no learned
transfer was measured.
Pooled and macro accuracy
The same weighting problem reappears at evaluation time, where it is easier to hide. Suppose slice s has c_s successes out of n_s cases. Two aggregates are in common use:
Three more definitions are needed. Retention in this chapter means a slice’s
accuracy under a candidate minus the same slice’s accuracy under a named
comparator. It is a difference in accuracy, not a ratio. (The raw records store
it under a field called retention; the cross-check in this chapter’s evidence
package confirms that the field is exactly this difference.) Interference is
a fall in one slice caused by a change aimed at another. Leakage is any path
by which the same underlying item appears on both sides of a split. In
multilingual data, the commonest leak is a translation: an English training
example and its Spanish rendering are the same item, and putting one in
training and the other in test turns a generalisation measurement into a
memorisation measurement. Chapter 17, “Deduplication, Splits and
Contamination”, introduced split hygiene; here it has to be extended to families
of renderings.
The executed fixture
FineTuneLab’s P8-43A experiment was executed at seeds 37 and 38 as FT-P8-INTEGRATED-02-P8-43A-s37 and FT-P8-INTEGRATED-02-P8-43A-s38. It reuses the docket task from Part VII, the constructed classifier whose outputs are one of three actions, complete, refuse or review. Two policy revisions, p1 and p2, define two tasks, because the same request has a different correct action under each revision. Each request is rendered in two declared languages, English and Spanish. Each fixture has 192 rendered rows: 48 for training, 48 for validation and 96 for test.
Every rendered row carries a task head, a language label, an origin, a family and a split. Renderings of one origin stay in the same family and the same split, so an English test item never has its Spanish twin in training. Labels are derived from the trusted policy revision, not inferred from predictions. The implementation refuses fixtures with duplicate identifiers, a family that crosses splits, a missing slice label, a task head that disagrees with the policy, or a pooled count that does not match the declared total.
There is one fact about this fixture that governs everything that follows. The structured feature encoder ignores the request text. The English text is “Fictional request test/37/0” and the Spanish text is “Solicitud ficticia test/37/0”, and the encoder sees neither. The tokeniser does see them, and they differ: the record tokenises the two bare prefixes, “Fictional request” and “Solicitud ficticia”, as 17 and 18 byte-level tokens. Those tokens never reach the classifier. Equal English and Spanish scores in this experiment therefore say nothing about translation, multilingual understanding or cross-lingual transfer. They test the evaluation plumbing: that language metadata is carried, that slices are counted separately and that family isolation holds.
The four test slices each have 24 cases. Four constructed conditions were evaluated on all 96:
| Condition | p1/en | p1/es | p2/en | p2/es | Pooled | Macro |
|---|---|---|---|---|---|---|
| Base constructed system | 8/24 | 8/24 | 4/24 | 4/24 | 24/96 | 0.25 |
| Shared adapter A | 12/24 | 12/24 | 12/24 | 12/24 | 48/96 | 0.50 |
| Specialist A+B | 24/24 | 24/24 | 24/24 | 24/24 | 96/96 | 1.00 |
| Interfering Spanish B | 24/24 | 12/24 | 24/24 | 12/24 | 72/96 | 0.75 |
Both seeds produced exactly these counts. The “adapters” are constructed by writing down their effect, not fitted by an optimiser, so this is a controlled demonstration of a mechanism, not a measurement of learning. The base system always answers complete. It is right on the 8 complete-gold cases under p1 and the 4 under p2, which is why p1 and p2 differ even before any adaptation. The shared adapter fixes every refusal but sends 4 of the 12 review cases to refuse and leaves the other 8 at complete. The specialist gets everything right. The interfering condition is the specialist with a conflicting change applied only to Spanish: on both Spanish slices, all 12 review cases fall back to complete.
Now read the table twice, with two different comparators.
Against the base system, the interfering condition improves every slice. Its change from base is +0.667 on p1/en, +0.167 on p1/es, +0.833 on p2/en and +0.333 on p2/es. A reviewer who only compares with the base would see four improvements.
Against the specialist, the interfering condition loses 12/24 on each Spanish slice and nothing on either English slice. Its pooled score is 72/96 instead of 96/96. A reviewer who compares with the best available alternative sees a language-specific regression.
Both statements are true. The word “regression” is meaningless until the comparator is named, and the honest report states both.
When the pooled score does not move
The four slices above are equal in size, so pooled and macro accuracy always agree on them. The second fixture removes that symmetry. It has 100 cases: a majority slice of 80 and a minority slice of 20.
| System | Majority | Minority | Pooled | Macro |
|---|---|---|---|---|
| Baseline | 64/80 | 16/20 | 80/100 | 0.80 |
| Candidate | 72/80 | 8/20 | 80/100 | 0.65 |
Pooled accuracy is 80/100 for both systems. Nothing changed, by that measure. Majority accuracy rose from 0.80 to 0.90. Minority accuracy fell from 0.80 to 0.40: the candidate gets half of the minority cases wrong that the baseline got right. Macro accuracy falls from (0.80 + 0.80)/2 = 0.80 to (0.90 + 0.40)/2 = 0.65.
These outcomes are constructed, like the four conditions above: the fixture assigns each row’s baseline and candidate result by rule, so the table is executed arithmetic over declared outcomes, not predictions from a system. The derivation is short. The candidate gained 8 majority successes and lost 8 minority successes. In the pooled numerator those cancel exactly. In the macro average, 8 cases are 0.10 of the majority slice and 0.40 of the minority slice, so the loss is four times as heavy as the gain.
If the minority slice is a language whose users depend on the system as much as anyone else, the candidate should be rejected even though its pooled score is unchanged. If the population genuinely is 80 to 20 and every case matters equally, the candidate is a wash. The arithmetic cannot choose between those readings. The deployment population and the declared weights can.
The figure makes the weighting visible: equal bars hide the minority halving under pooled accuracy, while the macro bar falls, and the same interfering condition reads as an improvement against the base but a regression against the specialist. It plots constructed arithmetic, not learned transfer; language text never reaches the classifier.
A useful exercise is to keep every prediction fixed and change only the weights. With population weights 0.5 and 0.5, the candidate scores 0.65 and is worse. With weights 0.8 and 0.2, it scores 0.8 × 0.90 + 0.2 × 0.40 = 0.80, exactly the baseline. With weights 0.9 and 0.1, it scores 0.85 and wins. The same predictions lead to three different decisions.
Uncertainty by slice
Chapter 38, “Statistical Confidence and Ablation”, introduced paired family-cluster bootstrap intervals. P8-43A reuses that machinery for the specialist against the base, slice by slice, with 400 replications, the family as the resampling unit, and a practical threshold of 0.1.
At seed 37, the p2 slices give an effect of 0.8333 with interval [0.7083,
0.9167]. The p1 slices give an effect of 2/3, but every bootstrap draw equals
2/3, so the interval collapses to a single point and the record marks it
degenerate_bootstrap. At seed 38, the p1 interval is [0.4167, 0.8333] and the
p2 interval is unchanged. A degenerate interval is not evidence of certainty. It
happens here because every family contributes the same paired difference, so
resampling families cannot produce variation. It means the resampling scheme
has nothing to say, not that the effect is exact.
Two further cautions apply. The 24 cases in a slice come from 8 families, so the effective sample size is closer to 8 than 24. And the English and Spanish slices share origin families: each English test item has a Spanish rendering of the same origin. The 96 test observations are not 96 independent samples, and the two languages are not independent replications. The experiment therefore keeps the intervals per slice and makes no pooled significance claim.
FineTuneLab’s slice contract
The implementation lives in ftlab.capstone.specialists. It reuses the byte-level
tokeniser, the Part VII docket task contract, its partition and label authority,
the composition adapters, the evaluator and the statistics module. The new
contract is that a rendered record must state its task head, language, origin,
family and split, and that every evaluation is reported per declared slice
before any aggregate is computed. A missing slice is an error, not a zero.
Objective weights that are negative, non-finite or do not sum to one are
refused.
The language label lives outside the strict request schema that the docket contract inherited from Part VII. That is deliberate. The task authority decides what the correct action is from the policy and the request fields. The language label decides only which slice the case is counted in. Mixing the two, for example by letting a language label change the gold action, would turn metadata into a hidden feature.
Negative controls
The implementation refuses a duplicate identifier, a family that crosses from train into test, a missing slice label, a task head that disagrees with its policy revision, a pooled count that does not match the slices, gold labels that disagree with the policy, and objective weights that are negative, non-finite or unnormalised. The focused tests exercise most of these refusals but not all: duplicate identifiers and negative or non-finite weights are refused by the code without a dedicated test, and only unnormalised weights are tested. A further control is built into the design: because the encoder ignores text, changing the language rendering cannot change a prediction. Any experiment in which it did would reveal a bug in the encoder, not multilingual ability.
The authoring cross-check for this chapter recomputed every slice count, pooled total and macro average from the 96 per-case outputs of each condition at both seeds, confirmed that each recorded success equals agreement between output and gold action, recomputed each paired difference from the case outputs, checked the 80/20 tallies from their 100 rows and recomputed the dot product and weighted loss. It did not import FineTuneLab.
What the experiment shows and does not show
It shows that FineTuneLab carries language and task metadata through evaluation, keeps rendering families on one side of each split, reports slices before aggregates, and refuses inconsistent fixtures. It shows, with executed arithmetic, that pooled accuracy can stay flat while a minority slice halves, and that the same constructed condition is an improvement or a regression depending on the comparator.
It does not show multilingual competence. The text never reaches the
classifier. It does not show learned transfer or learned interference, because
every condition is constructed. It does not show that PCGrad or any other
gradient-alignment method helps, because none was run. The optional learned
comparison of shared and task-specific adapters, P8-43B, is recorded as
NOT_RUN_OPTIONAL, and its scientific question remains open.
Interpretation
The fixture is small, but the pattern it isolates is the general one. Any aggregate is a weighted average of slice results, and the weights are a policy choice. When adaptation changes slices by different amounts, an aggregate can move in any direction while individual slices move in others. Gradient conflict is one mechanism that produces such divergence during training. Data imbalance, leakage of translations across splits and shared capacity are others. The evaluation discipline is the same in every case: declare the slices, declare the weights, name the comparator and report each slice before any summary.
Limitations
The language renderings are synthetic and ignored by the features. The adapters are constructed, not trained. Each slice has 24 cases from 8 families, and the English and Spanish slices share families. The gradient vectors are supplied, not measured. The bootstrap intervals are per slice and can be degenerate. The fixture is a docket task built for this book, not a benchmark, and it cannot grant permission to train an incident-triage model.
Practical consequences
- Declare slices and their weights before training, and write down which population the weights represent.
- Report every slice with its numerator and denominator before any pooled or macro figure.
- Name the comparator whenever a result is called an improvement or a regression, and report against both the starting point and the best available alternative.
- Keep all renderings of an origin in one family and one split. Treat a translation as the same item, not a new one.
- Resample at the family level, and do not pool repeated renderings into a significance claim.
- Treat a negative gradient dot product as a diagnostic that something deserves investigation, not as proof of harmful transfer on held-out data.
- Check whether language or task metadata actually reaches the features before interpreting equal scores as transfer.
FineTuneLab
ftlab.capstone.specialists validates rendered records and slices, computes
pooled and macro summaries, change from a named comparator and per-slice paired
intervals through the Part VII statistics module. The experiment driver for
P8-43A is the Part VIII batch script, which replays the frozen plan for the
FT-P8-INTEGRATED-02 namespace. The installed ftlab command does not expose this
experiment as a profile.
Where this leads
A docket classifier emits one structured decision per request. The next chapter, “Tool Use, Code, Structured Outputs and Agent Data”, moves to systems whose output is a sequence of actions: tool calls, their results and a final answer. There the question is not only whether each output is well formed, but whether the whole trajectory did what was asked, and nothing else.
References
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L. and Stoyanov, V. (2019). Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116.
- Ruder, S. (2017). An Overview of Multi-Task Learning in Deep Neural Networks. arXiv:1706.05098.
- Wang, Z., Tsvetkov, Y., Firat, O. and Cao, Y. (2020). Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models. arXiv:2010.05874.
- Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K. and Finn, C. (2020). Gradient Surgery for Multi-Task Learning. arXiv:2001.06782.
Further Reading
- Alexis Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale” (2019). Read the source. Large-scale evidence for the transfer-versus-capacity-dilution trade-off behind the chapter’s weighting argument; already cited in References and retained as the best source for the central trade-off.
- Linting Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer” (2021). Read the source. Shows how language sampling sets slice influence during training, the twin of evaluation-weight choice.
- Jonas Pfeiffer et al., “MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer” (2020). Read the source. Trained modular adapters as the real counterpart to the chapter’s constructed interference conditions.