Fine Tuning From First Principles cover
Programmer.ie Book In development

Fine Tuning From First Principles

Understand fine-tuning from gradients and LoRA to data, preference learning, evaluation and release. Build FineTuneLab and learn when not to train.

A model can be trained successfully and still be the wrong solution to the problem.

Fine-tuning changes the parameters of a model. It can help a language model follow a specialised format, learn a stable task, adapt to a domain or reproduce useful behaviour from examples and preferences. But a lower loss does not guarantee a correct answer. An adapter does not make an obsolete policy current. A small checkpoint does not automatically make a service cheaper. And a completed training run does not establish that training was needed in the first place.

Fine Tuning From First Principles starts with those distinctions. It opens the machinery of model adaptation—from tokens, targets and gradients to data provenance, low-rank updates, preference objectives, evaluation, checkpoint recovery and release decisions—and makes each step inspectable.

The question

When should we change a model’s weights rather than change its instructions, provide better information, use a tool or keep the existing system—and what evidence would justify that decision?

The answer is not simply “when fine-tuning improves a benchmark”. We must ask which task the benchmark represents, which examples the model was allowed to learn from, which behaviour was preserved, what alternatives were compared, and whether the trained result can be reproduced and operated. A method that improves one number while damaging the required outcome has not earned its additional complexity.

What the book builds

The cumulative project is FineTuneLab, an educational Python system that grows alongside the chapters. Its components specify task contracts, audit data and rights, identify models and tokenisers, calculate training budgets, construct batches and losses, train small models and adapters, compare outcomes, preserve checkpoints and evidence, and exercise evaluation and release workflows.

By the end, FineTuneLab has an installed command-line interface and a capstone that connects supported workflows. It also refuses combinations it cannot justify. Its examples include deterministic rules, constructed adapters, small CPU training runs, preference-objective calculations and a local release-and-rollback rehearsal. These are deliberately different kinds of evidence, not one fictional end-to-end trained model.

This is an engineering laboratory, not a claim of a production-ready fine-tuning platform. The purpose is to expose precisely what was computed, learned, simulated, measured or left unrun.

The eight parts

Part I: Choosing the Intervention — Chapters 1–6.
What changes, and should it change at all? Begin with a parameter update small enough to calculate, distinguish fine-tuning from prompting, retrieval, tools and distillation, and define the task contract before touching a trainer. Build a resource ledger, inspect candidate checkpoint identities and establish baselines that can defeat a fine-tune.

Part II: The Mechanics of Training — Chapters 7–12.
From text to a defensible update. Follow tokenisation, chat templates, masks and target positions into next-token loss and teacher forcing. Trace gradients through a small Transformer, derive optimiser behaviour and memory costs, then build a working fine-tune without hiding the operations behind a training framework.

Part III: Data and Supervision — Chapters 13–18.
The objective is only as meaningful as its examples. Examine rights, provenance and privacy; turn task descriptions into labelling contracts; build instruction and conversation datasets; decide which tokens receive loss; prevent contamination across splits; and assess synthetic data with quality controls and dataset records. A valid record is not automatically a trustworthy label.

Part IV: Building an Adaptation — Chapters 19–24.
From a reproducible environment to low-rank updates. Prepare local training infrastructure, identify candidate model baselines, examine supervised adapter training and derive LoRA from linear algebra. Design controlled LoRA comparisons and investigate QLoRA’s frozen quantised base, trainable adapters and real compatibility requirements. Estimated resource fit is not an executed model run.

Part V: Methods, Trade-offs and Recovery — Chapters 25–30.
Different ways to change a model, different costs to account for. Compare full, partial and parameter-efficient updates, explore methods beyond conventional LoRA, separate continued pretraining from instruction tuning, and examine hyperparameters, overfitting and forgetting. Finish with distributed-training cost models and the state needed to interrupt, resume and reproduce training.

Part VI: Preferences, Rewards and Safety — Chapters 31–36.
The signal changes; the need for evidence does not. Move from demonstrations to response comparisons, derive DPO and related objectives, then examine reward models, PPO, GRPO and verifiable rewards. Study refusals, specification gaming and reward hacking. A higher preference or reward score is not automatically a safer or more useful answer.

Part VII: Evaluation and Operation — Chapters 37–42.
What did the adaptation actually improve? Define task-level evaluation, paired statistical comparisons and ablations; challenge systems under distribution shift; test adapter composition and merging; and follow supported artefacts through export, local serving, auditing and rollback. Separate numerical agreement, task success, system behaviour and production-readiness claims.

Part VIII: Specialization and Capstone — Chapters 43–48.
Connect the pieces without concealing their limits. Examine multilingual and multi-task slices, structured tool trajectories, long-context mechanisms, multimodal projections and expert routing. Then design, run and inspect FineTuneLab’s installed workflow, including a training pipeline, decision handbook and local release rehearsal. The capstone can conclude that the right action is not to fine-tune.

How the book reasons

Each proposed adaptation begins with a failure to explain, not an optimiser to run. Which part of the system owns that failure: the model’s stable behaviour, current external information, the output contract, the tool boundary or the evaluator itself? What is the strongest admissible no-training baseline? What change is permitted, and what must remain unchanged?

The experiments distinguish parameter updates from constructed weights, training loss from generated-answer quality, a numerical estimate from measured resource use, and a passing test from the scientific claim it actually tests. Results remain conditional on their task, data, model, seed, environment, evaluator and authority to execute. Negative and inconclusive findings are part of the argument, not results to hide.

What the evidence supports—and what it does not

The book contains worked derivations, inspectable implementations, synthetic fixtures, local measurements, negative controls and independently checked numerical records. In the capstone, a randomly initialised tiny Transformer is genuinely trained on a small synthetic copy task and adapted towards a rotation task. Its loss changes, but its held-out outputs remain constant across the tested prompts: a lower loss is not evidence that it learned the intended behaviour. Another capstone branch finds a deterministic rule matching a constructed adapter on its small docket fixture. Neither example establishes a universal ranking of training methods.

Two book-level scientific milestones remain unexecuted and unauthorised: an end-to-end LoRA fine-tune of a pinned real open model, and a fixed benchmark comparing prompting, retrieval or tools, and training. The educational CPU capstone is not a substitute for those studies. Execution records and checksums also do not establish externally authenticated execution. The complete 48-chapter manuscript remains a draft, not publication acceptance.

Why first principles

A training framework can report success while the wrong tokens receive loss, the evaluation leaks across data families, the reference model changes, the frozen base moves unexpectedly, or a checkpoint cannot be reconstructed. The resulting numbers can look precise even when the experiment has stopped answering its original question.

First principles make these failures diagnosable. Identify the objects, derive the calculation, instrument the update, preserve its evidence, test the result against a meaningful baseline and refuse conclusions the experiment did not earn.

The aim is not to persuade every reader to fine-tune a model. It is to make the reader capable of explaining when fine-tuning is justified, what it changed, what it improved, what it cost—and when a simpler solution wins.

Contents

Chapters

01

What Fine-Tuning Actually Changes

Trace a real CPU parameter update, separate context from learned state, and inspect a frozen base with a trainable adaptation branch.

03

Write the Task Contract Before Training

Define success as a versioned executable contract, separate structural validity from policy correctness, and preserve evaluation rules before training.

04

Budget the Adaptation

Account for adaptation memory and lifecycle cost, preserve unknown runtime quantities, and separate analytical feasibility from permission and measured training fit.

05

Selecting a Starting Checkpoint

Pin the model, tokenizer and protocol; inspect four small Qwen candidates; and separate a reproducible investigation shortlist from evidence of task quality or training compatibility.

06

A Baseline Can Defeat the Fine-Tune

Build versioned baseline protocols, compare declared interventions on common cases, and preserve no-training and insufficient-evidence outcomes.

08

Next-Token Loss and Teacher Forcing

Derive stable next-token cross-entropy, check a numerical reference, and apply the objective to the exact labels and masks from Chapter 07.

09

Backpropagation Through a Transformer

Trace a small causal Transformer, derive its vector-Jacobian products, and check selected parameter gradients independently with finite differences.

11

The Memory and Compute Budget

Measure a real small Transformer step, reconcile tensor storage with analytical estimates, and distinguish allocator peaks, saved tensors and useful throughput.

16

Which Tokens Receive Loss?

Turn supervision policy into exact token positions, derive its gradient consequences, and verify causal alignment before training.

22

LoRA From Linear Algebra

Derive low-rank updates, factor gradients, initialisation, memory counts and merge behaviour, then verify them against PEFT.

23

Designing a LoRA Experiment

Turn adapter settings into identifiable paired experiments, distinguish training mechanics from quality, and reject incomplete evidence.

26

Beyond Conventional LoRA

Classify IA3, prompt tuning, prefix tuning, DoRA and AdaLoRA by what each changes, derive the IA3 and soft-prompt mechanisms actually implemented, and keep unrun methods labelled as literature.

27

Continued Pretraining Versus Instruction Tuning

Separate learning a domain's token distribution from learning an instruction-to-response mapping and from answering with facts that change, then read a controlled experiment's negative results as carefully as its positive ones.

28

Hyperparameters, Overfitting and Forgetting

Treat schedules, effective batch, epochs, regularisation and checkpoint selection as parts of one experiment, read raw validation curves without smoothing, and measure forgetting where it is possible but not forced.

29

Scaling Beyond One GPU

Derive what data parallelism, ZeRO, FSDP, CPU offload and activation checkpointing change in per-device memory and communication, and validate a planning ledger without mistaking it for a measurement.

30

Checkpoints, Reproducibility and Recovery

Define exactly what a training checkpoint must contain, distinguish reload, restart, resume and replay, and test exact CPU recovery against deliberate failures without claiming determinism across hardware or library versions.

31

Learning From Comparisons Rather Than Demonstrations

Treat a preference pair as a recorded judgement rather than ground truth, represent comparisons as a directed graph, detect duplicates, contradictions and cycles, model label noise, and split by family before any label exists.

32

Direct Preference Optimization From First Principles

Derive the DPO loss from the Bradley–Terry model and the KL-regularised reward objective, explain every term and its gradient, and run it on a tiny random policy where the training objective falls but held-out preference agreement does not move.

33

Alternatives to DPO

Derive ORPO, SimPO and KTO from the choices DPO makes, fix a real numerical defect in the odds calculation, and compare the objectives on one starting policy where SimPO's own objective improves while held-out agreement falls to 1/4.

34

Reward Models, PPO and Classical RLHF

Separate the reward model, the policy and the value baseline; derive the pairwise reward loss and PPO's clipped surrogate for both advantage signs; and test both on tiny fixtures where the reward model fails exactly on the input pattern its training never showed.

35

GRPO and Verifiable-Reward Fine-Tuning

Replace PPO's learned value baseline with group-relative advantages, compute them by hand including the zero-variance case, and show on a three-action bandit that a verifier drives true success up while a marker proxy drives it below 0.007.

36

Safety, Refusals and Reward Hacking

Account for every outcome of a refusal policy against its own denominator, reward a toy policy for refusing, and show that legitimate completion collapses from 2/2 to 0/2 while the refusal reward reaches 6/6 and overall correctness stays exactly where it was.

37

Evaluation That Measures the Task

Separate the model's probabilities from the system's delivered result, derive every task metric with its own denominator, and show a controlled postprocessor fault that lowers task success to 23/24 while the model's negative log-likelihood stays at 0.284779.

38

Statistical Confidence and Ablation

Put a paired, family-clustered bootstrap interval around a difference between systems, show the same 16/24 effect producing a zero-width interval at one data seed and a wide one at another, and keep data-seed, bootstrap-seed and training-seed variation apart.

39

Generalisation, Shift and Adversarial Tests

Derive 144 linked challenges from 24 test origins, separate inputs that only look new from patterns that are genuinely unseen, and show a stale policy source producing 0/24 task success while selecting the correct action in 20/24 cases.

40

Adapter Composition and Model Merging

Derive what adding, merging and sequencing rank-1 adapter deltas does to a weight matrix, measure which capabilities survive composition, show a conflicting delta erasing a retained behaviour, and check that merged and unmerged FP64 probabilities agree exactly in this tiny model.

41

Exporting, Serving and Measuring the Result

Export the composed model as identity-checked JSON artifacts, reload it, serve it on a bounded loopback HTTP interface whose 24/24 responses match the library, and read each local timing and memory figure against the boundary it actually measures.

42

Operating, Auditing and Rolling Back a Fine-Tune

Keep experiment completion, evaluation, review, authorisation, activation, health and rollback as separate states, then run a sandbox incident in which a release label stays correct while the loaded model changes, and a retried rollback restores all 24 baseline outputs and the original model identity.

43

Multilingual and Multi-Task Adaptation

Write a multi-task, multi-language objective as a weighted sum of slice losses, then show with executed fixtures how a pooled score can rise or stay flat while one slice gets worse, why the comparator must always be named, and why synthetic language renderings say nothing about multilingual competence.

44

Tool Use, Code, Structured Outputs and Agent Data

Define whole-trajectory success for a tool-using system as a conjunction of syntax, order, arguments, authority, results, terminal answer and side effects, then replay one valid and twelve defective traces through a strict validator: one succeeds, and eleven of the twelve failures pass the syntax check.

45

Long Context, Multimodal Models and Mixture of Experts

Treat long context, multimodal projection and mixture-of-experts routing as three separate mechanisms, derive what each changes in the forward pass, check them on tiny CPU fixtures, and show why a compatibility check that refuses unsupported combinations is the honest substitute for runs that were never made.

46

FineTuneLab: Design the Usable Solution

Describe the installed ftlab command as an architecture: a strict configuration, fixed profile and backend pairs, contracts that cannot be overridden, a three-node typed graph with sealed receipts, and separate meanings for planning, authorising, executing, evaluating and accepting. Every command and output shown was run.

47

FineTuneLab: Run and Compare the Training Pipeline

Run FineTuneLab's pipeline branch by branch: a contract that forbids training, a constructed docket system beside its rule control, a real tiny training run whose loss falls while every output stays constant, a preference objective computed by hand and an interrupted run that resumes exactly. Then decide, from that evidence, when fine-tuning is justified.

48

FineTuneLab: Release, Runbook and Decision Handbook

Follow FineTuneLab from installation to a rehearsed rollback with commands that were actually run, then collect the operating runbook, the limits of the release rehearsal and a decision handbook that separates what the code decides from what a person must judge. Fine-tuning is an engineering decision supported by evidence.