Fine Tuning From First Principles
Understand fine-tuning from gradients and LoRA to data, preference learning, evaluation and release. Build FineTuneLab and learn when not to train.
A model can be trained successfully and still be the wrong solution to the problem.
Fine-tuning changes the parameters of a model. It can help a language model follow a specialised format, learn a stable task, adapt to a domain or reproduce useful behaviour from examples and preferences. But a lower loss does not guarantee a correct answer. An adapter does not make an obsolete policy current. A small checkpoint does not automatically make a service cheaper. And a completed training run does not establish that training was needed in the first place.
Fine Tuning From First Principles starts with those distinctions. It opens the machinery of model adaptation—from tokens, targets and gradients to data provenance, low-rank updates, preference objectives, evaluation, checkpoint recovery and release decisions—and makes each step inspectable.
The question
When should we change a model’s weights rather than change its instructions, provide better information, use a tool or keep the existing system—and what evidence would justify that decision?
The answer is not simply “when fine-tuning improves a benchmark”. We must ask which task the benchmark represents, which examples the model was allowed to learn from, which behaviour was preserved, what alternatives were compared, and whether the trained result can be reproduced and operated. A method that improves one number while damaging the required outcome has not earned its additional complexity.
What the book builds
The cumulative project is FineTuneLab, an educational Python system that grows alongside the chapters. Its components specify task contracts, audit data and rights, identify models and tokenisers, calculate training budgets, construct batches and losses, train small models and adapters, compare outcomes, preserve checkpoints and evidence, and exercise evaluation and release workflows.
By the end, FineTuneLab has an installed command-line interface and a capstone that connects supported workflows. It also refuses combinations it cannot justify. Its examples include deterministic rules, constructed adapters, small CPU training runs, preference-objective calculations and a local release-and-rollback rehearsal. These are deliberately different kinds of evidence, not one fictional end-to-end trained model.
This is an engineering laboratory, not a claim of a production-ready fine-tuning platform. The purpose is to expose precisely what was computed, learned, simulated, measured or left unrun.
The eight parts
Part I: Choosing the Intervention — Chapters 1–6.
What changes, and should it change at all? Begin with a parameter update small enough to calculate, distinguish fine-tuning from prompting, retrieval, tools and distillation, and define the task contract before touching a trainer. Build a resource ledger, inspect candidate checkpoint identities and establish baselines that can defeat a fine-tune.
Part II: The Mechanics of Training — Chapters 7–12.
From text to a defensible update. Follow tokenisation, chat templates, masks and target positions into next-token loss and teacher forcing. Trace gradients through a small Transformer, derive optimiser behaviour and memory costs, then build a working fine-tune without hiding the operations behind a training framework.
Part III: Data and Supervision — Chapters 13–18.
The objective is only as meaningful as its examples. Examine rights, provenance and privacy; turn task descriptions into labelling contracts; build instruction and conversation datasets; decide which tokens receive loss; prevent contamination across splits; and assess synthetic data with quality controls and dataset records. A valid record is not automatically a trustworthy label.
Part IV: Building an Adaptation — Chapters 19–24.
From a reproducible environment to low-rank updates. Prepare local training infrastructure, identify candidate model baselines, examine supervised adapter training and derive LoRA from linear algebra. Design controlled LoRA comparisons and investigate QLoRA’s frozen quantised base, trainable adapters and real compatibility requirements. Estimated resource fit is not an executed model run.
Part V: Methods, Trade-offs and Recovery — Chapters 25–30.
Different ways to change a model, different costs to account for. Compare full, partial and parameter-efficient updates, explore methods beyond conventional LoRA, separate continued pretraining from instruction tuning, and examine hyperparameters, overfitting and forgetting. Finish with distributed-training cost models and the state needed to interrupt, resume and reproduce training.
Part VI: Preferences, Rewards and Safety — Chapters 31–36.
The signal changes; the need for evidence does not. Move from demonstrations to response comparisons, derive DPO and related objectives, then examine reward models, PPO, GRPO and verifiable rewards. Study refusals, specification gaming and reward hacking. A higher preference or reward score is not automatically a safer or more useful answer.
Part VII: Evaluation and Operation — Chapters 37–42.
What did the adaptation actually improve? Define task-level evaluation, paired statistical comparisons and ablations; challenge systems under distribution shift; test adapter composition and merging; and follow supported artefacts through export, local serving, auditing and rollback. Separate numerical agreement, task success, system behaviour and production-readiness claims.
Part VIII: Specialization and Capstone — Chapters 43–48.
Connect the pieces without concealing their limits. Examine multilingual and multi-task slices, structured tool trajectories, long-context mechanisms, multimodal projections and expert routing. Then design, run and inspect FineTuneLab’s installed workflow, including a training pipeline, decision handbook and local release rehearsal. The capstone can conclude that the right action is not to fine-tune.
How the book reasons
Each proposed adaptation begins with a failure to explain, not an optimiser to run. Which part of the system owns that failure: the model’s stable behaviour, current external information, the output contract, the tool boundary or the evaluator itself? What is the strongest admissible no-training baseline? What change is permitted, and what must remain unchanged?
The experiments distinguish parameter updates from constructed weights, training loss from generated-answer quality, a numerical estimate from measured resource use, and a passing test from the scientific claim it actually tests. Results remain conditional on their task, data, model, seed, environment, evaluator and authority to execute. Negative and inconclusive findings are part of the argument, not results to hide.
What the evidence supports—and what it does not
The book contains worked derivations, inspectable implementations, synthetic fixtures, local measurements, negative controls and independently checked numerical records. In the capstone, a randomly initialised tiny Transformer is genuinely trained on a small synthetic copy task and adapted towards a rotation task. Its loss changes, but its held-out outputs remain constant across the tested prompts: a lower loss is not evidence that it learned the intended behaviour. Another capstone branch finds a deterministic rule matching a constructed adapter on its small docket fixture. Neither example establishes a universal ranking of training methods.
Two book-level scientific milestones remain unexecuted and unauthorised: an end-to-end LoRA fine-tune of a pinned real open model, and a fixed benchmark comparing prompting, retrieval or tools, and training. The educational CPU capstone is not a substitute for those studies. Execution records and checksums also do not establish externally authenticated execution. The complete 48-chapter manuscript remains a draft, not publication acceptance.
Why first principles
A training framework can report success while the wrong tokens receive loss, the evaluation leaks across data families, the reference model changes, the frozen base moves unexpectedly, or a checkpoint cannot be reconstructed. The resulting numbers can look precise even when the experiment has stopped answering its original question.
First principles make these failures diagnosable. Identify the objects, derive the calculation, instrument the update, preserve its evidence, test the result against a meaningful baseline and refuse conclusions the experiment did not earn.
The aim is not to persuade every reader to fine-tune a model. It is to make the reader capable of explaining when fine-tuning is justified, what it changed, what it improved, what it cost—and when a simpler solution wins.
Chapters
What Fine-Tuning Actually Changes
Trace a real CPU parameter update, separate context from learned state, and inspect a frozen base with a trainable adaptation branch.
When Fine-Tuning Beats Prompting, RAG, Tools or Distillation
Diagnose model failures, separate hard constraints from hypotheses, and design matched experiments that can justify training or keeping the original model.
Write the Task Contract Before Training
Define success as a versioned executable contract, separate structural validity from policy correctness, and preserve evaluation rules before training.
Budget the Adaptation
Account for adaptation memory and lifecycle cost, preserve unknown runtime quantities, and separate analytical feasibility from permission and measured training fit.
Selecting a Starting Checkpoint
Pin the model, tokenizer and protocol; inspect four small Qwen candidates; and separate a reproducible investigation shortlist from evidence of task quality or training compatibility.
A Baseline Can Defeat the Fine-Tune
Build versioned baseline protocols, compare declared interventions on common cases, and preserve no-training and insufficient-evidence outcomes.
Tokens, Chat Templates and Special Tokens
Derive a byte tokenizer, inspect pinned chat templates, and construct checked input IDs, attention masks and unshifted assistant labels.
Next-Token Loss and Teacher Forcing
Derive stable next-token cross-entropy, check a numerical reference, and apply the objective to the exact labels and masks from Chapter 07.
Backpropagation Through a Transformer
Trace a small causal Transformer, derive its vector-Jacobian products, and check selected parameter gradients independently with finite differences.
Optimisers, Learning Rates and Precision
Derive SGD and AdamW updates, preserve token-weighted accumulation, and inspect clipping and mixed precision with checked arithmetic.
The Memory and Compute Budget
Measure a real small Transformer step, reconcile tensor storage with analytical estimates, and distinguish allocator peaks, saved tensors and useful throughput.
Build a Fine-Tune Without a Training Framework
Assemble tokenisation, causal loss, gradients, updates and resource checks into a measured tiny source-training and saved-weight adaptation experiment.
Dataset Rights, Provenance and Privacy
Separate possession from permission, bind evidence to intended uses, and propagate source restrictions through a synthetic data pipeline.
From Real Tasks to Labelling Schemas
Define annotation semantics, uncertainty and adjudication while separating stable incident attributes from policy-dependent routing.
Instruction, Completion and Conversation Datasets
Preserve semantic authority while rendering labelled records into deterministic completion, instruction and chat examples.
Which Tokens Receive Loss?
Turn supervision policy into exact token positions, derive its gradient consequences, and verify causal alignment before training.
Deduplication, Splits and Contamination
Protect evaluation with lineage-aware grouping, distinguish deliberate policy pairs from leakage, and test attacks on split authority.
Synthetic Data, Quality Assurance and Dataset Cards
Assemble provenance, annotations, formats, supervision and group audits into an immutable synthetic release candidate with an explicit freeze gate.
Set Up a Reproducible Local Training Environment
Bind an offline Docker runtime to source, checkpoint metadata and fresh hardware evidence before permitting model work.
Choose the Small Model and Measure Baselines
Choose an inspectable checkpoint and a comparison protocol whose controls can reveal when training adds nothing.
Supervised Fine-Tuning: A Verified Adapter Path
Connect supervised-token masks, frozen parameters, optimiser updates and adapter identities in a minimal training path.
LoRA From Linear Algebra
Derive low-rank updates, factor gradients, initialisation, memory counts and merge behaviour, then verify them against PEFT.
Designing a LoRA Experiment
Turn adapter settings into identifiable paired experiments, distinguish training mechanics from quality, and reject incomplete evidence.
QLoRA: Four-Bit Base, Trainable Adapter
Separate packed weight storage, dequantised computation and adapter gradients, with measured quantisation errors and explicit backend limits.
Full, Partial and Parameter-Efficient Fine-Tuning
Choose which parameters may change, count the gradient, optimiser and checkpoint state each choice creates, and compare policies without confusing capacity with merit.
Beyond Conventional LoRA
Classify IA3, prompt tuning, prefix tuning, DoRA and AdaLoRA by what each changes, derive the IA3 and soft-prompt mechanisms actually implemented, and keep unrun methods labelled as literature.
Continued Pretraining Versus Instruction Tuning
Separate learning a domain's token distribution from learning an instruction-to-response mapping and from answering with facts that change, then read a controlled experiment's negative results as carefully as its positive ones.
Hyperparameters, Overfitting and Forgetting
Treat schedules, effective batch, epochs, regularisation and checkpoint selection as parts of one experiment, read raw validation curves without smoothing, and measure forgetting where it is possible but not forced.
Scaling Beyond One GPU
Derive what data parallelism, ZeRO, FSDP, CPU offload and activation checkpointing change in per-device memory and communication, and validate a planning ledger without mistaking it for a measurement.
Checkpoints, Reproducibility and Recovery
Define exactly what a training checkpoint must contain, distinguish reload, restart, resume and replay, and test exact CPU recovery against deliberate failures without claiming determinism across hardware or library versions.
Learning From Comparisons Rather Than Demonstrations
Treat a preference pair as a recorded judgement rather than ground truth, represent comparisons as a directed graph, detect duplicates, contradictions and cycles, model label noise, and split by family before any label exists.
Direct Preference Optimization From First Principles
Derive the DPO loss from the Bradley–Terry model and the KL-regularised reward objective, explain every term and its gradient, and run it on a tiny random policy where the training objective falls but held-out preference agreement does not move.
Alternatives to DPO
Derive ORPO, SimPO and KTO from the choices DPO makes, fix a real numerical defect in the odds calculation, and compare the objectives on one starting policy where SimPO's own objective improves while held-out agreement falls to 1/4.
Reward Models, PPO and Classical RLHF
Separate the reward model, the policy and the value baseline; derive the pairwise reward loss and PPO's clipped surrogate for both advantage signs; and test both on tiny fixtures where the reward model fails exactly on the input pattern its training never showed.
GRPO and Verifiable-Reward Fine-Tuning
Replace PPO's learned value baseline with group-relative advantages, compute them by hand including the zero-variance case, and show on a three-action bandit that a verifier drives true success up while a marker proxy drives it below 0.007.
Safety, Refusals and Reward Hacking
Account for every outcome of a refusal policy against its own denominator, reward a toy policy for refusing, and show that legitimate completion collapses from 2/2 to 0/2 while the refusal reward reaches 6/6 and overall correctness stays exactly where it was.
Evaluation That Measures the Task
Separate the model's probabilities from the system's delivered result, derive every task metric with its own denominator, and show a controlled postprocessor fault that lowers task success to 23/24 while the model's negative log-likelihood stays at 0.284779.
Statistical Confidence and Ablation
Put a paired, family-clustered bootstrap interval around a difference between systems, show the same 16/24 effect producing a zero-width interval at one data seed and a wide one at another, and keep data-seed, bootstrap-seed and training-seed variation apart.
Generalisation, Shift and Adversarial Tests
Derive 144 linked challenges from 24 test origins, separate inputs that only look new from patterns that are genuinely unseen, and show a stale policy source producing 0/24 task success while selecting the correct action in 20/24 cases.
Adapter Composition and Model Merging
Derive what adding, merging and sequencing rank-1 adapter deltas does to a weight matrix, measure which capabilities survive composition, show a conflicting delta erasing a retained behaviour, and check that merged and unmerged FP64 probabilities agree exactly in this tiny model.
Exporting, Serving and Measuring the Result
Export the composed model as identity-checked JSON artifacts, reload it, serve it on a bounded loopback HTTP interface whose 24/24 responses match the library, and read each local timing and memory figure against the boundary it actually measures.
Operating, Auditing and Rolling Back a Fine-Tune
Keep experiment completion, evaluation, review, authorisation, activation, health and rollback as separate states, then run a sandbox incident in which a release label stays correct while the loaded model changes, and a retried rollback restores all 24 baseline outputs and the original model identity.
Multilingual and Multi-Task Adaptation
Write a multi-task, multi-language objective as a weighted sum of slice losses, then show with executed fixtures how a pooled score can rise or stay flat while one slice gets worse, why the comparator must always be named, and why synthetic language renderings say nothing about multilingual competence.
Tool Use, Code, Structured Outputs and Agent Data
Define whole-trajectory success for a tool-using system as a conjunction of syntax, order, arguments, authority, results, terminal answer and side effects, then replay one valid and twelve defective traces through a strict validator: one succeeds, and eleven of the twelve failures pass the syntax check.
Long Context, Multimodal Models and Mixture of Experts
Treat long context, multimodal projection and mixture-of-experts routing as three separate mechanisms, derive what each changes in the forward pass, check them on tiny CPU fixtures, and show why a compatibility check that refuses unsupported combinations is the honest substitute for runs that were never made.
FineTuneLab: Design the Usable Solution
Describe the installed ftlab command as an architecture: a strict configuration, fixed profile and backend pairs, contracts that cannot be overridden, a three-node typed graph with sealed receipts, and separate meanings for planning, authorising, executing, evaluating and accepting. Every command and output shown was run.
FineTuneLab: Run and Compare the Training Pipeline
Run FineTuneLab's pipeline branch by branch: a contract that forbids training, a constructed docket system beside its rule control, a real tiny training run whose loss falls while every output stays constant, a preference objective computed by hand and an interrupted run that resumes exactly. Then decide, from that evidence, when fine-tuning is justified.
FineTuneLab: Release, Runbook and Decision Handbook
Follow FineTuneLab from installation to a rehearsed rollback with commands that were actually run, then collect the operating runbook, the limits of the release rehearsal and a decision handbook that separates what the code decides from what a person must judge. Fine-tuning is an engineering decision supported by evidence.