When Fine-Tuning Beats Prompting, RAG, Tools or Distillation
Diagnose model failures, separate hard constraints from hypotheses, and design matched experiments that can justify training or keeping the original model.
A model sends an incident to the wrong team. An engineer collects examples, trains an adapter and observes a lower validation loss. The next policy change arrives, and the system routes the same kind of incident incorrectly again. It is tempting to prescribe more training. The more useful question is whether the model was ever given the current routing policy.
This is a hypothetical failure, but the engineering mistake is concrete: an intervention can optimise a component that does not own the missing information. A training run may execute correctly while leaving the application’s actual problem untouched. Chapter 01, What Fine-Tuning Actually Changes, showed that an update changes a permitted numerical transformation. It did not show that changing that transformation is the right way to repair an application.
The decision needs a causal explanation, a feasible intervention and a comparison that could reject it. If the model follows an ambiguous instruction literally, clarify the instruction. If the application withholds the relevant policy, supply it. If a stable distinction remains difficult after appropriate controls, investigate supervised adaptation. If the output is correct but too expensive to obtain, investigate execution changes, a smaller checkpoint or teacher-derived training. Each is a hypothesis about the source of the failure, and each needs different evidence.
The title of this chapter therefore poses an empirical question. Fine-tuning beats an alternative when a matched evaluation shows a worthwhile improvement under the task’s quality, retention, resource and operational constraints. We have not run that comparison for a language model in this book. What we will build here is an inspectable way to design it without pretending the answer is already known.
Diagnose the failure at the boundary where it occurs
Start with the observable error and the information available when it happened. Preserve the input, permitted context, policy revision, actual output and expected outcome. Then ask what would have to change for the answer to become correct. Do not begin with the assumption that the observed output reveals which part of the model is deficient.
A missing fact and a weak task distinction are different diagnoses. A classifier might recognise an access incident correctly while routing it according to an obsolete escalation table. Its stable classification and policy-dependent action should receive separate scores. If both are collapsed into “accuracy”, a policy correction can be mistaken for improved classification, or a stronger classifier can conceal stale routing. Giving the model the correct policy during diagnosis is a useful controlled intervention; looking up the test record’s gold route is privileged information and is not a deployable baseline.
Instructions and context selection are separate boundaries too. A prompt might never define whether a historical outage counts as a current emergency. Another prompt might define the rule accurately but bury it beneath irrelevant material. Compare a clarified instruction and a development-selected context policy before attributing either failure to insufficient learned representations. A tool failure is different again: a model cannot obtain an authoritative current status from a tool the application never exposes, and weight updates cannot repair a broken service endpoint.
Repeated malformed output suggests an output-control problem. Repeated confusion between stable technical categories suggests a discrimination problem. A failure requiring capabilities absent even under a carefully constructed development example may call for a different starting checkpoint or a narrower task. These diagnoses can overlap. Training data could teach a model to use supplied context more consistently, but that would not remove the obligation to supply the context or validate the action.
Cost and latency failures also need a boundary. Long prompts, repeated retrieval, serial tool calls, slow execution and a large checkpoint can all contribute. A smaller adapted model is one candidate remedy, not a diagnosis. Measure the request path before attributing the entire delay to model size.
Finally, inspect the evaluator. If reviewers disagree about the expected severity, if the current policy has no rule for an ambiguous case, or if a parser mistakes a valid answer for an invalid one, there may be no trustworthy target to optimise. Better training can reproduce inconsistent labels more faithfully. The appropriate response to unreliable ground truth is to repair the rubric and adjudication process, and to abstain from selecting a training method meanwhile.
Compare mechanisms without making false alternatives
The interventions below change different objects. This matters because they can be combined. An application may retrieve policy, constrain a structured response and use a fine-tuned student model in the same request.
| Intervention | What changes | Useful hypothesis to test | Recurring obligation |
|---|---|---|---|
| Zero-shot prompting | Instructions/context; weights fixed | The task was expressed poorly | Version prompts and re-evaluate edits |
| Few-shot prompting | Adds demonstrations to context | Examples resolve a stable ambiguity | Select examples on development data; account for context cost |
| Structured output and validation | Permitted generation and acceptance rules | Syntax or field boundaries cause failure | Maintain schemas, semantic checks and failure handling |
| RAG | Selected evidence and possibly a trained retrieval/generation pipeline | Relevant knowledge is absent from context | Refresh sources/index, test retrieval and evidence use |
| External tools/policy lookup | Access to authoritative functions or current state | The answer needs a computation or versioned fact | Enforce permissions, handle errors and record source versions |
| SFT | Trainable parameters fitted to demonstrations | A stable response mapping is not learned adequately | Curate labels, retain capability probes and manage checkpoints |
| LoRA / QLoRA | Factorised adaptation; QLoRA also quantises the frozen base | The SFT study needs a feasible update representation | Verify kernels, target modules, masks and actual resource use |
| Continued pretraining | Parameters fitted to a selected corpus objective | Domain representations need adaptation | Audit corpus and check task transfer plus forgetting |
| Preference optimisation | Parameters fitted to comparisons/rewards | Relative response quality is the relevant supervision | Audit preferences, reference/reward assumptions and specification gaming |
| Distillation | Student parameters learned from teacher-derived signals | A cheaper student can retain required teacher behaviour | Verify supervision rights, teacher errors and student generalisation |
| Hybrid system | A combination of these objects | Different failure causes require different repairs | Measure component interactions and total operational cost |
Zero-shot prompting supplies a task without in-context demonstrations; few-shot prompting includes selected examples. Both can change behaviour with the weights fixed. The GPT-3 paper provides a primary reference for evaluating these settings without task-specific gradient updates. Its results do not predict whether our particular checkpoint can solve triage. Our control must specify the prompt, examples, token budget, decoding and task scorer, and must count the cost of those examples at inference time.
Structured output deserves its own control. A decoder can restrict which continuations are permitted by a grammar; a validator can reject an answer that violates a schema or a semantic rule. These are different operations. Rejecting invalid output does not generate a valid replacement, while a grammar cannot establish that a syntactically valid severity is correct. Guided generation research supplies a mechanism for restricting generation. We still need tests for truncation, impossible schemas, unsupported runtime features and a safe response when generation or validation fails. Prompting alone offers no corresponding structural guarantee.
RAG supplies selected evidence to generation, with possible learned components. The original RAG paper studied a trained retrieval-augmented system, so “RAG versus fine-tuning” is not a universal architectural division. A frozen generator with refreshed documents is one useful control. If retrieval supplies the wrong policy, retrieval quality is a separate failure; if the correct policy is supplied but ignored, generation or instruction-following remains at issue. Document updates, access control, source quality and citations remain operational responsibilities even when no generator training occurs.
A policy tool can be simpler than retrieval. When a trusted category and policy version determine one route, a deterministic table lookup may answer the question directly. Its result should carry the revision used, and unsupported inputs should be rejected or escalated. This reduces the amount of policy interpretation entrusted to generation, but does not make the upstream category correct. A tool that returns the desired route by reading hidden evaluation labels has crossed the information boundary and invalidated the comparison.
SFT fits demonstrations using the trainable parameters defined by the experiment. LoRA is a way to represent some of those updates, rather than a separate reason to train. QLoRA trains low-rank adapters through a frozen quantised base; it is not ordinary optimisation of four-bit base weights. These methods introduce checkpoint lineage, serving compatibility and rollback obligations. A smaller saved adapter does not by itself establish shorter prompts, lower serving latency or a successful memory fit. Those are measurements, and a low-memory path that cannot reload or satisfy the task is not an accepted result.
Continued pretraining addresses another hypothesis: adapting representations to domain text before or alongside task-specific supervision. Gururangan and colleagues investigated this distinction. Their study supports considering a domain-corpus experiment; it does not establish that feeding our system more manuals will teach the correct operational routing rule. This candidate requires suitable corpus provenance and a comparison that includes subsequent task quality, training cost and retained capabilities. It is a more indirect intervention than supplying a missing policy to the current request.
Preference optimisation requires evidence about relative desirability or a reward specification. DPO, for example, learns from preferred/rejected responses under a particular objective. An invalid output schema is not sufficient reason to collect preferences, and lower preference loss does not certify the routing action. Preference collection, disagreements and opportunities to exploit an incomplete reward definition become part of the maintenance burden.
Distillation changes the source or transfer of supervision. The foundational distillation paper describes learning another model from predictive signals supplied by a teacher. A student starting from an existing checkpoint can be fine-tuned on teacher responses, using full updates or adapters. Distillation and fine-tuning are therefore compatible descriptions. The student must be tested against both the teacher and an unadapted student control; otherwise an acceptable small checkpoint may be mistaken for a benefit of teacher-derived training. Teacher errors, insufficient student capacity and unclear output-reuse rights can each defeat the proposal.
For every trained candidate, verify the model terms and the rights/provenance of the relevant data before using them. For every external component, establish where inputs travel and which operations are permitted. The decision component records declared constraints; it does not inspect licences, certify permissions or infer them from a public download link. A local-only requirement constrains retrieval, teacher generation and tools as well as the final model.
Knowledge freshness and behavioural stability
The useful distinction is between what should remain stable and what must be updated. “Return these fields” can remain stable while the policy defining a particular route changes weekly. Technical terminology may evolve slowly, yet an emergency escalation rule can change immediately. Treating all of these as one training target couples their update cycles.
A fine-tune can learn to classify an incident or to use an explicitly supplied policy. It can also encode a policy association that becomes obsolete. The former hypotheses preserve a path for current information; the latter requires considering another training and release cycle when policy changes. This is an engineering reason to test externalising policy, not a theorem that retrieval always wins. An unavailable source, insufficient permissions, poor retrieval or unreliable tool use can still make a proposed hybrid fail.
The strongest hybrid hypothesis for our running example separates a category/severity decision from a versioned policy lookup and validates the combined record. The classifier might be the base model, an adapted model or a deterministic rule. We do not need to decide that in advance. Compare each permitted classifier with the same policy source, then inspect policy-version changes independently. A combined score should never hide an unacceptable high-severity miss behind improved schema compliance.
Economics is conditional on acceptable quality
Training introduces fixed and recurring costs. Prompt-heavy inference, retrieval, tools, larger checkpoints, retries and human review introduce different recurring costs. A useful first model compares accepted systems over a stated horizon:
Let an adapted candidate add fixed cost \(\Delta F=F_1-F_0\) and reduce per-request cost from \(c_0\) to \(c_1\). For positive savings and nonnegative incremental fixed cost, equality occurs at
Quality cannot be traded away invisibly to make the denominator attractive. Define required quality, acceptable regression, resource ceilings and latency before accepting a candidate. A cheap model that sends severe incidents to the wrong team may be inadmissible. Nor can a monetary equation establish latency: a smaller model with more retries may cost less yet respond later. Measure end-to-end latency under an appropriate workload, including lookup and failure handling. Chapter 04 will develop the budget in detail; this chapter uses the small equation only to expose the assumptions needed to justify an efficiency experiment.
Evidence must be good enough to reject the proposal
An experiment is weak if any outcome can be interpreted as a reason to train more. State a falsifiable claim instead: after controls selected on development data, adaptation improves the specified stable classification measure while satisfying schema, policy, retention, latency and cost constraints on a locked evaluation protocol. State what would reject that claim, including insufficient uncertainty bounds or failure on a critical slice.
Before comparing methods, establish a label rubric and an independent scorer. Preserve ordinary cases, ambiguous cases, important error classes and meaningful distribution shifts. Keep policy-dependent routing apart from stable category/severity measures. Report invalid outputs and abstentions according to the same declared rules. A method that silently repairs its own responses is a different intervention from one whose raw outputs are scored; compare repair explicitly if it is part of the intended system.
Training-example count is an inventory, not proof of diversity, rights, consistency or sufficiency. Prefer a smaller auditable set to a larger collection that repeats a few templates, and test whether case families rather than near-duplicates determine the apparent gain. Select prompts, examples and hyperparameters on development data. Do not use final evaluation cases as few-shot examples, teacher-generation prompts for training, preference pairs or tuning feedback.
Record task and scorer versions, model/tokeniser identities, generation settings, permitted information and resources. Some interventions deliberately change context or model size; make those differences visible rather than forcing an artificial claim that everything was identical. Separate a mechanism comparison from a whole-system comparison. A smaller student versus its teacher is a legitimate deployment comparison, but the unadapted student control is needed to attribute a gain to distillation.
Chapter 03 will define the first task contract; Chapter 18 must freeze the dataset, evaluation and acceptance thresholds before real-model tuning. We are specifying obligations here, not freezing new task criteria or reporting held-out scores.
Four incident-triage studies with different explanations
The executable cases in examples/ch02/scenarios.json are analyst-declared synthetic scenarios. They contain no model predictions, privileged gold labels or empirical comparisons. Their purpose is to make the consequences of a diagnosis explicit.
A — Format failure. Assume correct semantic fields but inconsistent serialisation. Training is prohibited and the training budget is zero in this fixture. The admissible first study compares a clear zero-shot instruction, development-selected demonstrations, supported output constraints and deterministic validation. Measure semantic correctness as well as validity, and define rejection/escalation behaviour. If a no-training pipeline meets the contract, there is no need to make weight updates part of this solution. If it fails, the fixture’s budget and permission constraints still do not authorise training.
B — Changing policies. Assume stable categories but frequent policy revisions. Test current policy lookup with the original classifier and validation, then consider an adapted classifier with the identical lookup if stable classification itself needs improvement. Include unseen incidents under new policy versions. Record which version produced each route. The proposed hybrid separates stable labels from current actions; it is not a measured win. If a task simultaneously requires a live network policy and forbids network operation, the honest output is an explicit conflict, not a suggestion that enough training can supply future unseen policy.
C — Stable specialist behaviour. Assume recurring category confusion under a stable definition after zero/few-shot controls. This can justify designing an SFT study if rights, examples, retention coverage and resources are available. Keep a rule baseline and inspect whether labels actually distinguish the categories. Compare the base checkpoint with the adapted checkpoint under matched task conditions; use LoRA or QLoRA only after validating the relevant execution path. The fixture’s 200 available examples is a declared count, not a sample-size recommendation or evidence that the model will learn the distinction. The no-training option remains available if later evidence fails to justify adaptation.
D — Inference efficiency. Assume a stronger teacher meets quality needs but exceeds the serving budget. First compare the unadapted smaller student and feasible execution/prompt changes. Then investigate teacher-derived training, with verified teacher-output permissions, an adequate student starting point and a separately evaluated student. Count teacher-data production, student training and maintenance as well as serving. A student adapted using LoRA on teacher demonstrations is both distillation and fine-tuning. If it cannot retain the required quality, its lower nominal serving cost does not settle the decision.
The four cases do not establish an ordering of techniques. They show that different premises lead to different studies. Changing the premises, such as removing trusted labels or the right to use a teacher, must change what the planner is willing to suggest.
A decision component that can say what it does not know
The permanent FineTuneLab foundation now lives in examples/ftlab/src/ftlab/decision.py. It is a Python library with no third-party runtime dependency. The Chapter 02 runner supplies fixtures to that library; future callers can supply another task using the same versioned input contract.
DecisionInput records a description, observed-failure declarations, knowledge volatility, output requirements, training-example count, evaluation quality/coverage, base capability, declared prompt-control coverage, rights and privacy boundaries, teacher availability, budgets, latency and regression tolerance. Unknown or missing fields, invalid values and unsupported schema versions are rejected. Values such as evaluation_quality="adequate" are declarations from the caller. They are not verified experiment records.
The planner separates three levels. Hard constraints block candidates: no training permission, no budget, unverified rights or a prohibited remote teacher cannot be overcome by a favourable heuristic. Heuristics select a proposed study, such as investigating policy lookup for volatile knowledge. Evidence obligations remain attached to every candidate, including canary resource checks and matched evaluation. An unresolved diagnosis or deficient evaluation produces planning abstention; a contradictory operational requirement produces a conflict. No weighted score claims to identify the objectively best technique.
The heuristic priority is explicit: abstain on conflicts or evidence gaps first; then consider an admissible live-policy hybrid; then an efficiency-driven student comparison; then stable-discrimination adaptation after prompt controls; otherwise test no-training controls. Thus, when efficiency and discrimination occur together and a permitted teacher is available, the student study is proposed first. If that candidate is blocked, the adaptation branch can be considered. This orders investigations; it is not evidence that one intervention is optimal.
The priority ordering above is a decision procedure with blocking and abstention exits, not a ranking of methods:
flowchart TD
D["Diagnose failure + record evidence"] --> C{"Hard constraint blocks?"}
C -->|yes| BL["Candidate blocked"]
C -->|no| G{"Evidence gaps?"}
G -->|yes| AB["ABSTAIN"]
G -->|no| V{"Volatile knowledge,\nlookup admissible?"}
V -->|yes| HY["Propose TEST_HYBRID"]
V -->|no| E{"Efficiency binding,\nteacher permitted?"}
E -->|yes| ST["Propose DESIGN_STUDENT_COMPARISON"]
E -->|no| S{"Stable discrimination\nafter prompt controls?"}
S -->|yes| AD["Propose DESIGN_ADAPTATION_COMPARISON"]
S -->|no| NT["TEST_NO_TRAINING_BASELINES"]
Follow the chart from diagnosis to proposal: any hard-constraint block or evidence gap exits before a study is proposed, and every deployment verdict still waits for a matched evaluation.
Run the package’s source-tree demonstration from the repository root:
python -S -m examples.ch02.decision_demo --output experiments/FT-02/my-run.json
python scripts/run_tests.py --output .local/ch02/my-test-timings.json
Choose new report paths. -S disables normal site-package initialisation: the decision demo needs only the standard library, makes no network requests and loads no model. PyTorch remains necessary for the separate Chapter 01 tests in the combined suite. The test runner times each test, labels any duration over 60 seconds LONG, and excludes known long tests from routine chapter checks. An explicit --include-long run is required to include them. Per-test timing includes setup, the test body and teardown/cleanup. Suite execution timing also includes class/module fixtures, but excludes discovery/import and selection; discovery/import duration is reported separately. The full input fixtures, output requirements and source hashes are preserved with the evidence.
The executed demonstration emits these planning labels:
| Synthetic input | Executed output |
|---|---|
| A, format/budget | TEST_NO_TRAINING_BASELINES |
| B, changing policy | TEST_HYBRID |
| C, stable discrimination after controls | DESIGN_ADAPTATION_COMPARISON |
| D, teacher/student efficiency | DESIGN_STUDENT_COMPARISON |
| C with evaluation missing | ABSTAIN |
| B with conflicting offline/live-network requirements | ABSTAIN |
These are outputs of implemented planning rules, not discoveries about model performance. Every deployment verdict is ABSTAIN_UNTIL_MATCHED_EVALUATION. The report includes conditional/blocked candidates, required baselines, evidence gaps, risks, no-training fallback, possible hybrid components and reasons to reconsider. Declared completed controls are explicitly marked unverified. Latency, cost and regression limits appear as acceptance obligations, never as predicted achievements.
Automated tests check constraints, missing evaluation, separate policy coverage, rights, local-only teacher access, deterministic output and the illustrative cost equation. A report that names a successful LLM or claims a measured savings figure would exceed what this component knows. The durable next step is to ingest immutable evaluation records through a future interface, not to reinterpret analyst declarations as evidence. The broader ftlab training/evaluation/serving CLI remains unimplemented.
Use the framework as an experiment procedure
For a new task, first preserve a small set of observable failures and decide whether the expected answers are trustworthy. Identify the information missing at request time, the stable behaviour to be learned, and the constraints the deployed system must obey. Withhold a method decision if these are unknown.
Next establish admissible no-training controls, including a rule or tool when the task permits one. Select prompt variants on development data and inspect their failure slices. If the remaining stable errors support an adaptation hypothesis, plan a bounded training comparison with explicit retention and resource checks. If the remaining error is stale policy, test a current information path before asking weights to memorise another revision. If efficiency is the concern, include an unadapted small-model control before teacher-derived training.
Finally, compare only after fixing what counts as success and what evidence can reject the proposal. Record improvements and regressions together. Retaining the original model, using a rule-based solution or postponing automation because the evaluator is unreliable are valid outcomes. Training is justified by the behaviour of the complete accepted system, not by the presence of a new adapter file.
Status: TECHNICALLY_CHECKED_WITH_LIMITATIONS; author accepted with limitations; unpublished draft. The deterministic planner, constraints, abstention paths, hypothetical cost arithmetic and Chapter 01 remediation have been executed and tested. No real prompting, retrieval, distillation or fine-tuning comparison has run. Sources and exact commands are recorded in research/ch02/; original demonstration evidence remains in experiments/FT-02/, with targeted review repairs recorded separately in experiments/FT-02-R1/. Chapter 03, Write the Task Contract Before Training, turns the proposed quality boundaries into an explicit versioned contract before any claim of adaptation benefit.
Further Reading
- Tom B. Brown et al., “Language Models are Few-Shot Learners” (2020). Read the source. The primary reference for zero- and few-shot evaluation without gradient updates, which is the no-training control any fine-tuning claim must beat.
- Patrick Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020). Read the source. The original RAG paper, which studies a trained retrieval-augmented system and motivates treating retrieval and fine-tuning as combinable.
- Tim Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023). Read the source. Defines the frozen-quantised-base plus adapter boundary, so a small adapter file is never mistaken for a proven memory fit.