← Fine Tuning From First Principles

A Baseline Can Defeat the Fine-Tune

Build versioned baseline protocols, compare declared interventions on common cases, and preserve no-training and insufficient-evidence outcomes.

An adapted model routes most tickets correctly. Its training loss fell, its responses look polished, and the team can explain how the adapter was saved. Then someone implements a policy lookup: given a category, read the current routing table. The lookup never needs to relearn a changed team assignment. If classification was already adequate, the expensive adaptation may have solved a problem the system did not have.

That outcome would be useful. The purpose of the experiment is to discover which intervention earns its complexity. A fine-tune becomes valuable when it improves a meaningful outcome over relevant alternatives while meeting quality, latency, cost and maintenance constraints. Running an optimiser successfully cannot establish any of those comparisons.

Chapter 02, When Fine-Tuning Beats Prompting, RAG, Tools or Distillation, diagnosed which intervention to investigate. This chapter specifies how that investigation can produce a defensible comparison. We will implement baseline records, evidence transitions and compatibility checks, execute a small rule-and-lookup mechanism, and leave the four Qwen model baselines explicitly unexecuted.

Choose the question before the opponent

A baseline is a reference procedure whose inputs, implementation and evaluation are declared. It need not be a language model. A constant classifier can reveal class imbalance; a rule system can expose a synthetic dataset’s shortcuts; a policy tool can remove information staleness. Different baselines test different explanations of failure.

Suppose an unadapted model produces malformed JSON. Comparing it with a fine-tuned model that also receives a better prompt and a JSON grammar would mix three changes. The improvement might come from the prompt or output constraint. A useful comparison first asks whether the permitted non-training interventions resolve the diagnosed failure. Validation detects a malformed response; constrained generation changes which responses can be emitted. Neither alone establishes that a category or route is correct.

Likewise, comparing a small base model with a larger instruction model does not isolate fine-tuning. Parameters, architecture, pretraining, post-training and input protocol may all differ. Such a systems comparison can be useful if its purpose is choosing a deployable system. It must not be interpreted as a causal estimate of one training method.

Baseline family Question it tests This chapter’s execution boundary
Constant or development-majority diagnostic Does the task reward a trivial answer? Constant output evaluated on the public fixture; no majority estimated from test labels
Deterministic rules Can declared intake fields and limited text patterns solve the case? Executed, with explicit lexical limitations
Validation or constrained generation Is the observed failure structural? Schema validation executed; model constrained decoding planned
Zero-shot prompting Can the pinned checkpoint follow the task without examples? Four Qwen specifications planned
Few-shot prompting Do permitted development examples resolve the failure? Supported specification family; no model run
Retrieval-assisted generation Is missing external information the limiting factor? Index identity required; retrieval execution planned
Policy lookup tool Can changing policy be supplied explicitly? Deterministic lookup executed
Hybrid classification and lookup Can stable labels and changing routing be separated? Rule classification plus lookup executed; model hybrid planned
Fine-tuned checkpoint Do updates improve the residual failure? Hypothetical specification and constructed outputs only
Distilled student Can a smaller system retain a justified teacher’s behaviour? Optional future specification; no teacher/student experiment

The original retrieval-augmented generation paper combines parametric generation with external retrieved memory. That mechanism motivates an information-access control; its reported experiments do not predict performance on incident triage. A small local routing table may require a direct lookup rather than a document retriever. There is no requirement to execute every family when the task does not need it.

Separate a permitted tool from an oracle

Our canonical incident input includes text, policy revision, reported impact and a supplied information-status field. The last field is trusted intake metadata in this reference interface. The validator does not infer whether the ticket is genuinely complete. Giving that metadata to every eligible method is fair under this contract; silently giving it only to the rule baseline would change the question.

The stable outputs are category, severity and the decision to abstain. Route and policy citation depend on the active policy. A policy lookup may read the declared category-to-route table. It may not read a protected label file, infer a label from a record identifier, or use the evaluation split as an input feature.

An oracle that receives the correct category and severity could estimate a conditional ceiling for policy routing. It would not demonstrate end-to-end classification. FineTuneLab rejects a declared gold-label source or an oracle presented as a deployable baseline. Declared access rules cannot discover every hidden file read by arbitrary Python code: the eventual execution harness must enforce the boundary as well as record it.

The executable rule function receives only the incident, task contract and policy. Gold labels are held separately and passed to the fixture scorer. These are public development cases, so their labels are intentionally inspectable. They are not the future held-out benchmark.

Make fairness a versioned contract

A comparison needs a common target, not artificially identical bytes everywhere. Base checkpoints may use a completion protocol while instruction checkpoints use their native conversation template. The exact differences should be fixed before observing final outcomes and justified in the comparison record.

FineTuneLab’s BaselineSpec binds task ID, task version and contract digest; policy identity; dataset revision, source manifest and split digest; scorer identity and metric definitions; allowed information sources; checkpoint and tokenizer identities where applicable; prompt configuration, tools, retrieval index and dependencies. The execution contract records sequence and output budgets, decoding, seed, truncation, retries, repair, timing and resource boundaries.

The specification reuses TaskContract and the Chapter 05 checkpoint validator. It does not introduce another label taxonomy. Metric definitions must exactly match the task contract, including direction and undefined-value behaviour. Prompt configuration has its own content digest; checkpoint references must resolve to the existing pinned manifest, including the tokenizer and chat-template identities.

For a clean ablation, hold every relevant field fixed except the named intervention. For a broader systems comparison, disclose differences such as model identity, prompt protocol, generation settings or a retrieval dependency. Differences in task, policy, evaluation split, scorer, metric definitions or allowed information sources are hard incompatibilities in this first component. Timing/resource boundaries and evaluation rules must also agree. A policy-shift study needs its own explicitly versioned design, not an exemption hidden inside an ordinary comparison.

Our model specifications reserve 256 output tokens within a total sequence limit of 512, use deterministic decoding, reject over-length inputs and allow no retries or silent repairs. Those are proposed controls. No Qwen prompt was run on the triage fixture. A future repair procedure must be a named intervention whose extra attempts, tokens and elapsed time count towards its budget.

A specification is not an execution receipt

A valid configuration describes an experiment that might be possible. It does not establish that the data exist, dependencies work or any predictions were generated. The baseline record therefore has an evidence-backed progression:

    stateDiagram-v2
    [*] --> PLANNED
    PLANNED --> READY_FOR_EXECUTION: manifest + ledger + scope
    READY_FOR_EXECUTION --> EXECUTED: run records + predictions
    EXECUTED --> EVALUATED: scorer binds run + labels
    PLANNED --> INVALIDATED: reason recorded
    READY_FOR_EXECUTION --> INVALIDATED: reason recorded
    EXECUTED --> INVALIDATED: reason recorded
    EVALUATED --> INVALIDATED: held-out reuse
  

PLANNED requires no execution claim. Readiness requires a canonical split manifest, an access ledger and an explicit execution scope. The manifest binds case identity, content, role, permitted uses and state; an alias can point to it but cannot override those restrictions. Non-development permissions require frozen manifests bound by the task contract. Our provisional public fixture grants only mechanism-demonstration use. A real model additionally needs verified model rights and a runtime receipt binding its exact checkpoint, protocol and environment. Metadata alone cannot supply that receipt. An explicitly illustrative model scenario can become ready to process constructed predictions without claiming that a model is ready to run.

Execution requires a run identity, source/environment/seed provenance and exactly one prediction entry for every case in a non-empty input set. Evaluation must bind that run, prediction digest, label digest, scorer and label-access event. Missing task measurements appear as undefined, rather than disappearing from the record. Invalidation is terminal for that record and requires a reason; a corrected study needs new identity and lineage.

Prediction evidence is an ID-to-payload mapping, not a list of identifiers. Missing values and foreign IDs are rejected; identical responses for different cases remain legitimate, as the constant control illustrates. Actual malformed response text can be retained as a failure. Before it becomes an evaluated record, the scorer must supply explicit per-case failure accounting: malformed responses cannot count as primary successes or disappear from the denominator, and schema validity must agree with the payloads. The current failure-accounting adapter supports an all-case proportion primary metric; other aggregations need a verified task adapter. Record version 1.1 enforces these boundaries. Historical version 1.0 records remain evidence of earlier runs, not permission to promote them by editing a status.

Each transition refers to a hashed artifact and the previous event hash. Changing an old prediction or evaluation body breaks its reference. Changing a final status without the transitions is rejected. A constructed prediction bundle cannot be relabelled as a real model run or its invented latency called a measurement.

These checks establish consistency of supplied evidence. They do not authenticate an issuer or prove that a dishonest producer measured what it claims. The future registry must retain trusted producer receipts and raw artifacts. This chapter implements the small ingestion and comparison boundary, not a complete trainer, execution sandbox or final evaluation service.

Fresh runtime evidence has a different entry point

Chapter 05’s screen deliberately keeps historical free memory unknown and model runtime unverified. It remains appropriate for metadata-only investigation. A separate screen_fresh entry point now accepts an explicitly current, hashed hardware observation and independently verified receipt references. It distinguishes dependency checks, model initialisation, inference and bounded training instead of treating them as one compatibility flag.

The observation must be no more than 300 seconds old relative to the caller’s explicit UTC clock. Receipts must bind the same checkpoint manifest, environment and proposal; stage order and contradictory evidence are checked. Caller-supplied resource calibration remains analytical. Even successful receipt ingestion leaves launch permission false and task quality unmeasured. Synthetic unit fixtures exercise this future interface; no current GPU observation, model initialisation or canary was performed here.

The cached tokenizer inspection also now verifies every cached artifact’s bytes and digest before loading it. Missing, tampered or unexpected files cause a specific diagnostic. The code does not silently replace a file or fetch a dependency. Historical Chapter 05 inspection records are retained; the integrity-checked rerun has a separate remediation record.

Design the holdout before producing a winner

The complete synthetic dataset is still planned. The twelve cases below have real file/content identities as software fixtures; those hashes must never be reported as the future benchmark’s hashes. A meaningful held-out benchmark still needs independent label review, diversity, group separation and an adequacy rationale.

Partition Permitted role Separation requirement
Training Learn parameters and construct permitted demonstrations No protected evaluation cases or labels
Validation Select prompts, settings and stopping choices Record choices before final evaluation
Ordinary test Estimate the selected system’s task performance No prompt or hyperparameter repair from its outcomes
Family challenge Test transfer to held-out scenario families Separate family/template/entity groups
Policy-shift challenge Test stable labels under a changed routing policy Explicit policy/task revision; no hidden routing refresh
Retention probes Detect loss of separately specified abilities Frozen probe identity and constrained use

The scikit-learn leakage guidance explains why information from test data must stay outside model development. Here that includes selecting few-shot examples, constructing a retrieval index, changing routing rules and generating preference data. Leakage need not involve gradient updates.

The access ledger records dataset identity, split, purpose, phase, whether labels were accessed, and run identity. Permissions follow the resolved manifest and task-owned split purpose. Unknown authorities and conflicting aliases fail closed; a display name cannot turn test data into validation data. Exact case-content overlap with registered protected manifests is checked independently of case IDs and ordering. This does not replace the later family and near-duplicate audit. Every scored run needs its own label-access entry. If final outcomes lead to intervention selection or repairs, preserve that invalidating reuse; create a new untouched evaluation basis before claiming confirmatory performance. Repeatedly renaming a test split “validation” does not restore independence.

Chapter 18 still owns the benchmark freeze. Chapters 19–20 must validate the runtime and execute the frozen Qwen baselines. Nothing in this protocol authorises weights, inference or training now.

Run a deliberately limited incident mechanism

The public fixture contains obvious access, service, security and billing incidents; missing information; uncertain impact; a negated compromise; a resolved historical compromise; conflicting statements; unseen wording; multiple intents; and malicious text attempting to redirect routing. A separate policy-change probe revises the task contract with parent lineage and shows a route change while category and severity remain fixed.

The rule classifier recognises a small set of literal patterns. It has two explicit guards for the fixture’s negated and historical phrases. It abstains for insufficient or contradictory intake, unknown impact, no recognised category or multiple recognised categories. This is inspectable software, not a general language-understanding system. A different negation or historical construction could defeat its literal guards.

After classification, the lookup reads the exact policy table. Ticket text cannot name a preferred destination and override that lookup. The injection fixture is routed to security response because its recognised category is security. That one case establishes a deterministic software property on the fixture; it is not an adversarial robustness certification.

Run the offline demonstration from the repository root:

python examples/ch06/baseline_demo.py --output experiments/FT-06-FIX/new-run.json

Use a fresh destination. The rule-and-lookup output matches 11 of 12 authored routes. It abstains on “Authentication denies entry”, whose authored label is access. Its seven valid non-abstaining answers are correct on this fixture. Four other abstentions agree with authored human-review routes, so routing accuracy and answer coverage measure different things.

The constant access answer obtains 3 of 12 routes and misses all three high-severity cases. A hypothetical fine-tuned candidate has deliberately constructed outputs: three authored answers are changed to billing while the remaining fields illustrate a different error pattern. Its 9 of 12 route agreements are arithmetic on those constructed values, not output from a trained model.

Fixture quantity Executed rules/lookup Constructed model illustration
Route agreements 11/12 9/12
Structurally valid responses 12/12 12/12
Category macro-F1 0.9500 approximately 0.5179
Valid non-abstaining coverage 7/12 8/12
Selective route risk 0 3/8
High-severity misses 0/3 0/3

Rule baseline against the constructed candidate

The bars keep the two evidence states apart: teal marks the executed rule-and-lookup run on the public fixture, while the dashed series marks the constructed illustration that was never a model run. Read the denominators with the figure — coverage and selective risk measure different things, and an abstain-all system would score zero coverage rather than zero risk.

The comparator returns BASELINE_PREFERRED with ILLUSTRATIVE_ONLY scope and training_justified=false. The primary argument supplies no gain from adaptation. Retention remains undefined, so this is a reason to reject that training argument, not permission to deploy the baseline. Separate probes reject an undisclosed protocol difference, policy mismatch, missing prediction evidence and leaked test-label use.

Read the denominators before the score

For \(n\) evaluated cases, exact routing accuracy is

\[ A=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}[\hat r_i=r_i]. \]
Invalid task outputs count as failures rather than being removed from \(n\). This is the usual fraction-correct construction described by the [accuracy documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html), with invalidity governed by our task. The fixture scorer records structural validity separately from policy/abstention consistency. An output can satisfy the JSON schema and still violate the required human-review relation. Such an output cannot earn valid answer coverage. Invalid-output rate and schema-invalid rate are reported separately.

For category \(c\), precision asks how often a predicted positive is correct, while recall asks how many gold positives were recovered:

\[ \mathrm{Precision}_c=\frac{TP_c}{TP_c+FP_c},\qquad \mathrm{Recall}_c=\frac{TP_c}{TP_c+FN_c}. \]
The [precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) and [recall](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html) references distinguish these denominators. A rare critical class may have low recall despite high overall accuracy. Undefined denominators need an explicit convention.

The task contract defines

\[ F1_c=\frac{2TP_c}{2TP_c+FP_c+FN_c},\qquad \mathrm{MacroF1}=\frac{1}{4}\sum_{c\in\{access,service,security,billing\}}F1_c. \]
This gives each declared class equal weight, consistent with the [macro-averaging definition](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html). Our contract additionally requires the whole macro score to be undefined if any class's F1 denominator vanishes. It never silently omits an absent class. Invalid or abstained predictions contribute false negatives for known gold categories. An empty case set is rejected rather than assigned a perfect score.

Let \(S\) contain valid non-abstaining answers. Coverage and selective route risk are

\[ \mathrm{Coverage}=\frac{|S|}{n},\qquad \mathrm{Risk}=\frac{\sum_{i\in S}\mathbf{1}[\hat r_i\ne r_i]}{|S|}. \]
If every response abstains, coverage is zero and selective risk is undefined. Reporting zero risk would disguise the absence of answers. The high-severity miss rate uses a different denominator: all gold high-severity cases. Below-high predictions, invalid outputs and abstentions count as misses for that metric, with abstentions also reported separately. A system cannot satisfy this constraint merely by abstaining on every difficult incident.

These conventions are not a universal safety measure. They are the existing synthetic task’s definitions. Their importance is that every compared intervention uses the same definitions and reports undefined quantities openly.

Pair outcomes, then account for constraints

For comparable methods, define per-case utility differences \(d_i=u_i^{candidate}-u_i^{baseline}\). The paired mean \(\bar d=n^{-1}\sum_i d_i\) preserves the fact that both systems faced the same cases. Comparing unrelated averages would lose that information. Multiple tickets from one scenario family may be dependent; the future family bootstrap must respect that grouping rather than treat every paraphrase as independent evidence.

The twelve public cases demonstrate arithmetic and software failure paths. They cannot establish statistical significance, broad generalisation or the required confidence bound. Chapter 38 develops the full uncertainty methodology. The current comparator deliberately leaves non-illustrative superiority decisions insufficient until the frozen-benchmark and uncertainty service exists.

The canonical targets remain provisional: at least 99% schema validity, at least three percentage points of primary improvement, at most two points of retention degradation, and no increase in high-severity misses. A future acceptance also needs the declared uncertainty, latency and cost conditions. A higher primary average cannot compensate for a failed safety or retention gate.

The interface distinguishes baseline preferred, candidate preferred, insufficient evidence, incompatible comparison and acceptance constraints violated. Its point-estimate candidate-preferred branch is restricted to explicitly illustrative arithmetic. Neither decision grants training or launch authority. The fixture’s missing retention and canonical latency measurements remain missing.

Measure the whole service before making an economic claim

A request’s latency includes prompt preparation, retrieval or lookup, generation, validation, retries and failures according to the fixed boundary. Report a distribution over a declared workload, with warm-up, concurrency, hardware, timing synchronisation and repetitions specified. A median alone can conceal a slow tail; percentiles from twelve one-pass CPU calls cannot characterise a model service.

The demonstration records individual CPU rule-call durations and an observed arithmetic mean for that narrow workload. Its canonical task latency metric remains undefined because the representative measurement protocol is unfinished. The model illustration’s 25 ms value is an authored assumption. Its resource records cannot be relabelled measured.

If \(N\) is the future request volume, a useful accounting identity is

\[ C_{request}=\frac{C_{training}+C_{evaluation}+C_{maintenance}}{N} + C_{inference}+C_{lookup}+C_{human\ review}. \]
Chapter 04 supplies the broader lifecycle ledger. Here the crucial comparison issue is matching scope: amortised training cost cannot be compared with a baseline's GPU-only marginal cost while omitting its human review. No current tariffs or real model costs were measured. The fixture's zero versus 0.002 accounting units are labelled assumptions, not evidence of a free baseline or a priced deployment.

Preserve a stopping decision

Stop the training proposal when an admissible baseline already meets the real requirements, when the diagnosed failure needs information access rather than learned behaviour, or when expected residual gain cannot justify cost. That judgement needs evidence appropriate to the claim. A heuristic forecast can reject an uneconomic investigation; it cannot manufacture a measured quality win.

Also stop or defer when labels are unreliable, evaluation is inadequate, dependencies are unverified, resource requirements exceed limits, or safety and retention cannot be established. A hybrid may deserve the next investigation when stable classification is useful but routing changes frequently. Keep the no-training outcome available after seeing results rather than treating it as a failure of the project.

The four Chapter 05 Qwen identities and base/instruct distinctions are preserved in planned records. Qwen3 chat explicitly disables thinking; base completion does not borrow an instruction template by default. The unresolved Qwen3 stored/logical parameter discrepancy remains an investigation for later authorised weight inspection. Runtime compatibility, inference measurements, dataset freeze and model quality are still absent.

Part I now supplies parameter-update mechanisms, intervention planning, task contracts, resource ledgers, checkpoint provenance and a baseline comparison foundation. Part II will explain the learning machinery in greater depth. That mathematical work does not remove the obligations established here: a justified adaptation still needs trustworthy data, a strong baseline and comparable evidence before its updates can earn a claim of value.

Further Reading

  1. Scikit-learn developers, “Common pitfalls and recommended practices” (living documentation). Read the source. The reference treatment of data leakage behind the chapter’s access ledger, including leakage without any gradient update.
  2. Scikit-learn developers, “sklearn.metrics.precision_score” (living documentation). Read the source. Pins down the precision denominator, showing why a rare critical class needs its own view.
  3. Scikit-learn developers, “sklearn.metrics.recall_score” (living documentation). Read the source. Pins down the recall denominator behind the high-severity miss rate, where abstentions count as misses.