Choose the Small Model and Measure Baselines
Choose an inspectable checkpoint and a comparison protocol whose controls can reveal when training adds nothing.
Suppose a fine-tuned model classifies six out of ten incidents correctly. That sounds promising until a rule reading one permitted metadata field also gets six correct. Suppose it then classifies every family-challenge example correctly. That sounds stronger until a word-overlap lookup using training examples does the same.
Neither comparison proves that fine-tuning is useless. They show that the score alone does not identify what was learned or whether training was necessary. A small local model is attractive because we can inspect it, afford controlled experiments and preserve failures. Those advantages disappear if the task or baseline protocol rewards a trivial alternative that we never measured.
This chapter prepares the checkpoint decision and common evaluation protocol. Pretrained inference remains unexecuted. The numerical results below characterise the engineering data and its simple controls; they are not model-quality results.
Choose an investigation candidate, then admit an experiment
A checkpoint can be suitable for investigation without being admitted for execution. Its immutable identity, architecture, template and licence evidence are inputs to a decision. Admission additionally needs actual weight integrity, fresh compatible hardware, a bounded workload and the required authorisations.
The four previously captured candidates remain the investigation set:
| Candidate | Pinned revision | Configuration-derived logical parameters |
|---|---|---|
| Qwen2.5-0.5B | 060db6499f32faf8b98477b0a26969ef7d8b9987 | 494,032,768 |
| Qwen2.5-0.5B-Instruct | 7ae557604adf67be50417f59c2c2f167def9a775 | 494,032,768 |
| Qwen3-0.6B-Base | da87bfb608c14b7cf20ba1ce41287e8de496c0cd | 596,049,920 |
| Qwen3-0.6B | c1899de289a04d12100db370d81485cdf75e47ca | 596,049,920 |
These counts come from the reviewed dense architecture ledger applied to captured configurations. They are not rounded marketing names or newly downloaded tensor inventories. The Qwen3 captured/published inventory is 751,632,384, while the configuration ledger is 596,049,920. The difference is exactly
The selected checkpoint is stored in BF16. The planned execution converts it to an FP16 base with FP16 matrix compute, FP32 adapters, gradients and optimiser states, and FP32 loss reduction. That conversion is a declared intervention, not a neutral read: FP16’s finite range is ±65,504 against BF16’s approximately 3.4e38, and the exponent/mantissa trade changes rounding. A later CPU inventory must report, per tensor, whether all values are finite, the maximum absolute value, how many values fall outside the FP16 range and the maximum relative BF16-to-FP16 round-trip error.
The acquisition proposal chooses Qwen2.5-0.5B-Instruct for a bounded next investigation. Its prior metadata provides an instruct template and consistent configuration/storage counts. That is a pragmatic proposal, not a finding that it outperforms the base model or the Qwen3 candidates. The base Qwen2.5 candidate remains useful for a later controlled comparison of instruction tuning.
Understand what base and instruct comparisons change
A base model continues text under its pretraining distribution. An instruct model has undergone additional adaptation intended to respond to instructions or conversations. Comparing them can be useful, but it changes learned weights and usually the expected prompt convention together.
An adapter trained on an instruct model is not directly comparable to a prompt baseline using the base model if the stated question is whether the adapter adds value. The clean primary control uses the same base checkpoint, tokenizer, template, generation settings and policy input as the adapted condition. A base-versus-instruct comparison is a separately named intervention.
Chat formatting is part of this identity. The Transformers chat-template documentation describes how message structure becomes a token sequence. A missing assistant boundary or an extra control token can change behaviour without a weight update. A format-only improvement should not be attributed to the adapter.
Our cached-tokenizer inspection executes the already pinned assets locally, records vocabulary and EOS identities and renders a short artificial prompt. It does not instantiate a model. A working tokenizer establishes an input construction boundary; it cannot establish embedding compatibility or response quality for weights that have not been inspected.
Ask whether the dataset can measure the intended difference
Revision 0.2.0 corrected important data-engineering failures. Its class coverage, opaque incident identifiers and explicit role plan are useful. They do not make it a final benchmark. Revision 0.2.1 adds an explicit measurement-capacity audit and study constraints while preserving the incident content.
Each evaluation role has one distinct base incident per class. Its second surface adds the fixed prefix “Incident report: ” to the same text. Policy-shift pairs also reuse the same incident under two policies. Counting these as independent cases would overstate the evidence.
For a ten-row role, an accuracy of 80% means eight surface rows were correct. When the two surfaces agree, it may mean four of five underlying incidents were correct. Neither description supplies a representative sampling argument. One new case per class cannot reveal the diversity of failures that a deployment comparison needs to understand.
This does not make the set worthless. It is a controlled engineering study set: it can expose identity mistakes, broken masks, incorrect routing and inconsistent predictions on declared relationships. Its appropriate claims are narrower than “generalises to incidents” or “fine-tuning beats retrieval”.
The card revision is a change to disclosure and required controls. It creates no additional independent incident and no new evidence of model performance. Rights and model-study approval remain pending.
Measure the inexpensive controls before interpreting a model
The capacity audit fits its controls on training data only. The class-frequency control chooses the most frequent training category with a fixed tie-breaking rule. Because the data are balanced across five categories, its category accuracy is 20%.
The metadata-only control maps reported_impact values to the most frequent training category for that value. On these fixtures it reaches 60% category accuracy across the evaluation roles. This input is allowed by the canonical task contract. It is therefore legitimate evidence available to a model, rather than automatically a prohibited shortcut.
The important consequence is comparative. A model reaching 60% may have learned little beyond that field. A model reaching 70% has not necessarily demonstrated a reliable ten-point advantage: the cases are few, dependent and constructed.
The lexical control chooses the training incident with greatest word-set Jaccard similarity and transfers its category. Evaluation labels never enter the lookup. Ties use a deterministic record order. The family challenge reaches 100% under this control. Its designed compositions share sufficient vocabulary with the training mechanisms that the challenge does not isolate difficult compositional reasoning.
These observations constrain interpretation. They do not justify weakening the controls or removing inconvenient cases. If the eventual question requires a harder generalisation boundary, it needs a separately designed data revision and sampling protocol.
The figure fixes the shared evidence contract beside the three measured controls and the unrun model zone. Notice that the 60% metadata-only control uses permitted task evidence, so a model reaching 60% may have learned little beyond the impact field.
The study contract therefore requires four controls: class frequency, metadata-only prediction, train-only lexical or nearest-neighbour retrieval, and the existing deterministic rule baseline where applicable. The capacity audit does not newly evaluate the deterministic incident rule model; its future comparison remains subject to the same use gates as other model study work.
State which output is being compared
Category accuracy and routing accuracy are different measurements. Routing depends on both the stable category and the active policy. A method can classify an incident correctly and route it under yesterday’s policy, or route correctly through a lookup despite producing a poor free-text explanation.
The canonical task contract retains category, severity, route, abstention and citation semantics. We do not create a second ontology for the local experiment. The baseline protocol references that contract’s exact hash and the 0.2.1 manifest identity.
Policy shift is particularly small here. Only the access route changes under p2, affecting two of ten p2 surface rows. A method that ignores the supplied policy can therefore retain many correct answers. An aggregate policy-shift score needs a separate count of the cases on which the correct route actually changed.
Severity coverage also limits the question. Medium severity is absent from test, family challenge and policy shift. Perfect accuracy on those slices cannot establish medium-severity competence. A per-class or per-severity table should report the missing support rather than turn it into an implicit success.
Retention is still a set of future probes, not an established earlier capability. No adaptation-versus-before comparison can be inferred from its existence.
Give every method the same evidence contract
The planned comparison includes deterministic rules, policy lookup, prompt-only inference, permitted few-shot prompting, retrieval/tools, base and instruct checkpoints, an adapter condition and the no-training decision. These names describe different interventions, not a list of already measured competitors.
For an adapter-versus-unadapted comparison, fix the permitted incident and policy input, checkpoint revision, tokenizer, prompt template, generation settings, dataset role and scorer. Differences beyond adapter weights need to be declared and justified.
Few-shot examples must come from permitted development data. Retrieval sources must have their own identity and access history. A policy lookup should receive the same current policy available to the model. Degrading a control by withholding information would answer a different question.
The planned deterministic decoding uses no sampling, one beam and a bounded maximum of 128 generated tokens, with a tokenizer-bound EOS rule. These are protocol choices, not measured optimal settings. If truncation prevents a valid response, retain it as a failure instead of repairing the answer invisibly.
For every attempted prediction, record the raw response, record and target identities, model/tokenizer revisions, contract/policy, prompt, generation settings, dataset/split, scorer and execution-source digest. Record exceptions and invalid outputs alongside successful responses. A denominator containing only parseable answers would reward unreliable methods.
Latency is either actually measured under a stated boundary or explicitly absent. A fixture populated with invented milliseconds is not a baseline. The same principle applies to resource use, cost and confidence limits.
Validate predictions without pretending to authenticate them
The implemented prediction validator checks required fields, run-owned bindings and canonical output semantics. An output routing a valid access incident to the billing team is refused. A wrong task, dataset or policy identity fails the run binding rather than being accepted because its JSON is well formed.
Those checks do not create a trusted scorer receipt. A caller can supply an execution digest or scorer name without proving the corresponding code ran. Metric recomputation, authenticated execution/source receipts and production latency/cost enforcement remain open integration dependencies.
This is why the validator returns a deliberately narrow result: evidence shape validated, trusted scorer receipt not established, production comparison not authorised. It is an inspectable building block, not the completed evaluation service.
A no-training decision also needs evidence. If a deterministic policy tool already meets a properly approved task contract, adapting weights may introduce needless maintenance. Here the final benchmark and resource targets remain provisional, so we cannot declare that decision established either.
Use paired outcomes, then respect their unit
For matched incidents, a future comparison can record whether both methods are correct, both wrong, or only one correct. The mean paired difference is
With one base incident per class in an evaluation role, elaborate confidence intervals cannot manufacture population coverage. The honest report can provide paired outcomes, exact support counts, dependence and explicit undefined or unestablished uncertainty. A larger confirmatory study is a different design, not a prettier error bar.
Validation, not protected test performance, governs future prompt or hyperparameter choices. Every inspection changes the evidence about what remained unseen. All current engineering examples have already been inspected during development; no untouched protected benchmark exists.
Reproduce preparation and identify the missing experiment
The offline CPU batch in examples/part_iv/all_cpu.py verifies captured checkpoint sources, inspects approved tokenizer caches, produces the resource plan and writes the baseline protocol. It performs no pretrained inference. The acquisition proposal names exact file bytes and the captured expected LFS-content hash, together with time/disk limits and cleanup rules.
Acquisition itself remains awaiting an explicit author decision. Actual tensor inventory, a separately approved loading canary, dataset rights and trusted evaluation evidence are still prerequisites. We therefore report the real-model baseline experiment as NOT_RUN_AUTHORIZATION.
Once those prerequisites are satisfied, an adapter experiment will have a clear control and a denominator that includes failure. The next chapter builds the training mechanism at a scale where its state, masks and updates can be inspected, while preserving the distinction between that mechanism and a pretrained-model fine-tune.
Further Reading
- Qwen Team, “Qwen2.5 Technical Report” (2024). Read the source. Describes the 0.5B to 72B family with SFT and reinforcement post-training, informing checkpoint choice without authorising execution.
- Hugging Face contributors, “Chat templates” (living documentation). Read the source. Shows how messages become tokens with model-specific control markers, explaining why template alignment is a controlled condition.
- Hamish Ivison et al., “Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2” (2023). Read the source. Provides an open SFT-mix and evaluation recipe that models disciplined baseline reporting.