← Fine Tuning From First Principles

Set Up a Reproducible Local Training Environment

Bind an offline Docker runtime to source, checkpoint metadata and fresh hardware evidence before permitting model work.

A CUDA matrix multiplication succeeds, the GPU has several gigabytes free, and the package installer reports no broken requirements. We still do not know whether a particular language model can be loaded safely, whether its tokenizer matches its embeddings, or whether adapting it is an authorised experiment. Each observation answers a narrower question than the proposed run.

Part II made a small Transformer inspectable. This chapter prepares a different boundary: a reproducible environment in which a pinned pretrained checkpoint could eventually be investigated. The distinction matters because a model introduces external weights, architecture-specific kernels, tokenizer assets and additional rights and resource requirements. A working educational loop cannot certify those dependencies by inheritance.

Our executed work stops at environment and tensor verification. No pretrained weights are downloaded or loaded, and no incident-study model is evaluated. The result is useful precisely because the missing prerequisites remain visible.

Reproducibility is a relationship between identities

A package version identifies a release, while a wheel hash identifies one distribution artifact. Neither identifies the book implementation being executed. An experiment needs both. It also needs the configuration, input data, device, precision and relevant environment. We can represent a run identity schematically as

$$ I_{\mathrm{run}} = H(I_{\mathrm{source}},I_{\mathrm{environment}},I_{\mathrm{model}}, I_{\mathrm{tokenizer}},I_{\mathrm{data}},I_{\mathrm{configuration}}). $$
This hash binds declared inputs; it is not an authenticated account of execution. A process capable of inventing a report can also invent its hash. Trusted execution receipts and independent verification remain separate requirements.

Docker gives us a controlled userspace and a repeatable entry point. It does not put the Windows GPU driver inside the image, fix the amount of free VRAM, or make floating-point reductions identical across devices. The PyTorch reproducibility notes explicitly distinguish controlling randomness within an environment from guaranteeing results across releases and platforms.

For that reason, exact CPU replay and tolerant CPU/CUDA tensor agreement are different tests. We do not weaken an exact file-identity test because a numerical comparison is approximate, or demand identical GPU bytes merely because CPU replay was exact.

Preserve the reference environment when extending it

The existing CPU reference image remains available for ordinary regressions. Part IV adds a separate training stack rather than changing the meaning of an earlier recorded run. The two training profiles share these selected releases:

Component Selected version
Python 3.13.18
PyTorch 2.6.0, separate CPU and CUDA 12.4 builds
Transformers 4.57.3
Tokenizers 0.22.1
PEFT 0.18.0
Accelerate 1.12.0
Safetensors 0.7.0
bitsandbytes 0.48.2
Datasets 4.4.1
Hugging Face Hub 0.36.0

These are selected, tested versions, not a recommendation to install whichever release is newest. The previous reference image uses a different Transformers stack. Keeping both images makes that difference explicit instead of presenting old results as observations from the new environment.

The training wheel lock binds the resolved dependency changes to SHA-256 values. The inherited reference images retain their own locks and captured identities. The CUDA image additionally pins the compiler-package changes required by the tested runtime. Its package inventory and final image identity accompany the evidence. An unavailable historical package should stop a rebuild; replacing it silently creates a different environment.

A successful dependency resolver proves that declared package requirements can coexist. We also import the actual libraries, run the selected PEFT layer, compare its gradients and execute the relevant CUDA kernels. Those behavioural checks answer questions that dependency metadata cannot.

Keep source and generated state apart

The Compose configuration mounts the project read-only at /book and places new results beneath the ignored .local/docker-results directory. Runtime network access is disabled. Temporary state and compiled-kernel caches have explicit writable destinations; neither belongs in manuscript source or the model cache.

The entry-point guard refuses a stale installed FineTuneLab package. It verifies that the imported package comes from the mounted source and reports the current project version. Without that check, a test can appear to validate a repair while actually importing yesterday’s wheel.

The installed-wheel check is deliberately different. It builds a fresh wheel, compares packaged Python and schema bytes against source, installs into an isolated destination and imports from there. Source execution and installed-package execution therefore provide complementary evidence.

New result paths are required. A report is not an editable status display: overwriting an earlier failure would erase the evidence needed to explain an environment repair.

Separate Windows, WSL2 and Linux responsibilities

Our reference host has an NVIDIA RTX 2060 with 12 GB nominal VRAM. The measured software path is Docker Desktop using Linux containers and NVIDIA passthrough. The Windows driver supplies GPU access to the Linux workload. Docker documents its Windows GPU support, while NVIDIA describes the boundary and restrictions in its CUDA on WSL guide.

A native Windows Python installation is a different environment. A successful Linux-container kernel does not verify a Windows wheel, compiler or loader. Likewise, a command demonstrated here is not evidence for a separate Linux distribution or another GPU generation.

When Docker’s Linux engine is stopped, the correct first result is an unavailable environment, not a CPU approximation presented as CUDA. Once the engine is available, device visibility must still be followed by numerical execution. The reference GPU profile fails if CUDA is unavailable; it has no automatic CPU fallback.

Environment boundary: what the image owns

The figure puts each claim on its side of the userspace boundary. Notice that the passing kernel checks and the refused weight gates live on opposite sides: a working matrix multiplication coexists with a blocked loading plan.

This separation also clarifies installation advice. Updating the container’s Python packages cannot repair an incompatible host driver. Changing the driver does not establish that a particular quantisation wheel contains a suitable kernel.

Probe the operations that the experiment will need

A useful environment probe records the GPU name, driver, compute capability, CUDA runtime, compiled architectures, total memory and currently free memory. It then exercises representative operations rather than ending at enumeration.

Our bounded CUDA check uses FP32 and FP16 matrix multiplication, a tiny low-rank linear layer and backward propagation through its adapter path. It also executes bitsandbytes NF4 quantisation and dequantisation on a 128 by 64 tensor, followed by a small matrix multiplication and adapter-gradient calculation.

The reference reports compute capability 7.5 and a driver version of 591.44. The tiny NF4 reconstruction has an RMSE of approximately 0.00881 for that particular linearly spaced tensor. This is a kernel and numerical observation, not a language model result. Chapter 24 examines why its distribution and scale cannot predict the error of quantising a pretrained network.

The check requires at least 1 GiB free before execution, uses a 60-second outer timeout and rejects peak tensor allocation above 128 MiB. Those are probe limits, not a proposed model’s memory budget. Reported free memory is a fresh observation; another application can change it immediately afterwards.

We make no BF16 training claim. A capability boolean is insufficient to establish the availability and numerical behaviour of the complete training path. FP16 and FP32 are the tested paths here. The PyTorch mixed-precision documentation explains why operation precision and parameter storage are separate choices.

Retain the environment failures

The first extended CUDA check failed because Triton tried to create a cache below a read-only home directory. Giving the cache an explicit results-overlay location exposed a second dependency: the runtime lacked a C compiler.

Both failures are informative. Importing a library can initialise a backend that needs runtime compilation even when the requested calculation is small. Package installation alone did not reveal that requirement. The repaired CUDA image contains the pinned compiler dependencies and retains an explicit writable cache. The same bounded check then passes without changing the requested device or substituting a different quantiser.

A cached successful run does not by itself prove a fresh machine can compile the same helper. Reproduction therefore includes the Dockerfile, compiler package lock, cache policy and image evidence. Cache contents are generated state; they are not an undocumented prerequisite shipped in Git.

Connect the device observation to a checkpoint plan

Consider the previously inspected Qwen2.5-0.5B-Instruct configuration. Its logical ledger contains 494,032,768 parameters. At two bytes each, base weight payload is

$$ 494{,}032{,}768 \times 2 = 988{,}065{,}536\ \mathrm{bytes}. $$
That number is a derived payload, not a measured loading peak. The captured safetensors file is slightly larger because a file also carries structure and metadata. The checkpoint itself is stored in BF16. The proposed path converts it to an FP16 base and FP16 matrix compute, with FP32 adapters, gradients and optimiser states and FP32 loss reduction. This is a declared precision intervention: FP16's finite range is ±65,504, whereas BF16's is approximately 3.4e38. A future CPU inventory must report per-tensor finiteness, maximum absolute value, values outside the FP16 range and the maximum relative BF16-to-FP16 round-trip error before any load is authorised.

For the declared rank-eight query/value targets, Chapter 05’s architecture ledger gives 540,672 adapter parameters. A proposed FP32 adapter has 2,162,688 bytes of parameters, the same number of gradient bytes and 4,325,376 bytes for two FP32 Adam moments. These assumptions are explicit; another implementation could choose different storage or additional trainable modules.

The dry-run ledger also includes temporary-buffer assumptions of 256–1,024 MiB and allocator headroom of 512–1,024 MiB. It records checkpoint payload, host/disk requirements, sequence length 256, microbatch one and accumulation four. It does not pretend that these terms exhaust the loading or training peak.

Activations remain unknown for the actual checkpoint and execution path. Consequently the total has a conditional planning lower bound and no justified finite upper bound. Adding free VRAM to a table does not fill that missing term. The resource classifier correctly returns INSUFFICIENT_EVIDENCE, even though the weight payload looks modest beside nominal device capacity.

Gradient accumulation changes examples per update, not the simultaneous microbatch. It also does not automatically divide every source of memory by the accumulation count. The activation graph, vocabulary logits, temporary kernels and loading strategy must be considered in their own units.

Reproduce the permitted work

After building the profiles as described in docker/TRAINING.md, run:

docker compose -f docker-compose.book.yml --profile training run --rm book-training-cpu
docker compose -f docker-compose.book.yml --profile training-gpu run --rm book-training-gpu

These services use the standard timing runner. Discovery/import time is separate from per-test execution. An individual test taking more than 60 seconds is LONG and is excluded from routine chapter checks once registered. A suite lasting longer than a minute does not make every constituent test LONG.

Metadata, cached-tokenizer inspection and loading plans have separate services. Tokenizer inspection is local-only and verifies the already approved cache before using it. No service in the permitted environment path needs pretrained weights.

The model-dependent service entry points refuse acquisition, inference, training, evaluation and checkpoint inspection with NOT_RUN_AUTHORIZATION. They are gates, not executed model experiments. A future author decision must specify which operation is permitted, for which immutable revision and resource limits.

What this environment establishes

We have an inspectable pinned userspace, source-bound execution, actual CPU/CUDA tensor evidence and a conservative checkpoint plan that refuses missing evidence. We have not demonstrated pretrained loading, fit, throughput, model quality, BF16 training or real QLoRA compatibility. Dataset rights and release authority remain separate from every environment result.

That boundary makes the next question precise. Given a runtime in which the relevant small operations work, which checkpoint and baseline comparison are worth authorising? Hardware readiness is one input to that decision, rather than its answer.

Further Reading

  1. PyTorch contributors, “Reproducibility” (v2.6 documentation). Read the source. Distinguishes seeding one environment from cross-release or CPU/GPU guarantees, grounding the run-identity design.
  2. Docker contributors, “GPU support in Docker Desktop for Windows” (living documentation). Read the source. Defines the WSL 2 GPU passthrough boundary used by the reference runtime.
  3. NVIDIA contributors, “CUDA on WSL User Guide” (living documentation). Read the source. States the driver and limitation boundaries that prevent treating a kernel check as training admission.