QLoRA: Four-Bit Base, Trainable Adapter
Separate packed weight storage, dequantised computation and adapter gradients, with measured quantisation errors and explicit backend limits.
An adapter has few trainable parameters, but its frozen base can still dominate memory. If we store that base more compactly, can the adapter continue to learn? Yes, provided the computation and gradient paths are defined correctly. Compact storage, numerical precision and trainability are separate properties.
QLoRA combines a quantised frozen pretrained base with trainable low-rank adapters. The QLoRA paper describes NF4 storage, double quantisation and memory-management choices for that approach. This chapter separates those ideas before testing small pieces of the mechanism.
Our executed work includes a deterministic uniform four-bit CPU demonstration and a bounded bitsandbytes NF4 CUDA tensor check. Neither downloads or loads a pretrained model. A real Qwen QLoRA training experiment remains NOT_RUN_AUTHORIZATION.
Quantisation stores an approximation, not smaller real numbers
A four-bit code can represent at most sixteen distinct values. A floating-point weight can take many more values, so replacing it with a code loses information unless the original happens to lie exactly on the reconstruction grid.
A simple block quantiser uses a scale a and integer code q:
Codes are packed two per byte. Merely putting one small integer in each int8 element would use eight bits per weight, despite calling its numerical range “four bit”. We therefore measure both code packing and scale storage.
The reconstruction error e_i=hat(w_i)-w_i depends on the scale and weight distribution. Away from clipping, nearest rounding gives an error no larger than a/2. A large outlier can make a coarse scale for an entire block and increase errors for the many small weights beside it.
Block size trades metadata against local resolution
One scale for a whole model is cheap but usually too coarse. Smaller blocks adapt their ranges locally, at the cost of more scales.
If a block contains K weights and stores one FP32 scale, the idealised effective bits per weight are
Padding matters for small tensors. A matrix with an odd number of weights still needs a whole final packed byte, and block padding can add further codes. The implementation records the original shape so dequantisation discards padding rather than treating it as real parameters.
A backend can choose different block sizes, metadata layouts or scale precision. Those choices belong in its identity. Applying the formula for one layout to a different library’s allocation would produce a misleading memory claim.
Work through a small reconstruction
For weights [-1, -0.5, 0, 1] under a single symmetric block, a=1/7. The nonzero extremes reconstruct exactly. The half-valued weight lies between available grid points and is rounded to a neighbouring reconstruction, with error bounded by 1/14 under the chosen rounding rule.
The important object is the dequantised matrix used in computation. A code by itself has no weight meaning without its scale, mapping and block position. Losing or mismatching scale metadata can damage a model even when every code byte remains intact.
Our executable reference quantises a 16 by 16 FP32 matrix whose values are linearly spaced from -1.7 to 2.0, using blocks of eight. It measures:
| Quantity | CPU uniform four-bit demonstration |
|---|---|
| Original FP32 weight bytes | 1,024 |
| Packed codes plus FP32 scales | 256 |
| Weight RMSE | approximately 0.043918 |
| Maximum absolute weight error | approximately 0.101569 |
| Maximum forward-output error on the declared input | approximately 0.657780 |
The forward error can exceed the error of an individual weight. A linear output combines many errors, weighted by the input. This example is a numerical demonstration on a structured tensor, not a representative estimate of language model quality.
The storage comparison excludes the temporary dequantised matrix and the rest of the program. It is a persistent base-buffer count. In particular, this educational implementation materialises a floating reconstruction for its linear operation; it is not an optimised fused four-bit kernel.
Bound the forward effect without promising a task score
For a row-batch X and reconstruction error E=hat(W)-W,
In a deep network, normalisation, attention and nonlinearities can amplify, suppress or redirect these differences. Weight RMSE is not a substitute for an approved task evaluation. A finite small-tensor output is also not evidence that a complete quantised model remains accurate or stable.
This is why a real QLoRA comparison needs a floating-base adapter control with matching data, targets and training budget. Otherwise a difference may be caused by quantisation, a different adapter configuration or the surrounding numerical implementation.
Explain NF4 without calling every four-bit scheme NF4
NF4 uses a non-uniform reconstruction codebook designed around a normal-distribution assumption. Its codepoints allocate representational resolution differently from the equally spaced signed integers in our CPU exercise. Block scaling and actual weight distributions still influence the resulting error.
Uniform four-bit arithmetic is useful for deriving packing, scale overhead and gradient boundaries. It cannot establish numerical equivalence with NF4 merely because both store sixteen or fewer code values.
The distinction is testable. The CPU reference reports CPU_UNIFORM_SYMMETRIC_INT4_NOT_NF4. The separate CUDA check invokes the actual bitsandbytes functional quantiser with quant_type nf4, block size 64 and compressed statistics. Its report names the library version and the precise tensor.
We do not transfer an error threshold from one demonstration to the other as if the inputs were matched. They use different matrices and different quantisers.
A useful intuition is to imagine placing reconstruction levels using quantiles of an assumed weight distribution: denser probability regions receive more levels. That motivates a non-uniform codebook, but is not a recipe for reproducing the backend’s exact NF4 constants or zero treatment. The implemented library mapping, normalisation and metadata must still be inspected as a versioned system. A normal-distribution assumption also does not establish that every block in a particular pretrained checkpoint follows that distribution. Outliers and changing activation sensitivity remain reasons to measure actual errors.
Quantise the scales as a second approximation
Scales can consume a substantial fraction of low-bit storage. Double quantisation reduces that metadata by quantising scale information itself.
Our educational illustration maps the first-level FP32 scales to 8-bit values relative to one FP32 maximum. For the 32 scales of the CPU matrix, metadata changes from 128 bytes to 36 bytes: 32 code bytes plus four for the maximum. The measured maximum scale error is approximately 0.0005042.
That illustration measures scale reconstruction only. It does not replace the main demonstration’s forward calculation with a second quantised implementation, and it is not an exact reproduction of bitsandbytes’ nested representation.
Scale errors produce additional weight errors. For a fixed code q, replacing a by a+delta(a) changes its reconstructed weight by q delta(a). The code magnitude therefore controls how a scale approximation propagates locally.
A storage ledger must count both levels, their grouping and any offsets or codebooks. “Double quantisation” does not mean that code bits become free or that all metadata is exactly one byte per block.
Preserve the adapter gradient path
For a frozen quantised base, the evaluation map can be written
The input gradient still contains the frozen base contribution. Running the whole base operation under no_grad can sever paths to earlier adapters, even though the base weights themselves were never meant to change. Freezing storage and detaching a computation are different decisions.
The CPU demonstration attaches rank-two factors to the quantised 16 by 16 projection. It has 64 trainable adapter parameters. The first A gradient is zero because B starts at zero, while the B gradient is nonzero. Later A gradients become nonzero. Three updates leave the packed codes and scales unchanged.
This demonstrates the intended trainability boundary. Its squared-output loss is not a language-model objective or a claim that an adapter repaired the quantisation error. It simply makes the gradient flow and immutable buffers observable.
The figure separates stored bits, compute dtype and trainability. Notice that the two error numbers come from different tensors on different backends, so they must never be compared — and that a real QLoRA comparison would still need its floating-base control.
Storage precision does not determine compute precision
A packed four-bit base must be interpreted by a computation. A backend may dequantise tiles to a floating dtype and use specialised matrix kernels. Adapters, accumulations, loss reductions and optimiser states can have different dtypes again.
Consequently a configuration called “four bit” does not mean every tensor occupies four bits. Activations, gradients and vocabulary logits can remain major memory terms. Gradient checkpointing trades recomputation for activation retention; paged optimiser mechanisms address another part of the allocation problem. Neither creates a universal percentage saving for a model we have not run.
The planned real-model path uses explicitly reviewed FP16/FP32 choices. The checkpoint is stored in BF16 and would be cast to an FP16 base and matrix compute, with FP32 adapters, gradients and optimiser states and FP32 loss reduction. That cast is a declared precision intervention: FP16’s finite range is ±65,504 while BF16’s is approximately 3.4e38. BF16 training is not claimed for the reference RTX 2060. A single software capability flag cannot substitute for checking the required kernel and numerical path.
The quantised-base adapter must also be distinguished from training quantised base weights. Our mechanism freezes the codes and scales throughout. Learning or requantising the base would be a different method and require different gradient and identity rules.
Paged optimiser state addresses a different allocation
Adam’s moment arrays grow with trainable parameter count. A paged optimiser can use a memory-management mechanism that moves eligible state between host and accelerator rather than requiring every page to remain resident on the GPU. That can change the handling of transient pressure; it does not eliminate the state or guarantee that every required allocation can be served.
Host capacity, transfer behaviour and page faults can affect performance. Activations and temporary kernels also have their own allocation requirements. A paged optimiser is therefore not interchangeable with quantising the base or checkpointing activations. These techniques target different terms in the ledger.
Our CPU exercise uses ordinary AdamW. The CUDA check exercises NF4 tensors and adapter gradients, not a paged optimiser. No paged-optimiser performance or WSL memory-management compatibility result is claimed. A future selected optimiser requires an explicit backend/workload test and its own resource evidence.
Verify the actual backend on the actual device
The bitsandbytes 0.48.2 installation guide describes platform and accelerator requirements. A compatibility table motivates a probe; it is not proof that our installed wheel works with our driver and runtime.
The authorised CUDA tensor check runs on the RTX 2060, compute capability 7.5,
using the pinned CUDA 12.4 PyTorch build. It quantises and dequantises a 128 by 64
FP16 tensor, executes a matrix multiplication and backpropagates through tiny
adapter factors. The NF4 tensor reconstruction RMSE is about 0.00881. The book
path checks finiteness before calling the backend: nf4_quantise_checked refuses
NaN or infinity with QUANT_INPUT. The backend itself accepts a NaN and poisons
its entire 64-value block, so a later check cannot reconstruct the lost values.
The same check now attaches a LoRA adapter to a tiny bitsandbytes.nn.Linear4bit
layer (64 by 32, NF4, FP16 compute), runs forward and backward, compares the
output with a dequantised FP16 reference, confirms finite nonzero adapter
gradients, confirms that the four-bit weight has no gradient and records peak
allocation under a 128 MiB ceiling. This exercises the actual four-bit compute
path at tiny-tensor scale only; it is not Qwen QLoRA training.
That check initially exposed missing writable-cache and compiler dependencies. The repaired image includes the explicit kernel cache and pinned compiler packages. Failure was not converted into a CPU result or a different quantisation scheme.
The final report establishes execution of those small operations. It does not establish Qwen loading, end-to-end training, stable mixed-precision optimisation, peak model memory or useful adaptation. Those remain blocked until the model and workload are separately authorised and admitted.
Merging changes the approximation problem
A floating-point adapter can be merged by adding its effective update to a floating base. With a quantised base, the choices differ.
Keeping a floating merged matrix increases storage. Requantising hat(W)+Delta(W) introduces a new error. Retaining the unmerged adapter preserves separate base and factor paths but requires the corresponding runtime. These are different artifacts with different memory and numerical behaviour.
Our reference refuses the floating merge operation for QuantLinear. It does not silently dequantise, merge and requantise under a method name whose equivalence was tested only for floating matrices. A future quantised merge needs its own identity, output comparison and acceptance tolerance.
The same caution applies to save/reload. Codes, scale metadata, backend version, base identity and adapter configuration must remain compatible. An adapter file alone does not specify a complete QLoRA inference or training environment.
What remains before a real QLoRA comparison
The CPU arithmetic, adapter-gradient boundary and selected tiny CUDA kernels are executed evidence. The actual pretrained QLoRA experiment is pending. Its prerequisites include exact weight content verification, compatible model loading, data-use approval, fresh resource admission, fixed controls and valid evaluation evidence.
The 0.2.1 incident set is still a controlled engineering study set with strong simple baselines and no untouched protected benchmark. Lower-bit memory use would not repair its measurement capacity or grant its use. Nor would a successful training run establish superiority over prompting, tools or no training.
The permitted reference can be reproduced through examples/part_iv/all_cpu.py and the separate book-training-gpu service. Reports distinguish uniform four-bit arithmetic, actual NF4 tensor execution and unrun real-model work. That separation leaves a precise next decision: whether to authorise a specific checkpoint acquisition and bounded canary, rather than treating these mechanism results as a completed local fine-tune.
Further Reading
- Tim Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023). Read the source. Introduces NF4, double quantisation and paged optimisers for finetuning through a frozen quantised base; also recommended in Chapter 23 for experimental discipline.
- Tim Dettmers et al., “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale” (2022). Read the source. Develops outlier-aware 8-bit matrix multiplication, explaining why distribution and block scaling govern quantisation error.
- Bitsandbytes contributors, “Installation Guide” (v0.48.2 documentation). Read the source. Tables compute-capability support, showing why installation alone cannot certify the quantised path.