← Inference From First Principles

Quantization and Numerical Precision

Four different things are called quantisation quality: storage, reconstruction error, model behaviour and speed. Measured separately on GPT-2, they disagree.

I quantised the same matrix, the 38.6-million-entry embedding table of GPT-2, to four bits in two different ways. Applied to the input side, where tokens are looked up, it cost 0.7 per cent in perplexity. Applied to the output side, where the same numbers produce the scores for the next token, perplexity went from 61 to 8,123. The same matrix, the same error of about 13 per cent in its weights, and a difference of four orders of magnitude in the consequence.

Anyone who tells you that a model “loses 1 per cent” under quantisation has averaged over something. This chapter takes the average apart. It measures four things that are routinely collapsed into one, on a real model and a real text, and shows how each can disagree with the others: how much smaller the model is, how close the quantised weights are to the originals, how differently the model behaves, and whether any of it runs faster.

The headline results, which I will justify below: an 8-bit scheme with one scale per output channel is nearly indistinguishable from the original model; the same 8 bits with one scale for the whole tensor is not; 4-bit weights cost 8 to 13 per cent in perplexity and a fifth to a quarter of the model’s top-choice tokens with simple rounding; and on this machine, storing weights in fewer bits made the matrix product up to twenty-six times slower, not faster.

Where this sits

Chapter 7 measured that for a memory-bound operation, time follows bytes: the output projection took 1.9 times longer in 32-bit than in 16-bit. Chapter 11 turned that into a lever in the bottleneck model: narrower data reduces the byte term. This chapter pulls that lever and counts the cost. Chapter 4 supplied the reason to expect trouble: the residual stream of GPT-2 has coordinates a thousand times larger than typical, and Chapter 3 showed that its logits sit around −100, which makes every error in a dot product visible against a narrow spread.

The Inference Lab piece added here is the quantisation workbench: simulated quantisers with a configurable bit width and granularity, a storage account that includes scales and zero points, activation quantisers, an evaluator that reports perplexity, top-1 agreement and KL divergence against the unquantised model, and the speed probes. Every model artefact the Lab produces from now on carries the scheme that made it in its manifest, since a quantised model and its parent are different artefacts with different outputs.

For the Pi series, this is the chapter that decides whether a model fits on a local machine at all. Chapter 15 and the model-serving volume start from the sizes below.

Formats and what they can represent

A number format has a range, set by its exponent, and a resolution, set by its mantissa. The five floating-point formats in use, from torch.finfo:

Format Bits Largest value Smallest normal Relative step (epsilon)
fp32 32 \(3.4 \times 10^{38}\) \(1.2 \times 10^{-38}\) \(1.2 \times 10^{-7}\)
fp16 16 65,504 \(6.1 \times 10^{-5}\) \(9.8 \times 10^{-4}\)
bf16 16 \(3.4 \times 10^{38}\) \(1.2 \times 10^{-38}\) \(7.8 \times 10^{-3}\)
fp8 (e4m3) 8 448 0.0156 0.125
fp8 (e5m2) 8 57,344 \(6.1 \times 10^{-5}\) 0.25

The two 16-bit formats make opposite choices. bf16 keeps fp32’s range and has only 7 bits of mantissa; fp16 has less range and three more bits of precision. Casting 70,000 to fp16 gives infinity and to bf16 gives 70,144. Casting 2,956, the largest residual-stream value found in Chapter 4, gives 2,956 in fp16 and 2,960 in bf16.

For GPT-2, fp16’s range is adequate (that coordinate is 4.5 per cent of its limit), and so its extra precision is the better choice. For larger models whose activations can exceed 65,504, bf16’s range is the reason it is preferred. The choice between them is not a convention; it depends on the model.

Integer formats have no exponent. An integer of \(b\) bits has \(2^b\) evenly spaced levels, and a scale maps them to real values. For symmetric quantisation:

\[ s = \frac{\max|x|}{2^{b-1}-1}, \qquad q = \operatorname{round}(x/s), \qquad \hat{x} = q\,s \]
The error per value is at most \(s/2\), so it is proportional to the largest value in whatever group shares the scale. That one fact is the source of nearly every result below.

Granularity

A scale can be shared by the whole tensor, by each output channel (each column of the weight matrix), or by each block of \(g\) inputs within a channel (a group). Finer granularity means smaller groups, so each scale is set by a less extreme maximum, and also more scales to store. With a 16-bit scale per group, the storage per weight is

\[ \text{bits per weight} = b + \frac{16}{g} \]
A 4-bit scheme with groups of 128 therefore stores 4.125 bits per weight, and with groups of 32 it stores 4.5. Asymmetric schemes, which add a zero point, spend another \(b/g\) bits.

What the weights look like

The 48 linear layers of GPT-2 hold 84.9 million weights, 68 per cent of the model; the rest is embeddings (31 per cent), biases and norms. I measured the shape of their distributions:

Statistic Median over 48 layers Worst
Largest weight over RMS 20.8 186 (layer 3, feed-forward output)
Excess kurtosis (0 for a Gaussian) 3.2 790 (same layer)

The three most outlying layers are the feed-forward output projections of layers 1, 2 and 3, with peak-to-RMS ratios of 186, 162 and 158. A Gaussian matrix this size would have a ratio near 5. Weights in this model are heavy-tailed, with a few very large entries that set the scale for everything sharing it.

The consequence for a scale chosen from the maximum is direct. If the largest weight is 186 times the RMS, an 8-bit grid spaced to span it has a step of 1.5 RMS, so the typical weight is rounded to a handful of levels. Reconstruction error, as relative Frobenius norm between the quantised and original weights:

Scheme Bits per weight Median error Worst layer
int8, per tensor 8.00 0.047 0.421
int8, per channel 8.01 0.0089 0.020
int8, group 128 8.13 0.0068 0.0097
int4, per channel 4.01 0.160 0.279
int4, group 128 4.13 0.124 0.148
int4, group 64 4.25 0.110 0.128
int4, group 32 4.50 0.099 0.109
int4, group 128, asymmetric 4.16 0.104 0.119
int3, group 128 3.13 0.287 0.324
int2, group 128 2.13 0.771 0.792

Per-tensor int8 has a median error of 4.7 per cent and a worst of 42 per cent: in the outlying layers it destroys nearly half of the weight’s norm. Moving to one scale per channel cuts the median five-fold and the worst twenty-fold, for two extra bytes (one 16-bit scale) per output column. Each halving of the group size lowers 4-bit error by roughly 10 per cent, relative, and costs 0.125 to 0.25 bits per weight.

These are statements about weights. They say nothing yet about what the model does.

What the model does

I evaluated each scheme on 57 windows of 512 tokens, 29,127 tokens in all, taken from this book’s own chapters 6 to 11. The fp32 model’s perplexity on that text is 62.69. That figure is high because the windows are short and the prose is technical; it is a baseline and not a benchmark. Three numbers per scheme:

  • Perplexity, how well the model predicts the actual next token.
  • Top-1 agreement: how often the quantised model’s most probable token equals the original’s.
  • KL divergence from the original’s next-token distribution, in nats per position.

The first measures accuracy against the text. The second and third measure fidelity to the unquantised model, which is what matters if the aim is to serve the same model more cheaply.

These are simulated schemes: weights are quantised and dequantised and the model then runs in ordinary floating point. They measure behaviour exactly. They do not measure speed.

Scheme Perplexity Change Top-1 agreement KL (nats) Bits per weight (linear) Whole model*
fp32 (reference) 62.69 – – – 32 475 MiB
fp16 62.70 0.00% 98.3% 0.0002 16 237 MiB
bf16 63.74 +1.67% 87.9% 0.0212 16 237 MiB
int8, per tensor 69.73 +11.2% 78.0% 0.136 8.00 156 MiB
int8, per channel 62.61 −0.13% 97.6% 0.0012 8.02 157 MiB
int8, group 128 62.70 +0.02% 98.5% 0.0005 8.13 158 MiB
int4, per channel 88.34 +40.9% 65.6% 0.343 4.02 116 MiB
int4, group 128 71.05 +13.3% 76.3% 0.145 4.13 117 MiB
int4, group 64 69.79 +11.3% 78.6% 0.115 4.25 118 MiB
int4, group 32 67.99 +8.5% 80.6% 0.090 4.50 121 MiB
int4, group 128, asymmetric 68.67 +9.5% 79.5% 0.105 4.16 117 MiB
int3, group 128 274.65 +338% 34.5% 1.59 3.13 107 MiB
int2, group 128 18,575 collapse 4.4% 6.05 2.13 97 MiB

* Linear layers at the stated width; embedding table, biases and norms at fp16. The fp32 row is 475 MiB for the whole model.

Reading the table.

fp16 is the gold standard for fidelity, not 100 per cent. It changes the top choice at 1.7 per cent of positions (98.3 per cent agreement) while leaving perplexity unchanged to four digits. Even an essentially lossless change flips some top-1 tokens, wherever two candidates are within rounding of each other. That is a floor for agreement, and the relevant question for any scheme is how far below 98 per cent it sits.

Perplexity hides fidelity. bf16 raised perplexity by 1.7 per cent, which sounds like nothing. But it disagreed with the original about the best token at 12 per cent of positions, and its KL divergence is a hundred times fp16’s. A model served in bf16 and one served in fp16 would give visibly different greedy text on this model, with nearly the same score. Whether that matters depends on what the output is for. A reproducibility requirement (Chapter 39) would care, and a benchmark score would not notice.

int8 per channel is effectively free. Perplexity is unchanged within noise (the −0.13 per cent is the sort of difference a different window set would produce, and I did not test its significance), agreement is 97.6 per cent, KL is 0.0012. The model is a third smaller than fp16 at no measurable cost. The same eight bits with a single scale per tensor costs 11 per cent and 20 points of agreement, which is the heavy-tailed weights from the previous section arriving in the metric.

int4 with plain rounding costs 8 to 13 per cent. Smaller groups help: group 32 recovers a third of the loss of group 128 at 0.375 more bits per weight, and an asymmetric group-128 scheme beats the symmetric one for 0.03 bits. But at four bits every simple scheme here changes the top choice at one position in five. This is naive round-to-nearest. Methods that choose the rounding using calibration data, such as GPTQ and AWQ, exist to reduce that gap, and I did not implement or measure them. Nothing here says how much they recover, and larger models are reported to tolerate low-bit weights better, which I also did not test.

There is a cliff. Three bits cost 338 per cent in perplexity and two bits destroy the model. The slope from eight to four bits is gentle and the slope from four to three is not.

The matrix that is used twice

The embedding table is tied: the same 38.6 million numbers turn tokens into vectors at the input and vectors into scores at the output (Chapter 3). Quantising it at int4 with groups of 128 produced a perplexity of 7,862.

I separated the two uses on 12 windows of the same text:

Where int4 (group 128) is applied Perplexity (baseline 60.6)
Input lookup only 61.0 (+0.7%)
Output projection only 8,123
Both (tied) 7,983
Both, int8 per channel 64.7 (+6.7%)
Both, int8 group 128 61.9 (+2.1%)

The input side tolerates four bits easily. The output side cannot. The mechanism is visible in the logits. For a set of final hidden states, I computed the scores with exact and with quantised embeddings:

Value
Spread of exact logits across the vocabulary (std) 3.43
Error std across the vocabulary, int8 group 128 0.24
Error std, int4 group 128 4.74
Error std, int4 per channel 5.25
Mean logit −123.9
Hidden-state norm / embedding-row norm 264 / 4.0

With four bits the noise in the scores is larger than the difference between the scores. The common offset of the error is small (0.08), so it is not the large negative mean being disturbed; softmax cancels that. The noise varies across tokens and swamps a spread of 3.4. I attribute this to the hidden state’s norm of 264, against 4.0 for an embedding row, which multiplies every error in a row. That attribution is an inference from these numbers and I did not isolate it.

What follows is practical and not specific to GPT-2. The output projection is the layer to leave in higher precision, and a tied embedding is two matrices that happen to share storage. In the “whole model” column above I left it at 16 bits, which is why the int4 model is 117 MiB and not 70: the embedding table is 31 per cent of this model, and a model cannot shrink below it.

Quantising activations

Weight-only quantisation reduces memory traffic. Quantising the activations too (W8A8) lets the matrix product itself run in 8-bit integers, but activations are harder because their ranges are set by the input and cannot be examined in advance. I simulated int8 weights per channel with dynamic int8 activations, with a scale computed per tensor or per token:

Scheme Perplexity Change Top-1 agreement KL
W8A8, activation scale per tensor 85.95 +37.1% 65.2% 0.383
W8A8, activation scale per token 63.00 +0.49% 94.2% 0.0073
W8A8, per tensor, with smoothing (α = 0.5) 79.69 +27.1% 68.1% 0.300

A single scale per tensor is ruinous for activations: +37 per cent, with agreement down to 65 per cent. Per token it is nearly free. The reason is in the activations. Calibrating on Chapters 1 to 5, I recorded the largest magnitude of each input channel to every layer. For the input of the last feed-forward output layer, the largest channel is 28.9 times the median channel (48.9 against 1.7). One shared scale, set by that channel, leaves every other channel with a handful of levels. Scaling per token helps, I think, because large activations are concentrated in a few rows, as Chapter 4’s massive activations were at position 0, and a per-row scale confines the damage to those rows. I did not check where the outliers sit in this layer’s input.

SmoothQuant’s idea is to move the difficulty into the weights: divide each input channel of the activation by a factor \(s_j = \max|X_j|^{\alpha}/\max|W_j|^{1-\alpha}\) and multiply the matching row of the weights by the same factor, which leaves the product unchanged and flattens the activation outliers. I implemented this for the layers whose input comes straight from a layer norm. It helped, from +37 to +27 per cent, but did not fix the problem. I think the remaining damage is in the layers I did not smooth, because their input is the output of an attention operation or a GELU and the division cannot be folded into a preceding layer. That is a hypothesis. I used a single α, did not search for a better one, and did not smooth those layers.

Does it run faster?

Every result so far is storage or behaviour. Speed is a different measurement, and for quantisation it is the one most often assumed. On this GPU, for an output-projection-sized matrix (768 × 50,257) at one token, with the weights stored in the stated format and the activations in fp16, two runs agreed:

How the weights are used Time Stored size of the weights
fp32 weights 512 µs 147 MiB
fp16 weights 269 µs 74 MiB
int8 weights, converted to fp16 on each call 1,610 – 1,621 µs 37 MiB
packed int4 weights, unpacked and converted on each call 5,477 – 6,925 µs 19 MiB

The 16-bit model is the fastest and the smallest formats are the slowest, by a factor of six for int8 and twenty to twenty-six for int4. The conversion is done in PyTorch operations, not in a fused kernel, so each call has to write the entire dequantised weight matrix to memory and read it back: more traffic than the original fp16 weights, not less. The stored size shrank by four, the bytes moved grew, and the time followed the bytes. Chapter 11’s model applies: a lever on the byte term works only if the kernel actually moves fewer bytes.

A fused kernel, which reads the packed weights and converts them in registers before multiplying, avoids the round trip, and these are what production quantised inference uses. This machine has none for 4-bit weights, so what I can show is the cost of not having one. I did not measure what a fused kernel would achieve.

There is one real low-precision path here. PyTorch exposes 8-bit integer matrix products on tensor cores (torch._int_mm), which require both operands in int8. At 256 tokens:

Shape fp16 int8 × int8
Output projection (768 × 50,257) 1,150 – 1,178 µs 741 – 747 µs
Feed-forward up (768 × 3,072) 59 – 60 µs 60 – 61 µs
Square (4,096 × 4,096) 284 – 286 µs 289 – 291 µs

The int8 path is 1.55 times faster for the memory-heavy shape and no faster for the others. The comparison flatters it: the figure excludes quantising the activations and rescaling the result, and the output projection’s 50,257 columns are not a multiple of 8, so I padded them to 50,264. The padding is the same alignment trap as in Chapter 7.

The summary of speed is therefore narrow. Moving from fp32 to fp16 helps wherever bytes matter, as Chapters 7 and 11 showed. Beyond that, a speed-up from fewer bits needs a kernel that exploits it, hardware that supports it, and a shape that is bound by something the kernel improves.

What the numbers do not show

  • GPT-2 small, plain rounding. Larger models are reported to be more tolerant of low-bit weights. I did not measure any. Nothing here reflects GPTQ, AWQ or any calibrated weight rounding.
  • The book’s own prose as test text. 29,127 tokens in 57 short windows. Perplexity differences below about one per cent should not be read as real; I did not test significance. Task accuracy was not measured at all.
  • Simulated quantisation. The behaviour numbers are exact for the simulated arithmetic. A real kernel with different accumulation might differ slightly.
  • bf16 on this GPU has no native tensor-core path; only its numerical effect was measured.
  • One smoothing configuration. One α, applied only to layers fed by layer norm.
  • Unfused dequantisation. The slow rows say what a missing kernel costs, not what a tuned one achieves.
  • No KV-cache quantisation. Chapter 8 showed that at batch sizes above about seven the cache is the larger traffic. Quantising it has its own accuracy question, which this chapter does not answer.
  • No energy measurement.

Inference Lab increment

The Lab now has:

  • quantize_dequantize(w, bits, group, symmetric), returning the dequantised weights and the true bits per weight including scales, tested for level counts, grid exactness, granularity and storage accounting.
  • fake_quant_activation, smoothing_scales, pack_int4 and unpack_int4.
  • evaluate(model_q, model_ref, windows), returning perplexity, top-1 agreement and mean KL against the reference.
  • The formats, weights, quality and speed commands.

The rule it adds to the manifest idea of Chapter 2: a quantised model is a different artefact, and its manifest names the scheme, the granularity, the layers it was applied to and the layers it was not. A request record that says “GPT-2” for any of the rows above has said something untrue.

And the evidence rule that this chapter earns: report storage, reconstruction error, behaviour and speed as four measurements. A claim about one is not a claim about the others.

What you should now be able to do

  • Say what each of fp32, fp16, bf16 and the two fp8 formats can represent, and when each overflows.
  • Compute the storage of a quantisation scheme including its scales and zero points.
  • Choose a granularity and explain what it trades.
  • Measure weight reconstruction error and explain why it does not predict model behaviour.
  • Measure behaviour as perplexity, top-1 agreement and KL, and say what each misses.
  • Explain why per-tensor scales fail for heavy-tailed weights and for activations with outlier channels, and what per-channel, per-token and smoothing change.
  • Say why the output projection is the layer to protect, and what a tied embedding implies.
  • Predict whether a quantisation scheme will run faster, from the roofline and the kernels that exist.

Key terms

  • Quantisation: representing values on a coarser grid, usually with integers and a scale.
  • Scale / zero point: the factors that map integers to real values.
  • Granularity: how many values share a scale (tensor, channel, group, token).
  • Group size: the number of consecutive inputs sharing a scale.
  • Weight-only / W8A8: quantising only the weights, or weights and activations to 8 bits.
  • Outlier: a value far larger than typical, which sets a shared scale.
  • Calibration: choosing scales or smoothing factors from sample data.
  • Perplexity / agreement / KL: accuracy against text; same top choice; distance between output distributions.
  • Fused dequantisation: converting packed weights inside the matrix-product kernel.

Reader exercises

  1. Reproduce the group-size sweep for 4 bits, and find the group size at which the extra scale storage costs more than the accuracy gained.
  2. Quantise the embedding table’s output side at int8 with groups of 32. Where is the smallest width at which the logit noise falls below the logit spread?
  3. Smooth all four layer types in the W8A8 experiment, ignoring that some cannot be folded in practice. Does the damage disappear? That tests the hypothesis in the text.
  4. Search for the best α for the smoothing experiment on the calibration text only, then evaluate once on the test text. How much does it recover?
  5. Measure agreement between fp16 and bf16 on a second text. Is the 88 per cent stable?
  6. Implement round-to-nearest with a per-group search for the scale that minimises weight error, rather than using the maximum. How much of the int4 gap does it close?
  7. Predict, from Chapter 11’s model, the speed-up of a fused int4 weight-only kernel on the output projection at one token. What would you need to measure to check it?

References

Evidence in this chapter

Claim Category
Format ranges and resolutions in the first table Established background, read from torch.finfo
Quantisation error is proportional to the group’s largest value Established background
GPT-2’s linear weights are heavy-tailed (peak/RMS median 21, up to 186) Book result
Per-channel int8 is statistically indistinguishable from the original on this text Book result; significance not tested
Per-tensor int8 costs +11% and int4 rounding costs +8% to +41% depending on granularity Book result, one model, one text
bf16 changes the top choice at 12% of positions despite +1.7% perplexity Book result
Quantising the output side of the tied embedding at int4 destroys the model; the input side is nearly unaffected Book result, 12 windows
The cause is the hidden-state norm amplifying embedding noise Book hypothesis, supported by the logit-noise numbers
W8A8 per-tensor +37%, per-token +0.5%, smoothing +27% Book result
Remaining smoothing damage is in layers that were not smoothed Book hypothesis, untested
Unfused int8 and int4 dequantisation were 6× and 20–26× slower than fp16 Book result, two runs
int8 tensor-core matmul is 1.55× faster only for the memory-heavy shape Book result, partial costs excluded
GPTQ, AWQ, fp8 execution, fused int4 speed, KV-cache quantisation, task accuracy, larger models Not measured

Implementation status

  • Manuscript: content/books/inference/12-chapter.md
  • Runnable examples: examples/ch12/quantization.py (original), examples/ch12/quant_lab.py (quantisers, evaluator and probes)
  • Tests: tests/test_ch12_quantization.py, tests/test_ch12_quant_lab.py (storage accounting, level counts, grid exactness, granularity and outlier behaviour, asymmetric advantage, activation outliers, smoothing equivalence, int4 packing, statistics; no model or GPU needed)
  • Evidence: research/ch12/sources.md, research/ch12/results.json, research/evidence-register.md
  • Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about calibrated quantisation methods, larger models, task accuracy or the speed of fused low-bit kernels. Part III begins with Chapter 13, which returns to attention and the block-wise exact algorithm that Chapter 8 could only describe.