Memory Bandwidth and Compute Bottlenecks

I wrote down a prediction for how long 58 different GPU operations should take, using nothing but their size and three numbers measured on this machine. The median measurement came out 10 to 14 per cent above the prediction. At least nineteen in twenty were within a factor of two. And not one memory-bound operation ran faster than its bound, which is what a bound is for.

Then I used the same model to predict the time of a whole decode step as the batch size grew, and it was wrong by a factor of three. It said the step would stay at its floor until a batch of around 256. The measured step started to climb at 64, and at 256 it was 30 milliseconds against a prediction of 10.

A model that works for single kernels and fails for a composition is more useful to know about than a model that always works. This chapter builds the model, shows where it is accurate, and shows exactly where it breaks. It is the tool the next chapters use to say what an optimisation can and cannot be expected to achieve.

Where this sits

Chapter 7 gave the roofline: attainable rate is the lower of a compute ceiling and bandwidth times arithmetic intensity. Chapter 9 found that for GPT-2 small, neither phase was in its textbook regime, because both were bound by the cost of issuing operations. Chapters 8 and 10 added bytes: the score tensor, the cache. This chapter puts all three into one model, measures its ceilings with more care than before, and tests it.

The Inference Lab piece added here is the bottleneck model: predict_time(flops, bytes, launch, bandwidth, peak), which returns three lower bounds and names the largest; the ceilings probes that measure its inputs; and an evaluation harness that scores predictions against measurements. The rule it supports is one the book will use for the rest of Part II and Part III: before claiming that an optimisation helps, say which term of the model it moves, and by how much.

For the Pi series, this is the diagnostic step. A local model that feels slow is slow for one of three reasons, and each has a different remedy. The measurements here are how to find out which.

Three ceilings

An operation on a GPU takes at least as long as each of three things:

- \(t_{\text{launch}}\), the fixed cost of issuing the operation. Chapter 7 measured it at 35 microseconds for a tiny matrix product. It does not depend on the size of the operation. - \(\text{bytes}/B\), the time to move the data at the sustainable memory bandwidth \(B\). - \(\text{FLOPs}/P\), the time to do the arithmetic at the peak rate \(P\) for the precision.

The largest of the three sets the bound and names the regime: launch-bound, memory-bound or compute-bound. The classic roofline has only the last two. The first is missing from textbook treatments because they assume operations large enough to hide it, and for small models and small batches that assumption is false.

Each term is a lower bound. Real operations are slower, because kernels are not perfectly efficient and the three activities overlap imperfectly. A prediction that comes out below the measurement is the model working. A measurement below the prediction means one of the inputs was wrong: a ceiling set too low, or a byte count too high.

Measuring the ceilings

The model is only as good as its inputs, and each is a measurement that needs care.

Bandwidth. I measured the throughput of a copy (read and write), a sum (read) and a fill (write) over buffers from 4 KiB to 1 GiB, with CUDA events and batched calls. Two runs agreed closely:

Buffer	Copy	Read (sum)	Write (fill)	Time per call (copy)
4 KiB	1 GB/s	0 GB/s	1 GB/s	9 µs
256 KiB	67–73 GB/s	28–30 GB/s	35–42 GB/s	8 µs
1 MiB	261–269 GB/s	78–80 GB/s	128–162 GB/s	8 µs
16 MiB	274–275 GB/s	275 GB/s	288–293 GB/s	122 µs
64 MiB	276–279 GB/s	298 GB/s	290–297 GB/s	482–486 µs
256 MiB	277–279 GB/s	304–305 GB/s	292–299 GB/s	1.9 ms

Three things in the table matter. Buffers up to a few hundred kilobytes take 7 to 9 microseconds regardless of size, so their “bandwidth” is nothing but the launch floor. Bandwidth only saturates from about 16 MiB. And the achievable figure depends on the pattern: reads reached 305 GB/s, writes 299, and a copy 279, because mixing reads and writes costs. I take 305 GB/s as the sustainable figure for read-dominated work, which is most of inference, and note that a 1 GiB copy fell to 213–231 GB/s in both runs for a reason I did not investigate.

Arithmetic. The best matrix product at sizes from 4,096 to 8,192 reached 6.2 to 6.4 TFLOP/s in fp32 and 27.2 to 30.8 TFLOP/s in fp16. Chapter 7 measured up to 7.4 and 31.4. The spread is the clock and power state of a board that is also driving a display.

Launch. Per-call costs for the smallest operation of each kind:

Operation	Floor
8×8 matrix product	12.1 – 13.5 µs
1,024-element add	9.0 – 9.6 µs
Layer norm, 4×768	11.1 – 11.8 µs
Softmax, 4×1,024	8.7 – 8.9 µs

These are about a third of what Chapter 7 measured for the same matrix product earlier in the same day: 35.4 microseconds. Nothing in the GPU changed. The floor is the speed at which the host CPU can run PyTorch’s dispatch path, and the host was in a different state. It also explains something I flagged as unexplained in earlier chapters: the library’s cached decode step, 11 ms in Chapter 9, measured 5 to 8 ms in Chapter 10. The launch floor is a property of the host and of the session, not of the model or the GPU.

This forces a rule on the whole book. Ceilings have to be measured in the same session as the measurements they judge. A launch floor from this morning cannot be used to classify an operation measured this afternoon.

Counting the bytes

The model needs the minimum bytes each operation must move. For the operations in a transformer:

Operation	FLOPs	Minimum bytes
Matrix product \(M \times K \times N\)	\(2MNK\)	\((MK + KN + MN)\,s\)
Elementwise binary (add)	\(n\)	\(3\,n\,s\) (read two, write one)
Elementwise unary (GELU)	about \(8n\)	\(2\,n\,s\)
Layer norm over \(n\) elements	about \(8n\)	\(2\,n\,s\)
Softmax over \(n\) elements	about \(5n\)	\(2\,n\,s\)
Decode attention, \(B\) sequences, \(T\) cached	\(4BhTd\)	\(2BhTd\,s\) (keys and values)

The intensities show at a glance why these classes behave differently. Add is 0.08 FLOP per byte in fp32, GELU 1.0, layer norm 1.0, softmax 0.6. Every one of them is far to the left of the ridge point of about 21 FLOP per byte (6.4 TFLOP/s over 305 GB/s), so every one is memory-bound as soon as it is large enough to leave the launch floor. A matrix product is the only operation in the model with intensity that can be made high.

The bytes are a minimum: each operand read once and each result written once. If a kernel moves more, for instance because it makes an intermediate copy, the bound is optimistic. I tested one such suspicion in the batched-decode experiment below, and rejected it.

58 predictions

For each of 58 operations, in fp32 and fp16, I computed the prediction before reading the measurement, then measured it with CUDA events. The operations: matrix products from decode-sized (one row) to prefill-sized, square products from 512 to 4,096, adds and GELUs from 1,024 to 134 million elements, layer norms and softmaxes from one row to 16,384 rows. I ran the whole set twice.

Ratio of measured time to predicted time:

	Run 1	Run 2
Median	1.14	1.10
Within 25% of the prediction	60%	74%
Within a factor of 2	100%	95%
Faster than the bound by more than 5%	16%	10%

By regime:

Regime	Points	Median ratio	Faster than bound
Memory-bound	28–29	1.13 – 1.21	0%
Compute-bound	12	1.07 – 1.09	17 – 25%
Launch-bound	17–18	1.02 – 1.08	22 – 35%

A few of the individual points:

Operation	Measured	Predicted	Regime	Ratio
1×768 @ 768×3,072 (fp32)	34 µs	31 µs	memory	1.11
1×768 @ 768×50,257 (fp32)	644 µs	504 µs	memory	1.28
512×768 @ 768×3,072 (fp32)	355 µs	382 µs	compute	0.93
4,096² (fp32)	21.97 ms	21.73 ms	compute	1.01
add, 134 M elements (fp32)	6.83 ms	5.25 ms	memory	1.30
layer norm, 16,384×768	375 µs	328 µs	memory	1.14
layer norm, 1×768	12.3 µs	7.9 µs	launch	1.57

The pattern is what a correct bound looks like. Memory-bound operations never beat the bound and are typically 10 to 30 per cent above it: the bandwidth ceiling is sound. Compute-bound operations sit within about 10 per cent either way, and the cases below the bound, up to a quarter of that group, most likely come from the peak arithmetic rate having been estimated from a different, sometimes slower, measurement taken earlier in the same run. The launch-bound points scatter most, since the floor differs by operation and I assigned one floor to each class. A layer norm of a single row cost 12 microseconds against an assumed 8.

Two cautions on what the 58 points show. They are single operations, timed in isolation with batched calls, so they say nothing about whether a model built from them behaves as the sum. And I chose the shapes, including the ones GPT-2 uses, which is a point in favour of relevance and against any claim of generality.

The composition fails

The model’s first real test is the Chapter 9 question: what happens to the time of a decode step as more sequences are decoded together? I built the batched step from the hand-written forward pass of Chapter 10 at a context of 128 tokens, fp32, a static cache, and varied the batch from 1 to 512.

The model’s inputs for one step are: the weights read once (475 MiB), the cache read once per sequence (\(128 \times 73{,}728\) bytes each), and \(B\) times the per-token FLOPs. The launch floor I took as the measured step at batch 1, about 8.3 to 8.5 ms. The “max of totals” version of the model, which takes the largest of total launch, total bytes and total FLOPs, predicts a step that stays at the floor until a batch of about 128 and is compute-bound beyond about 256.

Measured, in two runs:

Batch	Step time	Tokens per second	Prediction (max of totals)	Measured / predicted
1	8.3 – 8.5 ms	118 – 121	8.3 – 8.5 ms	1.0
8	9.8 – 10.3 ms	774 – 816	8.3 – 8.5 ms	1.2
32	9.1 – 9.9 ms	3,224 – 3,523	8.3 – 8.5 ms	1.1 – 1.2
64	11.7 – 12.4 ms	5,160 – 5,470	8.3 – 8.5 ms	1.4 – 1.5
128	18.0 – 21.3 ms	5,997 – 7,117	8.3 – 8.5 ms	2.1 – 2.6
256	30.5 – 33.4 ms	7,673 – 8,406	9.9 – 10.1 ms	3.1 – 3.3
512	87.9 – 96.0 ms	5,331 – 5,823	19.8 – 20.1 ms	4.4 – 4.8

The practical finding is better than the model’s: on this GPU, decoding 32 sequences at once costs 7 to 20 per cent more time per step than decoding one, for 27 to 30 times the tokens per second, and 64 sequences cost 40 to 50 per cent more for about 45 times. Batching a small model is nearly free until the arithmetic arrives. Throughput peaked at 256 sequences, around 8,000 tokens per second, which is about 63 to 71 times the single-sequence rate, at 3.6 to 4.0 times the step latency, and then fell at 512.

The model’s failure is the instructive part. It predicted the knee too late by a factor of two to four. Two corrections fix some but not all of it.

Compose bounds per kernel. A step is a sequence of kernels, not one operation. The matrix products are compute-bound at large batch and the attention over the cache is memory-bound, and they run one after the other. Their bounds add. The sum of per-kernel bounds, with matrix products at \(\max(B \cdot 247\text{ MFLOP}/P,\; \text{weights}/B_w)\) and attention at \(B \cdot 128 \cdot 73{,}728\) bytes over the bandwidth, gives 8.9 ms at 128 sequences, 17.8 at 256 and 35.6 at 512. Against the measured steps, that is 2.0 to 2.4 times too low at 128, 1.7 to 1.9 at 256 and 2.5 to 2.7 at 512. Better than the max of totals (3 to 5 times too low), but a lower bound that is out by about two.

A rejected explanation. The static cache returns slices of a larger buffer, which are not contiguous in memory. I suspected the attention kernel was quietly copying them, which would triple the cache traffic. I timed one layer at batch 256: slices took 0.754 ms and contiguous copies 0.766 ms, both close to the 0.665 ms bandwidth bound. The slices are not slower. The suspicion was wrong, and the gap is not a hidden copy.

I did not find the rest. About half of the step at 256 and 512 sequences is unaccounted for by the sum of the bounds. Candidates are the many small elementwise operations in the unfused forward pass, which at these sizes are no longer free; the bias additions that follow each matrix product as separate kernels; clock throttling under a sustained heavy load; and the fall at 512, which I did not examine at all. Each is testable and I have tested none. The model gives a floor and the measurement says how far above it the real system sits.

Moving the dots

Each lever in this book moves a specific term of the model. Two can be demonstrated with a single measurement.

Fusion removes memory passes. Computing \(x \cdot y + z\) as two kernels reads and writes an intermediate: six memory passes over the tensors. One fused kernel reads three and writes one: four. The predicted ratio is 1.5. Measured over 64 Mi elements:

Precision	Two kernels	One fused kernel	Ratio	Fused throughput
fp32	5.89 ms	3.70 ms	1.59	290 GB/s
fp16	2.82 ms	1.83 ms	1.54	293 GB/s

The measurements follow the passes. Both are at about 95 per cent of the sustainable bandwidth, so the fused kernel is as good as a memory-bound kernel can be.

Narrower data removes bytes. The same operations in fp16 took 2.09 times (unfused) and 2.02 times (fused) less time than in fp32, for half the bytes. That is the Chapter 7 observation made exact: for a memory-bound operation, time is bytes divided by bandwidth, and halving the bytes halves the time. Chapter 12 applies the same lever to weights.

The full set of levers, by the term they move:

Lever	Term it moves	Evidence in this book
Larger batch	Raises intensity (weights reused); leaves launch cost per step unchanged	Chapter 7 (thin matmuls), this chapter (batched decode)
Cache	Removes recomputed FLOPs and bytes; adds cache bytes	Chapter 10
Narrower weights and cache	Reduces bytes	Chapter 7, this chapter; Chapter 12
Fusion	Reduces bytes (intermediates) and launches	This chapter; Chapters 13 and 18
Execution graphs, compilation	Reduces launch cost	Chapter 18
Block-wise attention	Reduces bytes (score tensor)	Chapters 8 and 13

A lever that moves a term that is not the binding one does nothing. Quantising a launch-bound step does not help; fusing a compute-bound matrix product does not help; adding a cache to a short context on a launch-bound GPU does not help (Chapter 10 showed this). The model’s job is to say which lever to reach for.

A procedure

For any operation or step whose time you want to understand:

1. Measure the ceilings now, in this session, for the precision you use: bandwidth for the access pattern, peak arithmetic, and the launch floor for this operation class.
2. Count FLOPs and the minimum bytes from the shapes.
3. Compute the three bounds and name the largest.
4. Compare with the measurement. A ratio within about 1.3 says the model explains it. A ratio of two or more says there is a missing term.
5. Find the missing term before applying a remedy. Extra copies, extra kernels, throttled clocks and poor kernel selection all look like “memory-bound” or “compute-bound” in a profiler that does not count bytes.

What I did not have was a hardware profiler. NVIDIA’s counters would show achieved occupancy, cache hit rates and the number of kernels directly. On this setup the PyTorch profiler recorded no GPU activity at all (Chapter 7), so every conclusion above is an inference from times, which is exactly the kind of evidence this model exists to discipline.

What the numbers do not show

• One GPU, one session’s ceilings. The launch floor in particular was a third of its value earlier in the day. Ceilings from another session or machine cannot be used.
• Single operations in isolation. The 58-point test does not test the composition, which is where the model failed.
• Batched calls. Operation times are steady-state queue times, not single-call latencies (Chapter 7).
• An unfused hand-written forward pass. Its batched step is slower than an optimised implementation’s would be, and the unexplained gap may be an artefact of that.
• fp32 only in the batch sweep, at one context length of 128, with a static cache.
• No counters. Explanations of the unaccounted time are untested hypotheses.
• The 1 GiB copy anomaly and the fall at 512 sequences were not investigated.

Inference Lab increment

The Lab now has:

• predict_time(flops, bytes, launch, bandwidth, peak), classify, the byte and FLOP counters for matrix products and elementwise operations, decode_step_bytes and decode_step_flops, and ratio_summary, all tested without a GPU.
• measure_ceilings(), which produces bandwidth against buffer size and pattern, per-class launch floors and per-precision peak arithmetic, in the session that is about to use them.
• The model, batch and fusion commands, which reproduce every table in this chapter.

The rule it adds: a performance claim names the term it moves. “Faster” is not a claim; “reduces bytes by half in a memory-bound step, predicted 2.0×, measured 2.0×” is.

What you should now be able to do

• State the three-ceiling bound and classify an operation by its largest term.
• Count the FLOPs and the minimum bytes of the operations in a transformer.
• Measure bandwidth, peak arithmetic and launch floors, and say why each must be taken in the same session as the measurement it judges.
• Score a prediction against a measurement, and say what a ratio below one means.
• Explain why the bounds of a sequence of kernels add, and why the max of totals underestimates a composed step.
• Predict the effect of fusion and of narrower data from the bytes alone.
• Choose the right lever for a given regime, and say which levers will do nothing.

Key terms

• Launch-bound, memory-bound, compute-bound: limited by issue cost, by bandwidth, or by arithmetic.
• Sustainable bandwidth: the measured rate for large buffers of a given access pattern.
• Peak rate: the best measured arithmetic throughput for a precision.
• Lower bound: a time no implementation can beat, as opposed to a prediction of what one will achieve.
• Fusion: combining kernels so that intermediates stay in registers or cache.
• Intensity: FLOPs per byte moved.
• Ridge point: the intensity where the compute and bandwidth ceilings meet.
• Steady-state time: time per call for a queue of back-to-back calls.

Reader exercises

1. Run ceilings twice, an hour apart. How much do the launch floors and bandwidth move? Which of the book’s earlier numbers would change?
2. Extend the 58-point test to an operation not covered: an embedding gather, or a concatenation. Compute the minimum bytes first.
3. Repeat the batch sweep at a context of 1,024. Where does the model predict the knee, and where is it?
4. Find the missing half of the step at batch 256: replace the unfused elementwise operations with fused ones one at a time and measure each change.
5. Examine the fall in throughput at batch 512. Is it clocks, memory, or kernel selection?
6. Do the same sweep in fp16. The model says the weights’ share halves; what does it say about the knee?
7. Write the three-ceiling model for a multi-GPU step with an all-reduce, adding a fourth term for the network. What would you need to measure?

References

• Roofline: An Insightful Visual Performance Model for Multicore Architectures — the two-ceiling model this chapter extends.
• NVIDIA Deep Learning Performance: Matrix Multiplication — vendor documentation on arithmetic intensity and the conditions for compute-bound execution.
• Efficiently Scaling Transformer Inference — an analysis of transformer inference cost that distinguishes memory and compute time for partitioned inference.

Evidence in this chapter

Claim	Category
An operation takes at least as long as each of its launch, byte and FLOP times	Established background (the roofline, plus the launch term from Chapter 7)
Sustainable bandwidth about 305 GB/s (read), 279 (copy) on this GPU	Book result, two runs
The launch floor was about a third of Chapter 7’s morning value	Book result; cause (host CPU state) is an inference
Median measured/predicted 1.10–1.14 over 58 operations; memory-bound points never below the bound	Book result, two runs
Batching 32 sequences costs 7–20% more per step for 27–30× the tokens	Book result, two runs, hand-written forward, fp32
The max-of-totals prediction misses the batched step by 3–5× at 256 and above	Book result
The sum of per-kernel bounds is a lower bound within 1.4–2.7×	Book result
A hidden copy of cache slices explains the gap	Hypothesis tested and rejected
What explains the rest of the gap	Unknown; hypotheses listed, untested
Fusion gives 1.54–1.59× against 1.5× predicted; fp16 halves memory-bound time	Book result
Hardware counters, L2 behaviour, int8, multi-GPU	Not measured

Implementation status

• Manuscript: content/books/inference/11-chapter.md
• Runnable examples: examples/ch11/roofline.py (original), examples/ch11/bottlenecks.py (model, probes and experiments)
• Tests: tests/test_ch11_roofline.py, tests/test_ch11_bottlenecks.py (counts, the three-bound prediction, regime classification, batching behaviour and summary statistics; no GPU needed)
• Evidence: research/ch11/sources.md, research/ch11/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about other hardware, hardware-counter data or production systems. Chapter 12 applies the “narrower data” lever to the weights themselves, and has to answer the question that this chapter could avoid: what does it cost in accuracy?