Matrix Multiplication and GPU Execution

The engine from Chapter 6 generates 32 tokens from GPT-2 in 6.4 seconds on one CPU thread. Move the same weights to the GPU in this machine and it takes 0.45 seconds. The output is the same 32 tokens. That is a 14-fold improvement, and it should feel too small.

The same GPU multiplies two large matrices at about 7,000 billion floating-point operations per second. One CPU thread manages about 53 billion. That is a ratio of roughly 130. So where did the other 90 per cent of the advantage go? Part of the answer is that a language model is not one large matrix multiplication. It is hundreds of small ones, and small ones behave differently. A single-token step through the output layer of GPT-2 uses about 2 per cent of the arithmetic this GPU can deliver. The chip spends the rest of that time waiting for weights to arrive from memory. Another part of the answer is that, below a certain size, a GPU call costs about 35 microseconds whatever it computes, and a CPU can multiply two small matrices in a tenth of that.

This chapter explains both effects, with measurements. It also deals with a prior problem: it is remarkably easy to get GPU timings that are wrong by a factor of several thousand, and I will show one.

Where this sits

Chapter 4 gave the cost of a forward pass in floating-point operations. Chapter 5 showed that the naive generation loop reprocesses its whole prefix. Chapter 6 built an engine that measured both on a CPU. Until now the only hardware question was how many operations there were. From here on, the question is also what the hardware does with them, and the answer depends on a quantity that the operation count leaves out: how many bytes have to move.

The Inference Lab piece added here is a measurement harness for accelerators: event-based timing, a clock ramp so measurements start from a warm device, a launch-floor probe, measured ceilings for bandwidth and arithmetic, and a roofline helper. It also makes the Chapter 6 backend device-aware, so the same engine runs on CPU or GPU and its manifest records which.

For the Pi series, this is the chapter where “local models” acquires a hardware meaning. The machine used here is an RTX 2060 with 12 GB, which is an ordinary developer GPU and not a data-centre part. The GPU computing volume later in the series goes below the library to write kernels, and model serving builds on the batching behaviour shown here. Both start from the measurements in this chapter.

What a matrix multiplication costs

A product of an \(M \times K\) matrix and a \(K \times N\) matrix produces an \(M \times N\) matrix. Each output element is a dot product of length \(K\), which takes \(K\) multiplications and \(K\) additions, so

In a transformer layer, \(M\) is the number of token positions being processed, and the \(K \times N\) operand is a weight matrix. GPT-2's linear layers are:

Layer	\(K\)	\(N\)	Weight elements
Query, key, value projection	768	2,304	1.77 M
Attention output projection	768	768	0.59 M
Feed-forward up	768	3,072	2.36 M
Feed-forward down	3,072	768	2.36 M
Output projection (logits)	768	50,257	38.6 M

The weights are the same whatever \(M\) is. A prompt of 500 tokens has \(M = 500\), and the generation step that follows has \(M = 1\). The arithmetic scales with \(M\); the weight bytes do not. That asymmetry is the whole chapter.

Arithmetic intensity

The ratio of arithmetic to memory traffic is called arithmetic intensity, in FLOPs per byte. Counting each operand once, with \(b\) bytes per element,

When \(M\) is much smaller than \(K\) and \(N\), the weights dominate the denominator, and this reduces to For \(M = 1\) in 32-bit floating point, \(I = 0.5\) FLOP per byte: for every four-byte weight read from memory, the processor does two operations. In 16-bit, \(I = 1\). Every weight is used once and discarded. The same weights used for 64 tokens give \(I \approx 32\) in fp32, because each weight loaded is now used 64 times.

The roofline

A processor has two limits: a peak rate of arithmetic, \(P\) FLOP/s, and a peak rate of memory transfer, \(B\) bytes/s. For an operation of intensity \(I\), the attainable rate is bounded by

This is the **roofline model**: a sloped line where memory limits the rate, and a flat ceiling where arithmetic does. The two meet at the **ridge point** \(I^* = P/B\). Operations to the left of it are memory-bound and those to the right are compute-bound. It is a bound and not a prediction. Real kernels fall short for many reasons, but a measurement far below the roofline is a sign that something other than the two ceilings is in the way, and finding it is the work of the next several chapters.

Timing a GPU without fooling yourself

GPU work is asynchronous. When Python calls a @ b, the CPU hands a request to the driver and returns immediately. The multiplication happens later, on the GPU, while the Python program carries on. A timer wrapped around the call measures how long it takes to ask.

I timed a 4,096 × 4,096 fp32 matrix multiplication, 137 GFLOP, three ways:

Method	Median time	Implied rate
Time the call only	0.18 ms	757,000 GFLOP/s
Time until cuda.synchronize() returns	23.3 ms	5,909 GFLOP/s
CUDA events around the call	23.6 ms	5,816 GFLOP/s

The first row claims 757 teraflops from a GPU whose real ceiling is under ten. It is not a subtle error. It is also the number you get from the most natural way to write the benchmark. The original version of this chapter’s benchmark did synchronise, which is correct, but it reported the fastest of five runs, and its first recorded GPU result, 3,242 GFLOP/s at 512³, was a minimum over a handful of iterations.

CUDA events are the better tool. They are timestamps placed in the GPU’s own work queue, so they measure the interval during which the device was executing, independent of what the CPU was doing.

Four conditions for a trustworthy number

1. Synchronise or use events. Never time an asynchronous call.
2. Warm up. The first calls load kernels and select algorithms. Discard them.
3. Bring the clocks up. An idle GPU runs slowly to save power. Before I added a three-second ramp, the 4,096 benchmark measured 5.1 TFLOP/s; afterwards 5.8. Querying the device showed why. At rest the core clock was 1,050 MHz and the memory clock 5,000 MHz, in a low-power state. Under load they were 1,980 MHz and 6,801 MHz. A benchmark that starts on an idle GPU measures the ramp.
4. Report the distribution and the environment. Medians with ranges, the device, driver, framework versions, the clock and power state, and what else the device was doing.

That last point has a specific application here. This GPU is also driving the display and a web browser, under Windows’ display driver model. Two consequences showed up in the data. Single-call event times for very short kernels were bimodal: the same 8 × 8 multiplication reported 4 microseconds or 60 to 100 microseconds, depending on how the work happened to be submitted. And the clock settled at different values at different times, so a given size varied by up to 40 per cent between runs. I deal with the first by timing batches of calls, and with the second by repeating whole runs in separate processes and reporting their range. Neither can be removed, and a reader with a dedicated compute GPU will see tighter numbers.

What this machine can do

Before judging any kernel, I measured the hardware’s own ceilings. I did not use datasheet figures, because I have not verified them for this exact card and driver, and because what matters is what a PyTorch program on this system can reach.

Ceiling	Measured
Memory bandwidth, large copy (read and write)	278 GB/s
Memory bandwidth, best read-dominated matmul	303 GB/s
fp32 matmul, 4,096 to 8,192 square	6.4 to 7.4 TFLOP/s
fp16 matmul, 4,096 to 8,192 square	29.7 to 31.4 TFLOP/s

Two things are worth noting. The bandwidth ceiling is not one number: a copy reads and writes and reached 278 GB/s, while a matrix product that mostly reads weights reached 303. And half precision was about four to five times faster than single, which is not what halving the bytes would predict for a compute-bound operation. It is what a dedicated matrix unit, the tensor core, gives: the fp16 path uses hardware that fp32 on this generation of GPU does not.

I will use the best observed values, 303 GB/s, 7.4 TFLOP/s and 31.4 TFLOP/s, in the roofline below. A ceiling should not be exceeded by a measurement, and using a dedicated test’s lower value would let kernels appear to beat it. The honest statement is that these ceilings are uncertain by perhaps 20 per cent. The ridge points follow: 24.6 FLOP/byte in fp32 and 103.7 in fp16.

Matrix size and the cost of a call

Square products from 32 to 8,192, each the median of three separate runs:

\(n\)	CPU, 1 thread (GFLOP/s)	GPU fp32	GPU fp16
32	10	2	2
64	34	14	12
128	45	113	96
256	53	895	787
512	53	6,285	6,419
1,024	54	6,317	25,461
2,048	54	4,986	22,828
4,096	–	7,217	29,672
8,192	–	6,445	31,073

Read the table in three bands.

Below 128, the CPU wins. At \(n = 32\) the CPU does the product in 6.6 microseconds and the GPU takes about 38. The crossover is between 64 and 128.

From 128 to 512, the GPU’s rate climbs steeply, but its time per call hardly moves. The GPU time for \(n = 32\) to \(n = 512\) was 36 to 43 microseconds, over a 4,000-fold range of arithmetic. In this band the GPU is not computing; it is being asked. A direct measurement of the floor: an 8 × 8 matrix product on the GPU cost 35.4 microseconds per call (range 34.3 to 37.0) over 2,000 back-to-back calls, a 1,024-element addition 19.6 microseconds, and the same 8 × 8 product on one CPU thread 4.2 microseconds. The CPU can issue and execute about 240,000 such calls a second; the GPU path about 28,000. The issuing itself, in Python through PyTorch’s dispatcher and the driver, is the cost, since the time to enqueue and the time to finish were identical.

From 512 upward, the GPU is compute-bound. fp32 reaches 6.3 to 7.2 TFLOP/s and stays there. fp16 takes longer to saturate, reaching 25 TFLOP/s at 1,024 and 30 at 4,096, because a faster unit needs a larger problem to stay busy.

The 2,048 row is the least stable: fp32 ranged from 3,791 to 5,987 and fp16 from 14,614 to 23,227 across the three runs. I believe this is the clock or power state changing under sustained load, as the board approached its power limit, but I did not isolate it. It is a reminder not to read a single size in a single run.

Transformer shapes

Square matrices are a convenience. The products inside GPT-2 are \(M \times 768\) by \(768 \times N\), with \(M\) the number of tokens in the step. I measured two of them as \(M\) grew from 1 to 1,024, each cell the median of three separate runs.

The output projection, \(N = 50{,}257\), is the cleanest case because its weights are large: 154 MB in fp32, 77 MB in fp16.

Tokens \(M\)	fp32 time (µs)	Share of fp32 peak	fp32 GB/s	Intensity (FLOP/B)
1	510	2.0%	303	0.5
8	595	13.9%	262	4.0
32	592	56.0%	272	15.4
64	781	85.0%	214	29.5
128	1,468	90.4%	123	54.7
1,024	11,859	89.5%	31	217.5

From 1 token to 32 tokens, the time barely changes: 510 to 592 microseconds, a 16 per cent rise for 32 times the arithmetic. The achieved bandwidth sits at 262 to 303 GB/s, within about 15 per cent of the best bandwidth observed. The layer is limited by how fast 154 MB can be streamed from memory, and the arithmetic is free. At \(M = 1\) the GPU delivers 151 GFLOP/s of a possible 7,445, 2 per cent. At \(M = 32\) it delivers 56 per cent. After that, time grows in proportion to \(M\), which is what a compute-bound operation does.

The model predicts the knee. With a ridge point of 24.6 FLOP/byte and an intensity of about \(M/2\), the product should leave the bandwidth-bound regime near \(M \approx 49\). The measured time was flat to 32, 32 per cent higher at 64, and nearly 2.5 times higher at 128. That is consistent with a knee between 32 and 64. It is a consistency check and not an independent test, since the bandwidth ceiling came in part from these same measurements.

The same layer in fp16 tells the same story with a different constant.

Tokens \(M\)	fp16 time (µs)	Share of fp16 peak	GB/s
1	271	0.9%	285
8	558	3.5%	140
32	394	20.0%	204
128	593	53.1%	152
1,024	4,436	56.7%	41

At \(M = 1\) the fp16 layer takes 271 µs against 510 in fp32: a factor of 1.9 for half the bytes. For a memory-bound operation, time follows bytes. That is the first point where quantisation (Chapter 12) shows its reason, before any question of quality arises. Two other things do not fit the idealised model. At \(M = 8\) the layer is slower than at \(M = 16\) in all three runs (558, 605 and 554 µs against 344 to 356), which looks like the library selecting a different kernel; I did not investigate. And at large \(M\) fp16 reaches only 57 per cent of its peak, against 90 per cent for fp32, which says that this particular shape does not feed the tensor cores well. The roofline is a bound, and here the gap between bound and measurement is informative.

The smaller feed-forward layer (\(N = 3{,}072\), 9.4 MB of fp32 weights) shows the other limit. Its fp32 time is 36 to 43 microseconds for \(M\) from 1 to 8, and about 40 microseconds in fp16 right up to \(M = 128\). That is the launch floor again. A 9.4 MB weight matrix would take 34 microseconds to stream at 278 GB/s, so bandwidth and launch cost are indistinguishable at this size. The weights of this layer are too small to be bandwidth-bound in any way that a launch-bound call would reveal.

Layout and alignment

Memory layout is one of the first things people suspect, and mostly it is not the problem. For a 2,048 × 2,048 product, passing a transposed view instead of a contiguous matrix changed the time by less than 10 per cent in either direction in both fp32 and fp16, because the library handles transposition inside the kernel. Forcing a contiguous copy of the transpose before the multiplication cost an extra 19 per cent in fp16. Slicing every other column, which forces a copy, cost 14 per cent more in fp16 than half the arithmetic would imply. Layout costs when it triggers a copy.

Alignment matters more, and only for the fast unit. Tensor cores operate on tiles, and sizes that are not multiples of 8 force the library to pad or fall back. For fp16, with \(n \times n \times n\) products:

\(n\)	Multiple of 8	GFLOP/s
4,088	yes	28,196
4,090	no	27,919
4,092	no	27,256
4,095	no	23,108
4,096	yes	29,427
4,097	no	21,185
4,100	no	25,589

One extra row, 4,097 instead of 4,096, costs 28 per cent of throughput. The pattern is not perfectly clean (4,090 and 4,092 are fine, 4,100 is not), because the library chooses among kernels with different tile shapes and the rule is not a simple divisibility test. In fp32 the effect is small: 4,097 ran 6 per cent slower than 4,096. These are single measurements, medians of 15 calls, and I would repeat them before relying on any one row. The practical consequence is already clear. Model dimensions chosen as multiples of 64 or 128, and vocabularies padded to a multiple of 8, are not arbitrary tidiness. GPT-2’s vocabulary of 50,257 is not a multiple of 8, so its output projection is a mildly awkward shape.

What reduced precision does to the answer

Faster is useful only if the answer survives. I multiplied \(1024 \times K\) by \(K \times 1024\) matrices of standard normal values, for \(K\) from 64 to 4,096, against a float64 reference, and measured the relative error in the Frobenius norm:

\(K\)	fp32 error	fp16 error	fp16 inputs rounded, exact product
64	\(1.5 \times 10^{-7}\)	\(3.6 \times 10^{-4}\)	\(2.9 \times 10^{-4}\)
512	\(4.1 \times 10^{-7}\)	\(3.6 \times 10^{-4}\)	\(2.9 \times 10^{-4}\)
4,096	\(5.7 \times 10^{-7}\)	\(3.6 \times 10^{-4}\)	\(2.9 \times 10^{-4}\)

The fp32 error is near its rounding limit and grows slowly with \(K\), as accumulated rounding does. The fp16 error is about a thousand times larger and does not grow with \(K\). The last column shows why. Simply rounding the inputs to 16 bits and then multiplying exactly gives most of the error, \(2.9 \times 10^{-4}\), so the hardware’s own accumulation contributes little. This is consistent with accumulating partial sums in 32 bits inside the tensor core. I infer that from the pattern, and have not confirmed it with the hardware documentation. The error reported is for random normal data; real weights and activations, with the outliers seen in Chapter 4’s residual stream, may behave differently, and Chapter 12 is where that is tested properly.

From one product to a model

The remaining question is what these effects do to a whole forward pass. Chapter 4’s hand-written GPT-2 is a sequence of exactly the products above, plus normalisation, softmax and activation functions. Measured with the library implementation, from two runs of the same command:

Tokens	GPU forward	CPU forward, 1 thread	Ratio
1	6.6 to 8.3 ms	34 to 42 ms	5×
16	7.0 to 16.6 ms	94 to 109 ms	7× to 14×
128	12.9 to 13.3 ms	312 to 752 ms	24× to 57×
512	27.7 to 32.4 ms	1.16 to 2.90 s	36× to 104×

I give ranges because the CPU numbers disagreed with themselves: the same forward pass at 128 tokens took 0.75 s in one run and 0.31 s in the next, a factor of 2.4. I did not find the cause, and it shows how little one CPU measurement can be trusted on a shared desktop machine. The GPU numbers were steadier.

The GPU column is the interesting one. The forward pass took about the same time for 1 token as for 128 tokens. A step that processes 128 positions costs roughly twice a step that processes one. This follows from the two effects above. At small token counts, the weight matrices are streamed once and the arithmetic is free; and the model is made of hundreds of operations, each paying the launch floor. I measured the second directly by comparing when the call returned to when the GPU finished:

Tokens	Call returns after	GPU finished after	GPU still working after return
1	4.9 ms	5.5 ms	0.6 ms
16	5.2 ms	6.0 ms	0.8 ms
128	4.6 ms	7.7 ms	3.0 ms
512	5.4 ms	29.8 ms	24.5 ms

At 1 and 16 tokens the GPU finished within a millisecond of the CPU finishing its submission: the forward pass was bound by the CPU issuing work, about 5 ms regardless of how much work it was. At 512 tokens the CPU still spent 5 ms issuing, and the GPU then took another 24 ms to finish; now the arithmetic dominates. I could not count the kernels per forward pass, because the PyTorch profiler recorded no GPU activity on this Windows setup. It did record 1,224 operator events for one 16-token pass (nested wrappers included, so an upper bound on launches); at even 5 microseconds of CPU time each, that is several milliseconds.

That fixed cost is large compared with the arithmetic of a decode step. Reducing it is the subject of Chapter 18, on compilation and execution graphs.

The engine on the GPU

Running the Chapter 6 engine with a device-aware backend, 32 greedy tokens, five runs on each device:

	CPU	GPU
Total time (median)	6.41 s	0.453 s
Tokens per second	5.0	70.7
Forward per step (median)	194 ms	11.9 ms
Selection per step (median)	5.2 ms	1.6 ms
Output identical across runs	yes	yes

The CPU and GPU produced the identical 32 tokens. That is a result for one prompt and 32 tokens. I should not read it as a guarantee, since the GPU computes a different sum order and a near-tie could resolve differently. The two manifests have different hashes, as they should: the device is part of what ran.

Two observations. Selection is now 12 per cent of a step on the GPU, against under 3 per cent on the CPU, because the sampler’s small operations pay the same launch floor as everything else. And the naive loop’s waste, a factor of 22 in positions processed at this length, is nearly invisible here: with a GPU step costing the same at 18 positions as at 48, recomputation is cheap until the arithmetic stops being free. That happens at a few hundred tokens, as the 512-token row shows. This means the benefit of the KV cache in Chapter 10 depends on where you are in this chapter’s curve, and I will measure it there instead of asserting it.

What the numbers do not show

• One consumer GPU, shared with a display. Results on a dedicated data-centre part, with a different tensor-core generation, memory system and driver model, can differ in every row. The shape of the curves (a launch floor, a memory-bound region, a ridge) is general; the positions of the knees are not.
• Noisy clocks. The same size varied by up to 40 per cent across runs. Where I give a range, believe the range.
• Ceilings are estimates. The roofline uses the best values seen, and I trust them to about 20 per cent.
• No kernel-level view. I could not see which kernels ran, their occupancy, or whether the fp16 path used tensor cores in each case. The explanations offered for the fp16 behaviour are inferences.
• CPU baseline is one thread. A multi-threaded CPU would narrow the gap in the middle sizes considerably. The comparison is between one CPU thread and the whole GPU, and it should not be read as CPU versus GPU in general.
• Random data. Matrix products were timed on normally distributed values. Data-dependent effects, such as power draw varying with the bit patterns, were not studied.
• No energy measurement.
• One model, one prompt. The engine comparison is for 32 tokens of GPT-2 from one prompt.

Inference Lab increment

The Lab now has what it needs to measure accelerated code:

• time_cuda(fn, warmup, iters, inner): event-based per-call time, optionally batched, and time_cpu for the comparison.
• ramp_clocks(): warm the device first.
• Ceiling probes and the helpers gflops, matmul_bytes, arithmetic_intensity, roofline_gflops and ridge_point, all unit-tested without a GPU.
• A device-aware LibraryBackend, whose manifest now says cpu or cuda:0.

The harness enforces a rule that will matter for every later chapter: a GPU result is recorded together with the device, the driver, the library versions and the clock state it was taken in, as a distribution, after a warm-up. A GPU number without those is a rumour.

What you should now be able to do

• Compute the FLOPs and the arithmetic intensity of a matrix multiplication from its shape and element size.
• State the roofline bound and find the ridge point from measured ceilings.
• Time an asynchronous GPU operation correctly, and say what is wrong with timing the call.
• Explain why a decode step uses a small fraction of a GPU’s arithmetic and why batching it up to the ridge point is nearly free.
• Identify the launch-bound regime, and say what dominates there.
• Say when a transposed or strided operand costs time and when it does not.
• Predict, from the roofline, how a memory-bound layer’s time changes when its weights go from 32 bits to 16.
• Say why a measured CPU-to-GPU speedup needs a range and a statement of the CPU configuration.

Key terms

• GEMM: general matrix-matrix multiplication; here, any product of two matrices.
• Arithmetic intensity: FLOPs per byte moved.
• Roofline: the bound that attainable throughput is the lower of the compute ceiling and bandwidth times intensity.
• Ridge point: the intensity at which the two ceilings meet.
• Memory-bound / compute-bound: limited by data movement or by arithmetic.
• Launch overhead: the fixed cost of issuing one GPU operation.
• Asynchronous execution: the CPU continues after queueing GPU work.
• CUDA event: a timestamp in the GPU’s work queue.
• Tensor core: a hardware unit for small matrix tiles in reduced precision.
• Clock ramp: the rise from idle to operating clock speed under load.

Reader exercises

1. Repeat the “time the call only” measurement and then correct it. By what factor does it overstate the throughput on your hardware?
2. Run launch on your machine. What is the per-call floor, and how many tiny operations per second can your CPU issue?
3. Use ceilings and then thin with your own numbers. Where does the knee of the output-projection layer fall, and does it match the ridge point you computed?
4. Find an \(M\) where fp16 is slower than fp32 for a thin product. What explains it?
5. Run the alignment sweep with values around 8,192. Is the penalty for a non-multiple of 8 the same size?
6. Add a multi-threaded CPU measurement of the square-matrix table. Where does the crossover with the GPU move?
7. Predict the time of an int8 weight-only output projection at \(M = 1\) from the roofline, then plan the measurement that would test it.
8. Pad GPT-2’s vocabulary from 50,257 to 50,264 and measure the output projection at \(M = 128\). Is the effect measurable?

References

• Roofline: An Insightful Visual Performance Model for Multicore Architectures — the paper that introduced the model used above.
• NVIDIA Deep Learning Performance: Matrix Multiplication — vendor documentation on tile sizes, alignment and arithmetic intensity.
• CUDA C++ Best Practices Guide — vendor documentation on asynchronous execution, timing and memory behaviour.

Evidence in this chapter

Claim	Category
A product costs \(2MNK\) FLOPs; intensity is FLOPs over bytes; the roofline bound	Established background
GPU launches are asynchronous, and the reference practice is events or synchronisation	Established background, from the cited documentation
Timing only the call implied about 757 TFLOP/s against 5.9 TFLOP/s with synchronisation	Book result
An 8×8 GPU matmul costs about 35 µs per call; a CPU thread does it in 4 µs	Book result, this machine
Output projection time was flat from 1 to 32 tokens in fp32 and used about 2% of peak at one token	Book result, three runs
The knee falls near the ridge point from the measured ceilings	Book result, consistent with the model
fp16 time follows bytes at one token (271 µs against 510 µs)	Book result
fp16 behaviours at \(M=8\) and at large \(M\)	Observed; cause not established
Alignment penalty of 28% at \(n=4097\) in fp16	Book result, single measurement
fp16 accumulates in 32 bits inside the tensor core	Book hypothesis, from the error pattern
GPU forward pass is bound by CPU issue at small token counts	Book result, one run
The engine is 14× faster on the GPU with identical output on 32 tokens	Book result, one prompt
Kernel counts, int8, multi-GPU, energy, dedicated-GPU behaviour	Not measured

Implementation status

• Manuscript: content/books/inference/07-chapter.md
• Runnable examples: examples/ch07/matmul_bench.py (original), examples/ch07/gpu_matmul.py (the measurement harness)
• Tests: tests/test_ch07_matmul.py, tests/test_ch07_gpu_matmul.py (helpers unit-tested; GPU tests skip when CUDA is absent; model commands are not exercised by tests)
• Evidence: research/ch07/sources.md, research/ch07/results.json, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It makes no claim about other hardware, dedicated compute GPUs, kernel-level behaviour or energy. Chapter 8 turns to the one operation in the model that is not a matrix multiplication by a weight: attention, whose cost depends on how long the sequence is.