GPU Utilization and Capacity Planning
Occupancy, memory, queue saturation, headroom.
Chapter 37 estimated cost. This chapter plans capacity.
Occupancy
GPU occupancy measures how many warps are active. Low occupancy can indicate underutilization or memory bottlenecks.
Memory limits
Memory limits cap batch size and context length. Exceeding memory causes OOM.
Queue saturation
Queue saturation occurs when arrival rate exceeds service rate. Latency grows without bound.
Capacity headroom
Capacity planning requires headroom for bursts. Extrapolating beyond measured load is unsafe. The model rejects loads beyond observed.
Research context
A single utilisation percentage cannot diagnose memory, tensor-core, scheduler, or queue bottlenecks. Roofline relates attainable performance to operational intensity and hardware ceilings. MLPerf separates scenarios, quality targets, latency constraints, throughput, and system metadata. Capacity should be linked to an arrival and token-length distribution plus latency SLOs, and saturation should be found with controlled load sweeps rather than extrapolation.
Experiment
The capacity model was built from 4 points. Max observed load was 8. Extrapolation to 100 was rejected. Tests verify build, rejection, demo, and serialisation.
Limitations
- Synthetic data.
- No real GPU.
- No confidence bands.
- No production planning.
Key terms
- Occupancy: active warps.
- Memory limit: max memory.
- Saturation: queue overload.
- Headroom: spare capacity.
- Extrapolation: predicting beyond data.
Reader exercises
- Add confidence intervals.
- Measure real GPU occupancy.
- Explain why extrapolation is unsafe.
- Design a capacity plan.
References
- Roofline: An Insightful Visual Performance Model for Multicore Architectures - peer-reviewed paper.
- NVIDIA Data Center GPU Manager documentation - official observability documentation.
- MLPerf Inference: Datacenter - benchmark specification and results portal.
Implementation status
- Manuscript:
content/books/inference/38-chapter.md - Runnable example:
examples/ch38/ - Tests:
tests/test_ch38_capacity.py - Evidence:
research/ch38/sources.mdandresearch/ch38/results.json - Verification command:
python -m unittest discover -s tests -v
The chapter is a draft. It does not claim production capacity. The next chapter will examine correctness.