Latency, Throughput and Time to First Token
TTFT, TPOT, latency, concurrency, percentiles.
Chapter 35 structured outputs. Part 7 now examines economics and evaluation, starting with metrics.
Definitions
- TTFT: time to first token.
- TPOT: time per output token.
- E2E latency: total request time.
- Throughput: tokens per second.
- Percentiles: tail behavior.
Why percentiles
Averages hide tail latency. Percentiles reveal user experience. TTFT p99 matters for responsiveness.
Denominators
Do not mix prompt and output tokens in throughput without stating. Prompt and output have different costs.
Research context
Metric names are not standardised enough for comparison by label alone. Define every latency metric from a timestamp pair. Report percentiles and offered load together with prompt/output length distributions. Use SLO-qualified goodput to expose the throughput/latency frontier. Chunked prefill and stall-free scheduling can control prefill/decode interference.
Experiment
The metrics collector aggregated 2 requests: TTFT p50 0.15 s, TPOT p50 0.025 s, throughput 41.67 tok/s. Tests verify percentile math, aggregation, rendering, and serialisation.
Limitations
- Synthetic data.
- No real workload.
- No hardware.
- No concurrency.
Key terms
- TTFT: time to first token.
- TPOT: time per output token.
- Throughput: tokens per second.
- Percentile: tail metric.
- Aggregation: combining measurements.
Reader exercises
- Add concurrency to the collector.
- Compare p50 and p99.
- Explain denominator separation.
- Design a benchmark report.
References
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving - peer-reviewed systems paper.
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve - peer-reviewed systems paper.
- vLLM Benchmark CLI - official runtime documentation.
Implementation status
- Manuscript:
content/books/inference/36-chapter.md - Runnable example:
examples/ch36/ - Tests:
tests/test_ch36_metrics.py - Evidence:
research/ch36/sources.mdandresearch/ch36/results.json - Verification command:
python -m unittest discover -s tests -v
The chapter is a draft. It does not claim production performance. The next chapter will examine cost.