Benchmarking Against Established Engines

Chapter 43 added backends. This chapter benchmarks fairly.

Apples-to-oranges

Comparisons must match workload, hardware, and configuration. Undocumented settings invalidate results.

What to compare

Compare output correctness, TTFT, throughput, peak memory, and task success. Report all.

Research context

Compare engines only under a shared workload, model artefact, quality target, arrival process, and cache policy. Record warm/cold state, versions, command lines, hardware, failures, and raw per-request data. Keep vendor/repository reference results in a separate table from book-reproduced measurements. MLPerf and vLLM document the metadata needed for comparability.

Unrun plans

If an engine is unavailable, publish a reproducible plan rather than fabricating results. The repository example lists vLLM and SGLang as unavailable.

Experiment

The example produced an unrun plan. Tests verify plans, serialisation, and availability flags.

Limitations

• No real comparison.
• No hardware.
• No licenses for some engines.
• No results.

Key terms

• Matched workload: same inputs.
• TTFT: time to first token.
• Throughput: tokens per second.
• Peak memory: max memory.
• Unrun plan: documented but not executed.

Reader exercises

1. Install vLLM and run matched workload.
2. Compare output correctness.
3. Explain unfair comparisons.
4. Design a comparison report.

References

• MLPerf Inference: Datacenter - benchmark specification and results portal.
• vLLM Benchmark CLI - official runtime documentation.
• DeepInfer benchmark and evaluation guide at 774ce244 - pinned first-party benchmark documentation.

Implementation status

• Manuscript: content/books/inference/44-chapter.md
• Runnable example: examples/ch44/
• Tests: tests/test_ch44_comparison.py
• Evidence: research/ch44/sources.md and research/ch44/results.json
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft. It does not claim benchmark results. The next chapter will conclude.