Designing the Inference Runtime

Capstone architecture, interfaces, safety.

Inference From First Principles Chapter 41 of 45

01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

Chapter 40 designed benchmarks. Part 8 now builds the Inference Lab capstone, starting with design.

Architecture

The runtime integrates tokenizer, model backend, sampler, cache, scheduler, API, and metrics. Each component has a typed contract.

ADRs

Architecture decision records document choices. The example includes two ADRs: CPU-first and typed contracts.

Contracts

Contracts enable swappable backends and testability. The contract test verifies a backend generates expected output.

Research context

Compare runtimes by explicit subsystems: request state, scheduler, cache manager, executor, sampler, protocol, and telemetry. Pinned implementations such as DeepInfer, PagedAttention, and SGLang distinguish reusable patterns from model- or vendor-specific choices. Define invariants at subsystem boundaries before optimising.

Experiment

The example produced 2 ADRs and a valid contract. Tests verify contract validation, demo, and serialisation.

Limitations

Toy backend.
No real integration.
No performance.
No safety audit.

Key terms

ADR: architecture decision record.
Contract: typed interface.
Backend: model implementation.
Capstone: final integration.
Non-goals: explicit exclusions.

Reader exercises

Add an ADR for caching.
Design a backend contract for streaming.
Explain non-goals.
Draw the integration diagram.

References

deepseek-ai/DeepInfer at 774ce244 - pinned first-party repository.
Efficient Memory Management for Large Language Model Serving with PagedAttention - peer-reviewed systems paper.
SGLang: Efficient Execution of Structured Language Model Programs - peer-reviewed systems paper.

Implementation status

Manuscript: content/books/inference/41-chapter.md
Runnable example: examples/ch41/
Tests: tests/test_ch41_contracts.py
Evidence: research/ch41/sources.md and research/ch41/results.json
Verification command: python -m unittest discover -s tests -v

The chapter is a draft. It does not claim production runtime. The next chapter will implement scheduling and caching.