Programmer.ie Book draft

Inference From First Principles

How a transformer becomes a request, how a request becomes tokens, and how engineering choices determine latency, throughput, cost, and correctness.

A model can know the answer; inference is the engineering that makes the answer arrive, at a cost, under constraints, and with evidence.

Inference is the missing link between a trained model and a useful system. It is not training, architecture research, prompting, or product UX. Inference owns tokenisation, tensor computation, memory, caching, batching, scheduling, quantization, distributed execution, protocol handling, and measurement.

Why this book exists

A trained transformer can be run in many ways: as a Python loop on a CPU, as an optimised GPU kernel, as a batched service, or as a distributed engine. The weights are the same; the behaviour, latency, throughput, and cost are not. The question is not only whether a model produces a token, but what computation produced it, how much memory and time it consumed, and whether the result is reproducible.

The central distinction

The book keeps these stages separate:

Model parameters - the learned weights and architecture.
Request - the user input, options, and constraints.
Tokens - the discrete symbols the model operates on.
Computation - matrix operations, attention, and feed-forward passes.
Generated tokens - autoregressive output.
Served response - streamed or batched API result.
Measured outcome - latency, throughput, cost, and correctness under a declared workload.

A correct token is not a fast system. A fast system is not a cheap system. Each transition has to be justified and measured separately.

The cumulative experiment

The book builds one small inference system progressively. It starts with a deterministic CPU toy generator, then adds a real model, matrix execution, attention, KV caching, quantization, batching, scheduling, and finally a multi-backend runtime.

Its workloads include exact prompts, long contexts, concurrent requests, tool calls, structured outputs, and declared benchmarks. Measurements are linked to hardware, software versions, model revisions, and input manifests.

Chapter plan

The forty-five-chapter manuscript spans eight parts:

Part	Chapters	Focus
I. Understanding Inference	01-06	Definition, training to execution, tokens, transformer computation, generation, first engine
II. Computational Foundations	07-12	Matmul and GPU, attention cost, prefill and decode, KV cache, bandwidth, quantization
III. Making Inference Fast	13-18	FlashAttention, PagedAttention, batching, prefix cache, speculative decoding, compilation
IV. Modern Model Architectures	19-24	MoE, routing, multi-head/latent attention, sliding window, long context, architecture versus runtime
V. Building an Inference Service	25-30	API, scheduling, concurrency and streaming, distributed inference, disaggregation, reliability
VI. OpenAI, Anthropic and Applications	31-35	Protocols, API-level optimization, reasoning, tool calling, structured outputs
VII. Economics and Evaluation	36-40	Latency and throughput, cost, GPU utilization, correctness, benchmarks
VIII. Inference Lab Capstone	41-45	Runtime design, scheduling and caching, multiple backends, benchmarking, lessons

Chapters are generated individually or in explicitly approved connected batches. Drafts receive independent technical and editorial review before the author decides readiness.

Status

Chapters 01–28 now have starter manuscripts, examples, tests, local result artefacts, and research-revised prose with three reader-facing primary references apiece. Chapters 29–45 remain prompt-only and must not be described as implemented; each has a three-source research dossier to constrain future work. All chapter flags remain drafts pending independent technical and editorial review.

Research map

The chapter research map records exactly three selected primary sources for every planned chapter, their evidential limits, and the editorial consequences for the draft.

Method

implement → measure → explain → write → verify. CPU-first baseline, optional hardware-specific integrations. Every result must have commands, versions, workload and evidence; otherwise it is explicitly NOT_RUN.

Contents

Chapters

What Is Inference?

Distinguish mathematical inference, LLM forward passes, token generation, and production serving, and measure where one request actually spends its time.

From Training to Execution

What training leaves behind, what must be true before it is executed, and what the first request pays that later ones do not.

Tokens, Embeddings and Probability

From text to token ids to a vector per token to scores over the vocabulary, and the rules that turn scores into a choice. Measured on a real model.

The Transformer as a Computation

A GPT-2 forward pass written out by hand, checked against the library, and used to count parameters, FLOPs and time component by component.

Autoregressive Generation

A forward pass becomes a sequence: the cost of the loop, why checking is cheaper than producing, and how to stop and stream without corrupting the text.

Building Our First Inference Engine

Assemble tokeniser, backend, sampler, loop and record into one engine with a stable interface, check each replaceable part against a reference, and measure the baseline.

Matrix Multiplication and GPU Execution

What a matrix multiplication costs, why one token uses about two per cent of a GPU, and how to time a GPU without fooling yourself.

Attention and Its Computational Cost

Attention is quadratic in arithmetic, but what hurts first is the score tensor. Measured memory, time and bandwidth for prefill-shaped and decode-shaped attention.

Prefill and Decode

A request does two different jobs. Measured on a real model: what each costs, where the textbook bottleneck fails for a small model, and how a long prompt stalls everyone else.

Key-Value Caching

A KV cache written by hand into GPT-2: why it is exact, what it saves, what it costs in memory, and how it fails when its state is wrong.

Memory Bandwidth and Compute Bottlenecks

A roofline with three ceilings, tested against 58 measured GPU operations, and then against a batched decode that it gets wrong.

Quantization and Numerical Precision

Four different things are called quantisation quality: storage, reconstruction error, model behaviour and speed. Measured separately on GPT-2, they disagree.

FlashAttention and Optimized Kernels

Block-wise exact attention is an algorithm; making it fast is a kernel. Both measured, plus an honest account of what could not be run here.

PagedAttention

A KV cache in fixed-size blocks: how much memory contiguous allocation wastes, what block tables, reference counts and copy-on-write buy, and what greedy admission costs.

Continuous Batching

A real GPT-2 server under static and continuous scheduling: latency, goodput and wasted work at four offered loads, plus what chunked prefill and queue policy do to fairness.

Prefix Caching

Reusing the work of a shared prefix: the identity it requires, what it saves on a real model, how hit rates depend on cache size and isolation, and the timing signal it leaks.

Speculative Decoding

Letting a small model guess and a large one check: the acceptance rules that keep the output exact, what the guesses are worth on real text, and where the arithmetic says it should and should not pay.

Compilation and Execution Graphs

Capturing a decode step as a graph: what Dynamo's guards promise, what a CUDA graph removes, what an Inductor-compiled kernel fuses, what each costs to build, and why the first claim in this chapter's earlier draft was wrong.

Mixture-of-Experts Inference

Storing many experts and computing with a few: what a batch of tokens makes a layer read, four ways to run the layer, what each costs on a GPU, and why capacity limits make a token's answer depend on its neighbours.

Expert Routing and Load Balancing

What a trained router actually does to a serving system: how three balancing methods differ, what a capacity limit costs in quality on learned routing, how placement from measured load cuts the straggler, and what communication would add.

Multi-Head and Multi-Latent Attention

How many keys and values a model keeps decides its cache, its decode bandwidth and how many requests fit: real cache sizes from cached models, why a naive grouped-query kernel saves nothing, what converting a multi-head model costs, and a latent-attention layer built from the paper and measured.

Sliding-Window Attention

A window bounds what attention reads, but not what a model needs: a rolling cache measured on Gemma-2, how far influence travels through stacked windows, what a plain window does to a model not trained for one, and why a masked window is not yet a speed-up.

Long-Context Inference

A context length is five claims, not one: that the model accepts the tokens, behaves at those positions, fits in memory, answers in time, and uses what it is given. Each measured on a model that advertises 128K, on a card that can hold a quarter of it.

Model Architecture Versus Runtime Architecture

Changing one thing at a time on a real model to find where the model ends and the runtime begins, what the software stack does when asked for something it cannot do, and an admission function that refuses before any work is done.

From Inference Loop to API

Putting the engine behind an HTTP interface and measuring what the boundary has to decide: where a request's time goes, how a character limit fails as a token budget, which errors a client can act on, how a schema may change, and what a shared backend does to concurrent callers.

Request Scheduling and Admission Control

What a queue in front of a one-at-a-time engine does when traffic exceeds its capacity, measured on a live service and a simulator checked against queueing theory: four admission policies, the deadline that makes an answer worth having, who gets turned away, and what stale estimates cost.

Concurrent Requests and Streaming

Thirty-two streams on one batched engine over real sockets: what overlap buys, why a stream's first token waits for its neighbours, what one slow reader does under four policies, and how long the engine goes on working for a client that has gone.

Distributed Inference

GPT-2 split across processes by tensor and pipeline parallelism, exact to the last token and no faster, and the cost model that explains why: what each strategy splits, what it must communicate, what that communication costs, and what a machine has to provide before splitting helps.