Concurrent Requests and Streaming

Thirty-two clients each asked for 64 tokens. Served one after another by the engine of Chapters 25 and 26, they would have taken 22 seconds in total, and the last would have waited 21 seconds for its first word. On the batched engine of this chapter they were all answered in about 1.5 seconds, 14 to 15 times faster, at 1,320 to 1,376 tokens a second against 90 to 94 for a single stream. Every one of the 189 concurrent streams I compared with its solo run had identical token ids.

Two other things were measured in the same service, and they are the reason for the chapter. One reader among eight that reads slowly made the other seven take 28 seconds instead of one, in the obvious design. And a client that disconnected mid-stream left the engine generating its 200 tokens for another two seconds when nothing listened for the disconnect, and for 10 to 19 milliseconds when Starlette’s own response class did.

The question

Chapter 25 answered a request with a response. Chapter 26 decided whether to start it. Both served one request at a time. A real service has many open at once, and for a language model an answer is not a value but a process that lasts seconds, so the service has to decide what to do with a request while it is running: let its neighbours share the engine, send its tokens as they are made, wait for a client that cannot keep up, and notice when the client has gone.

Four questions follow. What does overlap buy, and what does it cost the individual stream? What goes over the wire, and what can still go wrong after the first byte? What should happen when a client reads more slowly than the engine writes? And what does it take for the work, and the cache blocks, of an abandoned request to be released promptly and exactly once?

Where this sits

Chapter 14 built the paged cache and a decode step for sequences of different lengths. Chapter 15 put them in a continuous-batching scheduler and replayed traces against it with a virtual clock, which is a good way to measure a scheduler and not a service. This chapter takes the same pieces and runs them live: requests arrive over sockets while the engine runs, and tokens leave over sockets while it runs. Chapter 9 defined time to first token and said it must include what happens before the engine; Chapter 25 measured that for a one-token request; here it is measured for a stream as a client sees it.

Chapter 26’s gate decided, one request at a time, whether to start. That gate does not appear here; the service has a plain limit on the number of requests waiting for a slot, and Chapter 26’s finding that a count is not a bound on latency applies to it. Combining the deadline-aware gate with the batched engine is left as an exercise. Cancelling a request at the engine’s request, retrying, and recovering belong to Chapter 30; this chapter builds only the part of cancellation that streaming forces, which is releasing a stream’s resources when its client goes.

A step-driven engine

The engine of Chapters 25 and 26 ran a request from start to finish inside one call. For streams to overlap, the loop has to be inverted. A step-driven engine is called repeatedly and each call does one round of work for everything that is running:

1. apply what arrived since the last step: new requests join the waiting list, cancellations are applied;
2. admit waiting requests while there are slots and cache blocks (reserving the blocks a request can need, prompt plus the tokens it asked for, as Chapter 15 did), and run each admitted prompt’s prefill, which produces its first token;
3. decode one token for every other running sequence, in one batched pass over the paged cache;
4. return the events: tokens made, and sequences finished, with a reason.

Everything that touches the cache happens inside step(). The rest of the program talks to the engine through a thread-safe inbox: submit, cancel, pause and resume only record a wish, which the next step carries out. That is what makes cancellation safe: a sequence is never removed while its tensors are in use, and its blocks are released at one place, which fails loudly if called twice (the block table raises on a second free, and a test checks that it does).

Because a step decodes the whole batch together, its cost is not proportional to the batch. The median time of a decode step with no prefill in it, over a live run on GPT-2:

Thirty-two sequences cost 1.5 times what one costs; the endpoints give about 0.17 ms per added sequence on top of about 9.6 ms. (Chapter 11 found 7 to 20 per cent more per step for up to 32 sequences with a static cache. This step is the paged one of Chapter 14, which gathers each sequence’s blocks into a padded tensor, and I did not separate the two causes of the larger figure. The small growth is the launch-dominated regime of Chapter 18: on a model this size the step is mostly fixed cost.) The first-order model of a request in a full batch follows from it: its tokens come about one step apart.

Two choices in the loop matter later. A new sequence’s first token is made by its prefill, in the step in which it is admitted, and is delivered when the step ends. And events leave the engine in batches, one batch per step, so what a client sees is quantised by the step.

Isolation

Rows of a batch do not interact. That is the property that lets the service say a request’s answer does not depend on its neighbours, and it is a property to test and not to assume (Chapter 24 found that a padded batch changed which of two tied tokens won in 3 of 12 cases in half precision). Here the engine is float32 on the paged cache and the comparison is over the wire: every one of the 189 concurrent streams (three repetitions of 1, 2, 4, 8, 16 and 32 streams) had token ids identical to the same prompt run alone. That is a result for greedy decoding of GPT-2 in float32 on this cache; it says nothing about half precision, other models, or sampling under concurrency.

The wire

A stream is an HTTP response that does not end. Server-sent events are the framing most clients understand. The WHATWG specification defines it as text/event-stream, always UTF-8, as lines: a field is name: value, the names that matter are event, data, id and retry, a line that begins with a colon is a comment, and a blank line ends an event and dispatches it. An event without a blank line at the end of the stream is discarded. A client that reconnects sends the last id it saw in Last-Event-ID. The specification suggests a comment about every fifteen seconds to keep old proxies from dropping an idle connection.

The format says nothing about tokens. Above it sits a schema the service defines. The lab’s is small: a start event with the request id and the prompt’s token count; one token event per token with its index, its id and the text it adds; a done event with the finish reason and usage; and an error event in the Chapter 25 envelope. Anthropic’s streaming documentation, which describes an interface and nothing about the provider’s implementation, has the same shape at greater length: a named event with a JSON body that repeats the type, a ping event that may appear any number of times, and an error event, for which it gives overloaded_error as the example and says it would be a 529 outside streaming. It also says new event types may be added, so clients should handle unknown ones: Chapter 25’s rule about tolerant readers, one layer up.

The error event exists because of a fact about HTTP. Once the first byte of the body has been sent the status line is gone: 200 has been said. Everything that can be refused is therefore refused before the stream opens, as an ordinary error. The service validates the request, counts the prompt’s tokens against the budget, and checks the waiting limit before it returns the response object, so a request that is too long gets a 422 with application/json and the Chapter 25 envelope, and a stream only starts for a request that has been accepted. After that, a failure can only be an event.

The text in a token event is a delta made by Chapter 5’s incremental decoder, which holds back the bytes of a character that is not yet complete. A stop string is matched on the text, across token boundaries, and a match ends the sequence in the engine early and closes the stream with done and a finish reason of stop.

I wrote an SSE parser for the experiments from the rules above, so that what the server writes can be read back by something that does not know how it was written. It has unit tests for each rule (a leading byte order mark, all three line endings including a CR at the end of a chunk, comments, an event with no data, an id containing NUL, retry with digits only, an unfinished event at the end, and the same result for one chunk and for one byte at a time). It is not independent: no second implementation was used to check it, so a misreading of the rules would be in both writer and reader.

On 40 prompts of 32 tokens, streamed and answered in full by the same server, 40 of 40 streams had the text/event-stream content type, the request id in the header equal to the id in the start event, the events in the order start, tokens, done, token ids and text identical to the non-streaming answer, the same usage and finish reason, a last event id equal to the number of events, and nothing left unfinished at the end. A stream cost 73 to 76 bytes per token, counting its start and done events, so a 32-token stream was about 2.4 KB. None of the 1,280 token events carried an empty delta, which is what English prose gives GPT-2; text in a script that spreads a character over several tokens would.

Overlapping streams

Question. What does a batched engine give 1 to 32 simultaneous streams, measured by a client over HTTP?

Hypotheses (book hypotheses). (H1) Aggregate throughput grows nearly in proportion to the number of streams, while each stream’s gap between tokens grows by much less. (H2) Time to first token grows with the number of requests that arrive together, because their prefills run one after another inside a step. (H3) A request that arrives while others are running waits for the step in progress and then for its own prefill, and no more.

Held constant. GPT-2 in float32 on the paged cache, 32 prompts of about 200 characters taken from this book, 64 greedy new tokens each, the same server for every level. At each level N the first N prompts are sent at once from separate connections, three times; the makespan runs from the first request sent to the last stream finished. For comparison each prompt was also run alone, one after another.

What would falsify. H1 fails if 32 streams achieve less than twice the throughput of one, or if the gap between tokens rises in proportion to N. H3 fails if a joiner’s first token takes much more than a step plus a prefill.

(Ranges are over three repetitions. “Back-to-back” is the sum of the 32 solo runs’ durations, 22.05 s for 32; the solo medians were 16 ms to the first token, 0.68 s for the stream and 10.1 ms between tokens.)

H1 stands. Thirty-two streams ran at about fifteen times the rate of one, and a stream’s median gap went from about 10 to about 17 ms. The arithmetic of the step explains most of the aggregate: 63 decode steps at 15.1 ms is 0.95 s, plus about 0.45 s for the prefills, is 1.40 s, against a measured makespan of 1.49 to 1.55 s. At the tail, a stream’s 99th-percentile gap was 22 ms at 32 streams, but at 16 streams two of three repetitions had a p99 near 56 ms. I did not find the cause of those occasional gaps; the machine is shared.

H2 stands, and has a shape worth seeing. At 32 streams the median and the 99th-percentile time to first token are equal to within a millisecond (441 and 442 ms in one repetition). All thirty-two prompts are admitted and prefilled in the same step, one after another at about 14 ms each, and none of their tokens is delivered until the step ends. The first token of every stream arrives when the last prefill finishes. Delivering each stream’s first token as soon as its own prefill is done would give the early ones a better time and the last the same; the lab does not, and it is an exercise.

H3 stands, with a number. With 8 long streams already running, twelve 8-token requests arriving at random times between 0.6 and 2.1 seconds had a median first token of 41 ms (p99 62 ms); with 24 running, 47 ms (p99 77 ms); on an idle engine, 16 ms. A joiner pays the rest of the step in progress, its own prefill, and the wait for delivery, and nothing resembling the queue the other streams form.

What this does not include is memory pressure. The cache had 1,800 blocks and 32 streams reserved about 7 each; no run came near admission being limited by blocks, so the waiting limit and the reservation rule are exercised by unit tests and not by the experiment.

Where backpressure begins

Chapter 26’s service had no place for backpressure because it returned a whole response at the end. A stream has two: the engine produces tokens at its own speed, and a client reads at its own. Between them are several buffers: the hub’s queue for the stream, the server’s write buffer, the kernel’s socket buffers on both ends, and whatever sits between, none of which the application sees directly.

I measured the part nearest the server. A client connected and read nothing at all while the server wrote events as fast as it could. The server’s writes stopped being accepted after 252,434 bytes with 133-byte events, 262,395 with 1 KB events, 278,613 with 16 KB events, and 327,705 or 262,164 with 64 KB events, the same in each of three repetitions. So on this machine, a client that never reads absorbs roughly a quarter to a third of a megabyte before the server’s next write blocks.

A stream costs about 75 bytes a token. A 256-token stream, the largest the Chapter 25 budget allows, is about 19 KB: a thirteenth of the smallest figure above. A client with ordinary token events and the Chapter 25 token limit never exerts backpressure on the server at all. The slow client the starter chapter worried about is invisible on this stack unless events are large, because the whole answer fits in buffers before the first write blocks. Events carry more in practice (per-token log-probabilities, alternatives, tool-call deltas), and answers can be longer than 256 tokens, and then the slow reader appears. For the next experiment I made it appear by padding every token event with a 64 KiB SSE comment, so that a 64-token stream is about 4 MB; that is a laboratory device and is stated as one.

A client that cannot keep up

Four policies were built, each a different answer to the question of what to give up:

• Naive. The step loop hands each event to its stream and waits for the stream to take it before going on. This is what a straightforward implementation does if the code that writes to a client is called from the engine’s loop and awaited.
• Unbounded. Events are queued per stream without limit. The engine never waits for a reader.
• Pause. A stream whose queue reaches eight events is left out of the decode batch until its reader has caught up. Its cache blocks and its slot stay reserved.
• Drop. Like unbounded, but a stream that has been eight events behind for more than a stall timeout (0.25 s) is cancelled and told so.

The experiment: eight streams of 64 tokens with padded events; k of them (0, 1, 2 or 4) read one chunk and then sleep for 0.25 s, the rest read at once. I looked at the healthy streams’ completion time and gaps between tokens, and at when the slow streams’ sequences left the engine.

The requirement that one slow reader must not hold up the others fails for the naive policy and holds for the other three. With one slow reader among eight, the naive server finished the seven healthy streams after 28 seconds instead of one, and the 99th-percentile gap between their tokens went from 19 ms to about a second. The step loop was waiting for a reader. Nothing was wrong with the slow client except that it was slow; the defect is that the server’s progress depended on it.

The other three differ in what they cost. Unbounded released the slow stream’s cache after a second, because the engine finished generating and the client read the buffered output later: the cost is the events held. Here that is small, because the lab stores unformatted events and builds the 64 KiB padding as each event is written. A server that formatted eagerly would hold about 4 MB for each slow client of 64 tokens (64 events of 64 KiB; arithmetic and not a measurement), and the bound on it is the token limit, not any policy. Pause kept the engine’s resources for the slow stream’s own request, for 21 to 25 seconds: the cache blocks and a slot in the batch were held until the reader caught up. With a batch limit of 32 and a few such readers, that is slots that fast requests cannot use. I did not fill the batch, so the effect on admission is argued here and not measured. Drop gave the slot back after 0.6 to 0.8 seconds, and the client, when it eventually read to the end of what had been sent, received an error event of type slow_client at 18.7 to 19.7 seconds; its connection had been busy delivering the backlog the whole time.

None of this is the right policy for every service. The shape is that each policy chooses who bears the cost of a slow reader: everyone (naive), memory (unbounded), the slow request’s own cache (pause) or the slow request itself (drop). A sensible service pauses briefly and then drops, which is a combination I did not build.

When the client goes away

A stream is a request that outlives the function that started it, so something has to own what it holds. In this service the owner is the stream’s generator: whichever way it ends, a finally calls hub.close(session), which is idempotent: if the stream had not finished, it asks the engine to cancel the sequence, and removes the stream from the hub. The engine carries out the cancel at the next step boundary and releases the blocks in one place. Two parts of that sentence needed checking against the software I was using, and one of them I got wrong.

My first design assumed that Starlette would only learn a client had gone when a write failed, so I wrote a response class that also listened for the disconnect. Reading the installed code corrected me: uvicorn 0.54 advertises ASGI spec version 2.3, and below 2.4 Starlette’s StreamingResponse already listens for the disconnect and cancels the body. What uvicorn does do is return from send without error once the connection is lost. So a response that only writes, as Starlette’s is when the server speaks 2.4, would never learn. The experiment therefore compares two classes: Starlette’s own, and a write-only one that is Starlette’s 2.4 path without a failing write. The chapter reports the comparison, because the question is what happens if nothing is listening, which a reader may meet on another stack.

I measured the time from the client’s close, a clean one (FIN) or an abortive one (RST), to the release of the stream’s blocks as logged by the server, at five moments, six repetitions each, with a batch limit of four so that a stream could be made to wait for a slot.

With Starlette’s class the engine stopped within one step of the client leaving, having generated at most one token that nobody read, and a stream that was still waiting for a slot was never admitted at all. The largest single figure was 34.5 ms. FIN and RST showed no consistent difference. With a response that only writes, every one of the 48 abandoned sequences ran to its limit of 200 tokens, for clients that had read between 0 and 20.

The second experiment is a leak test. Three hundred streams from up to 48 concurrent clients on an engine with a batch limit of eight: 35% read to the end, half were disconnected after between 0 and 40 tokens (by FIN or RST), and 15% were slow readers disconnected after 3 or 10 tokens. 165 left before the end.

Neither design leaked a block: in both, every sequence was released once and the cache was whole afterwards, with no reserved, active, waiting or registered stream left and no error in the hub. The write-only design did not leak because it did not notice: every abandoned sequence finished its generation. What it cost was the work. For the 165 streams whose clients left early the engine produced 5.2 tokens for every token read, against 1.4 with Starlette’s class, and the same 300 streams took 47% longer. A token budget (Chapter 25) bounded the waste per stream; it did not remove it.

The mechanism that makes cleanup reliable is small and has four parts: one idempotent close that every exit path calls, a close that does not wait (the generator’s finally cannot await a cancelled task), an engine that applies cancellations only at step boundaries, and a release that raises if it is repeated. The rest is the test: the leak test checks the blocks, the registered streams and the number of releases per stream, and not only that the server is still up. In the Starlette run 294 engine events arrived for streams that had already closed (the finish of each cancelled sequence and the tokens made before the cancel took effect); they were dropped, and a counter records them.

What the numbers do not show

• One model on one machine. GPT-2 small in float32 on a shared RTX 2060, in its faster state; Windows; loopback. The step costs, the buffer sizes and the disconnect times are properties of this stack.
• Greedy decoding. The identity of concurrent and solo streams is for greedy decoding. Sampled streams use per-request seeds and were not compared.
• Padded events. The slow-client experiment inflates events to make real backpressure happen. A reader of 0.25-second pauses is also arbitrary. Each configuration was run once, not repeated.
• No saturation. The cache never limited admission, and the batch limit was only reached in the leak test. How a full batch treats a paused stream is argued and not measured.
• Two clients. httpx and a raw socket. A browser’s EventSource, a proxy, TLS and HTTP/2 were not used, and the specification warns that chunking can harm reliability.
• No resumption and no keep-alive. The service does not send comments to keep idle connections alive and does not honour Last-Event-ID.
• The parser is not independent. It was written from the specification by the same author as the writer.
• The assumption that corrected itself. I assumed the wrong thing about Starlette and found out by reading its source; the same could be true of something I did not read. The experiments measure behaviour for that reason.

Inference Lab increment

examples/ch27/stream_lab.py adds:

• sse_event, sse_comment and SSEParser: the wire format and a parser for it.
• BatchEngine: the step-driven engine over the Chapter 14 cache, with thread-safe submit, cancel, pause and resume, reservation-based admission, and exactly-once release.
• Hub: the step loop on asyncio, per-stream queues, the four policies, and an idempotent close.
• build_stream_app: the Chapter 25 service with POST /v1/stream and POST /v1/generate on the same engine, refusals before the first byte, a waiting limit, /stats and the lab routes.
• WriteOnlyStreamingResponse: the response that never listens, for the measurement.
• Commands sse, buffers, overlap, slowclient, disconnect and serve.

To run it and read a stream:

python examples/ch27/stream_lab.py serve --policy pause --device cuda --port 8000

curl -N http://127.0.0.1:8000/v1/stream -H "content-type: application/json" -d '{"prompt": "The capital of France is", "max_new_tokens": 8}'

The rule this adds: a stream owns what it holds, and every way it can end calls one idempotent close; the engine never waits for a client, and a slow client is handled by a policy that says who pays. Overlap is checked against solo runs, and cleanup is checked by the state of the cache afterwards.

What you should now be able to do

• Describe a step-driven engine and say why cancellation and admission happen at step boundaries.
• Read a server-sent-events stream by the specification’s rules, and say what can be refused before the first byte and what can only be an event.
• Measure the throughput and the first-token delay of overlapping streams, and verify that overlap did not change the answers.
• Say where backpressure begins on a given stack, and compare the policies for a slow reader by what each one makes someone else pay.
• Test that abandoned requests release their resources promptly and exactly once, by inspecting the cache and not only the response.

Key terms

• Step-driven engine: an engine called repeatedly, doing one round of work for every running request per call.
• Step boundary: the point between two steps, where requests join and leave.
• Server-sent events (SSE): a text framing of an unending HTTP response as events separated by blank lines.
• Backpressure: a slower consumer limiting a faster producer through a bounded buffer.
• Head-of-line blocking: one stalled consumer holding up work for the others.
• Idempotent close: a cleanup that can be called from every exit path and does its work once.
• Zombie generation: work done for a client that has gone.
• Exactly-once release: each resource is returned once, not zero times (a leak) and not twice (a corruption).

Reader exercises

1. Deliver each stream’s first token as soon as its own prefill finishes, and repeat the 32-stream experiment. Which percentile of the time to first token improves?
2. Combine pause and drop: pause a stream for a stall timeout, then cancel it. Measure how many batch slots paused streams hold when the batch is full.
3. Add comment keep-alives every 15 seconds and put a proxy in front of the service. What does the proxy do with an idle stream?
4. Implement Last-Event-ID resumption for a dropped stream. What must the server keep, and for how long?
5. Put the Chapter 26 deadline gate in front of the batched engine and replace the waiting limit with a deadline rule.
6. Format events eagerly and measure the memory held by the unbounded policy for four slow readers.
7. Repeat the identity check for sampled streams and in half precision, and report the first divergence by margin as in Chapter 24.
8. Replace the gather in the paged decode with a kernel that reads the blocks through the table, and measure the step cost at 32 sequences.

References

• WHATWG HTML Living Standard: Server-sent events — the event-stream format, reconnection and Last-Event-ID; read on 22 October 2026 (summary).
• Anthropic: Streaming messages — provider documentation of an interface (event types, ping and error events); not a statement about the provider’s implementation; read on 22 October 2026.
• Anthropic: Claude API errors — errors that arrive in a stream after a 200; read on 22 October 2026 (Chapter 25).
• Starlette 1.7.0 and uvicorn 0.54.0, read as installed (StreamingResponse.__call__; the HTTP protocol classes’ send and spec_version); the statements about them are about those versions as installed.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/27-chapter.md
• Runnable examples: examples/ch27/streaming.py (original toy stream manager, unchanged), examples/ch27/stream_lab.py (SSE, engine, hub, service, experiments)
• Tests: tests/test_ch27_streaming.py (toy), tests/test_ch27_stream_lab.py (50 tests: SSE formatting and every parser rule, with chunking and round trips; the engine against solo runs and the library, late joiners, single-token requests, eos, cancelling a waiting and a running request, double and late cancels, pausing, block-limited admission in order, the batch limit, thread-safe submission and cancellation, double free as an error, the step summary; the hub’s four policies including the stall of the naive one, an idempotent close, stop strings; the two response classes on a disconnect; the service’s events, refusals before the first byte, waiting limit, stop strings, cleanliness afterwards and concurrent streams; a tiny random GPT-2 on the CPU)
• Evidence: research/ch27/sources.md, research/ch27/results.json, research/ch27/raw/, research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It claims nothing about production streaming, about any server other than uvicorn on this machine, or about any client other than the two used. Chapter 28 moves from one GPU to several.