Request Scheduling and Admission Control

The same service was sent the same 279 requests in 40 seconds, at about twice the rate its engine could serve. A 16-token request took the engine about 0.3 seconds, and the arrivals brought about two seconds of work for every second of engine time. A client was happy with an answer within two seconds.

With an unbounded queue, which is what the Chapter 25 service had, 23 of the 279 requests were answered in time. The median request waited 26 seconds and the 99th percentile 57. Every request was eventually answered correctly; 256 of them were late. With a queue limited to eight, 48 were in time. With a queue that also dropped what could no longer finish by its deadline, 185 were. With an admission rule that refused, at arrival, what its own estimate said could not finish in time, 224 were, and no request that was served was late.

None of these services was faster. The engine, the model and the arrival trace were the same. What changed was which requests the service agreed to do, and the two things that follow from a refusal: it is quick (a median of 2.4 milliseconds, in this run), and it is the only part of overload the caller can act on.

The question

The Chapter 25 service has one engine, a lock, and a queue that nobody wrote: whatever is waiting for the lock is the queue, and it has no bound, no deadline and no way to say no. This chapter puts a queue there on purpose and asks what it should do.

Four questions. What does an unbounded queue do as the load approaches and passes the engine’s capacity, and does the textbook formula describe it? What is the minimum a service must do to protect its users from that: bound the queue, bound the wait, or predict the wait? Who pays when it says no, since requests are not alike? And what does the controller need to know, and what does it cost when it knows it wrongly?

Where this sits

Chapter 9 defined the latency metrics. Chapter 15 measured scheduling inside the batch, where a continuous scheduler decides at every step who runs; it reached loads up to about the engine’s capacity and defined goodput as the share of requests meeting an objective. This chapter is the layer in front: the one-at-a-time engine of Chapter 25, a queue, and what happens beyond capacity. Chapter 27 lets requests overlap and stream; Chapter 30 covers cancelling a request that is already running and retrying. The decision here is only whether and when to start a request.

Three papers frame the vocabulary, and none of their figures is reproduced here. DistServe (OSDI 2024, per the arXiv listing) defines the quantity to maximise as goodput, the rate of requests served within latency objectives on both the first token and later tokens. Sarathi-Serve (OSDI 2024) is about scheduling within the batch, the subject of Chapter 15. Mooncake, which is an arXiv preprint (the listing names no venue; the starter chapter called it peer-reviewed, which I cannot support), describes designing for overload with a prediction-based early rejection policy. The idea this chapter measures is the one in that last sentence: reject on a prediction, early.

A queue in front of one engine

Utilisation, and what an unbounded queue does with it

Requests arrive at a rate \(\lambda\) per second and each needs a service time \(S\). The utilisation is \(\rho = \lambda\,\mathbb{E}[S]\), the fraction of the engine’s time that the arrivals would fill. Below one, the queue is stable and the mean wait in a FIFO queue with Poisson arrivals is given by the Pollaczek-Khinchine formula,

These are standard queueing results; I did not read a textbook for this chapter. Instead I wrote a simulator, checked it against each formula, and used it for everything that follows. The workload is the one used in the live runs: new tokens of 4, 16 or 64 with probabilities 0.5, 0.35 and 0.15. With the service time fitted to the real engine (below) its mean is 0.146 s, so the engine’s capacity is 6.86 requests per second, and the second-moment term makes the coefficient of variation squared of the service time 1.41.

The last row is the overload fact in one line. At a load of 1.3 the longest wait is not a property of the system but of how long it has been overloaded; it is proportional to the duration of the run.

The simulator also corrects the starter. Its example, a burst of ten simultaneous requests against a queue of four and two concurrent slots, reported four accepted and six rejected. Counted correctly, two requests start at once and four wait, so six are accepted and four rejected. The starter’s loop admitted each arrival before it started anyone, so its two engines never counted. The starter’s code and tests are unchanged in the repository; the simulator has its own test for the correct count.

The policies

An admission policy answers one question at arrival: may this request join? The service has four, from least to most informed.

• None. Everything joins. This is Chapter 25.
• Bounded. At most eight requests may wait (the one in service does not count). The limit is a count.
• Bounded with a deadline. The same limit, and every request carries a deadline of two seconds from arrival. When a waiting request reaches the front, if it could not finish by its deadline even if it started now, it is dropped without being run.
• Predictive. No count. At arrival the service estimates the wait (the work in the queue plus what remains of the request in service) and rejects a request whose wait plus its own service time exceeds its deadline. Expiry at the front remains, as a second line.

All four are one pure function, shared by the simulator and the live service so that the one can be checked against the other:

def decide(policy, queued, queued_work_s, in_service_remaining_s, est_service_s, deadline_s, workers=1):
    wait = (queued_work_s + in_service_remaining_s) / workers
    if policy.max_queue is not None and queued >= policy.max_queue:
        return Rejection("queue_full", ..., retry_after_s)
    if policy.max_queued_seconds is not None and queued_work_s + est_service_s > policy.max_queued_seconds:
        return Rejection("work_limit", ..., retry_after_s)
    if policy.predictive and deadline_s is not None and wait + est_service_s > deadline_s:
        return Rejection("deadline_unreachable", ..., retry_after_s)
    return None

A rejection carries a name, the numbers behind it, and a Retry-After in whole seconds computed from the estimated wait. The status is 503, for the reason RFC 9110 gives for it: the server is temporarily unable to handle the request. The 429 of RFC 6585 is defined for a client that has sent too many requests in a given time, which is a statement about the client; global overload is a statement about the server, and the two ask for different behaviour. The service therefore has no 429, and per-client quotas, which would use it, are not built. Both overload responses are in Chapter 25’s error envelope with their own stable types, overloaded for a refusal at arrival and deadline_exceeded for a request dropped at the front.

The deadline is a new optional request field, deadline_ms. Adding an optional field to a request is the compatible change Chapter 25’s classifier calls compatible, and a test applies the classifier to the real pair of request models to confirm it. A request without one gets the policy’s default.

The live gate

The Chapter 25 service had the whole request path in one function. To insert a queue, that function is now three, prepare (parse, tokenise, budget), execute (the engine call) and finish (build the response), with the lock gone from the middle. The gate is an asyncio queue and one engine thread:

ticket = gate.submit(lambda: execute(p), est, deadline_s)   # raises RejectedError, never blocks
result = await ticket.future                                 # waits its turn, or raises Expired

The route is async def, so a thousand waiting requests cost a thousand futures and not a thousand threads. FastAPI runs plain def routes in a thread pool of limited size, and a request waiting for a thread would be waiting in a queue that no policy can see or bound; I did not measure that here, and it is one reason the gate does not use the pool for waiting.

One detail took a test to get right. A request admitted to an idle engine is in service from that moment, even if the worker task has not yet picked it up a microsecond later. If the gate counted it as waiting, a limit of eight would admit nine in a burst in the simulator and eight in the service. The gate’s view of its own state treats the head of the queue as in service when the engine is idle, and the unit tests check the same counts for a burst in the gate and in the simulator.

The estimates

The policies use a service-time estimate, and the estimate comes from a model fitted to the engine. Over 84 requests per round at 1, 2, 4, 8, 16, 32 and 64 new tokens, in two rounds, a line through the engine’s measured time had a slope of 8.4 and 8.6 ms per token and an intercept of a millisecond or so either way. The median error of the line on a request was 4.4% and 5.7%; the worst was 23% and 28%. One token took 8.7 ms, sixteen took 132 ms, sixty-four 534 ms.

Those are the speeds of the machine in its faster state. Chapter 25’s figures, at about 19 ms per token, are those of its slower one, which I infer to be another program sharing the card; I did not check what was running. During the experiments below the machine went between the two speeds, a 16-token request taking about 0.15 s or about 0.30 s, sometimes switching in the middle of a run. I did not control it, and the first design of the live experiment did not survive it, as the results explain. What it changed in the design is that the gate no longer trusts the calibration: it keeps a running ratio of the time requests actually took to the time the model predicted (an exponentially weighted average, weight 0.2) and multiplies every estimate by it. A fixed model is available behind a flag so that the cost of not doing this can be measured.

The experiment

Question. Which admission policy keeps the most requests inside a two-second objective as load passes capacity, who is refused under each, and what does the controller’s estimate cost when it is wrong?

Hypotheses (book hypotheses, to be falsified). (H1) With an unbounded queue the share served in time collapses once utilisation passes one, and degrades noticeably before it. (H2) A bound on the count of waiting requests does not bound the wait, because the wait is the count times the service time, and the service time depends on the machine and on the mix. (H3) A rule that rejects on a predicted deadline serves more requests than one that waits for the deadline to pass, and does so by preferring short requests. (H4) A controller that does not track the engine’s current speed will start requests it cannot finish in time.

Held constant. The model (GPT-2 small, the Chapter 10 cached backend), the prompt, the length mix, the two-second objective, and for each comparison the arrival trace: every policy sees the same seeded arrivals.

Load. An open-loop generator sends request \(i\) at its scheduled time whatever has happened to earlier ones, which is what independent clients do and what a closed loop of waiting clients would hide. Traces are Poisson at four nominal loads for 40 seconds, and one burst of 30 requests at once. A nominal load is relative to the calibrated capacity; because the machine’s speed changed, each run is reported at its effective load, the nominal load multiplied by the speed factor measured in that run (the median of observed engine time over the model’s prediction).

Measures. A request is good if its client sees an answer within two seconds. Requests are also counted as late (answered, but after two seconds), rejected (refused at arrival) or expired (dropped at the front). Goodput is good requests per second, and, because requests differ in size, good tokens per second.

What would falsify. H1 fails if the unbounded queue serves most requests in time at a utilisation of 1.3. H2 fails if a bound of eight gives the same share in time at two machine speeds. H3 fails if the predictive policy does not beat the deadline policy, or if its extra requests are not mostly short ones. H4 fails if a fixed-estimate gate finishes about as many requests late as a tracking one.

Results

An unbounded queue, at the calibrated speed

The first pass of the live runs ran each policy in turn. During it the machine ran at the calibrated speed for its first eight runs, which are the unbounded queue at all five traces and the bounded queue at three of them.

At a utilisation of 0.89 the unbounded queue, with nothing wrong with it and no request lost, served one in three within the objective. That is the Pollaczek-Khinchine formula and the high variance of the service times, not a bug: at 0.9 the simulator’s mean wait is about 1.6 s, its p99 latency 5.7 s, and 30% of requests miss two seconds. Above one, the behaviour is the overload row of the table above: the median wait is the time the overload has lasted. H1 stands, and its second half is the surprise: the collapse starts before capacity.

Four policies at an effective load of two

The first pass could not compare policies, because the speed changed between its runs. For the second pass every trace was run under every policy in turn, with the order rotated, all four servers resident, a speed probe before and after each run, and the gate tracking the speed. The machine was at its slower speed for most of it, which put the nominal 0.9 trace at an effective load of 2.0 for all four policies, with a speed that did not change during the runs (probes of 0.29 to 0.32 s throughout). That is the cleanest comparison in the chapter.

(279 requests each. Latencies are those of the requests that were answered.)

The bound of eight cut the damage and did not remove it: 86 answered requests were late, with a p99 of 4.6 s, because eight waiting requests at this speed is several seconds of work. The two deadline-aware policies answered nothing late. Dropping at the front and refusing at the door both do what the starter chapter said, trading rejected requests for stable latency for the admitted ones. The predictive policy served 39 more requests than the one that waited for the deadline, with fewer total refusals (55 rejected, against 61 rejected plus 33 expired), and, importantly, it refused them in 2.4 ms and not after two seconds in a queue.

H2 needs the same count limit at a second speed, and the live runs have it: the same arrival trace (279 requests) and the same bound of eight gave 233 requests in time with a p99 latency of 2.2 s when the machine ran at the calibrated speed (first pass), and 48 with a p99 of 4.6 s when it ran 2.2 times slower (second pass, above). The simulator, which can hold everything else fixed, shows the dependence across loads.

The sweep

The simulator ran each policy at ten loads, 20 seeds of 600 s each, at the calibrated speed and with every service time 2.25 times longer (the ratio the machine showed). The fraction of requests served within two seconds:

At the slower speed:

At a load of 1.0, the bound of eight serves 86% in time on the fast machine and 58% on the slow one. The deadline-aware policies hardly move: bounded-with-deadline 89% and 93%, predictive 97% and 93%. H2 stands: a count is a bound on the wait only for one speed and one mix, and would need retuning when either changes. A deadline expresses what the user cares about and did not need retuning.

Expressed as a share, the predictive policy at a load of 2 (0.83) beats the bounded deadline one (0.52) by a wide margin, and by more in the request goodput: 11.3 good requests a second against 7.2, with a capacity of 6.9 requests a second for the mix. A goodput above capacity is a warning sign, not a feat. It is possible because it serves a different mix: the cheaper requests. In good tokens per second the two are within half a per cent (118.4 and 117.8).

Who is turned away

Under every policy that refuses, the long requests pay. In the live run at an effective load of 2 (141 requests of 4 tokens, 93 of 16, 45 of 64), the share answered within two seconds by length class:

The predictive policy serves nearly every short request and 9% of the long ones. 41 of the predictive policy’s 55 rejections, and 37 of the deadline policy’s 94 refusals and expirations, were 64-token requests, which are 16% of the offered requests. The reason is in the arithmetic of the rule: a 64-token request needs about 1.2 s of the 2 s at this speed, so any queue longer than 0.8 s makes it infeasible, while a 4-token request needs 0.08 s. In the simulator at a load of 3, the predictive policy serves 97%, 65% and 2% of the three classes.

This is the consequence of optimising a count of requests that are alike under a rule that treats them as unlike, and it is a policy choice. A service that must not starve long requests needs a reserved share for them, or a longer deadline for them, or a different objective, and the choice belongs to whoever sets the objective. The measurement here is that the default behaviour of the best-scoring policy is to starve the most expensive class, and that a request count hides it.

A burst

A burst of 30 requests at once, at the slower speed (about 10 seconds of work), cannot be served in two seconds whatever the policy; about six requests of this mix fit.

Of the 30, the unbounded queue spent the next 11 seconds answering 27 requests that were already too late to matter, and the others declined them in about 50 milliseconds. A burst is also where the predictive policy gave the least: it refused 25, a few more than the simulator predicted for the same arrivals (19), and served four where the deadline policy served seven.

What the controller must know

An estimate of the service time is what the deadline-aware policies stand on. Two ways to be wrong were measured.

Stale. In the first pass the gate used the calibrated model and the machine slowed by a factor of 2.2. Of the requests completed under the two deadline-aware policies, 85 of 574 (14.8%) under bounded-with-deadline and 197 of 819 (24.1%) under predictive finished after the deadline: the service started work it could not finish in time, because it believed it was twice as fast as it was. In the second pass the gate tracked the measured speed and the same two policies completed 1 of 764 and 0 of 1,109 late. H4 stands. The runs differ in other ways (the speed was not constant, and the effective loads were not matched), so this is a comparison and not a controlled experiment, but the gap is two orders of magnitude.

Wrong about the work. A request is estimated from its max_new_tokens, and a caller who sets a stop string usually stops sooner. I sent 60 book prompts with a limit of 64 tokens and a stop string of a full stop. 56 stopped on the string and four at the limit; the median completion was 8.5 tokens (from 1 to 64), a median 13% of the maximum. If the predictive policy estimates from the maximum while the real lengths are those, it turns requests away that would have fit. In the simulator, at three loads defined by the work actually done (a true mean service time of 0.032 s, not the 0.146 s of full-length requests), the predictive policy serving on true lengths served 16,740, 23,472 and 32,939 requests within the objective in 600 s; estimating from the maximum, 16,003, 21,328 and 28,199, which is 4.4%, 9.1% and 14.4% fewer, with its rejections doubling or more (from 19 to 756 at the lowest load, and from 4,350 to 9,090 at the highest). The estimate is a bound on the work, and a bound is pessimistic when callers stop early. What a service does with that is learn the distribution of how much of the maximum a request uses and estimate with it. I did not build that.

An earlier version of this experiment held the nominal load fixed while early stops cut the real work to about a fifth, so it was not an overload at all, and it showed a difference of a fraction of a per cent. The data corrected the setup, as it should.

The simulator against the service

The sweep rests on the simulator, so it was checked against the live second pass. For each of the 20 runs, the simulator replayed the arrivals with the service times the engine had actually taken (and the model’s estimates scaled by the run’s measured speed). Across 5,680 requests the simulator and the service differed by 89 requests in the number good (1.6% of requests), 89 in the number rejected (1.6%) and 31 in the number expired (0.5%). The largest disagreements were the predictive burst (7 good and 6 rejected of 30), the bounded queue at nominal load 0.9 (17 good, 11 rejected of 279), and predictive at nominal 1.3 (12, 17 and 20 of 380), the last in a run during which the machine’s speed changed. I did not investigate the burst; my guess, a hypothesis, is that the gate’s running speed factor at the start reflected the previous trace.

The queue logic is therefore the same in both. What the simulator does not see is the machine, and the machine was the largest source of disagreement in the chapter.

What a refusal costs

A rejection in the sustained traces was answered in a median 2.3 to 2.5 ms, with a p99 of 3.5 to 4.8 ms. In the burst, when thirty requests arrive in the same instant, a median of 43 to 55 ms (p99 up to 65 ms). A request that waits two seconds in a queue to be dropped has held a connection, a client and a slot for two seconds to learn what the first would have told it at once; the predictive policy’s refusals are early for that reason, and the bounded-with-deadline policy’s expirations (33 here) are the late ones. The Retry-After values the service returned, computed from its estimate of the wait, ranged from 1 to 7 s. Whether clients obey them, and what clients that do not obey do to the load, is the retry problem of Chapter 30.

What the numbers do not show

• The machine changed speed during the experiments, and I did not control it. The first pass was not a fair comparison and is used only where the speed was constant. The second pass interleaves policies, probes the speed and reports effective load, which controls for the speed between runs and not within a run (four runs had probes before and after that differed by more than a factor of 1.5).
• One model, one engine. GPT-2 small, one request at a time. The simulator can run several engines; I tested that only for conservation and parallelism, not against theory or a live service.
• One arrival model and one mix. Poisson arrivals, one burst, three lengths. Real traffic is correlated and its lengths are not three numbers.
• Per-run counts, not confidence intervals. The live runs are single 40-second traces of 152 to 579 requests; the simulator’s sweeps are means of 20 seeds. The ordering of the policies is large compared with the disagreement between simulator and service, but I did not test small differences.
• No cancellation of work in progress. A request that passes its deadline while being served is finished. The deadline is checked before a request starts, not during it. That is Chapter 30.
• A fixed objective. Two seconds from arrival to the end of the answer is one choice. A first-token objective, which streaming makes possible in Chapter 27, would rank policies differently.
• No memory. Admission here is by estimated time. The KV cache memory a request needs (Chapters 10, 14 and 21) is the other resource, and it is not an input to the rules above.

Inference Lab increment

examples/ch26/scheduler_lab.py adds:

• Policy and decide: the admission rule as one pure function that names the rule and the numbers, and cannot_finish for the expiry test.
• ServiceModel and fit_service_model: service time from a line fitted to measurements.
• simulate: a discrete-event simulator of a FIFO queue in front of one or more engines, with the true service time allowed to differ from the estimate.
• Gate and build_gated_app: the live version, with the speed-tracking estimate, deadline_ms on the request, 503 responses with Retry-After, and /stats.
• run_trace_http: the open-loop generator, and GatedServer, which starts the service in a process.
• Closed forms for M/M/1, M/M/1/K and Pollaczek-Khinchine, against which the simulator is tested.
• Commands theory, starter, calibrate, real, compare, sweep, estimation and serve.

To run the gated service with the predictive policy:

python examples/ch26/scheduler_lab.py serve --policy predictive --device cuda --port 8000

The rule this adds: a service decides at the door, with a number it can explain, whether to start a request, and it checks that number against what the engine is doing now. The queue’s bound is a deadline and not a count; the estimate is tracked and not assumed; and the report separates good, late, rejected and expired, because a policy that scores well on one of them can be starving a class that another shows.

What you should now be able to do

• Compute the utilisation and the expected wait of a one-engine queue, and say what happens to the wait as utilisation passes one.
• Explain why a bound on queue length is not a bound on latency, and write the rule that is.
• Report an overload experiment with good, late, rejected and expired counted separately, and by request size.
• Say which status code a refusal should carry and why 429 and 503 differ.
• Check a simulator against a closed form and against a live service before trusting it for a sweep.

Key terms

• Utilisation (\(\rho\)): arrival rate times mean service time; the share of the engine’s time the arrivals would fill.
• Admission control: the decision, at arrival, whether a request may join the queue.
• Bounded queue: a queue with a limit, here on the count of waiting requests.
• Deadline: the latest time an answer is still worth having; a request that cannot meet it should not be started.
• Predictive admission: rejecting on an estimate of wait plus service.
• Goodput: requests (or tokens) per second that meet the objective.
• Open-loop load: arrivals that do not wait for earlier answers.
• Effective load: the nominal load times the speed factor measured during the run.
• Little’s law: the mean number in a system equals the arrival rate times the mean time in it.

Reader exercises

1. Add a reserved share to the predictive policy so that 64-token requests are served at least 25% of the time they are offered, and report what it costs the others.
2. Replace the estimate from max_new_tokens with one learned from the lengths callers actually use, and repeat the early-stop experiment.
3. Add a second engine (two backends) and test the multi-server simulator against the live service, or against the Erlang formulas.
4. Add per-client token buckets and a 429, and show that they and the 503 of global overload can be told apart by a client.
5. Make the gate cancel a request that passes its deadline in the middle of generation, and measure the work saved. (Chapter 30 builds the general mechanism.)
6. Admit by memory as well as time: add the KV cache bytes a request needs to the rule.
7. Repeat the live runs on an idle machine and report whether the speed still changes; explain how you checked.
8. Send the same trace as a burst every 10 seconds instead of Poisson arrivals, and find the load at which each policy first fails the objective.

References

• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Zhong et al.; the arXiv listing’s comments field says OSDI 2024; goodput as the objective; the authors’ figures are not reproduced here; read on 22 October 2026.
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Agrawal et al., OSDI 2024; scheduling within the batch, as in Chapter 15; read on 22 October 2026.
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Qin et al.; an arXiv preprint (the listing names no venue); prediction-based early rejection under overload, by the authors’ account; not reproduced; read on 22 October 2026.
• RFC 9110: HTTP Semantics — Internet Standard; 503 Service Unavailable and Retry-After.
• RFC 6585: Additional HTTP Status Codes — Proposed Standard; 429 Too Many Requests as a response to a client that has sent too many requests in a given time; read on 22 October 2026.

Evidence in this chapter

Implementation status

• Manuscript: content/books/inference/26-chapter.md
• Runnable examples: examples/ch26/admission.py (original toy simulator, unchanged), examples/ch26/scheduler_lab.py (policy, simulator, gate, load generator, experiments); examples/ch25/service_lab.py was split into prepare, execute and finish so that a gate can sit between them (its Chapter 25 tests pass unchanged)
• Tests: tests/test_ch26_admission.py (toy), tests/test_ch26_scheduler_lab.py (40 tests: the model fit and mix arithmetic; every admission rule with its numbers; the simulator on bursts, FIFO order, two engines, conservation, unbounded growth, deadline expiry, predictive rejection, true against estimated service time and determinism; the traffic generators; the closed forms and the simulator against M/M/1 and M/M/1/K; the live gate on a burst, on expiry, on predictive refusal and on speed tracking; the gated service on one request, on overload with Retry-After, on a request deadline, and the compatibility of deadline_ms; CPU only)
• Evidence: research/ch26/sources.md, research/ch26/results.json, research/ch26/service_model.json, research/ch26/raw/ (per-request rows of the two live passes, compressed), research/evidence-register.md
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft with draft = false. It claims nothing about production scheduling, about any system other than this one service on this one machine, or about the papers’ figures. Chapter 27 lets requests overlap and return a token at a time.