← Distillation From First Principles

Black-Box and API-Only Distillation

Restricted teacher access changes the signal, objective, query cost, and legal obligations. This chapter formalises top-1, top-k, sampled, and full-probability access, derives truncation and sampling bias, and reports a bounded local E1 extension.

In this chapter’s experiment, the training loop asked the teacher for 180,000 answers per arm and seed, and the teacher computed 1,200 of them. Both numbers are correct. They count different things: logical requests and actual evaluations after caching. A report that gave only one of them would misstate the teacher’s workload by a factor of 150. Restricted access to a teacher is like that throughout: what the student receives, what the teacher computes, and what the provider permits are separate quantities, and each has to be recorded.

Chapter 06 assumed that the teacher’s output signal was available. It compared probability matching, raw-logit matching, standardised logits, and decoupled target/non-target terms. That comparison quietly assumed a full classifier output: logits or a complete probability vector over the class space.

Many teachers do not expose that. An API may return one class, a short list of top-k alternatives, a sampled response, or a generated sequence. The training signal is then a restricted view of the teacher, not a smaller copy of the full distribution. The central question is:

What can a student learn when a teacher provides only restricted outputs, and how does that access change learning quality, query cost, uncertainty, and legal obligations?

The short answer is that restricted access can still support transfer, but it changes the objective and the evidence. A top-1 label is not a probability vector. A top-k list is not the full distribution. A sampled class is a draw, not a target. An API call is not permission to train a competing model. Each of these distinctions has a mathematical consequence and an engineering cost.

What “black-box” means operationally

Black-box is not a single interface. It is a statement about what the training process can observe. The useful description is operational:

Access What the teacher returns What is unavailable
Top-1 One predicted class or answer Full distribution, logits, confidence, gradients
Top-k labels The k highest-ranked classes Tail classes, calibrated probabilities
Top-k probabilities Returned classes and scores or probabilities Full normalisation, omitted classes, logits
Sampled class One draw from the teacher distribution The distribution itself, exact KL
Generated text A sequence of tokens or a structured response Token-level probabilities, hidden states
Full probabilities A complete distribution over the class space Usually logits, gradients, activations
Full logits Pre-softmax scores Weights, gradients, internal representations

The last two rows are references, not black-box capabilities. A provider that returns full probabilities is exposing more than one that returns top-1, but it still may not expose logits or weights. The word “API” does not tell us which row applies. The access record must.

Chapter 03 introduced the access-regime concept. Chapter 06 introduced response and logit distillation. This chapter applies both to restricted interfaces. It introduces no new canonical concept identifier; the relevant concepts remain KD-003 (access regime), KD-010 (response distillation), and KD-011 (logit distillation).

A deterministic interface returns the same output for the same input and parameters. A stochastic interface draws from a distribution. Top-1 and top-k are usually deterministic; sampled classes and sampled text are stochastic. Mixing the two without saying so makes the training target and the query count ambiguous.

Two interfaces can both be called black-box and still expose different information. A top-k list of unnormalised scores is not a probability vector; it needs a normalisation convention before it can be used as a target. A provider may expose log-probabilities for the tokens it sampled, which is not the same as the full next-token distribution. A generated sequence may be useful for supervised training without revealing any token-level probabilities. The access record should name the exact return type, not just the provider or the endpoint.

Top-k truncation is not the full distribution

Let the teacher’s full distribution over \(K\) classes be

\[ p=(p_1,\ldots,p_K). \]
Let \(S\) be the set of returned top-\(k\) classes, and define the retained mass
\[ Z=\sum_{i\in S}p_i. \]
If the interface renormalises the returned probabilities,
\[ q_i= \begin{cases} p_i/Z, & i\in S,\\ 0, & i\notin S, \end{cases} \]
then \(q\) is the teacher distribution conditioned on the returned subset. It is not the original distribution unless \(Z=1\). For a returned class,
\[ q_i-p_i = p_i\left(\frac1Z-1\right) = p_i\frac{1-Z}{Z} \ge 0. \]
Renormalisation increases every returned probability and assigns zero to every omitted class. The missing tail mass \(1-Z\) is discarded. The exact teacher-to-student KL is generally unavailable, because the student's probability on an omitted class is compared with a teacher probability that was never observed.

A four-class example makes the loss visible. Take

\[ p=(0.60,\ 0.25,\ 0.10,\ 0.05). \]
| Returned set | Retained mass \(Z\) | Discarded mass | Renormalised target | | --- | ---: | ---: | --- | | Top-1 | 0.60 | 0.40 | (1.0000, 0, 0, 0) | | Top-2 | 0.85 | 0.15 | (0.7059, 0.2941, 0, 0) | | Top-3 | 0.95 | 0.05 | (0.6316, 0.2632, 0.1053, 0) | | Top-4 | 1.00 | 0.00 | (0.6000, 0.2500, 0.1000, 0.0500) |

The top-2 target says that, conditional on the answer being in the top two, the first class is about 70.6% likely. It does not say that the first class is 70.6% likely in the full distribution. The difference is the discarded tail.

Renormalisation inflates the returned probabilities

Figure 1. The four renormalised targets from the table, drawn as bars. Narrower returned sets inflate the leading class (0.60 becomes 1.0000, 0.7059, 0.6316) while the discarded tail grows. Chapter arithmetic, not measured data.

An OTHER category can preserve the total omitted mass:

\[ q_{\mathrm{other}}=1-Z. \]
That is better than silently dropping the tail, but it still does not recover the individual omitted-class probabilities. The `OTHER` mass could hide one large class or many small ones. If the student has no `OTHER` output, the training objective must define how that mass is treated; it cannot be recovered by assumption.

The local E1 extension used the top-2 renormalised target in the table. It did not use an OTHER category. That is a declared truncation treatment, not a reconstruction of the full distribution.

Top-k labels without probabilities are weaker still. They identify a set, not a distribution. A learner can place a uniform target over the returned set, but that is a book-level choice rather than the teacher’s distribution. The same caution applies to rankings: an ordering does not determine the probabilities.

Sampling estimates a distribution at a cost

Suppose the teacher draws independent classes

\[ Y_j\sim p_T(\cdot\mid x),\qquad j=1,\ldots,n. \]
The empirical frequency estimator is
\[ \hat p_i=\frac1n\sum_{j=1}^n\mathbf 1[Y_j=i]. \]
Under a fixed teacher distribution, independent draws, identical sampling conditions, and a stable teacher version,
\[ \mathbb E[\hat p_i]=p_{T,i}, \qquad \operatorname{Var}(\hat p_i)=\frac{p_{T,i}(1-p_{T,i})}{n}. \]
For \(i\ne k\), the covariance is \(-p_{T,i}p_{T,k}/n\). More samples reduce variance, but each sample is another teacher query. The estimator is unbiased for the sampling distribution; it is not an exact reconstruction of the distribution at finite \(n\).

For a fixed student distribution \(p_S\), the Monte Carlo estimate of teacher-to-student cross-entropy is

\[ \hat L_{\mathrm{CE}} =-\frac1n\sum_{j=1}^n\log p_S(Y_j\mid x). \]
Its expectation is the full cross-entropy \(-\sum_i p_{T,i}\log p_{S,i}\). Its variance, for known \(p_T\), is
\[ \operatorname{Var}(\hat L_{\mathrm{CE}}) = \frac1n\left[ \sum_i p_{T,i}\left(\log p_{S,i}\right)^2 - \left(\sum_i p_{T,i}\log p_{S,i}\right)^2 \right]. \]
This is an estimate of cross-entropy, not KL. KL requires the teacher entropy term, which is not observed directly. Estimating a class distribution from independent draws is also not the same as estimating a sequence-level loss from correlated autoregressive tokens. The independent-class variance formula should not be extended to token sequences without a new argument.

A seeded illustration uses the same four-class distribution and independent draws:

Samples \(n\) Empirical frequencies Maximum absolute error Standard error, class 1
10 (0.80, 0.20, 0.00, 0.00) 0.20 0.155
100 (0.61, 0.25, 0.11, 0.03) 0.02 0.049
1,000 (0.596, 0.247, 0.110, 0.047) 0.01 0.016

The ten-sample estimate misses two classes entirely. The hundred-sample estimate is close on this draw, but that is one draw, not a guarantee. The thousand-sample estimate is closer still, at a hundredfold query cost relative to ten samples. The trade-off is information versus queries, not a free improvement.

A fixed query budget must also choose between coverage and repetition. Spending all queries on a few inputs improves the per-input frequency estimate but leaves most training examples unseen. Spending one query per input covers the training set but leaves high variance per input. The local experiment’s main sampled arm used one sample per input to match the coverage of the other teacher arms; it did not estimate a per-input distribution. That is a deliberate choice, not a claim that one sample is sufficient.

Query economics

A query count is ambiguous unless the unit is stated. The experiment records:

  • Logical requests. How many times the training loop asked for a teacher response.
  • Unique inputs. How many distinct examples were queried.
  • Actual evaluations. How many times the teacher computed a response after cache misses.
  • Cache hits. How many logical requests were served from a previously computed response.
  • Repeated samples. How many draws were made for the same input and sampling configuration.
  • Evaluation queries. Teacher queries used for held-out agreement, counted separately from training.
  • Latency and price. The time and monetary cost of the actual evaluations.

These are different quantities. A run with 1,000 logical requests, 800 cache hits, and 200 evaluations has a different teacher workload from one with 1,000 evaluations. A stochastic sample that is cached is not a new independent draw. A local zero-cost mock is not a benchmark of an external API price.

A simple cost model is

\[ C=N_q c_q, \]
where \(N_q\) is the number of billable queries and \(c_q\) is the assumed price per query. Token-priced services need a different calculation: input tokens, output tokens, cached tokens, and per-request minimums may all contribute. The chapter does not invent current provider prices. The local experiment reports operation counts, not a bill.

Query efficiency should be judged relative to learning quality. A method that uses half the queries but loses ten accuracy points is not automatically cheaper for the intended application. A method that uses more queries and improves nothing is not automatically better because it uses more information.

Caching also changes the meaning of a sample. Reusing a deterministic top-1 response is a valid cache hit. Reusing a stochastic draw is one fixed sample, not an independent observation. If the training loop resamples every epoch, the evaluation count grows with the number of epochs; if it caches, the target is fixed. Both designs can be legitimate, but they answer different experimental questions and must be reported separately.

A minimal teacher-client interface

The implementation is deliberately small. It has four pieces:

  • TeacherRequest records the input hash, access mode, temperature, top-k, sampling seed, and phase.
  • TeacherResponse records what was actually delivered: labels and/or probabilities, retained mass, cache status, teacher version, and rights record. It has no logits field, so a restricted response cannot accidentally expose full scores.
  • QueryLedger separates logical requests, evaluations, cache hits, rejected requests, and unique inputs.
  • TeacherClient enforces the access mode, budget, cache key, and rights decision. LocalMockTeacher keeps the full teacher inside the mock.

The cache key includes the teacher identity and version, input fingerprint, access mode, temperature, top-k, and sampling seed. Different teacher versions, temperatures, or sampling seeds therefore do not collide. A deterministic top-1 request may be served from cache. A sampled request is cached only for the same sampling seed and input; it is not presented as an independent draw.

The budget is checked before the teacher is called. If a request would exceed the authorised evaluations, the client raises an error and the teacher is not evaluated. This is the difference between a budget and a post-hoc count.

    flowchart LR
  Q[Teacher request] --> B{budget check}
  B -->|exceeds| E[raise error: teacher not called]
  B -->|within| C{cache lookup}
  C -->|hit| H[serve cached response]
  C -->|miss| T[evaluate teacher]
  T --> L[append to QueryLedger]
  H --> L
  

Figure 2. The client enforces the budget before any teacher compute, then separates logical requests from actual evaluations in the ledger.

A compact use of the interface looks like this:

from pathlib import Path

from examples.common.rights import check_action, load_manifest
from examples.common.teacher_client import LocalMockTeacher, TeacherClient

records = load_manifest(Path("experiments/E1-blackbox/rights.json"))
decision = check_action(records, "RM-102", "train_and_evaluate_e1_blackbox")
client = TeacherClient(
    mock_teacher,
    access_mode="topk",
    max_evaluations=2,
    phase="train",
    rights_decision=decision,
    top_k=2,
)
first = client.request(inputs)
second = client.request(inputs)
print(first.labels, first.probabilities, first.retained_mass)
print(second.cache_hit, client.ledger.summary())

The first request evaluates the teacher and returns a top-2 response. The second is a cache hit with the same access mode and input. The ledger reports one evaluation, one cache hit, and two logical requests. The response never contains the teacher’s full logits.

The bounded E1 black-box extension

The local extension asks a narrower question than a provider benchmark: when the teacher interface changes, what happens to a student trained on the same task, data, architecture, optimiser, and query count?

The design used the E1 teacher architecture and data generator, retrained the teacher locally because the original checkpoints are not stored, and froze it before student training. The retrained teacher reproduced the E1 teacher’s test metrics to four decimal places. The student was the same 228-parameter E1 student. All arms used ground-truth validation cross-entropy for early stopping, which differs from E1’s arm-specific validation objectives. The test set was used once for evaluation.

Arm Access Supervision Training teacher queries
A-hard None Ground-truth hard labels 0
B-top1 Top-1 Teacher top-1 pseudo-label 1,200
C-top2 Top-2 Renormalised top-2 probabilities 1,200
D-sample One sampled class One sampled teacher class 1,200
E-full Full probabilities Full teacher distribution 1,200

The query budget was 1,200 unique training inputs per teacher arm and seed. All student runs reached the 150-epoch cap, so each teacher arm made 180,000 logical requests, 1,200 actual evaluations, and 178,800 cache hits per seed. An additional 1,000 top-1 teacher queries per seed were used for agreement evaluation and counted separately.

The measured results were:

Arm Accuracy (%) CE ECE Agreement (%) Training s
A-hard 66.87 ± 3.62 0.8733 ± 0.0663 0.1461 ± 0.0189 66.97 ± 3.63 1.93 ± 0.03
B-top1 66.87 ± 3.62 0.8733 ± 0.0663 0.1461 ± 0.0189 66.97 ± 3.63 7.21 ± 0.17
C-top2 63.07 ± 2.62 0.8839 ± 0.0514 0.1270 ± 0.0232 63.17 ± 2.56 13.93 ± 0.13
D-sample 62.80 ± 1.15 0.8966 ± 0.0641 0.1276 ± 0.0164 62.90 ± 1.06 7.15 ± 0.10
E-full 64.30 ± 3.42 0.8877 ± 0.0562 0.1283 ± 0.0155 64.40 ± 3.35 14.14 ± 0.07

The baseline and top-1 arm are identical. The teacher is almost perfect on this task, and its top-1 training labels are consistent with the ground-truth labels; the exact label agreement on the training set was not separately recorded. That is a property of this task, not a general result. It shows that top-1 access can be sufficient when the teacher is both accurate and aligned with the task labels. It does not show that top-1 access is always equivalent to ground truth.

The top-2 and sampled arms were below the baseline, and the full-probability reference was also below the baseline. The full arm is not the E1 mixed-KL arm: it uses soft cross-entropy on full probabilities at temperature 1, with ground-truth validation selection. It is a reference within this protocol, not a strict continuation of E1. The experiment is a negative result for C, D, and E relative to the baseline; B-top1 is indistinguishable from the baseline. It does not establish that full probabilities are generally worse, or that top-2 or sampling are generally inferior.

The seed spread is large relative to the differences. The experiment reports descriptive means and standard deviations, not significance tests. The equal query counts across teacher arms also mean that the comparison is about information quality, not query efficiency: a top-1 label and a full distribution cost the same number of queries here, but they do not carry the same information.

The local operation counts are not API prices. A hosted service may charge per token, per request, or per unit of compute, and may cache or rate-limit differently. The experiment’s 1,200 evaluations per arm and seed are a local accounting unit.

Rights are part of the interface

Black-box access changes what is technically possible. It does not change the permission question. A workflow should distinguish:

  1. Can call. The account, endpoint, model version, and access mode are available.
  2. Can store. The terms permit retaining the response, prompts, and logs for the required period.
  3. Can train. The terms permit using the outputs as training data for the intended student.
  4. Can train a competing model. The terms permit the specific competing-model use, if that is what the student is.
  5. Can redistribute. The resulting student, adapters, or outputs may be distributed under the applicable terms.
  6. Can use the data. Prompts and outputs containing personal, confidential, or third-party data have a lawful basis and retention rule.

These are separate questions. The OpenAI Services Agreement restricts using Output to develop models that compete with OpenAI, subject to a narrow permitted exception (L-016; version effective 1 January 2026, retrieved 12 November 2026). The Anthropic Commercial Terms and the Gemini API Additional Terms contain competing-model restrictions in the recorded versions (L-017, L-019). Whether a returned probability vector is “Output” under a particular clause is an interpretive question that this book does not settle. The safe engineering response is to record the exact interface and the intended action, and to block when the answer is unknown.

The local extension used RM-102, a separate action-specific record for train_and_evaluate_e1_blackbox. It used only self-created data and models, no external outputs, and no distribution. It did not broaden RM-101, which remains limited to train_and_evaluate_e1. A local mock is not evidence of a provider’s terms or performance.

A practical decision table is:

Question If the answer is unknown
Is this endpoint permitted for training? Block
May the response be stored? Do not store
May the student be distributed? Do not distribute
Is the use a competing model? Escalate for specialist review
Does the prompt contain personal data? Remove or obtain a lawful basis
Is the teacher version pinned? Record it or do not rely on reproducibility

Chapters 36–39 develop the legal framework. This chapter’s rule is narrower: technical access is not permission, and a black-box interface does not make the permission question disappear.

Choosing a method under restricted access

If the teacher exposes only top-1 labels, the simplest objective is cross-entropy on the pseudo-label. It preserves the teacher’s chosen class and discards confidence and tail structure. It can be a strong baseline when the teacher is accurate and the task labels are aligned, as the local E1 extension showed. It is not a probability-matching method.

If the teacher exposes top-k probabilities, renormalise the returned subset only if that is the intended target. State the retained mass and the discarded tail. Use an OTHER category when the interface and student output space support it. Do not describe the result as full KL.

If the teacher exposes samples, treat the sample as a draw. Estimate frequencies or a Monte Carlo loss, report the number of samples per input, and do not cache a sample and call it an independent draw. If the samples are token sequences, do not apply the independent-class variance formula.

If full probabilities are available, use the Chapter 06 objectives. If full logits are available, the logit objectives become possible; if only probabilities are available, raw-logit matching is not. The access record determines the method, not the other way round.

The method-selection rule is therefore:

  1. Record the access regime.
  2. Record the query budget and cache policy.
  3. Choose the objective that the available signal supports.
  4. Evaluate correctness, agreement, calibration, and cost separately.
  5. Record the rights decision before training.
  6. Report negative and null results.

What the interface changes

The local E1 extension is a small piece of evidence, not a provider benchmark. Its main lesson is that the interface changes the signal, and the signal changes what the student can learn. A top-1 label and a full distribution can cost the same number of queries and still provide different information. The experiment also shows the value of a query ledger: the training loop made 180,000 logical requests per teacher arm and seed, but only 1,200 actual evaluations after caching. Without that distinction, the query cost would be misreported by a factor of 150.

Limits of this chapter’s evidence

  • Set-up: a local mock teacher, retrained from the E1 architecture and data generator because the original checkpoints are not stored, on one synthetic task. No provider interface, price, latency, or term was tested.
  • Measurements: three seeds; the ± values are descriptive standard deviations, and the seed spread is large relative to the differences between arms. Operation counts are a local accounting unit, not a bill.
  • Implementation: every arm used ground-truth validation cross-entropy for early stopping, unlike E1. The full-probability arm used soft cross-entropy at temperature 1, not E1’s mixed KL. The sampled arm drew one class per input. The exact agreement between the teacher’s top-1 labels and the training labels was not recorded.
  • Scope: top-1 matching the baseline is a property of this near-perfect, task-aligned teacher. The results do not show that full probabilities, top-2 targets, or sampling are generally worse. The truncation and sampling examples are illustrations. Whether a probability vector counts as “Output” under a provider’s terms remains unsettled.

What comes next

This chapter treated the teacher’s output interface as the constraint. Chapter 08 moves to a different signal: intermediate representations. Features may be more informative than outputs, but they are also less available, harder to align, and often subject to stricter access and privacy constraints. The same discipline applies: record what is accessible, derive the objective, measure the result, and separate technical access from permission.

References

Hinton, G., Vinyals, O. and Dean, J. (2015). Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop. arXiv:1503.02531. https://arxiv.org/abs/1503.02531

Menon, A. K., Rawat, A. S., Reddi, S., Kim, S. and Kumar, S. (2021). A Statistical Perspective on Distillation. Proceedings of the 38th International Conference on Machine Learning, PMLR 139:7632–7642. https://proceedings.mlr.press/v139/menon21a.html

Wang, D., Li, Y., Wang, L. and Gong, B. (2020). Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation From a Blackbox Model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1498–1507. https://openaccess.thecvf.com/content_CVPR_2020/html/Wang_Neural_Networks_Are_More_Productive_Teachers_Than_Human_Raters_Active_CVPR_2020_paper.html

Wang, Z. (2021). Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model. Proceedings of the 38th International Conference on Machine Learning, PMLR 139:10675–10685. https://proceedings.mlr.press/v139/wang21a.html

Anthropic Commercial Terms of Service. Effective 17 June 2025. https://www.anthropic.com/legal/commercial-terms (retrieved 12 November 2026).

Gemini API Additional Terms of Service. Effective 23 March 2026. https://ai.google.dev/gemini-api/terms (retrieved 12 November 2026).

OpenAI Services Agreement. Updated 1 December 2025; effective 1 January 2026. https://openai.com/policies/services-agreement/ (retrieved 12 November 2026).