Cost per Million Tokens

Chapter 36 defined metrics. This chapter converts them to cost.

Cost components

Cost includes hardware amortization, power, cooling, networking, storage, and operational overhead. API billing adds provider margin and abstraction.

Utilization

Utilization is critical. Low utilization raises cost per token. The estimator shows sensitivity to utilization.

Forecasts versus measurements

Estimates are forecasts based on assumptions. Measured spend requires actual billing and usage data.

Research context

Separate API list price from measured self-hosted cost. Build cost from token mix, cache hits, batch efficiency, utilisation, hardware time, energy, and rejected or aborted work. Do not compare providers without model, quality, region, contract, and input/output assumptions. Pope et al. model inference cost and latency for large models, but their measurements are for specific models and TPU v4 configurations.

Experiment

The estimator computed $11.11 per million tokens for 1M tokens at 100 tok/s, 50% utilization, $2/hour. Sensitivity varied utilization. Tests verify calculation, invalid inputs, and serialisation.

Limitations

• Synthetic assumptions.
• No real pricing.
• No measured spend.
• No egress.

Key terms

• Amortization: spreading capital cost.
• Utilization: fraction of capacity used.
• Cost per million tokens: normalized cost.
• Forecast: predicted spend.
• Measured spend: actual billing.

Reader exercises

1. Add GPU pricing.
2. Vary utilization and observe cost.
3. Explain difference between forecast and measured.
4. Design a cost dashboard.

References

• Efficiently Scaling Transformer Inference - research paper.
• OpenAI API pricing - official provider documentation.
• Anthropic API pricing - official provider documentation.

Implementation status

• Manuscript: content/books/inference/37-chapter.md
• Runnable example: examples/ch37/
• Tests: tests/test_ch37_cost.py
• Evidence: research/ch37/sources.md and research/ch37/results.json
• Verification command: python -m unittest discover -s tests -v

The chapter is a draft. It does not claim real cost. The next chapter will examine GPU utilization.