Economics5 min read

Cost per million tokens: the formula and what moves it

How to calculate self-hosted inference cost per million tokens from GPU price and measured throughput, with a worked example and the common mistakes.

Key takeaways

  • Cost per million output tokens = hourly GPU cost ÷ (peak tokens per second × 3,600) × 1,000,000.
  • Throughput at peak concurrency is the input that matters, not single-user speed.
  • On the same GPU, a 47% throughput difference between engines is a 32% difference in cost per token.
  • Real cost is higher than the formula's result unless the GPU is busy all the time; utilization is the usual gap.

If you run a model on your own or rented GPUs, the cost of a token is set by two numbers: what the GPU costs per hour, and how many tokens it produces in that hour. The formula is short, and most disagreements about inference cost come from which throughput figure goes into it.

The formula

Peak output tokens per second is the server's total output across all concurrent users at its best clean load level. Multiply by 3,600 for tokens per hour, divide the hourly cost by that, and scale to a million.

A worked example

We measured Qwen3.8-27B on one NVIDIA B300 at a peak of 5,947 output tokens per second with vLLM. Take an illustrative rental price of $6 per GPU-hour (an example for the arithmetic, not a quote). The GPU produces 5,947 × 3,600 = 21.4 million tokens per hour, so each million costs $6 ÷ 21.4 = $0.28.

On SGLang the same model peaked at 4,052 tokens per second. At the same example price that is $0.41 per million. Nothing changed but the engine, and the cost per token differs by 32%.

Common mistakes

  • Using single-user speed. Qwen3.8-27B streams 91.6 tok/s to one user. Putting that into the formula overstates cost by a factor of 65, because the GPU can serve many users at once.
  • Assuming the GPU is always full. The formula gives cost at peak load. At 50% average utilization, real cost per token is double.
  • Ignoring the workload. Long prompts with short answers produce fewer output tokens per second than chat. Use the scenario that matches your traffic.
  • Comparing different GPU counts. A model on four GPUs costs four times as much per hour. Compare cost per token, or throughput per GPU, never raw throughput.
  • Leaving out electricity when you own the hardware. For owned GPUs, energy per million tokens is a recurring cost; for rented GPUs it is already in the hourly price.

From cost per token to a business case

Cost per token tells you the margin on each token sold or the saving against an API price. Whether a deployment pays for itself also depends on hardware or reservation cost, utilization over time, power and staffing. The ROI calculator takes a configuration we measured, adds your prices and costs, and returns the break-even date and return, with a PDF you can share.

For a model or hardware we have not published, you can request a custom benchmark on your own workload.

Frequently asked questions

How do you calculate cost per million tokens for self-hosted inference?
Divide the hourly GPU cost by the number of output tokens produced per hour at peak throughput (peak tokens per second × 3,600), then multiply by one million.
Why not use single-user tokens per second?
A GPU serves many requests at once. Single-user speed measures one stream; total output at peak concurrency is typically tens of times higher, and that total is what the hourly GPU cost is spread over.
How does utilization affect cost per token?
The formula gives cost at full load. Real cost scales inversely with utilization: at 50% average utilization the cost per token doubles.

Measured by ModelPerf.ai on our own hardware. How each number is produced is in the methodology; current results are on the leaderboard.

All analysis