Key takeaways
- Tokens per second per user is how fast one response streams. It decides how the product feels.
- Time to first token (TTFT) is the wait before anything appears. It matters most for chat and voice.
- Peak throughput is the total output of the server under load. It decides cost per token.
- A model can stream 228 tok/s to one user and 18,468 tok/s in total at 256 users: both numbers are true and they answer different questions.
A single "tokens per second" figure hides more than it shows. The same model on the same GPU can honestly be described as doing 228 tokens per second or 18,468 tokens per second. Both are measurements of gemma-4-E4B-it on one NVIDIA B300 with vLLM. The first is what one user sees; the second is what the server produces for 256 users at once. Benchmarks that do not say which one they mean cannot be compared.
Tokens per second per user
This is the decode speed one person experiences: how quickly text appears once it has started. We report the median at concurrency 1, on a chat workload of 1,000 input tokens and 1,000 output tokens. For reading speed, anything above roughly 30 tok/s is faster than people read, so differences matter mainly for long outputs, code generation and agents that chain many calls.
Time to first token
TTFT is the delay between sending a request and receiving the first token. It covers queueing and processing the prompt. In our B300 batch it ranged from 19 ms to 322 ms at concurrency 1. TTFT grows with prompt length and with load, so a figure measured for one idle user is a best case. For results measured against a hosted endpoint, TTFT also includes the network round trip, and we say so on those rows.
Peak throughput
Peak throughput is the highest total output rate the server sustains across a concurrency sweep, with every request succeeding. We sweep the chat workload from 1 to 256 concurrent users and report the best clean level. This is the number that sets the economics: a GPU rented by the hour costs the same at 1 user or 256, so cost per million tokens is the hourly rate divided by the tokens produced in an hour at peak.
Three workloads, three behaviours
- Chat (1,000 in, 1,000 out): balanced. This is the headline scenario on the leaderboard.
- Prefill-heavy (4,000 in, 256 out): long prompts and short answers, as in retrieval and classification. Prompt processing dominates.
- Decode-heavy (128 in, 2,000 out): short prompts and long answers, as in drafting and code generation. Generation speed dominates.
A model and engine that lead on chat do not always lead on the other two. If your traffic is mostly long prompts or mostly long outputs, look at the matching scenario.
Questions to ask of any benchmark
- At what concurrency was each number measured?
- What were the input and output lengths?
- Which engine and version, on how many GPUs of which model?
- Were failed requests counted, and what happened at the levels that failed?
- Is energy measured or estimated, and at the GPU or at the wall?
Our answers to each of these are in the methodology.
Frequently asked questions
- What is time to first token (TTFT)?
- The delay between sending a request and receiving the first generated token. It includes queueing and prompt processing, and grows with prompt length and server load.
- What is the difference between tokens per second per user and peak throughput?
- Tokens per second per user is the streaming speed one person experiences. Peak throughput is the total tokens per second a server produces across all concurrent users. The first decides how the product feels; the second decides cost per token.
- Which metric determines inference cost?
- Peak throughput. A GPU costs the same per hour at any load, so cost per million tokens equals the hourly GPU cost divided by the millions of tokens produced per hour at peak.
Measured by ModelPerf.ai on our own hardware. How each number is produced is in the methodology; current results are on the leaderboard.
All analysis