Methodology

Benchmark methodology

The same method is used for every run, and each result states the engine version, hardware and workload it was measured with.

Metrics

What we measure

Speed

Output tokens per second for a single user (concurrency 1) on the chat workload.

Latency

Time to first token at concurrency 1, in milliseconds.

Throughput

Highest total output tokens per second across the chat sweep (1 to 256 users) with every request succeeding.

Energy

GPU power sampled at 10 Hz during runs, reported as watt-hours per million tokens.

Cost

Dollars per million output tokens from measured throughput and GPU rental rates.

Cold start

Time for the engine to load the model and begin serving requests.

Protocol

Testing protocol

Benchmarks run on NVIDIA (CUDA) and AMD (ROCm) GPU hosts under controlled conditions. We use vLLM and SGLang with their default configurations, with no custom tuning or optimization. This represents what most users will see in production. Published results today are on NVIDIA B300.

Inference engines

  • vLLM: latest stable release
  • SGLang: latest stable release
  • Standard configurations with no custom optimization
  • Exact versions published with each result

Workload scenarios

  • Chat (1k in / 1k out): balanced workload representing typical conversational AI
  • Prefill-heavy (4k in / 256 out): long context retrieval and analysis
  • Decode-heavy (128 in / 2k out): content generation and code completion

Concurrency testing

The chat workload is swept at 1, 2, 4, 8, 16, 32, 64, 128 and 256 simultaneous users; the prefill-heavy and decode-heavy workloads run at 1, 8, 32 and 128. Single-user speed and time to first token come from concurrency 1. Peak throughput is the best level at which every request succeeded; a level with failed requests is never used.

Energy measurement

GPU power is read from the GPU vendor's telemetry (NVML on NVIDIA, amdsmi on AMD), sampled at 10 Hz during the entire benchmark run. We record both idle power (model loaded, no requests) and peak power (at maximum throughput). Energy per million tokens is calculated from throughput and average power draw under load.

This measurement captures GPU power only, not full server power (CPU, fans, networking, storage). For total facility power, we apply a PUE (Power Usage Effectiveness) factor in TCO calculations. Typical data center PUE ranges from 1.2 to 1.6. Hosted endpoint results carry no power data.

Cost modeling

Cost per million output tokens is calculated from measured throughput and the GPU rental rate we maintain for that hardware, shown next to each cost figure. The calculation assumes full GPU utilization and includes only compute costs, not networking, storage, load balancing, or engineering overhead.

For owned hardware, we do not assign a cost. Users can model complete TCO including purchase price, electricity, colocation, and fixed costs using our ROI calculator.

Reproducibility

Every published result states:

  • The engine and its exact version
  • The GPU model, GPU count and parallelism
  • The workload, context length tested and measurement date
  • Whether the run was complete or partial, with notes on anything unusual

For every run we keep the run plan, per-request timings, 10 Hz power samples and server logs, and share them on request.

If you cannot reproduce a result, please contact us and we will investigate and correct any errors.

Limits

What we don't measure

  • Model quality or accuracy: We assume the model developer's published accuracy scores. Our benchmarks cover performance, cost and energy, not output quality.
  • Provider API latency: Most results are self-hosted inference. Hosted endpoints we benchmark over HTTPS are labelled, and their time to first token includes the network round trip.
  • Training performance: We measure inference only, not training or fine-tuning speed.
  • Theoretical peak performance: We report real measured throughput under realistic workloads, not theoretical FLOPS or memory bandwidth limits.

FAQ

Methodology questions

How do you measure inference speed?

We measure output tokens per second using vLLM and SGLang with standard configurations. There are three workloads: chat (1k in, 1k out), swept from 1 to 256 concurrent users, and prefill-heavy (4k in, 256 out) and decode-heavy (128 in, 2k out), run at 1, 8, 32 and 128 users. We report single-user speed (concurrency 1) and peak throughput at saturation.

How is energy measured?

GPU power is read from the GPU vendor's telemetry (NVML on NVIDIA, amdsmi on AMD), sampled at 10 Hz during benchmark runs. We capture idle power and peak power, then calculate watt-hours per million tokens based on throughput. This is GPU power only, not full server power. PUE is applied separately in TCO calculations.

How do you calculate cost per token?

Cost per million output tokens is calculated from measured throughput and the GPU rental rate we maintain for that hardware, shown next to each cost figure. The calculation assumes full utilization and does not include networking, storage, or engineering costs. Users can model complete TCO in our ROI calculator.

Are your results reproducible?

Each published result states the engine and version, the GPU model and count, the parallelism, the workload and the date, and the method on this page is the one used for every run. For every run we keep the run plan, per-request timings, 10 Hz power samples and server logs, and share them on request. If you cannot reproduce a result, tell us and we will investigate.

What hardware do you test on?

Our benchmark runner supports NVIDIA (CUDA) and AMD (ROCm) GPU hosts with the vLLM and SGLang engines. Published results today are on NVIDIA B300. Hosted endpoints can also be benchmarked over HTTPS; those results are labelled, and their time to first token includes the network round trip.

Questions about our methodology?

Read more about our approach on the About page, or commission a custom benchmark to test your specific use case.

Request a benchmark