About

About ModelPerf.ai

Independent benchmarking for AI model performance, cost, and energy efficiency.

Who we are

ModelPerf.ai is built and operated by Podstack , a Bengaluru-based GPU cloud and AI infrastructure startup founded by Saurav Kumar.

We run production GPU clusters and understand firsthand the critical importance of accurate, reproducible performance data when making infrastructure decisions.

Our mission

The AI industry needs transparent, independent benchmarking. Published API latencies don't reflect real-world performance. Marketing specs rarely match deployment reality. TCO calculations are guesswork without measured energy data.

We're building a trusted authority for AI inference performance: measured speed, cost per token, and energy consumption for open models on the vLLM and SGLang engines and on NVIDIA and AMD GPUs. Our goal is to become the SemiAnalysis of AI inference: deep technical analysis, reproducible methodology, and data you can cite.

Scope

What we measure

Inference speed

Tokens per second, time to first token, and peak throughput under realistic workloads.

Cost economics

Dollars per million output tokens based on measured throughput and current GPU rental rates.

Energy efficiency

GPU power draw and watt-hours per million tokens, sampled at high frequency during every run.

Concurrency behavior

How throughput, latency, and energy scale from single-user to multi-user load.

Results cover performance, cost and energy. We do not publish accuracy scores.

How we test

Benchmarks run on NVIDIA (CUDA) and AMD (ROCm) GPU hosts in controlled conditions, using vLLM and SGLang with standard configurations. Published results today are on NVIDIA B300. Workloads cover chat (1k in, 1k out), prefill-heavy (4k in, 256 out), and decode-heavy (128 in, 2k out) scenarios at concurrency levels from 1 to the clean saturation point. Hosted endpoints can also be benchmarked over HTTPS, and those results are labelled.

Energy is GPU power sampled at high frequency during every run. Cost calculations use the GPU rental rate we maintain for each hardware type. Each result states the engine version, hardware and workload it was measured with.

Read the complete methodology

Business model

The public leaderboard and live pipeline are free for everyone. Detailed cost and energy metrics are available for $10 per model, unlocking economics data across all engines and hardware configurations for that model.

We also offer paid benchmarking services: companies can commission custom runs on specific hardware, test private models or endpoints, and receive a full performance and TCO analysis. This work funds the infrastructure and analyst time behind the public research.

Independence

Transparency and trust

  • We are funded by Podstack and paid benchmark commissions. We do not accept payment from hardware vendors, cloud providers, or model developers in exchange for favorable results. Every paid benchmark is clearly disclosed.
  • Benchmark results reflect measurements taken under our methodology. Real performance will vary based on workload, engine configuration, hardware batch, and deployment environment. Use our data as a comparative baseline, not an absolute guarantee.
  • Each result states the engine version, hardware and workload it was measured with, and we keep the per-request timings and power samples for every run. If you find an error or can't reproduce a result, please let us know.
Contact

Get in touch

Questions about our methodology? Want to discuss a custom benchmark? Interested in partnering?