Operations4 min read
LLM cold start times on NVIDIA B300: 51 seconds to 28 minutes
Measured server start-up times for 18 model and engine combinations on NVIDIA B300, and what they mean for autoscaling and scale-to-zero.
Read the analysisAnalysis
Engine comparisons, cost per token, start-up times and guides to reading benchmarks, written from our own measured data.
Subscribe by RSSMeasured server start-up times for 18 model and engine combinations on NVIDIA B300, and what they mean for autoscaling and scale-to-zero.
Read the analysisHow to calculate self-hosted inference cost per million tokens from GPU price and measured throughput, with a worked example and the common mistakes.
Tokens per second per user, time to first token and peak throughput answer three different questions. Here is what each one means and when it matters.
We ran eight open models on both vLLM and SGLang on NVIDIA B300. SGLang was faster per user in 5 of 8; vLLM won peak throughput in 6 of 8.