Operations4 min read

LLM cold start times on NVIDIA B300: 51 seconds to 28 minutes

Measured server start-up times for 18 model and engine combinations on NVIDIA B300, and what they mean for autoscaling and scale-to-zero.

Key takeaways

  • Cold start ranged from 51 seconds (gemma-4-E4B-it on SGLang) to 1,700 seconds, about 28 minutes (DeepSeek-V4.1-Flash on four GPUs with vLLM).
  • Single-GPU models started in 51 to 476 seconds; models spread over two to seven GPUs took 447 to 1,700 seconds.
  • SGLang started faster than vLLM in 6 of the 8 models run on both engines.
  • At these start-up times, scale-to-zero is practical only for small single-GPU models.

Throughput gets the attention, but start-up time decides how you can scale. If a replica takes 20 minutes to become ready, autoscaling cannot follow a traffic spike and scale-to-zero means a 20-minute first request. We record cold start, the time the inference server took to start and be ready to serve, for every run.

Measured cold start

ModelGPUsvLLM (s)SGLang (s)
gemma-4-E4B-it115551
Qwen3.8-27B185158
gemma-4-31B-it1171126
Mistral-Small-4-119B1406202
Mistral-Medium-3.5-128B1476121
GLM-5.3-Flash2777599
GLM-5.34802447
DeepSeek-V4-Pro41,3491,480
DeepSeek-V4.1-Flash41,700-
Kimi-K371,414-
Cold start in seconds on NVIDIA B300, vLLM 0.32.0 and SGLang 0.5.22. A dash means the model did not run on that engine. Kimi-K3 was a partial run.

Size and GPU count dominate

Every single-GPU model was ready in under eight minutes, and the smallest in under one. Every model that needed more than one GPU took more than seven minutes, and three of them took more than 22 minutes. Larger weights take longer to load and to distribute across GPUs, and that cost is paid on every start.

SGLang usually started faster

Of the eight models run on both engines, SGLang was ready sooner in six. The difference was largest on Mistral-Medium-3.5-128B (121 seconds against 476) and gemma-4-E4B-it (51 against 155). vLLM was quicker on Qwen3.8-27B (85 against 158) and DeepSeek-V4-Pro (1,349 against 1,480).

This is the opposite of the peak-throughput result, where vLLM led in six of eight (see vLLM vs SGLang on NVIDIA B300). The engine that is cheapest per token at steady load is not always the one that scales out fastest.

What to do with these numbers

  • Scale-to-zero: realistic for small single-GPU models where a one to three minute first request is acceptable; not for multi-GPU models.
  • Autoscaling: set the scale-out trigger early enough to cover the start-up time, or keep warm spare capacity equal to the traffic growth you expect during one cold start.
  • Deployments: a rolling update of a model with a 25-minute cold start needs surplus capacity for the whole rollout.
  • Cost models: idle warm replicas are part of the real cost per token. The ROI calculator includes utilization for this reason.

Frequently asked questions

How long does an LLM inference server take to start?
In our NVIDIA B300 batch, from 51 seconds for a small single-GPU model to about 28 minutes for a model spread over four GPUs. Single-GPU models took 51 to 476 seconds.
Does vLLM or SGLang start faster?
SGLang started faster in 6 of the 8 models we ran on both engines (vLLM 0.32.0 and SGLang 0.5.22 on NVIDIA B300).
Is scale-to-zero practical for LLM inference?
Only for small models. With cold starts of one to three minutes on a single GPU it can work for non-interactive workloads; multi-GPU models with cold starts of 7 to 28 minutes need warm capacity.

Measured by ModelPerf.ai on our own hardware. How each number is produced is in the methodology; current results are on the leaderboard.

All analysis