Key takeaways
- Cold start ranged from 51 seconds (gemma-4-E4B-it on SGLang) to 1,700 seconds, about 28 minutes (DeepSeek-V4.1-Flash on four GPUs with vLLM).
- Single-GPU models started in 51 to 476 seconds; models spread over two to seven GPUs took 447 to 1,700 seconds.
- SGLang started faster than vLLM in 6 of the 8 models run on both engines.
- At these start-up times, scale-to-zero is practical only for small single-GPU models.
Throughput gets the attention, but start-up time decides how you can scale. If a replica takes 20 minutes to become ready, autoscaling cannot follow a traffic spike and scale-to-zero means a 20-minute first request. We record cold start, the time the inference server took to start and be ready to serve, for every run.
Measured cold start
| Model | GPUs | vLLM (s) | SGLang (s) |
|---|---|---|---|
| gemma-4-E4B-it | 1 | 155 | 51 |
| Qwen3.8-27B | 1 | 85 | 158 |
| gemma-4-31B-it | 1 | 171 | 126 |
| Mistral-Small-4-119B | 1 | 406 | 202 |
| Mistral-Medium-3.5-128B | 1 | 476 | 121 |
| GLM-5.3-Flash | 2 | 777 | 599 |
| GLM-5.3 | 4 | 802 | 447 |
| DeepSeek-V4-Pro | 4 | 1,349 | 1,480 |
| DeepSeek-V4.1-Flash | 4 | 1,700 | - |
| Kimi-K3 | 7 | 1,414 | - |
Size and GPU count dominate
Every single-GPU model was ready in under eight minutes, and the smallest in under one. Every model that needed more than one GPU took more than seven minutes, and three of them took more than 22 minutes. Larger weights take longer to load and to distribute across GPUs, and that cost is paid on every start.
SGLang usually started faster
Of the eight models run on both engines, SGLang was ready sooner in six. The difference was largest on Mistral-Medium-3.5-128B (121 seconds against 476) and gemma-4-E4B-it (51 against 155). vLLM was quicker on Qwen3.8-27B (85 against 158) and DeepSeek-V4-Pro (1,349 against 1,480).
This is the opposite of the peak-throughput result, where vLLM led in six of eight (see vLLM vs SGLang on NVIDIA B300). The engine that is cheapest per token at steady load is not always the one that scales out fastest.
What to do with these numbers
- Scale-to-zero: realistic for small single-GPU models where a one to three minute first request is acceptable; not for multi-GPU models.
- Autoscaling: set the scale-out trigger early enough to cover the start-up time, or keep warm spare capacity equal to the traffic growth you expect during one cold start.
- Deployments: a rolling update of a model with a 25-minute cold start needs surplus capacity for the whole rollout.
- Cost models: idle warm replicas are part of the real cost per token. The ROI calculator includes utilization for this reason.
Frequently asked questions
- How long does an LLM inference server take to start?
- In our NVIDIA B300 batch, from 51 seconds for a small single-GPU model to about 28 minutes for a model spread over four GPUs. Single-GPU models took 51 to 476 seconds.
- Does vLLM or SGLang start faster?
- SGLang started faster in 6 of the 8 models we ran on both engines (vLLM 0.32.0 and SGLang 0.5.22 on NVIDIA B300).
- Is scale-to-zero practical for LLM inference?
- Only for small models. With cold starts of one to three minutes on a single GPU it can work for non-interactive workloads; multi-GPU models with cold starts of 7 to 28 minutes need warm capacity.
Measured by ModelPerf.ai on our own hardware. How each number is produced is in the methodology; current results are on the leaderboard.
All analysis