A GPU workload can spend much of its cold start loading the model before it handles a single request.
NVIDIA’s November 10 technical article describes Snapshot, a Kubernetes system that saves an initialized workload’s CPU and GPU state so another instance can resume from it. The project’s v0.3.0 release is dated November 8. Reusing a saved state can cut the time spent rebuilding that state each time a workload starts.
What the restore measurement covers
NVIDIA reports restore times under 45 seconds in its tested configurations, which used B200 GPUs and VAST NFS storage. The measurement leaves out provisioning a node and loading the container image. A customer waiting on an inference endpoint may still wait for those steps. The destination needs matching GPUs and driver versions, and checkpoint storage throughput affects how quickly it can read the saved state.
A checkpoint can also age out of a deployment. Record the model revision and container image for each snapshot. If either changes, have a normal startup path available while the new checkpoint is prepared. Treat permission to read checkpoint storage as permission to read saved process state.
Time the first successful response
Run a normal start and a restore for the same model and container. Log when the request to start arrives, when Kubernetes allocates a node, when the image becomes available, and when the application returns its first valid response. Run the comparison on a node with the image cached and again on a node that needs to pull it.
The two runs can show whether the main delay is process initialization, storage, or capacity. Repeat after updating the model and measure how often the checkpoint needs rebuilding. Our NVIDIA Dynamo deployment coverage discusses other choices that affect model serving.
Abdessalam Alaoui
I'm Abdessalam Alaoui, founder of Musthave.AI. I write practical guides to AI models, tools, generative media.
