The most important default in an inference server is often not the model. It is what the server remembers after the response.
SGLang 0.5.20 was released on September 18, 2026 with 713 pull requests from 237 contributors. It adds support for eight model families, a CPU simulator and serving improvements. It also changes the Responses API storage model: server-side response state is now opt-in.
The privacy default changed in a useful direction
Operators who need stored response state must start the server with --enable-response-store. Without that flag, requests that depend on retained server state are rejected rather than quietly behaving as if storage exists. That turns retention into an explicit deployment choice.
| Operation | Storage disabled | Migration action |
|---|---|---|
| Ordinary stateless response | Can proceed | Verify the application does not assume later retrieval. |
| Retrieve a stored response | HTTP 400 | Enable storage or move state to the application. |
previous_response_id | HTTP 400 | Choose server state or replay explicit history. |
| Background response workflow | HTTP 400 | Enable storage and document retention. |
Decide who owns conversation state
There are two valid architectures. The server can store response objects so a client can refer to an earlier ID, or the application can own history and send the required context on each request. Server storage simplifies some workflows but creates retention, deletion, tenant-isolation and backup obligations. Application-owned state can be easier to govern but increases prompt construction and transfer.
- Define retention duration and deletion behavior before enabling the flag.
- Verify response IDs cannot cross tenants.
- Measure storage growth under retries and background jobs.
- Document whether backups include stored response state.
- Test a rolling restart and failover with state-dependent requests.
Eight model families expand the compatibility surface
The release notes list eight newly supported model families. That is useful for a shared serving layer, but a loaded checkpoint is only the first gate. Validate tokenizer behavior, chat templates, tool-call parsing, quantization, speculative decoding compatibility, multimodal preprocessing and maximum context for each exact model artifact.
Our AI GitHub repositories reference includes serving and evaluation projects that can help build a repeatable acceptance harness. The HyperPod inference gateway guide covers the separate routing and authentication layer above a model server.
Prefill CP v1 removal needs an explicit configuration audit
SGLang 0.5.20 removes prefill context parallelism v1. Search deployment templates, Helm values, environment files and launch scripts for the old option before upgrading. Do not wait for a production node to reveal a removed path. Rebaseline time to first token, throughput and memory with the supported implementation.
CUDA 12 artifact retirement changes the upgrade matrix
The project also retired CUDA 12 artifacts. A team pinned to an older driver or base image should treat the release as an infrastructure migration, not a package bump. Record driver, toolkit, container base, GPU architecture and extension versions. Build or choose artifacts only after checking the release’s supported matrix.
Use the CPU simulator as a contract test, not a performance proxy
The new CPU simulator can make configuration and API contract tests cheaper. It cannot predict GPU kernels, memory pressure or production latency. Use it to catch request-shape, routing and response-state failures in continuous integration, then run performance gates on representative GPU hardware.
A no-surprise migration sequence
- Inventory current response-state features and old prefill CP settings.
- Decide whether response storage is required and document retention.
- Run API contract tests with storage disabled.
- Confirm expected 400 responses for state-dependent operations.
- Rebuild the runtime matrix without retired CUDA 12 artifacts.
- Load every required model and run tool, template and context tests.
- Canary GPU performance before a fleet-wide rollout.
The practical verdict
The release notes include project-reported performance results. Preserve the exact hardware, model, sequence length, concurrency, quantization and sampling settings before comparing them with another server. A throughput headline measured on one kernel path does not predict the latency of a stateful Responses API application. During the upgrade, capture p50 and p95 time to first token, output-token throughput, GPU memory, queue time and error rate for the same request set. If one of the eight new model families needs a different chat template or multimodal processor, keep its result separate rather than averaging it into a fleet-wide score.
SGLang 0.5.20 is valuable both for what it adds and for the default it removes. Explicit response storage is easier to govern than silent retention. The upgrade still deserves a full migration because model compatibility, prefill configuration and CUDA artifacts all move at once.
Primary source
Checked September 20, 2026. Performance figures in the release notes are project-reported and were not independently reproduced by MustHave.ai.
