An AI endpoint can feel slow for two completely different reasons. One user may wait for the first token; an operations team may wait half an hour for a new model replica. AWS released a separate cache for each problem.
Two September 10 announcements address AWS LLM latency at different layers: prefix-aware routing for SageMaker real-time endpoints and model caching for HyperPod Inference. They share a word, cache, but they do not solve the same wait.
Diagnose the clock before choosing the fix
| Observed delay | What is waiting | AWS feature | Useful metric |
|---|---|---|---|
| Request starts, first token is late | Prompt-prefix computation | Prefix-aware routing | Time to first token and KV-cache hit rate |
| Traffic spikes, new pod is late | Container image and model weights | HyperPod model caching | Pod-ready and scale-out time |
Clock one: repeated prompt prefixes waste live compute
Many applications send the same long beginning with every request: system instructions, a retrieved document, conversation history or source code. Serving engines can reuse the computed key-value cache for that prefix. Random fleet routing weakens the benefit because similar requests land on different instances.
SageMaker’s new PREFIX_AWARE strategy hashes the beginning of the request and tries to send matching prefixes to the same instance. It also supports a concurrency threshold so a popular prefix can overflow to a less busy instance instead of creating a hotspot.
AWS reports the largest gain on long shared context
On Llama 3.1 70B Instruct across seven ml.p5.48xlarge instances, AWS reports that 8,000-token shared-prefix workloads reduced median time to first token by 71% to 77%. P90 improved 33% to 37%, KV-cache hit rate rose from about 25% to as high as 82%, and throughput increased 15% to 16%.
Short ShareGPT-style conversations showed smaller gains: 13% to 16% at P50 and 1.7% to 2% in throughput. That difference is the decision rule. Prefix-aware routing becomes more valuable as the reusable beginning becomes longer and more common.
Clock two: a pod cannot serve weights it has not downloaded
HyperPod’s problem happens before inference. A pod may spend five to seven minutes pulling its serving image and another 20 minutes or more downloading a 145 GB model. AWS says a 600 GB-plus model such as DeepSeek-R1 can push the download beyond 30 minutes.
The new weights cache preloads model files onto local NVMe, where AWS cites typical reads around 7 GB/s. The image cache pre-pulls the container. Each can be enabled separately, and preferred scheduling falls back to the original source when a warm node is unavailable.
The scale-out benchmark is promising, not universal
AWS reports roughly 60% faster scale-out for tested models between 57 GB and 145 GB. It says the image cache reduced fresh ECR pull time by as much as 97%. These are company benchmarks, and deployment behavior will vary with image size, weight source, storage capacity, cache readiness and node placement.
The cache also moves cost rather than making it disappear. Local NVMe must be large enough for the models, and preloading consumes time before demand arrives. Teams should measure cache occupancy, update churn and the percentage of pods that actually land on warm nodes.
A two-axis test reveals which feature earns its keep
- Separate steady-state request latency from scale-out readiness in your dashboard.
- Bucket requests by shared-prefix length and frequency.
- Record KV-cache hit rate, P50 and P90 time to first token for each bucket.
- Trigger a controlled scale-out and time image pull, weight load and readiness separately.
- Repeat with cold nodes, warm nodes and a model update that invalidates the old cache.
Our AI agent cost-control guide explains why latency measurements should sit beside accepted-task cost. The AWS cost-per-correct-answer analysis provides the complementary quality denominator.
My take: cache the computation at the layer where it repeats
Prefix-aware routing is for repeated computation inside live requests. HyperPod caching is for repeated transfer during replica startup. Treating them as one generic acceleration feature would hide the operational question that matters: which clock is hurting the user? Measure that first, then warm the right layer.
Primary sources
- AWS prefix-aware routing announcement and benchmarks
- AWS HyperPod model caching announcement and benchmarks
Checked September 13, 2026. Performance figures are AWS-reported. MustHave.ai has not independently reproduced these benchmarks.
