The fastest way to waste an AI budget is to fine-tune a knowledge problem or build retrieval for a behavior problem. AWS has published an eight-step customization spectrum. The useful reader question is simpler: what exactly is failing?
The AWS framework, published September 14, 2026, progresses from using a model as-is to prompt engineering, retrieval-augmented generation, prompt caching, distillation, fine-tuning, continued pre-training and custom training. AWS maps each step to its own services. This guide converts the staircase into a vendor-neutral diagnosis.
Start with the failure, not the technique
| Observed failure | First intervention | Escalate when |
|---|---|---|
| Wrong format, tone or missing instruction | Prompt engineering | Good examples still do not produce stable behavior |
| Missing private or changing facts | RAG | Retrieval is accurate but the model cannot use the evidence |
| Repeated long context is slow or expensive | Prompt caching | The model itself remains too large for the workload |
| A large model is correct but costly | Distillation | The smaller model cannot preserve required behavior |
| Knowledge is right but behavior is consistently wrong | Fine-tuning | The model lacks domain concepts, not examples of correct action |
| Domain language is not understood | Continued pre-training | No existing architecture fits the task |
| Architecture or reward environment is fundamentally new | Custom training | Only after the lower steps fail with evidence |
Prompt engineering fixes instruction failures
If the model knows the facts but returns the wrong shape, start with a clearer instruction, explicit constraints, an output schema and a few representative examples. The acceptance test should score adherence separately from factual quality. A long prompt is not automatically a good prompt; remove conflicting rules before adding more text.
RAG fixes access to knowledge
Retrieval is appropriate when answers depend on private documents, current policies, changing catalogs or evidence that must be cited. Measure retrieval recall, ranking quality and answer faithfulness separately. Fine-tuning cannot keep a frequently changing product table current without repeated training, while RAG can replace or reindex the source.
Caching and distillation solve different cost problems
Prompt caching removes repeated computation for shared prefixes such as system instructions, examples or large static context. Distillation teaches a smaller student to reproduce a validated teacher on a narrower task. Cache when requests repeat. Distill when the large model is consistently good but too expensive or slow even after prompt and retrieval optimization.
Fine-tuning changes behavior, not fresh facts
Fine-tuning is a fit when the model has the relevant knowledge but repeatedly misses tone, structure, decision boundaries or task execution. Supervised fine-tuning needs examples of correct behavior. Reinforcement fine-tuning needs a reward signal that reliably distinguishes good from bad outcomes. Neither method repairs a slow retriever or guarantees faster inference.
Continued pre-training is for comprehension gaps
Use continued pre-training only when a model fails to understand the language and structures of a domain even after it sees retrieved evidence and good examples. Raw corpus volume is not a reason by itself. The team needs evidence that domain comprehension, rather than access or instruction, is the limiting factor.
Custom training is an architectural decision
Training from early checkpoints or building a custom model belongs at the top of the staircase because it changes data governance, infrastructure, evaluation, safety and maintenance. It can be justified when no available architecture fits the modality, control requirements or proprietary reward environment. It should not be a prestige upgrade from an unmeasured prototype.
An evaluation ladder before each escalation
- Name one failure in observable terms.
- Build a frozen test set that contains that failure.
- Apply the lowest-cost intervention that could fix it.
- Measure quality, latency, total cost and operational burden.
- Escalate only if the current step misses a predeclared threshold.
- Keep the simpler system as a rollback path.
Our model selection guide helps create the test set. The cost-per-correct-answer analysis explains why a cheaper token price can lose once retries and review are counted.
A decision record prevents technique drift
For each step, record the failure, baseline, intervention, dataset, threshold, result, new operating cost and rollback. This turns customization into an engineering decision. It also makes it possible to remove an expensive technique when a later base model solves the original problem without it.
The practical verdict
The AWS staircase is useful because it makes escalation cost visible. Its strongest rule is vendor-neutral: start with the simplest system and move only when measured accuracy, latency, cost or domain failures justify more control. Most teams should prove prompt and retrieval limits before changing model weights.
Primary source
Checked September 14, 2026. The eight-step sequence comes from AWS. The failure-based mapping, evaluation ladder and vendor-neutral recommendations are MustHave.ai analysis.
