NVIDIA’s Switchyard can make Codex or Claude Code speak to a local or alternate model without teaching the agent a new API. The repository also prints the warning that matters most: pre-alpha, experimental, not for production.
NVIDIA NeMo Switchyard is an open-source Rust proxy and routing library for LLM traffic. It accepts OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages, chooses a configured backend, then translates the answer back into the format the client expects.
The practical promise is attractive: keep the coding-agent interface while changing the model behind it. The practical risk is equally clear: translation, routing, streaming, tools, sessions, and fallback now sit in a new critical layer.
What Switchyard puts between agent and model
| Layer | What the repository supports | What to verify |
|---|---|---|
| Client protocols | OpenAI Chat, OpenAI Responses, Anthropic Messages | Tools, streaming events, errors, and usage fields survive translation |
| Backends | vLLM, NVIDIA NIM, Ollama, OpenAI-compatible endpoints | Authentication, context limits, model IDs, and cancellation |
| Launchers | Claude Code, Codex CLI, OpenClaw | Environment variables and local configuration do not leak |
| Routing | Passthrough, random, classifier, stage, escalation, custom | The chosen route matches risk, latency, and tool capability |
| Metrics | Requests, errors, latency, tokens, routing overhead | Cardinality, redaction, retention, and cost accuracy |
Protocol compatibility is not behavior compatibility
A translated request can be syntactically valid and still behave differently. Models disagree about tool selection, JSON discipline, parallel calls, stop conditions, reasoning fields, prompt hierarchy, and how much context they can use reliably.
For a coding agent, those differences reach the filesystem and shell. A fallback that produces fluent prose but drops a required tool call has failed. The router needs task-level checks alongside a healthy HTTP response.
Choose a routing strategy for one measurable reason
- Passthrough is the safest first test because it proves translation without adding a routing decision.
- Random routing is useful for controlled A/B traffic, provided sessions do not jump unpredictably between incompatible models.
- Classifier routing adds another model decision before the worker model, so measure its accuracy and latency separately.
- Stage routing uses signals already present in the conversation, such as tool results and errors, to choose a tier.
- Escalation routing runs a weak tier first and asks a judge whether to retry on a stronger tier, which can improve coverage while increasing cost and latency.
Do not enable all of them during the first evaluation. Every additional branch makes a bad output harder to attribute.
The fallback test that matters
- Start with one client, one route, and one backend.
- Replay a fixed set of read-only tasks with streaming and tools enabled.
- Kill the backend during a streamed response and observe cancellation, retry, and duplicate tool behavior.
- Send a tool schema near the provider’s naming and size limits.
- Force context overflow, invalid credentials, rate limiting, and a malformed upstream response.
- Confirm that logs do not capture prompts, source code, secrets, or tool outputs unnecessarily.
- Require the fallback model to pass the same repository checks as the primary.
A router has failed if it returns an answer but loses the agent’s safety contract.
Treat the README warning as a deployment gate
The project says its API and algorithms may change significantly before version 1.0 and explicitly labels the software not for production use. That does not make it useless. It defines the correct setting: a disposable branch, a local benchmark harness, or an isolated evaluation environment.
Our article on Copilot Memory and Ollama controls shows why local models do not automatically create local privacy. Our AI agent security scorecard gives the permissions, secrets, logging, and rollback questions to ask before a proxy touches a real repository.
My verdict: a useful lab switch, not a production promise
Switchyard is interesting because it makes model routing inspectable. The protocols, targets, algorithms, and metrics live in an Apache-2.0 repository instead of behind a dashboard.
Use it to learn where your coding agent depends on a provider’s behavior. Start with passthrough, break the backend on purpose, and watch every consequential tool call. If the fallback has not survived that test, it is a demo route—not a resilience plan.
Read the repository
- Review NVIDIA NeMo’s Switchyard repository, quick start, routing strategies, maturity warning, and Apache-2.0 license.
Checked August 12, 2026 against the repository’s main branch. Features and maturity labels are project-reported and may change quickly before version 1.0.
