Nemotron 3.5 Lightning activates 3 billion parameters per token. The BF16 weights still occupy roughly 66 GB before your context window starts consuming memory.
NVIDIA has released Nemotron 3.5 Lightning, a 30-billion-parameter hybrid mixture-of-experts model with 3 billion active parameters. The model card advertises context up to one million tokens, publishes full BF16 reference weights under OpenMDW 1.1, and points deployment users toward a smaller NVFP4 variant.
The day-one shorthand needs repair. “3B active” describes the parameters used for a token’s compute path. It does not turn the release into a 3 GB model. The public BF16 weight files total about 65.8 GB, and NVIDIA’s own one-million-token examples move to eight H100 GPUs or one GB200.
Thirty billion total and three billion active answer different questions
The architecture combines Mamba-2, attention, and mixture-of-experts layers. Only part of the expert capacity is activated for each token, reducing compute relative to a dense 30B model. All expert weights still need to be stored and made available to the runtime.
Active parameters estimate per-token work. Total weights and KV cache determine whether the deployment fits.
That difference is why the BF16 card lists an 80 GB H100 or A100 as the single-GPU reference. Quantized community builds may fit elsewhere, but their memory use, speed, accuracy, and supported context must be measured as separate artifacts.
The one-million-token claim has a hardware footnote
NVIDIA says the model supports up to one million tokens. Its quick single-H100 deployment uses a 256,000-token maximum. The card’s one-million-token examples use eight H100s or one GB200. That is not a contradiction; it is a capacity plan.
| Configuration | Published context example | What it is for |
|---|---|---|
| 1× H100 80 GB | 256K | Single-GPU BF16 quick start |
| 1× A100 80 GB | Supported single-GPU class | BF16 research and evaluation with a workload-sized limit |
| 8× H100 | 1M | Long-context BF16 deployment example |
| 1× GB200 | 1M | Long-context Blackwell deployment example |
| NVFP4 release | Workload dependent | NVIDIA’s recommended optimized-inference path |
Long context also has an ongoing cache cost. A team deciding whether the model fits should test its real prompt length, concurrency, generation length, and latency target. Loading the weights is only the first memory check.
The BF16 release is a customization starting point
NVIDIA describes the full-precision release as reference weights for supervised fine-tuning, reinforcement learning, distillation, domain adaptation, and creating quantized or GGUF variants. It recommends the NVFP4 release for optimized inference.
That positioning should shape an evaluation. If you want to study the base artifact or build a domain-specific derivative, BF16 is useful. If you want an agent backend with practical latency and throughput, start with the optimized artifact and confirm that your serving stack supports it. Our guide to open weights versus hosted APIs covers the operational costs that arrive after the download.
NVIDIA’s benchmark table is useful, but still vendor-reported
NVIDIA reports 81.94 on MMLU Pro, 75.44 on GPQA Diamond without tools, 51.56 on SWE-bench Verified, and 24.58 on Terminal-Bench 2.1. It links benchmark-specific recipes, prompts, containers, parsing, and scoring settings through NeMo Gym and NeMo Evaluator.
Publishing the harness details is valuable because agent scores are sensitive to tool setup and retry policy. The results remain vendor-reported until another team reproduces them from the released weights. Do not compare one number with a competitor’s headline unless the harness, model variant, tool budget, and scoring rules match.
This is the same reason the Qwen 3.8 agent benchmark rerank mattered more than the first leaderboard position: the evaluation procedure can change the decision.
Switchyard can route agent traffic, but it is pre-alpha
NVIDIA also published Switchyard, a Rust proxy and library that translates OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. It can route traffic by a classifier, conversation stage, escalation decision, or random split. That makes it relevant to teams testing an open model behind tools built for another provider’s API.
The repository carries an unusually clear warning: Switchyard is pre-alpha, experimental, and not for production use. Its APIs and routing algorithms may change substantially before version 1.0. Treat it as an evaluation component. Put it behind synthetic traffic, capture protocol differences, and keep a known-good gateway ready.
A deployment test that separates four claims
- Weight fit: measure model load, runtime overhead, and free memory before sending a long prompt.
- Context fit: test the prompt lengths your application actually uses at realistic concurrency.
- Quality fit: reproduce a small task suite with the exact BF16 or quantized artifact you plan to serve.
- Agent fit: record tool success, protocol translation errors, retries, wall time, and cost per completed task.
- Fallback fit: force the router and backend to fail so you know which requests are retried, escalated, or lost.
A local or private deployment is not automatically simpler than an API. The Meta Muse Glimmer backend test showed how day-one runtime support can matter more than the nominal model size. Nemotron 3.5 Lightning adds an attractive compute profile, but the serving path still has to earn production trust.
My verdict: promising efficiency, serious infrastructure
Nemotron 3.5 Lightning is interesting because NVIDIA has paired a sparse hybrid model with reference weights, reproducibility recipes, optimized variants, and a routing experiment. The 3B active figure may translate into useful throughput, especially when the serving stack supports the intended quantization.
It is not a tiny local model and one million tokens are not free. Start with the optimized release, cap context to the workload, reproduce the coding tasks you care about, and keep Switchyard out of production until its own warning changes.
Read the primary artifacts
- Inspect NVIDIA’s Nemotron 3.5 Lightning BF16 model card and weights.
- Review the Switchyard repository and its production warning.
- Read the Switchyard routing documentation.
Which constraint will decide your deployment first: weight memory, context cache, runtime support, or task quality?
Checked August 11, 2026. Architecture, hardware examples, context limits, license, positioning, and benchmark results come from NVIDIA’s model card. Benchmark scores are vendor-reported. Switchyard maturity and capabilities come from its public repository and documentation.
