An agent completes a specialized task with a skill installed, fails the same task without it, and produces a beautiful improvement score. Then you learn that most tasks were tried once.
NVIDIA SkillEvaluator is still one of the more useful agent-evaluation releases this month. It turns a vague question, “Did this skill help?”, into a controlled comparison. NVIDIA’s first catalog results report a 41-point average lift in correctness and a 39-point lift in effectiveness across more than 300 verified skills for over 30 NVIDIA products.
Those are company-reported benchmark results, not independent proof that any skill will improve any agent by the same amount. NVIDIA also publishes the limitation that should shape how teams use the numbers: 85% of skills with results ran one attempt per task, 15% ran two, and the post provides no confidence intervals.
I like the method more than I trust the headline average. That is not a dismissal. It is exactly why the open-source framework is worth opening: teams can repeat the experiment on their own tasks instead of borrowing NVIDIA’s conclusion.
The paired test changes one main variable
SkillEvaluator’s live tier runs each evaluation case twice in separate Harbor sandboxes. The prompt, model, task inputs, and grading criteria stay the same. One run has the skill installed; the other does not. The score difference is reported as Skill Lift in points.
That design is much stronger than comparing last week’s agent with this week’s agent. Model changes, prompt edits, cached state, tool versions, and different graders can otherwise masquerade as skill improvement. A paired run does not eliminate randomness, but it gives the randomness fewer places to hide.
NVIDIA evaluates the comparison on two harnesses, Codex and Claude Code. It reports an average lift of 29 points across all dimensions for Codex and 34 for Claude Code. Excluding security, the reported lifts are 36 and 42 points. NVIDIA says product-level variation was much larger, ranging roughly from +2 to +46.
What the first benchmark actually measured
| Dimension | Without skill | With skill | Reported lift |
|---|---|---|---|
| Correctness | 46 | 87 | +41 points |
| Discoverability | 42 | 82 | +40 points |
| Effectiveness | 39 | 78 | +39 points |
| Efficiency | 43 | 78 | +35 points |
| Security | 97 | 98 | +1 point |
Correctness and effectiveness are the cleanest comparisons because they measure the result in both conditions. NVIDIA cautions that the scores are not pass-probability estimates. They are average scores on the specialized tasks in this evaluation set.
Discoverability and efficiency need different interpretation. Part of those scores concerns whether the agent finds and uses the relevant skill correctly. A run without the skill cannot load it, although it can still earn credit for useful tool use and for leaving irrelevant resources alone. The 40-point discoverability lift should not be read as a general measure of how much smarter the unaided agent became.
The one-attempt warning is not a footnote
Agent runs vary. A tool call times out. A model takes a different route. One grader interprets a borderline output differently. With one attempt, the reported difference can contain both the skill’s effect and the luck of that run.
NVIDIA says the catalog averages combine thousands of trials, which can make the overall direction informative. That aggregation does not make an individual skill’s single-attempt score stable. A team deciding whether to install one skill needs repeated evidence for that skill, on the tasks and failure costs it actually has.
This connects neatly with our review of DarwinX improving an agent harness while freezing the model. Holding the model steady helps isolate the system change. Repeating the run helps separate that change from variance. You need both.
Token savings can reverse direction
NVIDIA’s examples show why a single “efficiency” label is not enough. The jetson-optimize-memory skill reduced token use from 617,306 to 142,540 and execution time from 474.9 to 220 seconds. That is 474,766 fewer tokens and 254.9 seconds saved in the reported run.
The cuopt-install skill moved the other way. Tokens increased from 25,227 to 55,582, an extra 30,355 tokens, while execution time rose from 34 to 41.1 seconds. NVIDIA reports the relative changes as a 120.3% token increase and a 20.8% time increase.
Both are single-attempt examples. They do not establish durable cost ratios. They do establish a good measurement rule: a skill can improve correctness while making the path longer, or make the path shorter while activating on the wrong tasks. Track the outcomes separately.
A stronger skill trial for your own stack
- Choose real acceptance tasks. Use work that ends in a checkable artifact, not trivia written to match the skill documentation.
- Add negative cases. Include prompts where the skill must stay unloaded. Unnecessary activation can consume context and steer the agent toward the wrong workflow.
- Freeze the comparison. Pin the model, harness, prompt, tools, environment, grader, and task data.
- Run enough repetitions. Start with at least five paired runs for a cheap test and increase the count when variance or failure cost is high.
- Separate the metrics. Record correctness, accepted outcome, security failures, tokens, wall time, tool calls, retries, and human rework.
- Report uncertainty. Keep the individual results and calculate a spread. Do not hide five different outcomes behind one average.
- Inspect the failures. A score cannot tell you whether the skill supplied a wrong command, loaded at the wrong time, or merely encouraged a longer route.
- Set a release threshold. Decide beforehand what gain justifies the added context, maintenance, and supply-chain risk.
The held-out evaluation in our SHE agent-safety article adds another useful guard. Do not let the skill author see every final test. Otherwise the instructions can become a collection of answers to the benchmark rather than a reusable capability.
Static safety checks are necessary, not sufficient
SkillEvaluator’s first tier checks structure, secrets, personally identifiable information, licenses, scripts, prompt injection, and possible data exfiltration. Its second tier looks for duplicated guidance within a skill and overlapping coverage across the catalog.
Those checks reduce obvious risk before the live run. They do not prove that the skill is safe in every repository or connected account. A harmless instruction can become consequential when the agent receives production credentials. The permission scope still belongs outside the skill file.
That is also why the small +1 security lift should not be oversold. The baseline was already 97, leaving little room for movement. A rare severe failure can disappear inside a high average. Keep hard failure gates for secrets, destructive actions, and unauthorized network use even when the aggregate security score looks healthy.
My verdict: install the method before the catalog
NVIDIA’s benchmark is a useful argument for structured agent skills. It is an even better argument for paired evaluation. The with-skill and without-skill design gives teams a practical starting point that is far more honest than publishing a successful demo trace.
Do not treat “verified” as a permanent quality grade. Re-run the task after the model, tool API, repository, or skill changes. A skill earns its place by continuing to improve accepted work without creating a new cost or safety problem.
Run the evaluation yourself
- Read NVIDIA’s SkillEvaluator methodology and first results.
- Inspect the open-source SkillEvaluator repository.
- Review the August 12 benchmark snapshot.
- Read the Tier 3 live-evaluation documentation.
Which skill in your agent stack has been compared against the same task without it?
Checked August 24, 2026. Scores, task counts, harness comparisons, token use, and timing are attributed to NVIDIA’s August 19 technical post and its August 12 repository snapshot. The results have not been independently reproduced here.
