A 6.4x result sounds like a general product comparison. NVIDIA’s published evidence supports a narrower statement: one model, one Jetson, one harness and one highly specified inference configuration.
TensorRT Edge-LLM completed NVIDIA’s MLPerf Edge Agentic benchmark run in 24 minutes and 36 seconds on one Jetson AGX Thor Developer Kit. The company’s llama.cpp reference took 2 hours and 37 minutes, producing the reported 6.4x difference.
The benchmark scorecard
| Measurement | NVIDIA-reported result |
|---|---|
| Model | Qwen3.6-27B |
| Hardware | One Jetson AGX Thor Developer Kit with 128 GB unified memory |
| Completed turns | 1,007 |
| Total TensorRT Edge-LLM time | 24 minutes 36 seconds |
| llama.cpp reference time | 2 hours 37 minutes |
| Output throughput | 52.33 tokens per second |
| Median time to first token | 247.12 milliseconds |
| Median output-token latency | 14.68 milliseconds |
| BFCL accuracy | 87.94 percent |
These figures are company-reported. NVIDIA publishes the TensorRT Edge-LLM repository, an MLPerf-oriented release branch and configuration details, which makes the result more reproducible than an unsupported speed claim.
The 6.4x claim has a strict boundary
- Scenario: SingleStream with concurrency one.
- Power: Jetson running in MAXN mode.
- Weights and activations: NVFP4.
- KV cache: FP8.
- Decoding: tree MTP enabled.
- Cache state: approximately 96 percent hot.
- Sampling: temperature zero and seed 42.
- Reasoning: disabled.
Change any of those conditions and the ratio can move. Cold prompts may spend more time loading or processing context. Concurrent users change scheduling and memory pressure. Reasoning tokens can alter both latency and accuracy. A different quantization may improve fidelity while reducing throughput.
Why the hot-cache detail matters
A hot prefix cache is realistic when many requests share a stable system prompt, tool definitions or repeated document prefix. It is less representative when each edge task begins with a different camera frame, customer document or private context.
| Workload | Likely cache behavior | Test to add |
|---|---|---|
| Fixed factory assistant | High reuse of tools and instructions | Measure steady-state and restart behavior. |
| Mobile document agent | Shared system prompt, unique pages | Separate prefix speed from image or document processing. |
| Offline field support | Repeated manuals with varying questions | Measure cache size, eviction and retrieval accuracy. |
| Multi-tenant gateway | Lower reuse and more concurrency | Test isolation, fairness and tail latency. |
Accuracy and speed must move together
The reported 87.94 percent BFCL accuracy matters because aggressive quantization and speculative decoding can create a fast system that calls the wrong tool. A valid reproduction should confirm both throughput and task quality. Comparing only tokens per second would remove the acceptance boundary that makes an agent useful.
A reproduction plan for edge teams
- Use the published TensorRT Edge-LLM release branch and record the exact commit.
- Match the model, quantization, KV-cache format, power mode and harness.
- Reproduce the hot-cache run before changing anything.
- Run a cold-cache variant and report the new time to first token.
- Add concurrency at two, four and eight sessions if the product needs it.
- Turn reasoning on only if the production workflow uses it.
- Score function-call correctness and application-level task completion.
- Measure sustained power, thermal behavior and memory over a long session.
The MustHave.ai LLM latency and prefix-caching guide explains why cache state changes inference economics. Our AI GitHub repositories guide includes serving and evaluation projects useful for independent tests.
The practical verdict
NVIDIA’s result is meaningful because it demonstrates that a 27B-class tool-using model can complete a substantial agent benchmark on one edge device and because the company exposes a reproducible software path. The 6.4x number belongs to the published conditions. Buyers should repeat the run with their cache hit rate, concurrency, reasoning settings and accuracy threshold before sizing a deployment.
Primary sources
Checked September 19, 2026. Performance and accuracy figures are reported by NVIDIA for the stated configuration and have not been independently reproduced by MustHave.ai.
