Most synthetic tool-use data starts with a made-up request and asks an agent to find a path to the answer. ToolGrad turns the problem around: build a valid path first, then write the request that the path solves.
Google Research published ToolGrad on September 10, 2026. The official page links both the paper and an open repository.
Why query-first generation wastes work
In a query-first pipeline, a model invents a user instruction from a sampled API set. Another agent then searches for a working sequence of tool calls. Many searches fail because the request is impossible, the APIs do not compose or the path is too difficult to discover.
Those failures still consume model calls. The surviving paths can also favor what the search agent happened to find rather than a deliberately varied curriculum.
ToolGrad starts with a valid chain
ToolGrad proposes and evaluates APIs in small batches, extends a tool-use workflow and applies textual feedback to improve the chain. Once the answer path is valid, the system generates a corresponding user request.
| Stage | ToolGrad direction | Evidence to save |
|---|---|---|
| API proposal | Choose candidate tools for the chain | API version and schema |
| Workflow extension | Add calls that produce a valid result | Arguments, outputs and dependency |
| Textual evaluation | Describe what should improve | Feedback and accepted revision |
| Request generation | Write a query for the verified chain | Prompt, constraints and target result |
| Validation | Check execution and training record | Pass rule, failure reason and cost |
What the 99.8% number measures
Google researchers used ToolBench, which contains more than 16,000 real-world APIs, to compare ToolGrad with a query-first depth-first-search baseline. They report a 99.8% pass rate for generated examples, with fewer optimization steps and lower generation cost.
That is a researcher-reported generation result. It does not mean a model trained on the data succeeds on 99.8% of real user tasks. Pass criteria, API snapshots and generation conditions define the number.
The downstream test is separate
The team created ToolGrad-500 datasets and fine-tuned Gemma 3 models at 1B, 4B and 12B sizes. They evaluated the models on the Berkeley Function Calling Leaderboard, whose tool set differs from ToolBench. The researchers report improvements over the corresponding base models.
Using a different evaluation tool set is a useful design choice. Independent replication is still needed, especially for changing APIs, authentication failures and long workflows with irreversible effects.
A small reproduction plan
- Pin a public API schema and record its date or commit.
- Generate 100 short chains and keep every rejected example.
- Define pass before generation, including argument and output checks.
- Measure model tokens, tool calls, retries, wall time and human review.
- Create a query-first baseline with the same models and API pool.
- Evaluate trained models on different tools and on changed schemas.
Our Cohere agent-tools dataset analysis shows why tool availability and complete task automation are different measurements. The OpenAI Agents API guide maps the runtime choices around real tool execution.
Dynamic APIs are the hard case
An answer-first chain can be valid when generated and stale a month later. APIs rename fields, change authentication, alter rate limits and return new error shapes. A reusable dataset needs versioned schemas and a revalidation policy.
Safety also belongs in generation. A chain can be technically valid and still teach a model to call a sensitive endpoint without the right consent or approval.
A reusable release should include a failure ledger beside the accepted dataset. For each rejected chain, record the API versions, failed step, error class, validator response, retry count and final disposition. Researchers can then see whether the method avoided impossible workflows or merely filtered them after spending the calls.
The ledger also exposes curriculum gaps. If most rejections involve authentication handoffs, pagination or state-changing calls, a high overall pass rate may still hide weak coverage of the workflows that production agents find hardest.
My take: backwards is the sensible direction
For synthetic data, starting from something known to work is appealing. It spends less time asking an agent to rescue impossible prompts and gives the generator more control over the skills represented.
The method will be most convincing when other teams reproduce the efficiency gain on live, changing tool catalogs. The open code makes that test possible.
Read the primary material
Checked September 12, 2026. Method and results are attributed to the ToolGrad researchers. No independent reproduction is claimed.
