A training change can improve repeated success while narrowing the set of tasks solved at least once. Read both measurements before deciding what improved.
The new release turns database checks into rewards
The October 22 Microsoft and Hugging Face post introduces ThinkingBox Training. It uses checks on the final database state as post-training rewards for the agent.
The public training repository provides the integration and reference configuration. The underlying ThinkingBox preprint was first submitted in August and revised on October 1.
Keep that chronology clear. A new training integration and additional analysis can be news without presenting the older benchmark paper as a new publication.
Three measurements tell different stories
The authors evaluate 507 tasks with 20 attempts per task, giving 10,140 recorded attempts. Their Qwen3.8-27B comparison follows 50 GRPO training steps.
| Measurement | Reported change | How to interpret it |
|---|---|---|
| Average attempt success | 9.08 percentage points higher | 51.70% before training; 60.78% after. |
| Correct in all 20 recorded attempts | 69 additional tasks | The observed count rises from 38 to 107. |
| Solved at least once | 7 fewer tasks | The count falls from 453 to 446. |
All-attempt success describes the trials actually recorded. It does not prove a task will always succeed in deployment. At-least-once coverage answers a different question.
Check the resulting records, including extra effects
I would apply the same distinction to a business assistant before considering a training project. Define which records should change and which must remain untouched.
For a booking task, the acceptance condition could include the reservation, payment, and required approval state. A plausible final message cannot establish that those records agree.
- Save the starting database state and expected changes.
- Inspect required records after each trial.
- Count missing, incorrect, and unintended additional changes.
- Separate repeated success from occasional coverage.
- Review failures that end with an apparently normal completion message.
Use harmless fixtures and authorized test services for such an evaluation. We have not run this proposed workflow.
Treat the reference run as a specific configuration
The full-parameter reference run distributes 24 H100 GPUs across three nodes. The stack also depends on backend services and simulated-user or judge endpoints.
That reference configuration should not become a minimum requirement for every smaller experiment. Nor does a published training recipe establish that its gains transfer to another model or workflow.
The authors say they excluded the benchmark tasks from the training pool. Their reported results still need independent reproduction before they become a general reliability claim.
The GKE agent sandbox article covers infrastructure for a different training workflow. Our agent monitoring analysis explains why you need to review an action trace alongside its final answer.
Sources
- Training release and repeated-trial results
- Code, reference hardware and integration requirements
- Underlying preprint and version history
Which persistent record would reveal that your assistant’s completed task was actually wrong?
Checked October 25, 2026. Prepared with AI-assisted research and drafting, with claims checked against the linked sources. MustHave.ai has not performed the proposed product tests.
