ThinkingBox expands agent evaluation and trains for repeatable task completion
Microsoft’s ThinkingBox update checks the records an agent actually changes to assess task completion and policy compliance.
The expanded evaluation distinguishes average success from tasks completed successfully on all 20 attempts and compares their estimated inference costs.
Researchers also used final-state checks as reinforcement-learning rewards. On the reported 507-task benchmark, Qwen3.8-27B increased tasks passed on every attempt from 38 to 107 after training on separate tasks.
The article notes possible overlap in the surrounding setup and explains that repeated-test results do not guarantee future performance. Public framework, training-code and paper links accompany the methodology.
Partner briefing
- What it is
An expanded ThinkingBox study of agent task completion, repeatability and training.
- What’s changing
Adds model comparisons, repeated-success cost analysis and reinforcement learning using final-state verdicts.
- Why it matters
Shows why completion messages and successful tool responses can miss incorrect business records.
- Who it affects
Developers evaluating and training agents that change business-system records.