A pull-request reviewer that can read files is useful. A reviewer that can build the branch, run its tests and execute a focused diagnostic can test whether its criticism survives contact with the code.
GitHub changed Copilot code review on September 11, 2026 in four connected ways: it can use the full shell-tool set from the Copilot SDK behind the Copilot agent firewall, Lite reviews now combine an ensemble of agents, addressed comments can close automatically during rereview, and applied suggestions get context-specific commit messages.
The review loop now reaches executable evidence
GitHub says the review agent can run build commands, tests and targeted scripts, and retrieve information from available tools and APIs. That does not turn a review into a complete continuous-integration run. It does give the reviewer a way to check claims that static reading alone cannot settle.
- A suspected type error can be checked against the compiler.
- A claimed regression can be challenged with a focused test.
- A generated file or schema can be inspected through the repository’s own tooling.
- A suggestion that breaks the build has a better chance of being rejected before a developer follows it.
Before and after: what actually changed
| Review stage | Previous practical boundary | New behavior described by GitHub |
|---|---|---|
| Analysis | File reading and model reasoning | Builds, tests, targeted scripts, tools and APIs can contribute evidence |
| Lite effort | One reviewing agent | An ensemble contributes perspectives to one combined review |
| Rereview | Developers may close stale threads manually | Copilot resolves a comment when a later commit addresses it |
| Autofix | Generic commit-message text | A message is proposed from the specific change |
GitHub’s experiment is encouraging, but it is not an independent benchmark
In GitHub’s company-run experimentation, the Lite ensemble increased the average number of addressed comments per review by 47% for high-severity findings, 31% for medium-severity findings and 11% for low-severity findings. GitHub also reports that review cost fell by about 8%.
Those figures are directional evidence, not a universal accuracy score. GitHub did not publish enough detail in the changelog to reconstruct the sample, repository mix, false-positive rate or statistical uncertainty. An addressed comment can also reflect developer behavior rather than ground-truth defect detection. Teams should measure accepted findings and escaped defects on their own repositories.
The agent firewall matters as much as the shell
Executing repository commands expands what the reviewer can learn, but also what untrusted repository content may try to influence. GitHub says the shell tools run behind the Copilot agent firewall. Administrators should still decide which workflows are safe to exercise, especially when builds normally expect secrets, network access or package-install hooks.
- Use test credentials and isolated runners for review-time execution.
- Block production secrets and write access from the review context.
- Log commands, exit codes and tool calls so a finding can be reproduced.
- Treat repository instructions, fixtures and issue text as untrusted inputs.
- Require normal CI and human approval before merge.
Auto-resolution reduces noise, with one condition
During rereview, a Copilot comment now closes when a later commit addresses the underlying feedback; unresolved feedback stays open. This can make the remaining queue more meaningful. The condition is traceability: teams need to be able to connect the resolved thread to the commit and confirm that a broad refactor did not merely hide the original symptom.
The same principle appears in our Codex and Claude Code comparison: agent output is most useful when the path from action to evidence is reviewable, not just when the answer sounds confident.
A seven-day evaluation that produces useful numbers
Select repositories with different languages and build systems. Record every high- and medium-severity comment, whether a developer accepted it, whether the referenced command reproduces the issue, and whether the eventual fix survives CI. Compare that with review latency and human minutes saved.
Do not optimize for comment volume. A smaller review containing reproducible defects is better than a longer review full of plausible observations. Our list of AI GitHub repositories worth using can supply varied open-source test cases, but any evaluation should respect each project’s contribution and automation policies.
The practical verdict
Copilot code review is moving from text inspection toward evidence-producing analysis. The valuable part is not that more agents can comment. It is that a reviewer can attempt the build, run a relevant test and leave fewer stale threads behind. The remaining question belongs to each engineering team: does that extra execution produce reproducible findings without widening the review environment too far?
Primary source
Checked September 13, 2026. Product behavior and experiment figures are attributed to GitHub. MustHave.ai has not independently reproduced the experiment.
