5 AI code reviewers, 1 repository, and what developers did next

TL;DR

I see an engagement question in this export, not a model ranking. Before paying for another reviewer, I would check who reads the findings and what they change, while keeping silent fixes and unresolved quality outside the counts I can observe.

Question	What 3 months of data show
How many AI reviewers ran on the same code?	5 produced review output; on 506 of 1,011 pull requests, at least 2 of them reported a finding
How much of their output was a finding?	Far from all of it: most Friendly Reviewer comments were status notices, and every Greptile review body was empty or a trial notice
What did developers do with inline comments?	392 of 8,728 (4.5%) received a reply or a resolution from a person
Does “resolved” mean someone acted?	Not here: 3,746 of CodeRabbit’s 3,946 resolved threads were resolved by CodeRabbit itself
Which reviewer finds the most real defects?	Unknown: no comment in this dataset has been judged against the code yet

Between June 28 and September 28, 2026, a production application repository I work on received review comments from 5 AI reviewers on the same pull requests. I exported everything they posted through the GitHub API and counted what happened next.

The headline is uncomfortable. Of 8,728 inline review comments, 392 received a reply from a person or a thread resolution by a person. The other 95.5% show no visible human response. Some were probably fixed silently, and some were probably wrong, but the record alone cannot say which.

This part reports the historical repository export in place of the reviewer benchmark originally planned for the series. A frozen corpus with an independent reference set is still the right way to rank reviewers, and the last section explains what it would take. A separate Luna/Astra coding pilot is now prepared without model calls, as the October 14 preparation update in part 8 explains. It adds no new observations to this export. What follows is narrower: what the reviewers produced, how fast, and what the team visibly did with it.

The dataset and its limits

I exported the data on September 28, 2026, between 06:20 and 07:45 UTC, with read-only GitHub API calls. The window covers the 1,011 pull requests created from June 28 to September 28 inclusive, and the outputs created in the same window. The scripts, the per-file SHA-256 manifest, and the raw exports stay private, because comment text can contain personal data. This article publishes aggregates only.

Four limits apply to every number below:

• 1 repository, 1 team. Nothing here generalizes to other codebases or review cultures.
• Moving treatments. On September 25, the Claude review workflow ran claude-opus-4-8 at high effort, and the workflow file changed in 14 commits during the window. Greptile stopped posting on August 4, when its trial ended, and Copilot on July 16. The reviewers did not see the same pull requests for the same period.
• Snapshot state. Thread resolution and reactions reflect the moment of the export, not the moment of the review, and the last day is partial.
• Visible actions only. A developer who fixed a problem without replying or resolving the thread leaves no trace in these counts.

This is a closed historical window, retained as exported. Publication on October 14 does not refresh the counts or describe the repository’s current reviewer configuration.

Disclosure: Marc Hugon gives me access to Friendly Reviewer. I test it on real projects, report problems, and discuss product and editorial ideas with him. The collaboration is informal, and I have not been paid to rank the product.

5 reviewers, overlapping on half the pull requests

7 bot accounts appeared on the repository. 5 produced review output: Claude through a GitHub Actions workflow, CodeRabbit, Friendly Reviewer, and Greptile through their GitHub Apps, and Copilot through GitHub’s native integration. Codex was configured but posted nothing, and Cursor Bugbot posted only “not enabled” notices on 13 pull requests.

Reviewer	Pull requests with at least 1 finding	Active period in the window
Friendly Reviewer	539	June 29 to September 28
CodeRabbit	531	June 29 to September 28
Claude	336	June 29 to September 28
Greptile	136	June 28 to August 4
Copilot	52	June 29 to July 16

On 506 of the 1,011 pull requests, at least 2 reviewers reported a finding; on 277, at least 3; on 66, at least 4. Nothing assigned the reviewers distinct roles. The repository’s review policy classified each one only as current, optional, canary, or historical.

Most output was not a finding

Counting comments overstates what the reviewers said. I classified every output with a rule written for each reviewer’s format, such as a severity table with at least 1 entry or an actionable-comment count of at least 1. Everything else counts as a summary, a status notice, or an empty carrier for inline comments.

• Friendly Reviewer: 1,078 of its 4,380 comments contained a finding. 3,055 were status notices such as “Started”, “Aborted”, or “Complete”, because the service posts an acknowledgement and then edits it.
• Claude: 917 of its 2,374 formal reviews contained a finding; 1,031 said there was nothing to report and 400 only carried inline comments.
• CodeRabbit: 1,376 of 1,573 formal reviews reported actionable or nitpick comments, and its 1,025 general comments were walkthrough summaries and notices.
• Greptile: none of its 149 formal reviews carried a finding in the body, since 136 were empty carriers and 13 said the trial had ended. Its findings live in 248 inline comments.

A reader scanning a pull request sees all of this as review activity. A developer who has learned that most bot comments are notices has a reason to stop reading them.

What developers visibly did

Inline comments are the only output where a response can be counted per finding, through replies and thread resolution. Friendly Reviewer posts general comments without threads, so it is absent from this measure.

Reviewer	Inline comments	Human reply	Resolved by a person	Either
Claude	1,119	63 (5.6%)	98 (8.8%)	100 (8.9%)
CodeRabbit	7,037	142 (2.0%)	182 (2.6%)	278 (4.0%)
Greptile	248	2 (0.8%)	13 (5.2%)	14 (5.6%)
Copilot	324	0 (0%)	0 (0%)	0 (0%)
All	8,728	207 (2.4%)	293 (3.4%)	392 (4.5%)

Reactions tell the same story. Across 17,443 inline and general comments from all bots, 8 received any reaction, and none received a thumbs up or a thumbs down.

About 40% of the threads for Claude, CodeRabbit, and Greptile became outdated, which means the code at that location changed afterward. That is compatible with a fix, and equally with an unrelated edit, so it does not count as a response.

A resolved thread is not a human decision

Uber’s August 12, 2025 uReview account uses different definitions: usefulness ratings from interacting engineers, and an automated address rate obtained by rerunning the reviewer 5 times on the final commit. A comment counts as addressed when no rerun reproduces a similar issue. That is a model-based proxy, not independent confirmation of a correct fix, and neither measure has the denominator of our visible-response rate. Uber also describes filtering low-value categories and duplicates before publication. These are useful operational comparisons, but its reported gains cannot be ranked against this export as if both measured defect detection.

The first version of this analysis reported that 56.1% of CodeRabbit’s threads were resolved, against 8.8% for Claude. That comparison would have made CodeRabbit look 6 times more “acted on”. Checking who resolved each thread changed the picture:

Reviewer	Resolved threads	By a person	By the reviewer itself	By another bot
CodeRabbit	3,946	182	3,746	18
Claude	98	98	0	0
Greptile	18	13	0	5
Copilot	16	0	16	0

CodeRabbit and Copilot resolve their own threads, under rules I have not inspected. Whatever those rules are, a self-resolved thread records the reviewer’s judgment, not a developer’s. Any dashboard that reports “resolved threads” across reviewers needs the resolver’s identity next to the count.

Response times in the export

Latency is the one dimension where the reviewers looked alike. I measured it from each push, identified by the first run of the review workflow for a pull request and commit, to the reviewer’s first substantive output, over 4,110 pushes on 964 pull requests.

Reviewer	Pushes with an output	Median	90th percentile
Copilot	102	4 min 04 s	6 min 47 s
CodeRabbit	1,818	4 min 18 s	13 min 54 s
Claude	2,261	5 min 11 s	25 min 31 s
Greptile	123	5 min 16 s	47 min 21 s

Friendly Reviewer’s latency stays between 2 bounds. Its acknowledgement appears after a median of 5 min 58 s, and the edit that carries the review is dated by a last-update time whose median is 25 min 01 s, which can also include later edits. These timestamps describe response timing, not a causal explanation for the engagement rates. Testing whether timing affects engagement would require linking each finding to when it became available, the PR’s subsequent activity and the developer’s response.

What this data cannot say

It cannot rank the reviewers on quality. A low response rate can mean the comments were wrong, trivial, or duplicated by another reviewer, or that developers fixed problems without saying so. The counts cannot tell those cases apart.

It cannot give recall. The main branch has no revert commit in the window, and no incident record found in the repository links a production defect to a reviewer comment. Without such a reference, defects that every reviewer missed stay invisible.

It cannot give a cost per reviewer. The subscriptions are billed per seat or per month, and the CI runners are shared with every other job, so no reviewer’s cost can be isolated from the records available.

What a quality comparison would add

The smallest step toward quality is adjudication. For the 10 pull requests where 3 or 4 reviewers were active at the same time, the export lists 320 outputs with a finding. Judging each one as true, false, non-actionable, or duplicate, blind to the reviewer’s name, would give precision and overlap on a small sample. It would still not give recall.

Recall needs a reference set built before anyone reads the reviewer outputs: defects found by an exhaustive human review, defects reconstructed from later fixes, and injected defects with an executable oracle. LangChain’s ReviewBench and c-CRAB show 2 ways to build such a reference; both also show how much curation it takes. When a reviewer finds a real defect the reference missed, the reference gets a new version, and old results are never rescored silently.

Until then, the actionable result is operational. Before adding a reviewer, check whether anyone reads the ones you have. Part 6 shows the same repository’s review policy listing a reviewer as historical while it was the most active one, and part 8 starts the next evaluation from 1 reviewer instead of 5.

The WirelessCar study by Aðalsteinsson and colleagues, submitted May 22, 2025, asks another question. Ten participants evaluated proactive and on-demand review assistants, with qualitative interviews about trust, context and preferred interaction. It offers evidence about workflow preferences in that organization, rather than a measured recall or delivery-speed comparison. A follow-up to this export should collect those reasons directly instead of interpreting silence as approval or rejection.

---

YSNK

(You should now know)

• On this repository, 392 of 8,728 inline AI review comments (4.5%) got a visible human response
• Status notices and summaries made up much of the reviewers’ output, so raw comment counts overstate findings
• A resolved thread needs its resolver: most CodeRabbit resolutions were made by CodeRabbit
• Response timing is observable, but its effect on human engagement remains unmeasured
• Ranking reviewers on quality still requires adjudicated findings and an independent reference set