When an independent reviewer earns its place

TL;DR

Before adding a reviewer, I want evidence that it catches a verified defect or reduces rework enough to justify its cost. A fresh context or another provider changes the treatment; it does not by itself establish better judgment.

Decision	Evidence required
Test the value of another reviewer	Compare the same frozen change with and without the additional review treatment
Treat independence as a protocol	Record the model, provider, context, prompt, tools, rules, budget, and authority for every pass
Separate disagreement from quality	Adjudicate findings against the code without revealing the product name when possible
Keep product claims bounded	Friendly Reviewer is an author’s statement of design intent; Greptile’s independence language is a vendor statement
Defer any ranking	Recall, shared misses, and product superiority require an independent reference set

The agent that wrote a change already has a theory of why the change is correct. Asking it to review its own work can expose a typo, a failed test, or an inconsistency, but it does not create an independent challenge to that theory. Self-review preserves many of the conditions that produced the change: the same interpretation of the requirement, the same selected context, the same tools, and often the same model. A confident second pass can repeat the first pass’s blind spot.

A separate reviewer can apply a different test, and its contribution has to show up in the protocol and in the accepted work. A new product name, a fresh chat, or another model provider is not sufficient evidence on its own.

4 treatments and 1 adjudication

An evaluation needs more than “reviewed” and “not reviewed.” Give the same frozen diff and review criteria to 4 distinct treatments, then send their findings to a human adjudicator:

1. Self-review in the authoring session. This measures what the authoring agent can challenge while retaining its working context.
2. The same model in a fresh context. This removes conversational memory while preserving the model family and much of its training.
3. Another model or provider. This changes one potential source of error, but may retain the same requirement, repository instructions, and tool limits.
4. Deterministic controls. Tests, type checks, linters, static analysis, and policy checks evaluate explicit properties without relying on an LLM judgment.

The human adjudicator is not a fifth treatment. That person decides whether each finding is supported by the frozen evidence and whether the proposed disposition fits the product and risk context.

The treatments do not vote. Deterministic checks can prove that a specified assertion passed on a named revision, but they cannot infer every missing business rule. A model can notice a cross-file inconsistency that no current test encodes. The adjudicator can resolve ambiguity and accept a trade-off, with attention that is scarce and fallible.

What the comparison looks for is useful contradiction. Agreement can reflect independent convergence, and it can also reflect a shared bad specification.

Independence has several dimensions

SWR-Bench, by Zhengran Zeng and colleagues, supplies a complementary comparison on 1,000 manually verified GitHub PRs with repository context. The June 5, 2026 version evaluates whether generated reviews cover annotated change-actions and reports gains from aggregating reports. Its reference includes maintainability changes as well as functional defects, and the grading uses an LLM, validated against human judgment with roughly 90% agreement. This supports testing aggregation on a defined corpus. It does not establish independent errors, a current-model ranking or less human review work in our repository.

Even strategy names can conceal shared judgment. The current elek documentation, checked on October 14, says its crosscheck and council strategies use different lenses inside one model session. They therefore do not supply the separate sessions required by the comparison proposed here. That is a documented implementation boundary, not a quality result.

Reviewers running in separate processes can still make correlated errors. Inspect at least these dimensions before calling the second pass independent:

Dimension	Correlation risk
Requirement	Both reviewers receive the same incomplete or false acceptance criteria
Repository context	Both see the same selected files and miss the same dependency or operational constraint
Tools	Both lack test execution, dependency inspection, network access, or production evidence
Model and training data	Separate prompts to the same model can reproduce the same learned shortcut
Prompt and rules	Shared review instructions can exclude the same class of issue
Budget	Similar token, time, or file limits can truncate both reviews at the same boundary
Incentives and authority	Both may optimize for a green result or avoid blocking the author

Independence is therefore graduated. A fresh context separates conversational memory. A different model separates part of the inference path. A reviewer with different tools and explicit adversarial criteria separates more. None of those moves repairs a false requirement that every participant treats as authoritative.

Read-only permissions reduce the actions a reviewer can take. They do not establish independent judgment. Likewise, a fresh context can inherit a biased handoff. Record which evidence the reviewer obtains independently and which assumptions remain shared.

Record the dimensions rather than compressing them into a badge. That makes a later result interpretable. If a second reviewer rescues an access-control defect, you can ask which difference mattered. If both miss it, you can see what they still shared.

Preserve a judgment before the synthesis

A confident review can influence the next reader before they have examined the evidence. In AI Agents Push Humans Out of the Loop, Margaret Mitchell, Avijit Ghosh, and Samir Passi argue that current agent systems erode the attention that oversight depends on, through anchoring and approval fatigue. For selected comparison tasks, retain the adjudicator’s initial verdict before exposing another review, then record any revision and its justification. Whether this helps in a review workflow has not been measured.

Count a suggestion that introduces a regression as a harmful correction, even if someone accepted it at first. Report rescued defects, false positives, harmful corrections, and triage effort beside the final accepted result.

A completion evaluator has a narrower job

Claude Code’s goal evaluator reads the authoring conversation to decide whether the recorded evidence meets the goal condition. It does not independently rerun the commands or inspect the repository. Addy Osmani makes this distinction in Practical Loop Engineering: the completion check does not replace review of the resulting work.

If the transcript records a passing desktop performance check, that evidence does not establish acceptable mobile behavior. The specification must name the required environment, the worker must retain the corresponding measurement, and the assigned verifier must inspect or reproduce it. Keep the evaluator’s completion decision separate from that verification result and from the decision to accept the change.

Binding verdicts still need outcome evaluation

Liza provides an inspectable example of doer/reviewer separation backed by a deterministic supervisor. The guide’s Liza profile records code evidence at commit a22c12381c5d884d2586a48aaaa517bca184f9cf: role boundaries, task transitions and supervised integration. This is a pinned architecture review, not a claim about the latest release or measured production quality.

Those controls establish which verdicts may advance the workflow. They do not establish how often an accepted result satisfies an incomplete requirement. A separate reviewer can share the author’s assumptions. Measure false acceptances against an independent reference, then exercise detection, containment, and recovery after a false acceptance. A binding verdict gives the reviewer authority; the evaluation determines whether that authority is justified within the tested scope.

Friendly Reviewer is where this question became concrete

During a call on September 1, 2026, Marc Hugon described Friendly Reviewer as deliberately separated from the harness that produced the code. At about 00:53:28 in the transcript, he said the reviewer should be “très décorrélé” from the authoring system because its memory and context can bias the review.

That is an author’s statement of design intent, recorded in an automatic transcript that is not public. The product’s public page describes parallel agents whose findings an aggregator deduplicates. Neither source is an independent inspection of the service, and neither establishes better defect detection.

Disclosure: Marc Hugon gives me access to Friendly Reviewer. I test it on real projects, report problems, and discuss product and editorial ideas with him. The collaboration is informal, and I have not been paid to rank the product. Access is still a material relationship that readers should know about.

The product discussion raised a sound question: should the code producer also be its own judge? Answering it takes a frozen comparison and adjudicated outputs.

Greptile shows why a market label is not the result

Greptile’s independence page, accessed September 27, 2026, presents the service as “the independent code validator”, an agent-agnostic validation layer for code produced by multiple authoring agents. Its configuration reference documents repository-level rules, files to read for context, and cross-repository context settings.

These vendor statements support what Greptile says the service is designed to do. I did not inspect an execution, its effective context, or its comparative results, and the documentation does not show that its errors are independent of an authoring agent’s errors.

The same boundary applies to every reviewer marketed as independent. Verify which dimensions changed on the actual run, and treat quality as unmeasured until the output is adjudicated.

Run a frozen comparison before adding another bot

Use 1 immutable review packet for all treatments:

schemaVersion: 1
repository: example/service
baseSha: 71d4c9a
headSha: c30f610
diffSha256: "<sha256>"
criteriaVersion: review-criteria-v3
allowedContext:
  - src/auth/**
  - tests/auth/**
forbiddenContext:
  - prior-reviewer-outputs/**
limits:
  wallClockSeconds: 600
  maxCostUsd: 1.00

For every pass, retain the provider, model or tool version, prompt and rules hashes, accessible context, enabled tools, start and end times, cost when available, terminal state, and raw output. Normalize findings into stable fields such as location, severity, claim, evidence, and proposed remediation. Preserve the original output beside the normalized record.

Then adjudicate findings in a shuffled order. Hide the reviewer identity when the wording does not reveal it. The adjudicator should inspect the frozen code and criteria, not decide from the fluency of the comment.

Report 3 bounded observations:

• Unique observed finding: 1 treatment reported an adjudicated defect absent from the selected baseline outputs; this is an output-derived observation, not recall against all defects
• False signal: a treatment asserted a defect that the frozen evidence did not support
• Overlap: 2 or more treatments described the same underlying issue, even if their wording or severity differed

Do not report defects missed by all reviewers from this procedure alone. That claim needs an independent reference set assembled before reviewer outputs are revealed. Part 7 shows what 3 months of reviewer output on 1 repository can and cannot establish without one.

More reviewers can increase the wrong cost

Natural-Language Agent Harnesses, RQ2 table 3, supplies a counterexample. On its 125-task SWE-bench Verified subset, adding a verifier changed the score from 75.2% to 74.4%, although it rescued some cases the baseline missed. That is a net difference of 1 resolved task under the tested runtime and budget. In task sympy__sympy-23950, the verifier reported success while the official evaluator still failed. This small comparison motivates testing the verifier’s contribution on the intended workload; it does not establish a general penalty from independent review.

Naresh Jain’s harness-cost experiment keeps the same independent acceptance gate whether the agent receives development feedback or not. That separation is useful here: the review treatment can change while the reference deciding whether the resulting work is acceptable stays fixed. The experiment compares service-generation configurations, not code-review products.

A fleet without assigned roles often produces duplicate comments, inconsistent severity, and several summaries that a human must reconcile. Another reviewer can improve coverage while making the system slower to understand.

Assign each treatment a purpose. For example, deterministic checks own syntax, types, and executable invariants, and a contextual reviewer looks for requirement and cross-file failures. A security reviewer focuses on trust boundaries. A human owns ambiguous product trade-offs and the final disposition. The aggregator preserves disagreement instead of manufacturing a majority verdict.

Set a budget before execution, and stop when the assigned treatments finish, fail, time out, or exhaust their budget. A missing reviewer is a gap in the evidence, not an implicit vote that the change is clear. A reviewer that returns 0 findings proves only that it returned 0 findings on that input.

Part 3, Designing a bounded review loop, turns this comparison into an observable workflow with execution states, retry limits, handoff evidence, and escalation.

---

YSNK

(You should now know)

• Self-review and independent review are different experimental treatments
• Independence depends on context, tools, model, rules, budget, and authority
• Friendly Reviewer provides a declared design case, not comparative proof
• Greptile’s independence language is a vendor claim until actual runs are inspected
• Reviewer rankings require frozen inputs, blinded adjudication, and an independent reference set