Operating a multi-reviewer system safely

TL;DR

I would start with a reviewer whose scope and permissions I can inspect, then add another only for a defined role. The operating contract below is a design to evaluate; it does not certify any vendor installation or a deployed fleet.

Operational boundary	Required control
Repository input	Treat code, comments, configuration, and tool output as untrusted
Reviewer execution	Read-only by default, bounded filesystem and network, task-scoped credentials
Aggregation	Preserve source, overlap, contradiction, omission, and execution state
Publication	Use a separate identity and explicit approval for irreversible effects
Operations	Budgets, SLOs, alerts, runbook, kill switch, rotation, rollback, and failure drills

Adding a second reviewer adds another trust boundary. Adding 3 reviewers adds several data paths, credentials, failure modes, and outputs that somebody or something must reconcile.

At the Scale stage, the task is to operate the smallest set of reviewers that covers the assigned roles while keeping every input, permission, output, and degraded state visible.

This guide covers a local house reviewer, an external CLI reviewer, and a remote pull-request service. The same control model applies whether the implementation is Friendly Reviewer, CodeRabbit, Greptile, or an internal tool. Product names identify boundaries to inspect; they do not establish that a specific installation is safe.

Grow the fleet from 1 baseline reviewer

Start with 1 house reviewer whose model, effort, prompt, and tools are pinned, and keep its session separate from the authoring session. A separate session reduces shared context and instruction exposure. Running the same model twice does not make its errors independent. Part 8 explains why Opus 5.5 is the baseline I test next.

Add a reviewer only for a stated purpose, such as an isolated security analysis, a different retrieval path, or a defect class the baseline misses. After deduplication, measure its unique verified findings, false alarms, and adjudication time, and compare the combined system’s total cost with the single-reviewer baseline. Retire a pass that shows no worthwhile contribution, unless it is an independently required control.

A fleet also drifts from its declared policy. On a repository I work on, the review policy read on September 25, 2026 classified Friendly Reviewer as historical and Codex as current. Over the 3 months ending September 28, Friendly Reviewer posted output on 899 pull requests and Codex posted nothing. Compare the policy with observed outputs on a schedule, not only when a reviewer is added.

Record the requested model and the model that actually answered each attempt. If the provider falls back to another model, the aggregate report says so. A remote service whose model is undisclosed stays a product-level treatment; its product label does not tell you which model ran.

Map the system before assigning permissions

Start with roles and flows:

Author -> immutable review packet -> orchestrator
                                      |-> local house reviewer
                                      |-> external CLI reviewer -> remote provider
                                      |-> PR service through GitHub App

reviewers -> raw outputs -> normalizer -> aggregator -> human adjudicator
                                                    -> publisher -> GitHub

Each component has a narrow responsibility:

• Author produces the candidate change and requirements
• Orchestrator freezes inputs, applies budgets, dispatches reviewers, and records execution states and stop causes
• Reviewers inspect assigned properties and return evidence without merge authority
• Normalizer maps outputs into a shared schema while preserving originals
• Aggregator clusters related findings and exposes disagreements
• Human adjudicator resolves ambiguity and owns risk acceptance
• Publisher writes approved results to GitHub with a separate credential
• GitHub stores comments, reviews, checks, and branch-policy state
• External providers receive only the data and capabilities allowed by their contract and runtime configuration

Do not combine reviewer, aggregator, publisher, and merge authority in a single model session. A prompt-injected reviewer should not be able to approve its own output or write to the repository.

Treat every review input as untrusted

Source code can contain instructions in comments, strings, documentation, fixtures, generated files, commit messages, or pull-request text. Tool output can repeat those instructions. A reviewer with shell, filesystem, network, and publication access can turn an indirect prompt injection into an external action.

Classify inputs as data. Tell the reviewer which sources may define policy and which sources may only be inspected. Version trusted review criteria outside the candidate diff or fetch them from a pinned, reviewed revision.

Use a threat model with at least these assets:

Asset	Failure to prevent
Source and proprietary context	Unauthorized provider transfer or retention
Repository credentials	Push, status, issue, or pull-request writes outside the task
Provider keys	Exfiltration, reuse, or unbounded spend
Review policy	Candidate diff changes the rules used to judge itself
Findings and adjudication	Source erased, severity altered, or disagreement hidden
CI and developer host	Untrusted code escapes the declared execution boundary

For every data-flow arrow, record producer, consumer, data classes, transport, credential, retention, region, logging, and deletion mechanism. Mark an unanswered field UNKNOWN.

Apply least privilege at every role

Reviewers should be read-only by default. Give them only the repository paths, tools, and network destinations required for the assigned review role. Use a scratch directory for outputs. Deny access to production credentials and unrelated repositories.

Separate secrets by component:

• a reviewer credential may read the selected repository revision
• a provider API key may invoke the configured review endpoint within a cost budget
• a publisher credential may create the approved comment or status but cannot modify code
• merge authority remains outside reviewer and publisher identities

Require explicit approval before any reviewer-generated fix, branch update, issue creation, status override, or merge. If automatic fixes are allowed in a controlled path, use a distinct worker identity, a new commit, deterministic validation, and the normal merge policy.

Bound network access with an allowlist and deny local metadata, internal address ranges, and unrelated services. Record attempted denied connections. A process that runs on a developer workstation is not automatically local in the data-governance sense if it sends code to a remote model.

A worktree separates Git branches and edit state. It does not isolate the process, filesystem, credentials, environment variables, or network. Put an operating-system or container boundary below the reviewer when untrusted code may execute.

Eugene Yan’s Using LLMs to Secure Source Code, at 12:40, published by AI Engineer on July 17, 2026, demonstrates a separate verification agent reproducing a suspected SQL injection in a fresh container. This illustrates the distinction between finding a plausible issue and obtaining execution evidence. It does not establish the safety of our runner, its containment against hostile code or its coverage of other vulnerability classes.

Inspect vendor boundaries instead of assuming them

Current vendor documentation can guide the inventory, but effective installation state needs inspection.

The open-source elek review mode, checked on October 14, documents read/search tools for the model and host-posted comments. Its GitHub workflow still needs permission to write PR comments; the documented legacy agent mode exposes a broader tool surface. Inspect the installed mode and host credentials separately. This is a concrete permissions design to examine, not a runtime audit or proof of reviewer independence.

CodeRabbit’s GitHub installation documentation, checked on October 14, 2026, lists read-only access to actions, discussions, members, metadata, and merge queues, and read-write access to checks, code, commit statuses, issues, pull requests, and workflows. Write access to workflows lets the app change files under .github/workflows/, so include it in the permission review. The page also describes optional review caching. Requested permissions and documented retention do not establish the grants, cache settings or actions permitted for a particular installation.

Greptile’s security page, checked on October 14, 2026 and dated January 2026, describes cloud and customer-hosted deployments, code caching until access is revoked, stored embeddings and third-party or customer-hosted inference. It states hard deletion within 24 hours after an administrator’s request and backup destruction within 30 days, with an extension during an ongoing incident investigation. Support access to chat logs can be disabled. The network-rules page did not expose its current subnet in this check, so retrieve it from the vendor before configuring an allowlist. These statements do not verify any tenant’s settings or isolation.

Friendly Reviewer’s public page, checked on October 14, 2026, describes parallel review agents and aggregation. It prices an analyzed PR or MR at $0.50, including correction validations, plus $1 to initialize the repository’s synthetic Knowledge Base. It states “No persistent source code storage” while retaining that summary. Marc Hugon described the planning and aggregation design during our September 1 call. The public description does not establish the models used, effective permissions, provider retention or process isolation of a particular review.

Disclosure: Marc Hugon gives me access to Friendly Reviewer. I test it on real projects, report problems, and discuss product and editorial ideas with him. The collaboration is informal, and I have not been paid to rank the product.

A local house reviewer is no different. Its boundaries stay unverified until you inspect its actual process tree, inherited environment, readable paths, network calls, and credential access.

Trace every execution without overstating the SHA

Use the repository and commit SHA as correlation fields across dispatch, review, aggregation, adjudication, and publication. Also retain:

• run and attempt identifiers
• reviewer role and source identity
• provider and model or tool version
• prompt, rules, and configuration digests
• allowed context and observed file coverage
• tool and network policy
• input and output artifact digests
• start, end, cost, execution state, and stop cause
• publisher identity and receipt

The SHA lets operators join records. Part 4 owns the separate proof that a review is fresh for the current pull-request head and meaningful under the branch rule. Do not turn the mere presence of a SHA field into merge evidence.

Link sessions, resources, changes, and results

A single revision field cannot describe the whole provenance chain. Retain 4 kinds of objects and explicit links between them:

Object	What it identifies	Boundary to preserve
Session	An authoring or review work session and its run attempts	Requested model versus observed model; an unavailable identity remains UNKNOWN
Resource	Instructions, skills, context, or configuration made available to that session	Source digest versus digest of the content actually loaded; availability does not prove loading
Change	A commit, pull request, or candidate integration revision	Exact input revisions and explicit links to the sessions that contributed
Result	Review output, test report, decision, or publication receipt	Producing run, input bindings, execution state, scope, and retained artifact

For a constructed PR, session S1 implements the change and session S2 repairs tests. Link both to their actual commits and to the PR. If the PR is squashed, retain the association between those original commits, the PR, and the resulting commit. A session may contribute to several commits, and a PR may contain contributions from several sessions.

If a session reference is missing, record that gap. Assigning the most recent session would manufacture provenance. Likewise, a configured model name records intent; only provider or runtime evidence can establish which model produced a particular result. If the runner does not expose the loaded resource content, its digest is unknown even when the source file can be hashed.

Keep enough metadata to reconstruct these links without retaining unrelated private prompts or user data. This is a proposed evidence contract, not a claim of complete session capture.

Normalize findings without flattening disagreement

Use a shared finding schema:

{
  "schemaVersion": 1,
  "findingId": "local-reviewer:run-42:7",
  "source": {
    "reviewer": "local-reviewer",
    "runId": "run-42",
    "attempt": 1
  },
  "location": {"path": "src/auth/switch.ts", "startLine": 84},
  "category": "authorization",
  "severity": "high",
  "confidence": "medium",
  "claim": "The mutation does not bind the target account to the actor",
  "evidence": ["src/auth/switch.ts:84", "src/policy/accounts.ts:31"],
  "proposedAction": "Verify ownership before changing session scope",
  "executionState": "COMPLETED"
}

Keep the raw output and a normalization log. The normalizer should not upgrade severity, invent evidence, or discard hedging.

Cluster findings by underlying claim and location similarity. Give every cluster these fields:

• member finding identifiers
• overlap count and source set
• contradictory claims or severities
• unique evidence contributed by each source
• expected reviewer inputs that never arrived
• adjudication status and owner

Deduplication reduces reading cost, but the aggregate still has to show that, say, 3 reviewers agreed, 1 disagreed on severity, and 1 failed before producing output. A missing output is operational data. It says nothing about what that reviewer would have found.

Bound cost, latency, and retries

Set budgets at fleet and reviewer level:

Budget	Stop condition
Maximum reviewers	Do not dispatch unassigned roles
Wall-clock time	Mark the reviewer TIMEOUT and close its attempt
Cost	Stop new paid calls before the configured ceiling
Maximum attempts	Do not turn a persistent failure into an infinite retry loop
Maximum review rounds	Escalate repeated author-reviewer disagreement
No-progress window	Stop when another round changes no accepted evidence or disposition

Retry only classified transient failures, with bounded exponential backoff and jitter. A malformed response, permission denial, missing context, or unexplained process exit needs investigation, not another identical call.

Report every reviewer that did not complete with its execution state and stop cause from part 3, for example TIMEOUT, FAILED with UNAVAILABLE or PARTIAL_SCOPE, or ABORTED with OPERATOR_STOP after a kill switch. The aggregator may finish while 1 reviewer is degraded, but the combined report names the missing source and the policy consequence. An absent review never counts as a successful vote.

Reserve attention as well as execution budget

An affordable reviewer call can still produce work that nobody is available to verify. Apply the admission rule from part 1 to the fleet: name the verifier and the capacity already reserved before dispatching more work, and treat a stale capacity signal as unknown rather than as an available slot.

The handoff to that verifier exposes the relevant revision, evidence, disagreements, unresolved questions, and permitted next actions, with links to the original findings. A shorter summary that hides contradictions can raise the real verification cost. Measure decision accuracy, recovery, human effort, and satisfaction separately.

Separate uptake from verified corrections

In Building uReview, from 05:11, Uber’s Will Bond and Ameya Ketkar describe classifying the sentiment of developer replies and then tracking an address rate. For your fleet, distinguish reactions, changes made after a finding, and verified defect correction. Include regressions introduced by accepted advice. These are proposed evaluation distinctions, not independently measured results from the talk.

Test a removal before changing the fleet

Removing a reviewer is the same experiment as adding one, run in reverse. Compare the current pipeline with a version that omits 1 reviewer on the same frozen review units, as in the protocol from part 2. Keep permissions and acceptance criteria fixed, retain failed runs, and run the comparison on an isolated path so it cannot disable a required production control. If the removed reviewer’s budget goes to another reviewer, evaluate that configuration separately, because it changes resource allocation as well as membership.

Name the risk classes each reviewer owns. A rare rescue of a severe defect and frequent duplicate style comments have different consequences. Define the acceptable loss before inspecting results; an inconclusive small sample is not evidence that removing the reviewer preserves coverage. Part 7 explains why this decision needs adjudicated findings and an independent reference set.

Operate the fleet as a service

Define service objectives from the workflow’s needs, not from vendor marketing. Useful SLO dimensions include:

• percentage of eligible runs ending with an explicit execution state and stop cause
• latency distribution to first finding and terminal result
• percentage of outputs that validate against the schema
• cost distribution per reviewed change
• publication success for approved outputs
• percentage of current-head runs with complete trace correlation
• human adjudication queue age for blocking findings

Set alerts on error budget consumption, not every isolated timeout. Alert immediately on credential misuse, unexpected network destinations, cross-repository access, publication by the wrong identity, or a sharp cost increase.

The runbook should name owners and steps for provider outage, malformed output, prompt-injection suspicion, leaked credentials, contradictory blocking findings, stuck publication, and budget exhaustion. It should include:

1. disable dispatch for the affected reviewer
2. revoke or rotate its credentials
3. preserve redacted logs, artifact hashes, and provider receipts
4. identify repositories, revisions, and data classes exposed
5. remove untrusted published output if policy permits
6. route affected reviews to the documented degraded path
7. fix or replace the component and validate it on a frozen canary
8. restore gradually and watch the error budget

Use a kill switch at 3 levels: stop 1 run, disable 1 reviewer, or disable the entire review fleet while deterministic CI and human review continue. Test the switch before an incident.

Rollback changes to prompts, rules, normalizers, and routing through versioned releases. Preserve the prior version and its evaluation artifacts. A rollback of configuration does not delete data already sent to a provider, so incident handling and provider deletion procedures remain separate.

Recover when a reviewer accepts the wrong change

A reviewer outage and an incorrect approved change need different runbooks. For an incorrect change, record how the failure is detected, how exposure is contained, which application and data state can be restored, and who owns that decision. A Git revert may restore code while leaving data mutations or external effects untouched.

Recovery action	Decision owner to name	Evidence to retain
Resume an interrupted review task	Review service operator	Checkpoint, input revisions, ownership and duplicate-effect checks
Roll back reviewer configuration	Review policy owner	Previous configuration, permissions and verification of the restored behavior
Restore an application version	Service owner or incident commander	Deployment revision, compatibility check and service health after restoration
Restore data or compensate external effects	Data or domain owner	Affected records, recovery point or compensation ledger, reconciliation outcome

Extend the drill with a deliberately accepted defective change in an isolated environment. Verify detection, containment, the chosen restoration or compensation path, and consistency afterward. Retain who made each decision and the observed result. No production recovery is demonstrated by this written procedure.

Less deep human review is defensible only within a defined risk policy supported by review-quality evidence and an exercised recovery path. Reclassify combined PRs and changed dependencies, sample accepted work, and restore closer oversight when false acceptances or recovery failures exceed that policy’s limits.

The control and recovery exercise provides a local simulation for receipts and effect replay. For a real reviewer fleet, additionally test the provider’s identity, revision binding and permissions. A passing simulation does not establish those integration properties.

Run 1 reproducible failure drill

Create a frozen canary repository containing benign files plus inert prompt-injection text. Do not include real credentials or proprietary code. Record the expected boundaries, then simulate:

• 1 reviewer timeout
• 1 malformed structured output
• 1 unavailable provider
• 1 cost-budget stop
• 2 contradictory findings
• 1 reviewer instructed by repository text to publish or reveal a secret

Verify that the orchestrator records distinct execution states and stop causes, the normalizer rejects malformed output, the aggregator preserves contradiction, the publisher performs no unapproved action, network policy blocks the forbidden destination, and the kill switch stops subsequent dispatch.

Not yet demonstrated: this guide gives the threat model and the operating contract. I have not yet produced a fleet-specific permissions matrix or failure-drill artifacts, so the effective permissions, secret injection, egress, retention, region, subprocessors, and process isolation of Friendly Reviewer, CodeRabbit, Greptile, and local runtimes remain uninspected.

For a separate Luna/Astra coding pilot, local isolation checks were recorded on October 14, 2026. They covered hidden-source access, network and subprocess restrictions, and stopping a looping process; 12 confined evaluations used operator-written controls. Part 8 describes the preparation. It establishes behavior of that local fixture runner, not the permissions or retention of this reviewer fleet. A memory ceiling and live-provider behavior remain unverified.

Part 7, 5 AI code reviewers, 1 repository, and what developers did next, measures what a real fleet produced and what developers visibly did with it.

---

YSNK

(You should now know)

• Every additional reviewer adds a trust boundary and an operational failure mode
• Reviewers should be read-only while publication and merge authority stay separate
• Normalization must preserve source, overlap, contradiction, and missing output
• Worktrees separate Git state but do not provide process or network isolation
• Production operation needs budgets, degraded states, SLOs, drills, rollback, and a kill switch