Designing a bounded review loop

TL;DR

I would put the stop decision in a controller with retained inputs, counters and results. The model can propose a correction, but it should not reset its own budget or turn a missing result into a completed review.

Contract element	Required decision
Inputs	Freeze repository, base, head, diff, criteria, and budget
Execution state	End in COMPLETED, FAILED, ABORTED, TIMEOUT, or UNKNOWN
Outcome	Record CLEAR or FINDINGS only for a COMPLETED run
Stop cause	Name why the run ended, separately from the state
Rework	Cap correction iterations and stop when the evidence shows no progress
Handoff	Retain versioned inputs, configuration, findings, state, and stop cause

A review loop that asks an agent to keep reading and correcting until it feels satisfied has no end condition. This guide defines what may run, what counts as progress, what ends the run, and who decides what happens next.

The contract spans a local pass and a later CI stage. The local loop catches cheap failures before a push and emits evidence. CI consumes that evidence and runs its own controls. The handoff does not authorize a merge: provider identity, current-SHA proof, required checks, branch policy, and merge authority belong to part 4.

The workflow is vendor-neutral. A house reviewer, a CLI service, or another bounded verifier can implement the same interface. Part 8 maps the current Opus 5.5 API behavior onto it.

Admit recurring work before freezing a run

A scheduled check can select candidates for review. In Practical Loop Engineering, Addy Osmani describes an hourly check that summarizes new repository issues, followed by a bounded task. The admission policy below applies that pattern to review work. It does not assume that nesting a built-in goal command inside a scheduled prompt is supported.

Before creating a run, check that the candidate meets the queue’s eligibility rule, that its current revision has not already been processed, and that no active worker owns it. Admit work only while the shared review queue and the total budget allow another run. Record the admitted candidate with a run identifier, owner, input revision, and reserved budget, then freeze its inputs. If another check sees the same active candidate, it reports the existing run instead of launching a duplicate.

For example, 2 consecutive checks can both see the same open pull request. The second check reports the run that already covers that head revision. A new head requires fresh review evidence, and the admission policy still decides whether capacity is available. Exercise both cases before enabling automatic dispatch.

Freeze the inputs before the first pass

Every run starts with an immutable packet. At minimum, record:

• repository identity
• base and head commit SHAs
• a digest of the diff
• the requirement version or digest
• review criteria and their version
• allowed and excluded context
• deterministic commands to execute
• iteration, time, and cost budgets
• the person or role that owns escalation

If the diff changes, create a new run or a new numbered iteration. Do not replace the previous input in place. A finding attached to an earlier revision remains useful history, but it says nothing about the new head.

The requirement belongs in the packet because a reviewer can inspect code perfectly against an incomplete acceptance criterion and still approve the wrong behavior.

Revise incomplete criteria without moving the goalposts

Consider a constructed example: an export must contain all visible rows. The first tests check the column names and a single page. Work then reveals that the export silently omits later pages. Passing the original checks does not settle whether the task is complete.

Record a new requirement revision, the discovery that motivates it, the person who authorized the change, and the affected tasks and tests. Invalidate acceptance based on the superseded criteria and re-run the affected verification against the new frozen input. Budgets, role boundaries, and merge authority stay fixed unless their owner changes them. An agent may discover missing work within that policy; it may not weaken the requirement to turn a failure green.

Separate execution, outcome, and stop cause

A single status field cannot say both what the reviewer found and whether the reviewer finished. This series uses 3 fields and never merges them. Parts 4 and 6 reuse the same values.

Execution state records whether the declared procedure finished:

• COMPLETED: the procedure reached its terminal output
• FAILED: the procedure ended with a known technical, validation, or policy failure
• ABORTED: the controller or an authorized person stopped it under policy
• TIMEOUT: the wall-clock limit expired
• UNKNOWN: the retained evidence cannot establish any other state

Outcome exists only for a COMPLETED run:

• CLEAR: the reviewer returned 0 findings within its declared scope
• FINDINGS: the reviewer returned 1 or more findings

Stop cause explains why the run ended. The values used in this series are:

Stop cause	Typical execution state	Meaning
ACCEPTED_TERMINAL	COMPLETED	A valid terminal record was accepted
NO_PROGRESS	ABORTED	2 consecutive iterations met no progress condition
BUDGET_EXHAUSTED	ABORTED	Iterations, wall time, or cost reached the ceiling
OPERATOR_STOP	ABORTED	A person or kill switch stopped the run
UNAVAILABLE	FAILED	The provider was down or rate limited
REFUSED	FAILED	The model declined the request
MALFORMED_OUTPUT	FAILED	The output failed schema validation
CONTEXT_DENIED	FAILED	Required context or permissions were not available
PARTIAL_SCOPE	FAILED	The reviewer covered only part of the declared scope
WALL_CLOCK	TIMEOUT	The time limit expired
MISSING_TERMINAL_RECORD	UNKNOWN	The process ended without usable terminal evidence

A process exit code of 0 does not override a missing or malformed terminal record. A reviewer may exit 0 after its wrapper swallowed a timeout; without a valid completed record, the controller keeps TIMEOUT or UNKNOWN. CLEAR means 0 findings in the declared scope. It does not mean the change is correct. A partial scope keeps its findings as evidence, but it carries no outcome.

Model the loop as explicit transitions

Use a small state machine:

PREPARED
  -> CHECKING
  -> REVIEWING
  -> RECONCILING
  -> COMPLETED

CHECKING | REVIEWING | RECONCILING
  -> FAILED | ABORTED | TIMEOUT | UNKNOWN

RECONCILING
  -> CHECKING, only when budget remains and progress is demonstrated

The stages have distinct responsibilities:

1. Produce. The author or agent creates a candidate revision.
2. Check. Deterministic commands evaluate explicit syntax, type, test, security, or policy properties.
3. Review. A reviewer evaluates the frozen change against the criteria within its declared context.
4. Reconcile. The author accepts a finding, rejects it with evidence, or escalates the decision.
5. Stop or repeat. The controller evaluates budget and progress before authorizing another revision.

Only the transition controller changes the iteration counter. The reviewer cannot grant itself another attempt, and the author cannot turn a failed check into COMPLETED by editing the report.

Define progress before the loop runs

Keep 3 counters apart. A correction iteration is a new candidate revision after a review. A retry repeats a failed technical operation on the same revision. A continuation asks the same model to resume within 1 review pass; provider-specific rules for it belong in the adapter. The handoff example below sets maxIterations: 3, which allows the first review plus 2 correction iterations.

More activity is not progress. A correction iteration qualifies only when at least 1 of these conditions holds:

• a previously failing deterministic check now passes without a new required failure
• an accepted finding is fixed and its verification evidence is retained
• a disputed finding is resolved by the named human owner with a recorded reason
• the diff is reduced or split according to a predefined complexity rule

Rewritten comments, another model explanation, or a lower self-reported severity do not count.

When 2 consecutive iterations satisfy none of these conditions, the controller stops with NO_PROGRESS. When the maximum iterations, wall time, or cost is reached, it stops with BUDGET_EXHAUSTED. Both escalate with the latest evidence instead of asking the same reviewer to try again.

The limit of 2 correction iterations is a configuration example, not a universal optimum. Set it from your risk and cost constraints, then check how often it forces a useful escalation.

Tejas Kumar’s Harnesses in AI demonstration at 13:21, published by AI Engineer on May 17, 2026, shows the limit owned by an outer loop: no more than 3 attempts, with a deterministic verifier separate from the agent attempt. It is a small demonstration involving an upvote, not a review benchmark or evidence that 3 is an optimal limit.

A requested budget and an enforced limit need separate evidence. The controller must own the counters across restarts, enforce the process deadline and stop dispatch before its call budget is exhausted. If provider usage or pricing is unavailable, do not claim that the monetary ceiling was enforced; retain that gap and the limits actually enforced. A time clause in a model-evaluated completion condition is not a process timeout.

Make retries narrower than rework

Rework changes the candidate code or its disposition; a retry does not.

Retry only failures classified as transient, such as an unavailable endpoint or a rate-limit response that supplies a retry boundary. Cap retries, add backoff, and retain every attempt. Do not automatically retry malformed output, insufficient context, failed authentication, a refusal, or an unknown state. Those need configuration repair or a person.

TDD with Claude Code shows the cost of a retry limit without a visible stop. Its automatic test loop was capped at 10 iterations and gave up silently at the cap, until a second Stop hook added an explicit escape condition and a notification.

Failure	Execution state and stop cause	Automatic retry
Reviewer timeout	TIMEOUT, WALL_CLOCK	No, unless policy reserves 1 bounded retry
Malformed structured output	FAILED, MALFORMED_OUTPUT	No
Service temporarily unavailable	FAILED, UNAVAILABLE	At most the configured transient retry count
Required context inaccessible	FAILED, CONTEXT_DENIED	No
Model refusal	FAILED, REFUSED	No
Cost budget exceeded	ABORTED, BUDGET_EXHAUSTED	No
Process ended without terminal evidence	UNKNOWN, MISSING_TERMINAL_RECORD	No

The absence of a result never becomes CLEAR. If an advisory reviewer fails, the delivery workflow may continue under a documented degraded policy, and the review state stays visible as failed or unknown.

Resume an interrupted run without repeating its effects

After an interruption, load the last durable checkpoint and confirm the repository, base, head, criteria, and policy revisions. Verify that no stale worker still owns the task. Keep the run identifier and the remaining budget; a scheduler tick must not silently reset them. If ownership or the result cannot be established, record UNKNOWN and escalate.

External writes need more care. If a comment was published but its reply was lost, the loop cannot infer failure from an empty local receipt. Keep a stable effect identifier, reconcile it against the destination before any new attempt, and keep the publication unresolved when the destination cannot confirm it. A file rename alone does not provide exactly-once behavior across services.

LangGraph’s interrupt rules, checked on October 14, 2026, give a concrete replay hazard: resuming restarts the interrupted node, so an external write before the interrupt can execute again. The documentation recommends idempotent operations or separating side effects from the interrupt. A durable checkpoint preserves graph state; it does not, by itself, deduplicate a review comment already posted to another service. This guide has not exercised that integration.

Error messages should support that recovery. Michael Hablich’s Chrome DevTools talk at 13:14 describes rewriting tool errors so an agent can recover on its own. Applied here, a publish timeout should say whether the effect may have occurred and which reconciliation step is allowed.

Anthropic’s long-running agent account reconstructs work across sessions from an initializer script, a progress file, and Git history. That is a documented design, not proof that a new-context reviewer is independent or that external writes can be replayed safely.

The local control exercise tests permitted and rejected transitions, durable budgets, concurrent callers, and response loss against a simulated effect service. It demonstrates its own contract only. It does not validate a GitHub merge policy or a provider’s retry behavior.

Map provider responses at the adapter

Each provider has its own response shapes, and the adapter translates them into the fields above before the controller sees them. A text-only end of turn is not a completed review; a refusal returned with HTTP 200 is FAILED with REFUSED; a valid findings object with incomplete scope is FAILED with PARTIAL_SCOPE; a result produced by a configured fallback model records that model. Part 8 lists these mappings for Opus 5.5. Test each mapping with a retained fixture before connecting the adapter to publication.

Emit a portable handoff

The local stage writes a versioned manifest inside an authenticated envelope. CI validates the envelope, the manifest, and the artifacts before using any field.

schemaVersion: 1
runId: review-20260902-0042
producer:
  identity: "urn:example:workstation:review-runner-042"
  authentication:
    method: organization-keyless-dsse
    issuer: "https://identity.example.dev"
    subject: "urn:example:workstation:review-runner-042"
binding:
  repository: example/service
  baseSha: 71d4c9a
  headSha: c30f610
  criteriaVersion: review-criteria-v3
  producerIdentity: "urn:example:workstation:review-runner-042"
inputs:
  diffSha256: "<sha256>"
  requirementsSha256: "<sha256>"
configuration:
  loopPolicy: bounded-review-v1
  reviewer:
    kind: external-cli
    provider: example-reviewer
    version: "2026-09-02"
  budgets:
    maxIterations: 3
    maxWallClockSeconds: 900
    maxCostUsd: 2.00
execution:
  iteration: 2
  state: COMPLETED
  startedAt: "2026-09-02T08:40:00Z"
  endedAt: "2026-09-02T08:47:12Z"
  stopCause: ACCEPTED_TERMINAL
result:
  outcome: FINDINGS
  accepted: 2
  rejected: 1
  escalated: 0
artifacts:
  findings:
    path: findings-v1.json
    sha256: "<sha256>"
  deterministicChecks:
    path: checks-v1.json
    sha256: "<sha256>"
  rawReview:
    path: reviewer-output.txt
    sha256: "<sha256>"

A DSSE signature covers the canonical manifest, including the binding, the input digests, and every artifact digest. CI verifies the envelope first, then checks the signed binding against the expected repository and current revision, then recomputes every digest. It rejects an impossible combination such as state: TIMEOUT with outcome: CLEAR. Keep the raw reviewer output, because normalization can lose qualifiers or location detail.

Trust in the producer comes from a policy stored outside the candidate change, for example in the protected base revision. In this workstation model, an organization-controlled identity service authenticates an enrolled device before issuing a short-lived signing identity, and CI requires the authenticated subject, producer.identity, and binding.producerIdentity to match exactly. An allowlist added by the candidate head cannot authorize that same candidate. Part 6 covers the credentials involved.

Retention needs a decision too. In the repository measured in part 7, the review workflow keeps its CI artifacts for 1, 7, or 14 days. A handoff that a later audit or incident review will need has to be copied to storage with a retention you chose.

A self-supplied or unauthenticated manifest is untrusted. A person may read it as context, but it cannot skip a fresh review, satisfy a required policy, clear findings, or authorize a merge. When CI cannot authenticate the producer, it reruns the review on a trusted runner or records that no trusted handoff exists. An authenticated, fully matched artifact can spare CI an identical optional pass or display earlier findings. Authentication establishes provenance and integrity, not review quality.

Instrument the loop before optimizing it

Retain these signals for every run:

Signal	Purpose
Total and stage duration	Separate reviewer latency from local checks and human waiting
Cost by attempt	Find retries or reviewers that consume budget without progress
Iteration count	Detect unstable requirements and repeated correction cycles
Accepted findings	Show which comments affected the candidate change
Rejected findings	Inspect noise and disputed criteria without calling every rejection false
Escalations	Identify decisions that the loop cannot safely automate
Execution state and stop cause	Distinguish completion, failure, timeout, unknown, and policy stops

An accepted finding is not automatically a true defect: an author may accept a harmless suggestion to end a discussion. Quality claims need adjudication against the code and the requirement.

Running selected review before CI is a workflow hypothesis. Any reduction in remote runs, latency, or cost stays unmeasured until a comparison links equivalent changes to both local and remote execution records, failed attempts included.

Ask 4 different questions of verification

The tests that feed the controller need evaluation too. Separate the behavior being checked from the evidence that the checks can detect a defect:

Verification method	Question	Useful retained evidence
Independently approved examples	Does the implementation do what the requirement says for these cases?	Inputs and expected outcomes approved without copying the candidate implementation’s answer
Properties over generated inputs	Does an invariant hold beyond the examples already chosen?	The property, generator limits, and a replayable failing input and seed
Mutation checks	Do the tests detect an introduced fault for the intended reason?	Baseline result, mutation, tests executed, and the failing assertion or other classified outcome
Controller fixtures	Does the harness interpret the report and take the correct next action?	Valid clear report, confirmed finding, empty output, stale revision, 0 tests, and infrastructure failure

In a constructed account-update example, approved cases cover an owner and a different actor. Generated cases vary identities while keeping the rule that a non-owner cannot update the account. Removing the ownership check is a candidate mutation. Controller fixtures then verify that an empty test report cannot authorize the change. These are 4 checks of different boundaries, not 4 votes for correctness.

Start mutation evaluation from a passing baseline and a nonempty set of relevant tests, and compare candidates on the same code boundary, mutations, and exclusions. A timeout or a broken test environment has not detected the intended fault; classify it separately. A surviving mutation needs investigation, since the fault may be equivalent, outside the claimed scope, or missed by the tests.

Keep a positive fixture as well: a valid harmless change with usable evidence must be able to advance. A harness that blocks everything and a run that executes 0 relevant tests both fail to demonstrate useful verification.

Exercise failure before trusting the loop

Run the same frozen change through these scenarios:

1. Nominal completion with CLEAR
2. Nominal completion with structured FINDINGS
3. Reviewer exits with a nonzero code and a retained error
4. Reviewer exceeds its wall-clock budget
5. Reviewer exits 0 but omits the terminal manifest
6. Reviewer returns malformed findings
7. Reviewer returns a refusal, or a verdict whose text contains not approved
8. 2 correction iterations produce no defined progress
9. The cost budget is exhausted before completion
10. A receipt for an earlier revision is presented for the current head
11. A valid harmless change reaches its intended terminal result without being blocked
12. An interrupted publication is reconciled without duplicating its external effect

For each scenario, verify the execution state, stop cause, retained artifacts, retry count, and escalation target. Confirm that no missing evidence becomes CLEAR, that a verdict is read from a structured field rather than a substring, and that a later CI stage can reject a malformed handoff.

Not yet demonstrated: this guide specifies the contract. The nominal, FAILED, TIMEOUT, and UNKNOWN exercises have not yet been recorded on 1 frozen input, so treat every runtime behavior described here as a design to test in your repository.

An October 14, 2026 preparation exercise for a separate Luna/Astra coding pilot did verify an interruption through the installed Codex client against a fixed-response local server: the held TCP stream disconnected and continuation was refused. Part 8 records the scope and remaining unknowns. That local check does not validate this review controller, a live provider’s final usage counters or a hard spending limit.

A worktree keeps Git changes separate from another checkout. It does not provide process, secret, filesystem, or network isolation. Apply the controls in part 6 when the reviewer executes untrusted code or sends repository data to another provider.

Part 4, A green review is not a merge gate, explains what CI and GitHub must establish before a review signal can influence merge authority.

---

YSNK

(You should now know)

• Freeze the diff, requirement, criteria, context, and budgets before a review loop starts
• Keep execution state, outcome, and stop cause in separate fields
• Retry transient operations and escalate malformed, refused, unknown, or stalled runs
• Preserve a versioned handoff without treating it as merge authorization
• Exercise failure paths, and one permitted path, before claiming the loop works