Turn recurring review comments into tested controls

TL;DR

I would turn a recurring, confirmed defect into the narrowest control that can detect it, then test that control on valid alternatives as well. Another instruction is useful only when the missing context contributes to the failure.

Decision	Promotion evidence
Cluster recurring comments by cause	Human-reviewed examples, not keyword counts alone
Pick the earliest reliable control	Lint, type, test, static analysis, example, repository instruction, checklist, or architecture decision
Promote only stable corrections	Recurrence, impact, correction stability, and acceptable false-positive risk
Test before rollout	Training cases plus held-out positive and negative cases
Maintain the rule	Owner, reason, telemetry, review date, and removal path

If a reviewer writes the same accepted correction every week, the team is buying the same lesson repeatedly.

Before writing a longer prompt or adding another reviewer, ask whether this failure class can move into the earliest reliable control in the development harness.

Sometimes the answer is a linter or a test. Sometimes the comment depends on product intent and has to stay with a person. Telling the two apart takes evidence, because repetition alone does not make a rule correct.

Separate the symptom from the cause

Review comments describe symptoms at different levels:

• “Use the shared date helper” may indicate a style preference, a localization invariant, or a timezone defect
• “Check the account owner” may indicate a single missing condition or a broader access-control model failure
• “Add a test” may indicate weak coverage, an undocumented requirement, or an implementation that is difficult to exercise

Promoting the sentence directly into a rule often encodes the symptom. A string check for the shared helper may reject correct alternatives without proving timezone behavior. A prompt that says “check permissions” may reproduce the same vague judgment on every pull request.

Build a cluster around the underlying cause. Retain the original comment, diff location, accepted disposition, final code, and requirement. Ask a human reviewer to confirm that the examples belong together. A single recurring phrase can cover several causes, and a single cause can appear through many phrases.

A review can teach the production process

Yacine Hmito’s IFTTD discussion supplies a useful distinction: inspection identifies a bad piece, while improvement changes how the next piece is produced. He uses the example of defining a good unit test for an agent. Writing the instruction forces its author to make tacit quality criteria explicit; distributing that instruction does not establish that every operator can evaluate the result.

That supports 2 separate checks before promoting a review comment. Test whether the new control distinguishes the failure from a valid alternative, and check whether the people operating it understand its boundary. Hmito’s account motivates this method; it does not prove a reduction in rework for the system described here.

The cost can also move inside the automated loop. In Tokenomics, Salim and colleagues traced 30 ChatDev tasks using a GPT-5 reasoning model; the iterative code-review stage consumed an average of 59.4% of tokens. That result measures tokens in one multi-agent framework, not human review time or the cost of promoting a rule in this workflow. It motivates tracking automated review and correction as separate costs when a control is evaluated.

François Zaninotto’s Marmelab survey, based on 246 open-source harness repositories, reaches a related maintenance recommendation: test the harness itself, move executable constraints into scripts, observe usage, and remove obsolete components. Its inventory describes what those repositories contain. It does not measure the effect of the rule-promotion process proposed here.

Send each class to the right control

Birgitta Böckeler’s harness engineering framework (April 2, 2026) calls this human-led maintenance the steering loop: recurring failures inform changes to guides and sensors. It does not establish autonomous learning.

Use the narrowest mechanism that can verify the property:

Destination	Good fit	Poor fit
Formatter or linter	Syntax, naming, imports, banned APIs, local structural patterns	Business intent and cross-service behavior
Type system or schema	Invalid states, required fields, boundary shapes, exhaustive variants	Runtime authorization and temporal behavior
Unit or contract test	Deterministic behavior with stable inputs and outputs	Ambiguous product trade-offs
Static analysis	Data flow, taint, dependency, and known security patterns	Requirements absent from code and configuration
Example or fixture	A preferred pattern that developers and agents can copy	A property that must block invalid code
Repository instruction	Context, conventions, ownership, and procedures needed during work	Mechanical invariants that a command can enforce
Human checklist	Rare, high-impact questions requiring contextual judgment	Frequent deterministic checks
Architecture decision	Durable trade-offs, boundaries, and exceptions with rationale	Per-line implementation mistakes

An instruction can teach an agent where a boundary lives. A deterministic test can establish that an implementation respects an explicit property. Use both when the rule requires context and enforcement.

Place the control at the earliest loop where it has enough information. A formatter belongs inside the author’s inner loop. A contract test may run locally and again in CI. A cross-service risk decision may remain at the pull-request boundary with a named owner.

Use a promotion threshold

A candidate rule should satisfy 4 conditions before rollout:

1. Observed recurrence. The same adjudicated failure class appears across multiple changes or authors in a stated window.
2. Stable correction. Reviewers agree on the valid outcome, and the correction does not depend on an undocumented case-by-case preference.
3. Significant impact. The class causes defects, rework, security exposure, or enough repeated attention to justify maintenance.
4. Acceptable false-positive risk. The control can distinguish valid exceptions or route them through an explicit suppression process.

Do not set a universal occurrence count. A second instance of a severe authorization gap may justify a control, while 10 minor wording preferences may justify nothing. Record the threshold chosen for the failure class and the reason.

The central hypothesis is not measured yet: no linked before-and-after dataset shows that a promoted review rule reduced rework. The method below is designed to produce that evidence.

A proposed lesson still needs a source and an owner

AutoHarness separates capture, proposals that have no write access, and a single promoter that writes, which is a useful maintenance pattern to examine. On September 26, 2026, its supplied suite passed 410 tests, and 16 targeted reproductions on the pinned revision still found gaps that the evaluation record links to the code. Fabricated evidence was accepted without any link to a transcript, and creating a skill could overwrite a user-owned skill. Those cases were chosen to probe boundaries and say nothing about how often the project fails in use.

Keep observation, interpretation, candidate rule, and promoted rule distinct. Resolve contradictions against a retrievable source, name who may promote the rule and who maintains it, and validate the complete candidate before distribution.

For each recurring comment, decide whether to automate a stable invariant, retain contextual human judgment, or retire advice that adds no demonstrated value. Retention and removal both need reasons; the comment count alone supplies neither.

Improve the author and the reviewer separately

A review can reveal a code defect or a defect in the reviewing procedure. Give those observations separate improvement loops:

Target	Candidate change	Reserved evaluation	Decision evidence
Author’s harness	Rule, example, skill, or executable test addressing a confirmed code defect	Tasks held out before the candidate is tuned	Independently checked produced code, missed requirements, valid alternatives, and rework
Reviewer’s harness	Context, prompt, or procedure addressing false alerts, missed defects, or duplicates	Held-out diffs with independent annotations, including valid changes	Precision, recall where the reference set supports it, duplicate rate, and human triage cost

For a constructed authorization defect, an author-side candidate could teach where account ownership is enforced and add a behavioral test for another actor. Evaluate whether new tasks still omit the check and whether valid implementations remain possible.

Now suppose the reviewer falsely reports missing authorization because it never inspected the shared service that already enforces ownership. The candidate belongs in the review procedure: inspect the called authorization boundary before raising that finding. Evaluate it on annotated diffs with and without the protection. Requiring every caller to duplicate the service check would turn a reviewer error into unnecessary production code.

Keep separate candidate versions and results for these loops. An author improvement does not establish reviewer accuracy, and a cleaner reviewer report does not establish better produced code. Each candidate needs a named owner, a keep/revise/remove decision, and new reserved cases if its previous evaluation set becomes part of tuning.

Train on examples, validate on held-out cases

In the AI Engineer interview with Cat Wu and Thariq Shihipar, at 16:27, published July 15, 2026, the speakers describe adding PRs implicated in incidents to a code-review evaluation set so later changes can be checked for regressions. This is an attributed practice, without a public dataset or measured effect in that passage. It gives a concrete source for cases to retain, rather than treating every recurring comment as a new instruction.

Keep prompt maintenance distinct from model retraining. Song, Cai and Zhao’s When AI Reviews Its Own Code, first submitted June 26 and revised October 12, 2026, studies recursive fine-tuning on generated code. Its latest abstract reports a self-gate whose acceptance rises while benchmark correctness falls. That result concerns their retraining regimes, not the rule-promotion workflow here or today’s frontier reviewers. My proposed maintenance loop keeps externally checked cases and reserved evaluation separate from the system proposing its own improvement.

Split the corpus before implementing the rule:

• positive training cases contain the target failure
• negative training cases look similar but are valid
• held-out positive cases test whether the rule finds unseen instances
• held-out negative cases test whether it rejects valid code

Use the training subset to write the rule and its messages. Do not tune against the held-out subset after every failure. If you revise the rule after inspecting held-out results, version the dataset and reserve a new validation subset.

A minimal rule record can look like this:

schemaVersion: 1
ruleId: auth-owner-scope
owner: platform-security
reason: "Mutations must bind the target resource to the authenticated owner"
sourceCluster: review-cluster-2026-014
implementation: tests/security/owner-scope.test.ts
trainingSet: datasets/auth-owner-scope/train-v1.yaml
validationSet: datasets/auth-owner-scope/held-out-v1.yaml
approvedFalsePositiveRate: 0.02 # FP / (FP + TN) on adjudicated negative cases
reviewAfter: 2026-12-01
removal: "Delete the rule only after an approved replacement passes the same held-out set"

The rate in this example is a policy input, not a measured recommendation. Choose a threshold that matches the cost of a missed defect and the cost of interruption.

Run the candidate on both positive and negative cases. Record misses, false positives, execution failures, and runtime. A rule that catches every training example and rejects valid held-out code is not ready.

Encode the behavior, not a single acceptable shape

Consider a constructed bridge test. Requiring a flat deck rejects a safe arch; checking width, height and length can approve 2 disconnected halves. Both rules miss the purpose: safe passage under the required load. In software, a rule that mandates a specific helper name can reject a valid alternative, while a shape-only test can accept an implementation that fails the user’s task.

Write the behavioral invariant first. Include a valid alternative as a negative case for the detector and a superficially compliant defect as a positive case. If the control cannot distinguish them, narrow its claim or keep it advisory. Do not promote a recurring reviewer preference into a blocking invariant simply because it appears often.

A test promoted from a review comment needs the same check. TDD with Claude Code requires a regression test to fail on the unchanged implementation for the expected behavioral reason, then pass with the fix. A test that never failed has not shown that it detects the defect.

Keep metric denominators explicit. The fraction of alerts judged false, FP / (TP + FP), measures review noise. The false-positive rate, FP / (FP + TN), requires adjudicated negative cases, including cases where the rule stayed silent. They answer different questions and must not share a label.

Give the rule an owner and an exit

Rules age. APIs change, architecture moves, and a valid exception can become the common path. A rule without an owner becomes harness debt.

Store:

• the failure class and original evidence
• the exact property enforced
• the implementation and configuration version
• the owner authorized to change it
• the suppression policy and required justification
• telemetry fields
• the next review condition or date
• the removal or replacement procedure

A suppression is data. Capture where it occurred, who approved it, and whether the rule message or scope needs repair. Repeated unexplained suppressions can show that the control moved review noise into CI.

Low invocation frequency alone is insufficient reason to remove a control. A guard for a rare, high-impact action can remain valuable even when ordinary runs never trigger it. Distinguish undiscovered, loaded but unused, invoked and ineffective, and effective only on the exceptional path. Replay the original incident and permitted alternatives before changing its scope.

Removal needs the same discipline as addition. Run the replacement against the preserved held-out set, compare the behavior, and retain the decision. Do not delete a noisy rule without checking whether it was also preventing a high-impact failure.

Measure the meta loop

Compare equivalent windows before and after promotion. Report the change distribution and any concurrent policy change.

Measure	Definition
Recurrence rate	Adjudicated instances of the failure class per eligible change
Avoided review findings	Rule failures corrected before the reviewer would normally run
False alerts among rule failures	FP / (TP + FP): invalid rule failures divided by all adjudicated rule failures
False-positive rate on negative cases	FP / (FP + TN): false alarms divided by all adjudicated negative cases, including those that passed
Suppression rate	Approved suppressions divided by all rule failures
Escaped defects	Reference-set defects in the same class not detected before the selected boundary
Review load	Human dispositions and time spent on the class

“Avoided finding” requires an explicit counterfactual boundary. A local rule failure corrected before review shows that the rule intervened. It does not prove that a particular reviewer would have posted the same comment, so report it as an intervention count rather than as review work avoided.

An accepted review comment is not automatically a true defect. A suppression is not automatically a false positive. Adjudicate both against the rule’s declared property and the product requirement.

The meta loop succeeds when a versioned control performs better on held-out cases and the operational measures improve within the declared window. Installing the rule proves only that the rule exists.

Keep judgment where the invariant is unstable

Do not mechanize a comment when the correct answer depends on an unresolved product choice, an ethical or legal judgment, a one-off migration, or an irreversible action. A rule can ensure that a named person reviews a payment or access-control change. It should not fabricate the person’s decision.

Keep humans responsible for:

• conflicting requirements and unclear user impact
• risk acceptance and exceptions to security policy
• architectural trade-offs with several valid outcomes
• destructive migrations and external side effects
• changes where the available evidence cannot establish the invariant

The dividing line runs between stable, executable properties and contextual judgment, whoever handles them today. Reviewers surface candidates of both kinds, and the harness absorbs a property only after the control passes its own test.

Not yet demonstrated: a measured case needs a versioned comment corpus, human-reviewed clusters, at least 1 rule evaluated on held-out positive and negative cases, and a defined before-and-after window. Until those exist, reduced rework, fewer escaped defects, and lower review load remain unmeasured.

Part 6, Operating a multi-reviewer system safely, extends the same discipline to provider boundaries, untrusted inputs, normalization, budgets, incident response, and shutdown.

---

YSNK

(You should now know)

• Cluster review feedback by cause before encoding its wording
• Promote only recurring, stable, significant classes with acceptable false-positive risk
• Validate candidate controls on held-out positive and negative cases
• Give every rule an owner, telemetry, review condition, and removal path
• Preserve human authority for ambiguity, trade-offs, and irreversible decisions