I don’t trust my AI-generated pull requests · PR #4518
I walk through Méthode Aristote’s PR #4518, from preflight to merge, then show how to replay a defect to verify a harness fix.
What you'll set up
- ✓ Tie the diff, checks and review feedback to the right commit
- ✓ Choose tests that cover the changed user journey
- ✓ Bound corrections and identify who decides whether to merge
- ✓ Replay a defect and verify the effect of a harness change
Prerequisites
- → Have opened or reviewed a pull request
At Méthode Aristote, we had started producing much more code with AI. Colleagues on the product side were submitting their own PRs and, for a long time, I was the one pressing the merge button. Even without reviewing every change the way I used to, I was struggling to keep up.
I was lucky to work there. My managers had told me to go all in, both in the product and in how we developed it. If I needed more licences, I could get them. We had the means to produce code faster, but someone still had to follow the tests, the bot feedback and the PRs waiting for a human decision.
For my talk at GenAI Days Nantes on November 17, 2026, I deliberately chose a trivial PR. We needed to move an entry up in the admin menu and rename a link. You can understand the change without knowing the whole application. It still took 3 commits and 2 corrections to merge it.
PR #4518 brings together local tests, CI, review and the rules given to agents. Along with the tools and permissions that govern their work, these make up what I call the harness. In From code to merge, I describe how we organise agents’ work as a team. Here, I go through what happened on this PR and what I would change after reviewing it.
One menu change, 3 commits
The first commit changes the menu data, but a component test still looks for the link under its old name. CI fails. The author works on the product side. I had not trained them in TDD, and I expected the agent to handle the tests. When something breaks, I also look at what is missing from the harness I prepared for them.
The author fixes the test and the story, then pushes a second commit. CI passes, but review still finds 2 discrepancies: the destination page displays a different label, and the spec still describes the old menu. The test checks the link’s text and destination in the menu. It does not check what the destination page displays. The third commit aligns the menu, the page title, its empty state and the spec.
hrefBefore merging, I check the CI results and review comments on the latest commit. Results from the second commit are not enough to verify the third. An earlier review can still help me understand an unchanged part of the diff, provided I check what has changed since and whether the base has moved.
The team merged the third commit. The final review raised searching by the old term again, but I could not find the details of the team’s decision on that suggestion.
The 3 commits and their checks in detail · Slides 06 to 15.
Run the right tests before CI
Each correction can mean another round of tests, a build and a review. We had discussed AI licences. The bill for GitHub Actions and Namespace runners was more of a surprise. I try to catch failures on the workstation, using tests that can run there, before sending the code to CI.
At Aristote, preflight selects checks from the diff. I can ask it to explain its selection, run the selected tests and produce a report. CI has its own trigger rules. PostgreSQL integration tests run remotely, and E2E tests use a separate workflow triggered on demand.
On #4518, the menu test could run locally. Yet when I replay the first diff through the selector, it selects tests for the navigation catalogue and leaves out the component test. That tells me which tests it would have selected. For the author’s execution, I have their account: they report an inconclusive preflight because of environment files. I cannot count that as a pass.
I would test 2 changes using this diff: map the menu data to the component test locally, and trigger Storybook in Actions when that data changes. I can check the selection with the first diff, then run the tests to see what they detect. I would also replay an unrelated diff to check that I am not triggering these jobs unnecessarily. These changes still need testing.
Preflight commands and reading the report · The 2 proposed selection rules · Slides 08, 16 and 17.
Fix the PR, then the harness
Once the PR is fixed, I want to stop the next one from repeating the same mistake. I can make a rule more explicit about what to check when changing the menu, or add a hook that runs a test at the right point. But to know whether I have fixed the harness, I need to return to the scenario that caused the problem.
That is why I keep as many traces as I can: the ticket, instructions, conversations when I can retrieve them, the model, and the skills and hooks used. In an independent clone of the repository, I can replay the scenario with the corrected harness and see whether the error occurs again.
Issue #3338 provides a narrower example. The CI summary was red, but did not name the failing test. #3922 adds that name and a contract test. When I replay the tests, the summary names the failing test, and the previously passing case still passes. I checked those 2 cases. Whether this summary saves time on later PRs remains to be measured.
At Aristote, I estimate that I spent at least 50% of my time changing the harness. I was doing R&D, with time and budget to compare and replay cases, while the product team worked with the tools and rules I prepared. That senior engineering time is part of the cost of working this way.
The CI summary fix and replay conditions · Slides 27 to 29.
Which PRs still need a human?
I would like roughly 2/3 of PRs to need no human intervention during review. That is the goal I state in the talk, and I am still working on routing. Eventually, I would also like the agent to decide whether to merge PRs it can assess.
To tune the classification, I asked a CTO to review PRs and identify where they had helped, where they had nothing to do, and where the AI had classified a PR incorrectly. I can use that feedback to revise the rule. Help text, a median and a calculation formula call for different checks. The label identifies the expertise needed. The diff, dependencies, CI and discussions then help determine how far the review should go.
I also want to understand the PR without starting by reading all the code. The description should explain what changes in the product, then provide the technical details, tests and local harness traces. A diagram or screenshot can help. The final description of #4518 contains that information. I do not know how the agent used it.
On a CRM for statistics, for example, I see technically correct PRs waiting for an answer about what a data point means. The question belongs with product or business colleagues, but stays in a GitLab discussion they do not have time to follow. I am trying to bring it into Slack, with the context and available choices, so they can answer and unblock the PR. At the time of the talk, I am still working on that integration.
Review bots can be wrong too. I collect their comments, group duplicates and check their findings against the diff before making corrections. On #4252, CodeRabbit spotted an overly permissive test. But its proposed patch assigned the same code to 2 different errors. Replaying the parser showed distinct codes for those errors. The suggestion remained a comment.
If I authorise the agent to make corrections, I specify what it can change, its allowed attempts and the conditions for pushing. It must hand back control if it lacks the required results, stops making progress or exhausts its budget. Today, when it spots a harness defect, it often creates an issue with its diagnosis and suggestion. I revisit those issues and compare cases before choosing what to fix and replay. I have not established that this loop ran end to end on #4518.
Business cases, reviewers and correction limits · Slides 18 to 23.
Rules in Claude Code, Cursor and Codex
At Aristote, rules start in versioned sources in the repository. I then regenerate the files for Claude Code, Cursor and Codex according to the client and workstation profile. A backend developer and a frontend developer do not necessarily need the same MCPs. A colleague new to the repository may have restrictions or questions before a commit that I do not apply to myself under the same conditions.
A hook can run at session startup or before a tool call. With RTK, for example, it can request rtk git status so the agent receives filtered output. Git hooks respond to Git operations such as a commit or pull. In the talk, I also discuss running a BM25 search when a prompt arrives, to find skills or agents relevant to the request. That lets a colleague discover part of the harness without knowing every command.
I then check where the rule ended up. If it appears in the generated file, I know it is on disk. If it appears in the session trace, I know that session loaded it. To see what difference it makes to the result, I still need to replay a case.
- Is the rule declared?
- Static diagnostics from
doctoron the project files. - Did the session load the rule?
- A session trace, such as
/contextin Claude Code. - Does the new mapping select the test?
- A proposed replay of diff A, compared with the expected selection.
On November 14, 2026, a new Claude Code session listed the pilot rule in /context, in an isolated copy of the repository. I can confirm that it was loaded. Cursor and hot reloading still need verification. Likewise, ctxharness doctor inspects project files. To find out whether the menu test would be selected, I need to replay the diff through the selector.
When a colleague does not get the expected behaviour, I can check where it stops. Does the rule apply to their task? Did the session load it? Was the check supposed to run, and did it run? I adjust the configuration using those traces, then check what happens on their workstation.
File generation, profiles, hooks and rule loading · Slides 24 to 28.
My 99% confidence in my PRs
Today, I would say I trust my PRs 99%. That is my personal assessment. I have not measured a 99% reliability rate. Before merging, I check the CI results, review feedback and the commit they refer to.
On #4518, the second commit had passing CI, but the destination page still needed a correction. If that discrepancy happens again, the team can use the case to write a check and replay it to see whether it detects the problem.
The conclusion and related guides · Slide 31.
Exercise · Replay a check on your PR
I would start with a merged PR that waited for a check or review. I check that the CI results, comments and logs are still available. Give the prompt to an agent with read access to the repository, such as Claude Code with authenticated gh, then replace [URL_PR] with the link. If a run is missing or a comment does not identify its revision, note that in the report.
Analyse this GitHub PR in read-only mode: [URL_PR].
Use gh pr view, gh pr diff and gh pr checks. Read the available review comments and CI logs.
1. Record the dates of opening, commits, checks, review and merge. Associate each result with its SHA.
2. Find the longest observable wait. Cite the traces and distinguish facts, hypotheses and missing data. Separate elapsed time from human working time.
3. Propose 1 harness improvement. Specify the measure and expected result when replaying this case and a comparable case that already passed.Read the report with the Commits and Checks tabs open. Ask the agent to distinguish SHA changes from reruns on the same SHA, and to cite the two events that bound the longest observed wait. If the report attributes that wait to a reviewer without evidence, bring it back to the interval between events. Choose one improvement, record the expected result, and replay both the failing case and a comparable passing case. The report should let someone else repeat the check.
Example report for #4518
| Field | What I would record |
|---|---|
| Scope | Revision A and the base used for the diff. |
| Trace | The Vitest job fails on A. The admin menu component test looks for the old accessible name. |
| Limit | The replay shows which tests the local selector would have selected. The author reports an inconclusive preflight. |
| Elapsed time | Unknown in this example. To calculate it, record the two events and their GitHub timestamps. |
| Change to try | Map the admin catalogue to the admin menu component test locally. |
| Expected result | Diff A selects the admin menu component test. A nearby unrelated diff keeps the same selection. I then run the test to verify the result. |
Browse all slide decks
All 50 slides from the final V11.17 deck are available here: the 31 talk slides, 15 technical appendix slides and 4 Dev With AI reference slides. Click a thumbnail to enlarge the slide within this page. Use the arrows to move to the next or previous slide. The original slides are in French, with English captions and alternative text.
The talk · 31 slides
Technical appendices · 15 slides
The 15 A slides detail CI traces, test selection, rule loading and review cases.
Dev With AI references · 4 slides
The 4 B slides revisit the agent’s conversation, sharing a rule and replaying a check.
These reference points come from Dev With AI. The response-rate examples are fictional. Marek Kalnik’s Conversation Review · Brevo’s experience · Tugdual Grall and Xavier Noya’s talk.
Check the details
These sections retain the guide’s detailed commands, examples, available traces and limits of the results. Slide links take you to the corresponding material.
Jump to a detailed section:
- What the PR waits for before merging
- The admin menu and preflight · The cost of local/CI round trips
- The three commits on #4518
- Two triggers to revisit
- Route review and bound corrections · Get the question to the business contact
- From a generated rule to its effect · The proposed harness
- Try the exercise on your PR
- Why I say 99%
- Technical appendices · Dev With AI references
Context, waiting and how I count rework
What the PR waits for before merging
Before merging, a PR may wait for a reviewer or a runner. A correction can restart checks and review.
I distinguish waiting for a reviewer from the time they spend reading the diff and addressing feedback. In CI, I make the same distinction between queue time and run duration. Review and CI can progress in parallel. When a correction arrives, I compare revisions to determine which results still cover the diff.
To count rework after a CI failure, I group eligible attempts by SHA. I count a rework cycle when the CI Success check fails at least once on one SHA and GitHub then evaluates another SHA, whether that next one passes or fails. I use the order of the first observed executions. A rerun on the same SHA is counted separately. Even if it passes, I retain the initial failure when another revision follows. I exclude attempts without usable jobs and attempts that were only cancelled. This count tells me neither why a failure occurred, how it was fixed nor how much work it took. Counting method and scope, A13.
My work at Méthode Aristote
I returned to development at Méthode Aristote after leading tech teams. Today, I help teams bring AI into their work. I maintain the Claude Code Ultimate Guide and contribute to RTK. I chose our tutoring application’s menu because you can assess the result without knowing its architecture.
At Aristote, my managers gave me considerable freedom to try AI, both in the product and in how we developed it. I could buy more licences and spend time experimenting. Product colleagues started submitting their own PRs. In this case, I had not trained the author in TDD. I expected the agent to handle the tests. As these PRs accumulated, I also needed to improve the checks surrounding their work.
At Aristote, I could spend time and budget on harness R&D, comparing and replaying cases. I estimate that this took at least 50% of my time. The product team worked with the tools and rules I prepared. That senior engineering time is part of the cost of working this way.
During the talk, I also announce that I am joining the Claude ambassador network. I plan to organise or co-organise events, mainly in Paris.
On the workstation, the Claude Code hook can call rtk git status to filter git status output before returning it to the agent. RTK open source · RTK Pro offering.
The admin menu, terminology and preflight commands
Two changes in the admin menu
#4518 moves Utilisateurs up and renames a link. The label change also needs checking in the destination page’s title and empty state.
On October 23, 2026, the PR moves Utilisateurs up in the admin and super-admin menu. It also renames the link to the list of students who moved to paid tutoring after their trial. The route, permissions and inclusion criteria stay the same. When I read the diff, I check where Utilisateurs appears, then follow the renamed link to the page. The reconstructions hide the exact labels.
The talk’s terminology
A PR proposes a change to the team before integrating it into the target branch. GitLab calls this an MR. The diff shows changes against a base. The SHA identifies the commit being assessed. Here, preflight means checks run on the workstation. Checks are the results published on the Git hosting platform. I call the instructions, tools, permissions and checks that govern agents’ work the harness.
The cost of local/CI round trips
Agent licences had been part of our budget discussions. The GitHub Actions and Namespace runner bill was more of a surprise. With PRs and their corrections, tests, builds and reviews run again. The cost adds up even on a menu change.
I try to catch these failures locally, before CI, using checks that can run on the workstation. With selection based on the diff, I also try to reserve frontend deployments for changes that warrant them. To find out whether this helps, I need to look at local time, CI waiting time, avoided runs and defects that slip through.
During the talk, I describe peaks of 80 to 100 Claude Code or Codex sessions open on my workstation, along with containers. With this workload, even my M5 Max with 128 GB of memory reaches capacity. That explains my choice of hardware.
What preflight selects on the workstation
In the Aristote repository, I run explain to see which checks preflight selects and why. quick runs the selected local checks, including targeted Vitest tests and Node contract tests. Selection can also include ESLint or typechecking. On an application diff, full runs the Vitest suite. In this setup, PostgreSQL integration tests run remotely. E2E tests use a separate workflow that the team triggers on demand.
pnpm ci:local explain --base origin/develop
pnpm ci:local quick --base origin/develop
pnpm ci:local report --format markdownThe selector reads changes since the merge-base, including staged, unstaged and untracked files. The mapping associates changed paths with their corresponding tests. The engine adds modified tests it recognises. Even with a PASS, I read the covered scope, including when the engine reuses an eligible report. FAIL means a failure, and SKIPPED means exclusion. With UNKNOWN, the engine could not reach a conclusion, for example because the base is missing or environment files prevent isolation.
I also check how many tests ran. In the Aristote preflight inspected on October 22, if the Vitest report is missing or records zero executed tests, the result becomes UNKNOWN, even when the command returns exit code 0.
I run this preflight on demand. Git hooks have their own triggers, and GitHub Actions selects jobs according to the PR workflow rules. An inconclusive local report and a separate Actions run, A01.
The three commits, their tests and the merge decision
Three commits, two corrections and a merge decision
On A, the admin menu component test fails. After the test and story are fixed on B, review identifies two discrepancies in the page and spec. C fixes them and is the commit the team merges.
Commit 1 · The admin menu component test looks for the old label
On commit A, the author moves Utilisateurs and renames the link in the menu data. When I replay this diff’s four paths through the local selector, it selects two catalogue unit test files. Those tests can check entry order and label values in the data. The Gestion component test, which can run on the workstation, is left out.
The replay shows which tests the selector selects. For the author’s execution, I have their PR report: they describe running targeted tests directly, and a quick preflight that remained inconclusive because of dotenv files.
In Actions, Vitest renders Gestion in isolation. The test looks for the link by its old accessible name, while the component already displays the new label. The job merging Vitest reports surfaces this failure. The workflow leaves out Storybook on A. Claude points out that the story also looks for the old text.
| Check | What I want it to verify for this change |
|---|---|
| Catalogue unit test | Check entry order and label values in the data. |
| Gestion component test | Render the screen in isolation, find the link by its accessible name and check its href. A Storybook interaction can check the same contract in the story. |
| Destination page component test | Render the page and check its title and empty state. |
| E2E test | Open the application with an admin account, click through the menu, then check the page actually reached and what it displays with the expected data. This journey still needs to be established for #4518. |
Looking at the screen, I also check that the label is understandable and that the page uses the same words as the link. Choose a test, A02 · Replay the initial selection, A09.
Find the component test failure in Actions
Gate reads the diff and selects jobs. Setup prepares their execution, then their results appear in the PR checks. To find the component test failure, I open CI Success on commit A and read the merged Vitest report result. When Gate excludes a job, I read the condition responsible. When an assertion fails, I open the test and diff.
On the final run, the AI context job compares facts stated in our instructions with repository sources. For example, it would flag an instruction mentioning Next.js 16.2 if package.json declares 16.3.6. On this run, the Next.js references matched the repository. CI produced a plan for Claude and Codex. I check rule loading in a session trace. The Actions job in appendix A01 · How I check loading.
Commit 2 · CI passes, review finds two discrepancies
On commit B, the author replaces the old locator in the admin menu component test and story. Both look for the new accessible name and verify the link address. The author reports 8 targeted component tests passing after the correction. Actions passes on B, including Storybook interactions. The story change triggered that job on this commit.
Review of B notes that the destination page still displays a different label and that the spec still describes the old menu. Both tests check the text and href of the link in the admin menu. This feedback leads to the second correction, in the page and spec. The test and story fix in detail, A10.
Commit 3 · The author aligns the menu, page and spec
On commit C, the author aligns the menu label, page title and message shown when the list is empty. They also update the spec. The admin menu component test covers the link. Another test covers the title and empty state. In the final description, the author reports 127 targeted unit tests and 35 Storybook interactions passing. In Actions, I find C’s passing run and the Storybook job result.
In Files changed, GitHub shows the PR diff against its base. The final PR changes 9 files, and the slide shows some of them. To reproduce the initial selection, I return to the first commit and its 4 paths. An enlarged view of the final diff, A11.
Which checks and reviews cover the commit being merged?
In Commits, I take the SHA of the head to merge. In Checks, I find the CI and Claude Code Review workflows for commit C. The develop ruleset recorded for the talk requires CI Success, gitleaks and Changelog Gate. A successful status indicates that the Claude Code Review workflow completed. I then open the review to see what Claude found on that SHA.
I check required checks and feedback that still calls for a correction or decision. On C, the final review raises searching by the old term again. The team merged this commit. In the discussions I consulted, I cannot find the details of its decision on that suggestion.
After a new commit or rebase, I compare what changed since the reviewed SHA and also check whether the base moved. A review can still help explain an unchanged part of the diff. I separately check its scope, the evidence for the new commit and whether the approval remains valid on the Git hosting platform. Check the revision to merge, A06 · Review and revision, A07 · Results of all three runs, A12.
The two proposed selector changes
Two triggers to revisit after #4518
I would clarify the journey in the instruction and test two triggers on the first diff, one locally and one in Actions.
I would ask the agent to check entry order, the link’s accessible name and its destination in the admin menu, then read the destination page’s title and empty state. In its report, I want to see which checks it could perform and learn, before review, whether it lacked admin access.
I would also change 2 selectors. The local mapping would associate the admin navigation catalogue with the admin menu component test. In Actions, menu data paths would trigger Storybook even when the story file has not changed. To verify the proposal, I would give both selectors the 4 paths from diff A and expect the link test to be selected locally and Storybook to trigger in Actions. I would also replay an unrelated diff to detect unnecessary triggers. Replaying the selector shows which checks would start. To know what they detect, I run the tests on the relevant commit.
In configuration mode, ctxharness doctor --host both --scope project inspects project files for Claude Code and Codex. To find out whether the admin menu component test or Storybook interactions would run, I replay the diff through the relevant selector. Configuration diagnostics in ctxharness v0.6.0 · Configuration, session and replay.
Routing, business cases and bounded corrections
Route review and bound corrections
I give reviewers the journey and limits, then address their feedback on the same revision. The team defines the authorisation and stopping point.
Describe the change and tests in the PR
In the summary of #4518, I look for both menu changes and who they affect. The technical block gives the entry point, tests run, preflight result and remaining checks. The reviewer can use it to guide their reading of the diff and find results to verify. The final description contains that information. I have no trace of how the agent used it.
The description is for the team and the agent picking up the work. I want to understand the product change at a glance, with a diagram of the journey or exchanges between components when that helps. I also keep the technical details, tests and local harness traces, with screenshots when the rendered result needs assessing.
Who reviews help text and a median?
Two statistics MRs on another product helped me refine our review rule. The first changed web help text. An API MR already addressed the formula. The text and rendered result needed checking, followed by a business opinion if the wording was ambiguous. After the rebase, I found an equivalence check and a passing pipeline on the new head.
The second MR displayed a median added by the API. Business colleagues needed to check what the data meant. The technical reviewer needed to follow the value across MRs, including when it was missing or zero. The frontend tests used a mocked API. After correction, the pipeline passed and the team requested another review. I have not reconstructed what happened through to the merge of that web MR.
The label identifies the expertise needed. Review depth is decided from the diff, dependencies, CI and discussions. In the recorded example, a manually run merge-gate report could recommend a full review in both cases. Someone still had to decide who would do it. When a reviewer corrects the classification, I keep their feedback so I can revisit the rule using that case. Four routing cases, A15 · Change text or a denominator, A08 · Read the conversation if the request changed along the way, B01.
If the calculation itself changes, I ask business colleagues to validate the definition and expected cases. For access permissions or a migration, I ask for review of the effects on the relevant permissions, data or integrations.
Reduce the share of PRs waiting for human review
During the talk, I state my goal: roughly 2/3 of PRs would no longer need human intervention during review. I am still working on that routing. Eventually, I would also like the agent to decide whether to merge PRs it can assess.
To tune the classification, I asked a CTO to review PRs and identify where their intervention helped, where it was unnecessary and where the AI classified a PR incorrectly. This human work gives me a basis for correcting the rule. Changing a label, changing a business definition and running a migration require different expertise.
Get the question to the person who can answer
On the statistics CRM discussed in the talk, a PR can be technically correct and still wait for an answer about what a data point means. That question belongs with product or business colleagues. It can remain stuck in a GitLab discussion they do not have time to follow.
I am trying to bring the question into Slack, with the available choices and enough context to decide. An answer about a rate’s denominator or a median’s meaning would let the agent continue. At the time of the talk, I am still working on this integration.
Read feedback with the diff open
CodeRabbit and FriendlyReviewer are two external services that publish reviews on the PR. Claude can be called on demand. On #4518, Claude spotted the story locator on the first round, then the discrepancy between labels and the spec on the second. CodeRabbit’s quota limited its first two attempts. Its final summary covered 7 of 9 files and requested no action. I found at least 2 FriendlyReviewer passes. Some comments have changed since publication, which prevents me from recounting the passes with certainty.
tech:pr-feedback can gather checks and reviews for a SHA and its base. I group duplicate feedback, verify findings in the diff and pass disagreements to the person who can decide. I do not know whether anyone used this skill on #4518. Claude’s feedback appears in the PR history. FriendlyReviewer.
I collect feedback weekly or every 15 days, in batches of 50 or 100 PRs, to identify recurring issues. I look at rework and comments about tests or security. A recurring problem gives me a case to replay and a possible change to a rule, skill or check.
When the agent finds a harness defect, it often creates an issue with its diagnosis and suggestion. I then revisit those issues and compare cases before choosing a fix. Similar reports can guide the analysis. I check their cause and the fix’s effect by replaying the cases.
When the agent makes corrections and when it hands back control
With pr-feedback-fix, I ask the agent to verify the feedback and propose a plan. Before authorising it to act, I specify what it can change, how many attempts it has and the conditions for pushing. Depending on the feedback, I may ask it to correct code, run a check to resolve uncertainty or pass the question to the business contact.
After correction, I rerun the check that can settle the issue. If the SHA or base changes, I collect results for the new revision. The agent hands the diff and evidence to the team when the check confirms the fix. It can continue while it makes progress on an authorised correction and has attempts left. If it stops progressing, cannot access evidence or exhausts its budget, it hands back control with its attempt history and open questions. Pushing comes after checks and authorisation. Approval belongs to the authorised reviewer, and the team decides whether to merge. I have not established that this process ran end to end on this case.
The versioned protocol from October 22 allows 3 pushes per task by default, with a limit the person delegating can adjust. If evidence for the new SHA is still missing after 15 minutes, the agent hands back control. The review protocol, A14.
On #4252, CodeRabbit proposes the same code for two errors
On #4252, CodeRabbit noticed that a test merely expecting an exception also accepted an unrelated problem, such as a missing binary. Its proposed patch expected a more specific error but assigned the same code to two input families. In the November 11 parser replay, staged and multi-agent returned distinct codes. 2 of 4 inputs returned a code different from the proposed one. The suggestion remained a comment.
To check that the new assertion distinguishes the cases, I would make the parser return the staged code for the multi-agent case. The test should then fail. I propose this exercise based on the parser replay results. Mutation exercise, B04.
Rules, hooks and replay cases
From a generated rule to its effect
I check that the repository generated the rule and that the session loaded it. The #3922 replay covers the defect and a previously passing case. Measurement on later PRs remains to be done.
The repository generates instructions for Claude Code, Cursor and Codex
At Aristote, I edit the rule in the repository’s versioned sources. pnpm ai:sync regenerates instructions for Claude Code, Cursor and Codex according to the client and workstation profile. During a pull with merge or rebase, Git hooks can rerun synchronisation if the guides change. They compare the starting point with the new head when that comparison is available. A branch change can also trigger synchronisation. If it fails, the hook displays a warning. In CI, ai:ctxtest checks facts stated in the instructions.
The profile also changes what I generate. A backend developer and a frontend developer do not need the same MCPs for their tasks. A colleague new to the repository may have restrictions or pre-commit questions that I do not apply to myself under the same conditions.
The profile determines the generated files and settings. I then check them on disk and in the session using them. To confirm that a protection is active, I check whether it intervenes on the relevant task.
What /context confirms in a Claude Code session
- On disk. In the common sources, I find the rule and its conditions. After generation, I check which output file contains it for the client and profile used. A generated rule example, A04 · Post-merge hook, A03.
- In the session. On November 14, in an isolated repository copy, a new Claude Code session listed the pilot rule under Memory files in
/context. I can therefore confirm that this session loaded the rule. Loading in Cursor and hot reloading still need verification. - In the result. To check behaviour, I choose a case that distinguishes two answers. The provider-context comparison script can ask how many
UserRoleroles exist and which source is authoritative, then check the number and reference against the Prisma schema. The #4518 CI run stops at the plan for Claude and Codex. Evaluating a session is a separate check.
If a colleague does not get the same behaviour, I check whether the rule applies to their task, whether the session loaded it, whether the check was supposed to run, whether it ran and what difference its result made to the decision. Distribution and replay, 28 · Find where it stops, A05 · Share a rule with its test, B02.
In CI, diagnostics concern the files available in the checkout. When reading an export, I check its format separately from the results it contains. A valid report can contain static failures.
Session hooks and Git hooks
A session hook can prepare context at startup or when a prompt is sent. Before a tool call, PreToolUse can block an action. After the call, PostToolUse can run a check. Git hooks respond to Git operations, for example before a commit or push. To find out whether a hook ran, I look for the event and its result in the session or Git output.
With RTK, the hook can request rtk git status before git status executes, so the agent receives filtered output. At the same interception point, a hook can refuse an action according to the configuration. I can verify that it ran from the event and executed script.
I also discuss a BM25 search when the prompt arrives. The request text is used to find skills or agents relevant to the task. That lets a colleague discover part of the harness without knowing every command name. I distinguish the suggestion, loading the skill and running the check. Verifying the whole sequence requires a trace at each step.
#3338 / #3922 · The CI summary names the failing test
Issue #3338 describes a red CI summary that omitted the failing test’s name. #3922 adds that name to the summary and ships a contract test. In the replay, the summary names the test in the failing case, and the previously passing case still passes. The replay covers these two cases. Elapsed time on later PRs remains to be measured.
On comparable PRs that use this summary, I would record, before and after the change, whether the name appears, how long it takes from publishing the failure to its first diagnosis, and how often reviewers need to open the logs. GitHub timestamps give elapsed time. To measure human work, I would ask reviewers to record their reading and diagnostic time. If I also change a selector such as Storybook’s, I record its trigger rate, rework on other SHAs and the cost of additional jobs.
I keep the defect trace, the replay that demonstrates it and the decision made with the team. If the defect returns or false signals accumulate, I have the material needed to adjust the configuration. Keep, adjust or remove a check, B03.
Get the instruction onto a colleague’s workstation
If the correction concerns an instruction, I change its source and regenerate the files for the relevant profile and client. On the colleague’s workstation, I look for the loaded rule, then replay the defect and a case that already worked. The team keeps, adjusts or removes the correction according to the result. At Aristote, generation is documented. Whether each client picks it up, hot reloading and time savings still need verification.
The proposed harness at Aristote
In the setup I propose, I start from versioned sources and give the agent a scope whose result the team can verify. I tie the diff, checks and reviews to the SHA and base I need to assess. If evidence is missing, I look for the blocker. An authorised person decides whether to merge.
Replay the conditions that produced the error
To understand what led to the defect, I want to keep the model used, skills, hooks and agents involved. I mention Git AI during the talk and explain that I am trying to complete that trace with harness information.
I describe an independent clone of the repository where I can revisit the ticket, instructions and, when available, colleagues’ conversations. I want to know whether the same scenario would still have produced the problem with the corrected harness. The documented #3922 replay in this guide covers the CI summary and its 2 test cases.
To evaluate a harness change, I replay the same PRs before and after, with the same model and budget. I take at least one PR with a confirmed defect and one clean PR. For each, I record the expected verdict and the evidence needed to check it. I automate checks with a determinate result and leave ambiguous judgements to a reviewer.
I record missed defects, false blocks and review time. Before keeping the change, I check whether the result holds on PRs set aside for evaluation. A script and CI report are enough to start. In the proposed setup, a log would also connect sessions, models and Git operations to track run latency and cost.
The 99% conclusion and related links
Why I say 99%
Today, I would say I trust my PRs 99%. That is my personal assessment. Before merging, I check which checks ran, cross-check review findings and make sure the evidence covers the commit I am about to accept.
On #4518, passing CI on the second commit tells me that the selected tests pass. Review of that same commit takes me to the destination page, where the label still differs. That is the uncertainty I keep in mind before merging. If the discrepancy returns, the team can use the case to write a check it can replay.
From code to merge, the Dev With AI guide · Du code au merge, its French version · My bounded review loop.
Go Further in the Claude Code Guide
Practical resources selected to help you take the next step.
Open-source galaxy

















































