Skip to main content
Home: Florian Bruniaux Engineering & AI Contact

Florian Bruniaux / Explore

Search the site

Search in English or French, or jump to a section.

↑ ↓ to navigateEnter to openEsc to close

EN
Colour theme

I don’t trust my AI-generated pull requests · PR #4518

I walk through Méthode Aristote’s PR #4518, from preflight to merge, then show how to replay a defect to verify a harness fix.

30 min read Updated

What you'll set up

  • ✓ Tie the diff, checks and review feedback to the right commit
  • ✓ Choose tests that cover the changed user journey
  • ✓ Bound corrections and identify who decides whether to merge
  • ✓ Replay a defect and verify the effect of a harness change

Prerequisites

  • → Have opened or reviewed a pull request

At Méthode Aristote, we had started producing much more code with AI. Colleagues on the product side were submitting their own PRs and, for a long time, I was the one pressing the merge button. Even without reviewing every change the way I used to, I was struggling to keep up.

I was lucky to work there. My managers had told me to go all in, both in the product and in how we developed it. If I needed more licences, I could get them. We had the means to produce code faster, but someone still had to follow the tests, the bot feedback and the PRs waiting for a human decision.

For my talk at GenAI Days Nantes on November 17, 2026, I deliberately chose a trivial PR. We needed to move an entry up in the admin menu and rename a link. You can understand the change without knowing the whole application. It still took 3 commits and 2 corrections to merge it.

PR #4518 brings together local tests, CI, review and the rules given to agents. Along with the tools and permissions that govern their work, these make up what I call the harness. In From code to merge, I describe how we organise agents’ work as a team. Here, I go through what happened on this PR and what I would change after reviewing it.

One menu change, 3 commits

The first commit changes the menu data, but a component test still looks for the link under its old name. CI fails. The author works on the product side. I had not trained them in TDD, and I expected the agent to handle the tests. When something breaks, I also look at what is missing from the harness I prepared for them.

The author fixes the test and the story, then pushes a second commit. CI passes, but review still finds 2 discrepancies: the destination page displays a different label, and the spec still describes the old menu. The test checks the link’s text and destination in the menu. It does not check what the destination page displays. The third commit aligns the menu, the page title, its empty state and the spec.

Revision B · CI passes, labels differ
GestionNew link labelAdmin menu test and story · Accessible name and href
Destination pageDifferent label on the destination pageReview · Differs from the menu. The spec still describes the old menu.
Revision C · Matching labels
GestionNew link labelAdmin menu test · Link and destination
Destination pageMatching title and empty statePage test · Title and empty state
The path from Gestion to the destination page, reconstructed from the diff and review comments. The arrows follow the menu link.

Before merging, I check the CI results and review comments on the latest commit. Results from the second commit are not enough to verify the third. An earlier review can still help me understand an unchanged part of the diff, provided I check what has changed since and whether the base has moved.

The team merged the third commit. The final review raised searching by the old term again, but I could not find the details of the team’s decision on that suggestion.

The 3 commits and their checks in detail · Slides 06 to 15.

Run the right tests before CI

Each correction can mean another round of tests, a build and a review. We had discussed AI licences. The bill for GitHub Actions and Namespace runners was more of a surprise. I try to catch failures on the workstation, using tests that can run there, before sending the code to CI.

At Aristote, preflight selects checks from the diff. I can ask it to explain its selection, run the selected tests and produce a report. CI has its own trigger rules. PostgreSQL integration tests run remotely, and E2E tests use a separate workflow triggered on demand.

On #4518, the menu test could run locally. Yet when I replay the first diff through the selector, it selects tests for the navigation catalogue and leaves out the component test. That tells me which tests it would have selected. For the author’s execution, I have their account: they report an inconclusive preflight because of environment files. I cannot count that as a pass.

I would test 2 changes using this diff: map the menu data to the component test locally, and trigger Storybook in Actions when that data changes. I can check the selection with the first diff, then run the tests to see what they detect. I would also replay an unrelated diff to check that I am not triggering these jobs unnecessarily. These changes still need testing.

Preflight commands and reading the report · The 2 proposed selection rules · Slides 08, 16 and 17.

Fix the PR, then the harness

Once the PR is fixed, I want to stop the next one from repeating the same mistake. I can make a rule more explicit about what to check when changing the menu, or add a hook that runs a test at the right point. But to know whether I have fixed the harness, I need to return to the scenario that caused the problem.

That is why I keep as many traces as I can: the ticket, instructions, conversations when I can retrieve them, the model, and the skills and hooks used. In an independent clone of the repository, I can replay the scenario with the corrected harness and see whether the error occurs again.

Issue #3338 provides a narrower example. The CI summary was red, but did not name the failing test. #3922 adds that name and a contract test. When I replay the tests, the summary names the failing test, and the previously passing case still passes. I checked those 2 cases. Whether this summary saves time on later PRs remains to be measured.

At Aristote, I estimate that I spent at least 50% of my time changing the harness. I was doing R&D, with time and budget to compare and replay cases, while the product team worked with the tools and rules I prepared. That senior engineering time is part of the cost of working this way.

The CI summary fix and replay conditions · Slides 27 to 29.

Which PRs still need a human?

I would like roughly 2/3 of PRs to need no human intervention during review. That is the goal I state in the talk, and I am still working on routing. Eventually, I would also like the agent to decide whether to merge PRs it can assess.

To tune the classification, I asked a CTO to review PRs and identify where they had helped, where they had nothing to do, and where the AI had classified a PR incorrectly. I can use that feedback to revise the rule. Help text, a median and a calculation formula call for different checks. The label identifies the expertise needed. The diff, dependencies, CI and discussions then help determine how far the review should go.

I also want to understand the PR without starting by reading all the code. The description should explain what changes in the product, then provide the technical details, tests and local harness traces. A diagram or screenshot can help. The final description of #4518 contains that information. I do not know how the agent used it.

On a CRM for statistics, for example, I see technically correct PRs waiting for an answer about what a data point means. The question belongs with product or business colleagues, but stays in a GitLab discussion they do not have time to follow. I am trying to bring it into Slack, with the context and available choices, so they can answer and unblock the PR. At the time of the talk, I am still working on that integration.

Review bots can be wrong too. I collect their comments, group duplicates and check their findings against the diff before making corrections. On #4252, CodeRabbit spotted an overly permissive test. But its proposed patch assigned the same code to 2 different errors. Replaying the parser showed distinct codes for those errors. The suggestion remained a comment.

If I authorise the agent to make corrections, I specify what it can change, its allowed attempts and the conditions for pushing. It must hand back control if it lacks the required results, stops making progress or exhausts its budget. Today, when it spots a harness defect, it often creates an issue with its diagnosis and suggestion. I revisit those issues and compare cases before choosing what to fix and replay. I have not established that this loop ran end to end on #4518.

Business cases, reviewers and correction limits · Slides 18 to 23.

Rules in Claude Code, Cursor and Codex

At Aristote, rules start in versioned sources in the repository. I then regenerate the files for Claude Code, Cursor and Codex according to the client and workstation profile. A backend developer and a frontend developer do not necessarily need the same MCPs. A colleague new to the repository may have restrictions or questions before a commit that I do not apply to myself under the same conditions.

A hook can run at session startup or before a tool call. With RTK, for example, it can request rtk git status so the agent receives filtered output. Git hooks respond to Git operations such as a commit or pull. In the talk, I also discuss running a BM25 search when a prompt arrives, to find skills or agents relevant to the request. That lets a colleague discover part of the harness without knowing every command.

I then check where the rule ended up. If it appears in the generated file, I know it is on disk. If it appears in the session trace, I know that session loaded it. To see what difference it makes to the result, I still need to replay a case.

Is the rule declared?
Static diagnostics from doctor on the project files.
Did the session load the rule?
A session trace, such as /context in Claude Code.
Does the new mapping select the test?
A proposed replay of diff A, compared with the expected selection.
I look for evidence where it is produced: in files, in the session or in the selector’s output.

On November 14, 2026, a new Claude Code session listed the pilot rule in /context, in an isolated copy of the repository. I can confirm that it was loaded. Cursor and hot reloading still need verification. Likewise, ctxharness doctor inspects project files. To find out whether the menu test would be selected, I need to replay the diff through the selector.

When a colleague does not get the expected behaviour, I can check where it stops. Does the rule apply to their task? Did the session load it? Was the check supposed to run, and did it run? I adjust the configuration using those traces, then check what happens on their workstation.

File generation, profiles, hooks and rule loading · Slides 24 to 28.

My 99% confidence in my PRs

Today, I would say I trust my PRs 99%. That is my personal assessment. I have not measured a 99% reliability rate. Before merging, I check the CI results, review feedback and the commit they refer to.

On #4518, the second commit had passing CI, but the destination page still needed a correction. If that discrepancy happens again, the team can use the case to write a check and replay it to see whether it detects the problem.

The conclusion and related guides · Slide 31.

Exercise · Replay a check on your PR

I would start with a merged PR that waited for a check or review. I check that the CI results, comments and logs are still available. Give the prompt to an agent with read access to the repository, such as Claude Code with authenticated gh, then replace [URL_PR] with the link. If a run is missing or a comment does not identify its revision, note that in the report.

Analyse this GitHub PR in read-only mode: [URL_PR].

Use gh pr view, gh pr diff and gh pr checks. Read the available review comments and CI logs.

1. Record the dates of opening, commits, checks, review and merge. Associate each result with its SHA.
2. Find the longest observable wait. Cite the traces and distinguish facts, hypotheses and missing data. Separate elapsed time from human working time.
3. Propose 1 harness improvement. Specify the measure and expected result when replaying this case and a comparable case that already passed.

Read the report with the Commits and Checks tabs open. Ask the agent to distinguish SHA changes from reruns on the same SHA, and to cite the two events that bound the longest observed wait. If the report attributes that wait to a reviewer without evidence, bring it back to the interval between events. Choose one improvement, record the expected result, and replay both the failing case and a comparable passing case. The report should let someone else repeat the check.

Example report for #4518

FieldWhat I would record
ScopeRevision A and the base used for the diff.
TraceThe Vitest job fails on A. The admin menu component test looks for the old accessible name.
LimitThe replay shows which tests the local selector would have selected. The author reports an inconclusive preflight.
Elapsed timeUnknown in this example. To calculate it, record the two events and their GitHub timestamps.
Change to tryMap the admin catalogue to the admin menu component test locally.
Expected resultDiff A selects the admin menu component test. A nearby unrelated diff keeps the same selection. I then run the test to verify the result.

Browse all slide decks

All 50 slides from the final V11.17 deck are available here: the 31 talk slides, 15 technical appendix slides and 4 Dev With AI reference slides. Click a thumbnail to enlarge the slide within this page. Use the arrows to move to the next or previous slide. The original slides are in French, with English captions and alternative text.

The talk · 31 slides

An agent-proposed PR goes through checks and review before a human decision.
Slide 01 · The talk’s cover uses navigation PR #4518 as its running example.
A four-step outline, from the menu change to improving the harness, followed by two sets of appendix slides.
Slide 02 · The talk outline, with slide numbers for each section.
Review waits for an available person, and CI waits for a runner. Both can progress in parallel.
Slide 03 · Time to merge combines queues and work. Reconstructed for this guide.
Florian Bruniaux, his background as a developer and technical leader, and the Claude Community Ambassador announcement.
Slide 04 · My work at Aristote and joining the Claude ambassador network.
The hook redirects git status to rtk git status. The slide explains where to find RTK OSS and the Pro offering.
Slide 05 · RTK shortens the output returned to the agent here. The Pro inset links to the current offering. Reconstructed for this guide.
Two reconstructed changes in the admin menu. Utilisateurs moves up, and a link is renamed without changing its destination.
Slide 06 · The journey to check consists of moving an entry and changing a label. Reconstructed for this guide.
Six cards define PR and MR, harness, preflight, checks, SHA, diff and base.
Slide 07 · The terms I use to read the PR’s results.
The workstation selects checks based on the profile and diff. Actions selects its own jobs. The replay leaves Gestion out.
Slide 08 · The local report describes a selection. Remote jobs have their own triggers.
In Actions, the admin menu component test fails because it looks for the link by its old accessible name.
Slide 09 · The first run fails on the link’s accessible name. Reconstructed for this guide.
The GitHub Actions Vitest report shows the failing admin menu component test.
Slide 10 · The Vitest report associates the failure with the first commit’s check. Reconstructed for this guide.
On revision B, the test and story check the link. Review finds the old label in the page and spec.
Slide 11 · Review follows the link to the destination page. Reconstructed for this guide.
The menu, title and empty state are aligned. Four entries from the final diff are reconstructed, and CI passes.
Slide 12 · The second correction adds the page and spec to the change. Reconstructed for this guide.
Three reconstructed revisions. A changes the menu, B fixes the locators, and C aligns the page and spec.
Slide 13 · The first correction updates the locators. The second aligns the labels and spec. Reconstructed for this guide.
The Checks tab reconstructed for revision C. CI and the review workflow each have their own results.
Slide 14 · On C, I find the jobs’ run and then the associated review. Reconstructed for this guide.
Three questions before merging: which checks ran, what remains from review, and is this the right commit?
Slide 15 · I compare check results, review feedback and the commit I need to decide on. Reconstructed for this guide.
The instruction follows the menu journey. The proposed checks cover Gestion locally and Storybook in Actions.
Slide 16 · I specify the expected result before adjusting test selection.
Two proposed triggers applied to the initial diff. The QR code opens the public ctxharness repository.
Slide 17 · Replaying the first diff’s paths should select both checks.
A reconstructed PR description with a summary, technical details, tests and preflight results.
Slide 18 · The PR description gives a summary of the change and details of the checks. Reconstructed for this guide.
Two MRs need different expertise: a text change and a median supplied by the API.
Slide 19 · Help text requires checking the rendered result. A median also needs business agreement and end-to-end review.
CodeRabbit, FriendlyReviewer and Claude cards, with a process for cross-checking feedback on the same SHA and base.
Slide 20 · The current diff helps establish whether feedback calls for a correction.
A process involving the author, checks, business contact and authorised reviewer. Merging remains manual.
Slide 21 · The loop includes a handoff when the agent can no longer progress.
After collecting feedback, the agent checks it, proposes a plan and waits for authorisation. Depending on the result, it continues, passes the issue on or hands back control.
Slide 22 · A new SHA or base triggers a fresh collection of results.
Two parser inputs return distinct codes, while the proposed patch expected the same code.
Slide 23 · CodeRabbit’s diagnosis calls for a more precise assertion. The replay distinguishes the error codes.
An adapted generation diagram. One common source feeds Claude Code, Cursor and Codex. CI compares the instructions with repository versions.
Slide 24 · The profile and client determine the generated outputs. Reconstructed for this guide.
Three rows distinguish local generation, CI validation and evaluation. An adapted context trace confirms that a pilot rule was loaded.
Slide 25 · The November 14 trace shows the rule loaded in a Claude Code session. Reconstructed for this guide.
A diagram of Claude Code and Codex session hooks, followed by Git hooks.
Slide 26 · A hook responds to a session event or a Git operation. Reconstructed for this guide.
A CI summary omits the failing test’s name. After the fix, replay confirms that the name appears and that the passing case still passes.
Slide 27 · The CI summary fix was shipped. Its effect on later PRs remains to be measured.
The corrected source passes through generated files, the session, two test cases and a team decision.
Slide 28 · Rule installation and replaying its effect are checked on the workstation concerned.
Versioned sources, agents, checks, review and a human decision. Confirmed defects feed into harness replays.
Slide 29 · The proposed setup links the merge decision to the SHA and lessons from past work to a verifiable correction.
Three steps to choose a PR, paste the prompt and replay an improvement. The QR code contains the full prompt in French.
Slide 30 · The copyable block above provides an English translation of the prompt in the QR code.
A 99% conclusion with three reference points: checks, revision and a team decision. The QR code opens Du code au merge.
Slide 31 · The QR code opens the French guide Du code au merge.

Technical appendices · 15 slides

The 15 A slides detail CI traces, test selection, rule loading and review cases.

An adapted example. A local attempt remains inconclusive. A separate Actions run executes context validations and aggregates their results.
Slide A01 · The workstation attempt and the November 10 run each have their own trace. Reconstructed for this guide.
Four ways to check the menu: unit, component and E2E tests, then a human check of the rendered result.
Slide A02 · The check depends on where the defect could be hiding.
A hook compares versions after a merge, attempts installation if needed and synchronises instructions through a separate trigger.
Slide A03 · The four reported scenarios used a fake pnpm. Pulling with the real binary still needs testing.
An excerpt from an architecture rule in the common source and in AGENTS.md after generation.
Slide A04 · I found this rule in a Codex session’s context. Hot reloading still needs verification.
Five questions check whether a rule applied, was loaded, was expected, was followed and helped the decision.
Slide A05 · I look for the first missing trace when a colleague gets a different result.
A fictional example where the commit changes or the target branch advances. The team reassesses tests, review and approval.
Slide A06 · A new head or base can change the evidence needed.
One review concerns an earlier revision. Another concerns the current PR commit.
Slide A07 · On the latest version, I distinguish an identifiable review, a bot summary and feedback I cannot tie to the diff. Reconstructed for this guide.
Two fictional examples. Changing the text leaves the rate unchanged. Excluding a cancelled case changes the rate from 8 out of 10 to 8 out of 9.
Slide A08 · Defining the denominator requires a business decision.
On A, preflight leaves out Gestion and Actions leaves out Storybook. The reported preflight remained inconclusive.
Slide A09 · The two selectors leave out different checks. Reconstructed for this guide.
On B, the admin menu component test and story look for the new label and still verify the destination.
Slide A10 · The first correction updates the locators. Reconstructed for this guide.
A reconstructed list of four of the nine files in the final diff: story, destination page, catalogue and admin menu component test.
Slide A11 · Files changed shows the final PR diff against its base. Reconstructed for this guide.
A timeline of the three commits. The first CI run fails and the next two pass. Review then identifies the page and spec discrepancies.
Slide A12 · Each CI result belongs to a different revision.
A CI Success failure followed by evaluation of another SHA, with reruns on the same SHA counted separately.
Slide A13 · I count a rework cycle when a failure is followed by evaluation of another revision. Reconstructed for this guide.
A bounded review protocol, from read-only collection to a human decision.
Slide A14 · The versioned protocol describes handing back control when results are incomplete or a limit is reached. Reconstructed for this guide.
A grid of four fictional cases. A label, a business calculation, an API field and access permissions require different review expertise.
Slide A15 · The label describes the expertise needed. Review depth is decided separately.

Dev With AI references · 4 slides

The 4 B slides revisit the agent’s conversation, sharing a rule and replaying a check.

These reference points come from Dev With AI. The response-rate examples are fictional. Marek Kalnik’s Conversation Review · Brevo’s experience · Tugdual Grall and Xavier Noya’s talk.

A fictional conversation where a request to clarify text leads to a change in the denominator.
Slide B01 · Drawing on Marek Kalnik’s Conversation Review, I check where the business decision was made.
A fictional calculation rule, with its entry point in the service and two test cases.
Slide B02 · A colleague picking up a rule can find its implementation and replay both cases.
A case compared with and without a check, followed by a decision to keep, adjust or remove it.
Slide B03 · The team’s decision records what the check contributes.
The correct parser returns two distinct codes. The mutation exercise swaps one code to check that the test fails.
Slide B04 · The contract was replayed. The mutation remains a proposal.

Check the details

These sections retain the guide’s detailed commands, examples, available traces and limits of the results. Slide links take you to the corresponding material.

Jump to a detailed section:

Context, waiting and how I count rework

What the PR waits for before merging

Before merging, a PR may wait for a reviewer or a runner. A correction can restart checks and review.

I distinguish waiting for a reviewer from the time they spend reading the diff and addressing feedback. In CI, I make the same distinction between queue time and run duration. Review and CI can progress in parallel. When a correction arrives, I compare revisions to determine which results still cover the diff.

To count rework after a CI failure, I group eligible attempts by SHA. I count a rework cycle when the CI Success check fails at least once on one SHA and GitHub then evaluates another SHA, whether that next one passes or fails. I use the order of the first observed executions. A rerun on the same SHA is counted separately. Even if it passes, I retain the initial failure when another revision follows. I exclude attempts without usable jobs and attempts that were only cancelled. This count tells me neither why a failure occurred, how it was fixed nor how much work it took. Counting method and scope, A13.

My work at Méthode Aristote

I returned to development at Méthode Aristote after leading tech teams. Today, I help teams bring AI into their work. I maintain the Claude Code Ultimate Guide and contribute to RTK. I chose our tutoring application’s menu because you can assess the result without knowing its architecture.

At Aristote, my managers gave me considerable freedom to try AI, both in the product and in how we developed it. I could buy more licences and spend time experimenting. Product colleagues started submitting their own PRs. In this case, I had not trained the author in TDD. I expected the agent to handle the tests. As these PRs accumulated, I also needed to improve the checks surrounding their work.

At Aristote, I could spend time and budget on harness R&D, comparing and replaying cases. I estimate that this took at least 50% of my time. The product team worked with the tools and rules I prepared. That senior engineering time is part of the cost of working this way.

During the talk, I also announce that I am joining the Claude ambassador network. I plan to organise or co-organise events, mainly in Paris.

On the workstation, the Claude Code hook can call rtk git status to filter git status output before returning it to the agent. RTK open source · RTK Pro offering.

The admin menu, terminology and preflight commands

Two changes in the admin menu

#4518 moves Utilisateurs up and renames a link. The label change also needs checking in the destination page’s title and empty state.

On October 23, 2026, the PR moves Utilisateurs up in the admin and super-admin menu. It also renames the link to the list of students who moved to paid tutoring after their trial. The route, permissions and inclusion criteria stay the same. When I read the diff, I check where Utilisateurs appears, then follow the renamed link to the page. The reconstructions hide the exact labels.

The talk’s terminology

A PR proposes a change to the team before integrating it into the target branch. GitLab calls this an MR. The diff shows changes against a base. The SHA identifies the commit being assessed. Here, preflight means checks run on the workstation. Checks are the results published on the Git hosting platform. I call the instructions, tools, permissions and checks that govern agents’ work the harness.

The cost of local/CI round trips

Agent licences had been part of our budget discussions. The GitHub Actions and Namespace runner bill was more of a surprise. With PRs and their corrections, tests, builds and reviews run again. The cost adds up even on a menu change.

I try to catch these failures locally, before CI, using checks that can run on the workstation. With selection based on the diff, I also try to reserve frontend deployments for changes that warrant them. To find out whether this helps, I need to look at local time, CI waiting time, avoided runs and defects that slip through.

During the talk, I describe peaks of 80 to 100 Claude Code or Codex sessions open on my workstation, along with containers. With this workload, even my M5 Max with 128 GB of memory reaches capacity. That explains my choice of hardware.

What preflight selects on the workstation

In the Aristote repository, I run explain to see which checks preflight selects and why. quick runs the selected local checks, including targeted Vitest tests and Node contract tests. Selection can also include ESLint or typechecking. On an application diff, full runs the Vitest suite. In this setup, PostgreSQL integration tests run remotely. E2E tests use a separate workflow that the team triggers on demand.

pnpm ci:local explain --base origin/develop
pnpm ci:local quick --base origin/develop
pnpm ci:local report --format markdown

The selector reads changes since the merge-base, including staged, unstaged and untracked files. The mapping associates changed paths with their corresponding tests. The engine adds modified tests it recognises. Even with a PASS, I read the covered scope, including when the engine reuses an eligible report. FAIL means a failure, and SKIPPED means exclusion. With UNKNOWN, the engine could not reach a conclusion, for example because the base is missing or environment files prevent isolation.

I also check how many tests ran. In the Aristote preflight inspected on October 22, if the Vitest report is missing or records zero executed tests, the result becomes UNKNOWN, even when the command returns exit code 0.

I run this preflight on demand. Git hooks have their own triggers, and GitHub Actions selects jobs according to the PR workflow rules. An inconclusive local report and a separate Actions run, A01.

The three commits, their tests and the merge decision

Three commits, two corrections and a merge decision

On A, the admin menu component test fails. After the test and story are fixed on B, review identifies two discrepancies in the page and spec. C fixes them and is the commit the team merges.

Commit 1 · The admin menu component test looks for the old label

On commit A, the author moves Utilisateurs and renames the link in the menu data. When I replay this diff’s four paths through the local selector, it selects two catalogue unit test files. Those tests can check entry order and label values in the data. The Gestion component test, which can run on the workstation, is left out.

The replay shows which tests the selector selects. For the author’s execution, I have their PR report: they describe running targeted tests directly, and a quick preflight that remained inconclusive because of dotenv files.

In Actions, Vitest renders Gestion in isolation. The test looks for the link by its old accessible name, while the component already displays the new label. The job merging Vitest reports surfaces this failure. The workflow leaves out Storybook on A. Claude points out that the story also looks for the old text.

CheckWhat I want it to verify for this change
Catalogue unit testCheck entry order and label values in the data.
Gestion component testRender the screen in isolation, find the link by its accessible name and check its href. A Storybook interaction can check the same contract in the story.
Destination page component testRender the page and check its title and empty state.
E2E testOpen the application with an admin account, click through the menu, then check the page actually reached and what it displays with the expected data. This journey still needs to be established for #4518.

Looking at the screen, I also check that the label is understandable and that the page uses the same words as the link. Choose a test, A02 · Replay the initial selection, A09.

Find the component test failure in Actions

Gate reads the diff and selects jobs. Setup prepares their execution, then their results appear in the PR checks. To find the component test failure, I open CI Success on commit A and read the merged Vitest report result. When Gate excludes a job, I read the condition responsible. When an assertion fails, I open the test and diff.

On the final run, the AI context job compares facts stated in our instructions with repository sources. For example, it would flag an instruction mentioning Next.js 16.2 if package.json declares 16.3.6. On this run, the Next.js references matched the repository. CI produced a plan for Claude and Codex. I check rule loading in a session trace. The Actions job in appendix A01 · How I check loading.

Commit 2 · CI passes, review finds two discrepancies

On commit B, the author replaces the old locator in the admin menu component test and story. Both look for the new accessible name and verify the link address. The author reports 8 targeted component tests passing after the correction. Actions passes on B, including Storybook interactions. The story change triggered that job on this commit.

Review of B notes that the destination page still displays a different label and that the spec still describes the old menu. Both tests check the text and href of the link in the admin menu. This feedback leads to the second correction, in the page and spec. The test and story fix in detail, A10.

Commit 3 · The author aligns the menu, page and spec

On commit C, the author aligns the menu label, page title and message shown when the list is empty. They also update the spec. The admin menu component test covers the link. Another test covers the title and empty state. In the final description, the author reports 127 targeted unit tests and 35 Storybook interactions passing. In Actions, I find C’s passing run and the Storybook job result.

In Files changed, GitHub shows the PR diff against its base. The final PR changes 9 files, and the slide shows some of them. To reproduce the initial selection, I return to the first commit and its 4 paths. An enlarged view of the final diff, A11.

Which checks and reviews cover the commit being merged?

In Commits, I take the SHA of the head to merge. In Checks, I find the CI and Claude Code Review workflows for commit C. The develop ruleset recorded for the talk requires CI Success, gitleaks and Changelog Gate. A successful status indicates that the Claude Code Review workflow completed. I then open the review to see what Claude found on that SHA.

I check required checks and feedback that still calls for a correction or decision. On C, the final review raises searching by the old term again. The team merged this commit. In the discussions I consulted, I cannot find the details of its decision on that suggestion.

After a new commit or rebase, I compare what changed since the reviewed SHA and also check whether the base moved. A review can still help explain an unchanged part of the diff. I separately check its scope, the evidence for the new commit and whether the approval remains valid on the Git hosting platform. Check the revision to merge, A06 · Review and revision, A07 · Results of all three runs, A12.

The two proposed selector changes

Two triggers to revisit after #4518

I would clarify the journey in the instruction and test two triggers on the first diff, one locally and one in Actions.

I would ask the agent to check entry order, the link’s accessible name and its destination in the admin menu, then read the destination page’s title and empty state. In its report, I want to see which checks it could perform and learn, before review, whether it lacked admin access.

I would also change 2 selectors. The local mapping would associate the admin navigation catalogue with the admin menu component test. In Actions, menu data paths would trigger Storybook even when the story file has not changed. To verify the proposal, I would give both selectors the 4 paths from diff A and expect the link test to be selected locally and Storybook to trigger in Actions. I would also replay an unrelated diff to detect unnecessary triggers. Replaying the selector shows which checks would start. To know what they detect, I run the tests on the relevant commit.

In configuration mode, ctxharness doctor --host both --scope project inspects project files for Claude Code and Codex. To find out whether the admin menu component test or Storybook interactions would run, I replay the diff through the relevant selector. Configuration diagnostics in ctxharness v0.6.0 · Configuration, session and replay.

Routing, business cases and bounded corrections

Route review and bound corrections

I give reviewers the journey and limits, then address their feedback on the same revision. The team defines the authorisation and stopping point.

Describe the change and tests in the PR

In the summary of #4518, I look for both menu changes and who they affect. The technical block gives the entry point, tests run, preflight result and remaining checks. The reviewer can use it to guide their reading of the diff and find results to verify. The final description contains that information. I have no trace of how the agent used it.

The description is for the team and the agent picking up the work. I want to understand the product change at a glance, with a diagram of the journey or exchanges between components when that helps. I also keep the technical details, tests and local harness traces, with screenshots when the rendered result needs assessing.

Who reviews help text and a median?

Two statistics MRs on another product helped me refine our review rule. The first changed web help text. An API MR already addressed the formula. The text and rendered result needed checking, followed by a business opinion if the wording was ambiguous. After the rebase, I found an equivalence check and a passing pipeline on the new head.

The second MR displayed a median added by the API. Business colleagues needed to check what the data meant. The technical reviewer needed to follow the value across MRs, including when it was missing or zero. The frontend tests used a mocked API. After correction, the pipeline passed and the team requested another review. I have not reconstructed what happened through to the merge of that web MR.

The label identifies the expertise needed. Review depth is decided from the diff, dependencies, CI and discussions. In the recorded example, a manually run merge-gate report could recommend a full review in both cases. Someone still had to decide who would do it. When a reviewer corrects the classification, I keep their feedback so I can revisit the rule using that case. Four routing cases, A15 · Change text or a denominator, A08 · Read the conversation if the request changed along the way, B01.

If the calculation itself changes, I ask business colleagues to validate the definition and expected cases. For access permissions or a migration, I ask for review of the effects on the relevant permissions, data or integrations.

Reduce the share of PRs waiting for human review

During the talk, I state my goal: roughly 2/3 of PRs would no longer need human intervention during review. I am still working on that routing. Eventually, I would also like the agent to decide whether to merge PRs it can assess.

To tune the classification, I asked a CTO to review PRs and identify where their intervention helped, where it was unnecessary and where the AI classified a PR incorrectly. This human work gives me a basis for correcting the rule. Changing a label, changing a business definition and running a migration require different expertise.

Get the question to the person who can answer

On the statistics CRM discussed in the talk, a PR can be technically correct and still wait for an answer about what a data point means. That question belongs with product or business colleagues. It can remain stuck in a GitLab discussion they do not have time to follow.

I am trying to bring the question into Slack, with the available choices and enough context to decide. An answer about a rate’s denominator or a median’s meaning would let the agent continue. At the time of the talk, I am still working on this integration.

Read feedback with the diff open

CodeRabbit and FriendlyReviewer are two external services that publish reviews on the PR. Claude can be called on demand. On #4518, Claude spotted the story locator on the first round, then the discrepancy between labels and the spec on the second. CodeRabbit’s quota limited its first two attempts. Its final summary covered 7 of 9 files and requested no action. I found at least 2 FriendlyReviewer passes. Some comments have changed since publication, which prevents me from recounting the passes with certainty.

tech:pr-feedback can gather checks and reviews for a SHA and its base. I group duplicate feedback, verify findings in the diff and pass disagreements to the person who can decide. I do not know whether anyone used this skill on #4518. Claude’s feedback appears in the PR history. FriendlyReviewer.

I collect feedback weekly or every 15 days, in batches of 50 or 100 PRs, to identify recurring issues. I look at rework and comments about tests or security. A recurring problem gives me a case to replay and a possible change to a rule, skill or check.

When the agent finds a harness defect, it often creates an issue with its diagnosis and suggestion. I then revisit those issues and compare cases before choosing a fix. Similar reports can guide the analysis. I check their cause and the fix’s effect by replaying the cases.

When the agent makes corrections and when it hands back control

With pr-feedback-fix, I ask the agent to verify the feedback and propose a plan. Before authorising it to act, I specify what it can change, how many attempts it has and the conditions for pushing. Depending on the feedback, I may ask it to correct code, run a check to resolve uncertainty or pass the question to the business contact.

After correction, I rerun the check that can settle the issue. If the SHA or base changes, I collect results for the new revision. The agent hands the diff and evidence to the team when the check confirms the fix. It can continue while it makes progress on an authorised correction and has attempts left. If it stops progressing, cannot access evidence or exhausts its budget, it hands back control with its attempt history and open questions. Pushing comes after checks and authorisation. Approval belongs to the authorised reviewer, and the team decides whether to merge. I have not established that this process ran end to end on this case.

The versioned protocol from October 22 allows 3 pushes per task by default, with a limit the person delegating can adjust. If evidence for the new SHA is still missing after 15 minutes, the agent hands back control. The review protocol, A14.

On #4252, CodeRabbit proposes the same code for two errors

On #4252, CodeRabbit noticed that a test merely expecting an exception also accepted an unrelated problem, such as a missing binary. Its proposed patch expected a more specific error but assigned the same code to two input families. In the November 11 parser replay, staged and multi-agent returned distinct codes. 2 of 4 inputs returned a code different from the proposed one. The suggestion remained a comment.

To check that the new assertion distinguishes the cases, I would make the parser return the staged code for the multi-agent case. The test should then fail. I propose this exercise based on the parser replay results. Mutation exercise, B04.

Rules, hooks and replay cases

From a generated rule to its effect

I check that the repository generated the rule and that the session loaded it. The #3922 replay covers the defect and a previously passing case. Measurement on later PRs remains to be done.

The repository generates instructions for Claude Code, Cursor and Codex

At Aristote, I edit the rule in the repository’s versioned sources. pnpm ai:sync regenerates instructions for Claude Code, Cursor and Codex according to the client and workstation profile. During a pull with merge or rebase, Git hooks can rerun synchronisation if the guides change. They compare the starting point with the new head when that comparison is available. A branch change can also trigger synchronisation. If it fails, the hook displays a warning. In CI, ai:ctxtest checks facts stated in the instructions.

The profile also changes what I generate. A backend developer and a frontend developer do not need the same MCPs for their tasks. A colleague new to the repository may have restrictions or pre-commit questions that I do not apply to myself under the same conditions.

The profile determines the generated files and settings. I then check them on disk and in the session using them. To confirm that a protection is active, I check whether it intervenes on the relevant task.

What /context confirms in a Claude Code session

  1. On disk. In the common sources, I find the rule and its conditions. After generation, I check which output file contains it for the client and profile used. A generated rule example, A04 · Post-merge hook, A03.
  2. In the session. On November 14, in an isolated repository copy, a new Claude Code session listed the pilot rule under Memory files in /context. I can therefore confirm that this session loaded the rule. Loading in Cursor and hot reloading still need verification.
  3. In the result. To check behaviour, I choose a case that distinguishes two answers. The provider-context comparison script can ask how many UserRole roles exist and which source is authoritative, then check the number and reference against the Prisma schema. The #4518 CI run stops at the plan for Claude and Codex. Evaluating a session is a separate check.

If a colleague does not get the same behaviour, I check whether the rule applies to their task, whether the session loaded it, whether the check was supposed to run, whether it ran and what difference its result made to the decision. Distribution and replay, 28 · Find where it stops, A05 · Share a rule with its test, B02.

In CI, diagnostics concern the files available in the checkout. When reading an export, I check its format separately from the results it contains. A valid report can contain static failures.

Session hooks and Git hooks

A session hook can prepare context at startup or when a prompt is sent. Before a tool call, PreToolUse can block an action. After the call, PostToolUse can run a check. Git hooks respond to Git operations, for example before a commit or push. To find out whether a hook ran, I look for the event and its result in the session or Git output.

With RTK, the hook can request rtk git status before git status executes, so the agent receives filtered output. At the same interception point, a hook can refuse an action according to the configuration. I can verify that it ran from the event and executed script.

I also discuss a BM25 search when the prompt arrives. The request text is used to find skills or agents relevant to the task. That lets a colleague discover part of the harness without knowing every command name. I distinguish the suggestion, loading the skill and running the check. Verifying the whole sequence requires a trace at each step.

#3338 / #3922 · The CI summary names the failing test

Issue #3338 describes a red CI summary that omitted the failing test’s name. #3922 adds that name to the summary and ships a contract test. In the replay, the summary names the test in the failing case, and the previously passing case still passes. The replay covers these two cases. Elapsed time on later PRs remains to be measured.

On comparable PRs that use this summary, I would record, before and after the change, whether the name appears, how long it takes from publishing the failure to its first diagnosis, and how often reviewers need to open the logs. GitHub timestamps give elapsed time. To measure human work, I would ask reviewers to record their reading and diagnostic time. If I also change a selector such as Storybook’s, I record its trigger rate, rework on other SHAs and the cost of additional jobs.

I keep the defect trace, the replay that demonstrates it and the decision made with the team. If the defect returns or false signals accumulate, I have the material needed to adjust the configuration. Keep, adjust or remove a check, B03.

Get the instruction onto a colleague’s workstation

If the correction concerns an instruction, I change its source and regenerate the files for the relevant profile and client. On the colleague’s workstation, I look for the loaded rule, then replay the defect and a case that already worked. The team keeps, adjusts or removes the correction according to the result. At Aristote, generation is documented. Whether each client picks it up, hot reloading and time savings still need verification.

The proposed harness at Aristote

In the setup I propose, I start from versioned sources and give the agent a scope whose result the team can verify. I tie the diff, checks and reviews to the SHA and base I need to assess. If evidence is missing, I look for the blocker. An authorised person decides whether to merge.

Replay the conditions that produced the error

To understand what led to the defect, I want to keep the model used, skills, hooks and agents involved. I mention Git AI during the talk and explain that I am trying to complete that trace with harness information.

I describe an independent clone of the repository where I can revisit the ticket, instructions and, when available, colleagues’ conversations. I want to know whether the same scenario would still have produced the problem with the corrected harness. The documented #3922 replay in this guide covers the CI summary and its 2 test cases.

To evaluate a harness change, I replay the same PRs before and after, with the same model and budget. I take at least one PR with a confirmed defect and one clean PR. For each, I record the expected verdict and the evidence needed to check it. I automate checks with a determinate result and leave ambiguous judgements to a reviewer.

I record missed defects, false blocks and review time. Before keeping the change, I check whether the result holds on PRs set aside for evaluation. A script and CI report are enough to start. In the proposed setup, a log would also connect sessions, models and Git operations to track run latency and cost.

The 99% conclusion and related links

Why I say 99%

Today, I would say I trust my PRs 99%. That is my personal assessment. Before merging, I check which checks ran, cross-check review findings and make sure the evidence covers the commit I am about to accept.

On #4518, passing CI on the second commit tells me that the selected tests pass. Review of that same commit takes me to the destination page, where the label still differs. That is the uncertainty I keep in mind before merging. If the discrepancy returns, the team can use the case to write a check it can replay.

From code to merge, the Dev With AI guide · Du code au merge, its French version · My bounded review loop.

Arrow keys to navigate · Escape to close