Context engineering and code review with Bruno Soulez

TL;DR

Question from the conversation	Where I landed
What belongs in the harness?	Context, tools and controls need to fit the task, the operator and the delivery workflow
Who builds it?	Engineering expertise still matters, and product colleagues contribute business meaning
Can we stop reviewing code?	I inspect less code line by line on a web product, while keeping attention on sensitive changes; Bruno challenges the limits of that practice
What should a bug change?	Restore the service when necessary, then investigate the process and evaluate the repair
Will stronger models remove the need for controls?	Some controls may become unnecessary, while product context, delivery authority and cost still need decisions

Context, harnesses and CI

In IFTTD episode 383, published on November 4, 2026, Bruno Soulez and I start with context engineering. From there, we get into tooling, team responsibilities, code review and bugs. Whether I still read generated code comes up halfway through, once we have discussed the system around it.

The full conversation is on YouTube in French. The italicized excerpts below are translated from the supplied French transcript. I follow the conversation’s order, adding explanations and practical extensions between the excerpts.

Bruno asks: “Very concretely, how could we distinguish context engineering from harness engineering?”

In the episode, I put context engineering at the broader end: selecting the information an agent needs at the right moment. I describe harness engineering as the tools and systems that help us get acceptable technical and business quality from its output. Elsewhere, I sometimes place context inside the broader harness. The definitions still overlap. In practice, I want to know what information and tools the agent receives, and which checks apply. My Claude Code guide separates the model, runtime harness, repository harness and orchestrator by the behavior each one owns.

As I put it in the conversation, “The harness will be able to dynamically generate context adapted to the operator and the task at hand.”

That operator matters. A product colleague, a frontend developer and a backend developer can work in the same repository while needing different guidance and permissions. A profile that selects useful context does not, by itself, grant permission to act on a database or publish a change. The L0-to-L5 playbook turns those context choices into scoped rules, loading checks and shared configuration.

Bruno asks whether changes to CI also belong in the harness. I include them because locally generated code still has to enter the team’s delivery workflow. Tests, review and integration need to check how the change fits with everyone else’s work. More generated changes can move the queue into CI and multiply resource consumption. The episode does not establish a measured throughput gain or cost reduction.

In Build Evals That Actually Matter, at 35:46 on AI Engineer, Lyft’s Nick Ung and Akshay Sharma describe how an evaluation suite can be reused locally, after a prompt change and in a pre-commit hook. They also discuss CI/CD as an area they are exploring. This is a concrete example of sharing checks across a workflow, without a measured delivery gain attached to it.

His next angle is determinism: how much of a coding agent’s usefulness comes from the tools around the model? A linter can reject a mechanical violation. A test can exercise a specified behavior. A permission boundary can deny an external effect. Those tools give us specific checks, without making every output of the model deterministic.

Revisiting our internal process and harness notes brought up a useful distinction. A control can be present in the repository, have a client configuration generated from it, be loaded, run on an action and affect the protected outcome. Each claim needs checking. Generation applies to setups that distribute derived configurations; it does not mean every project needs a generator.

A hook file can exist while the active client never loads it. A loaded hook can report a warning while the workflow still permits the action. To examine a blocking claim, I would test the control and its permitted path: try a deliberately invalid case, check that the protected action stops, and verify that a valid case can proceed. Removing the control should change the invalid case’s outcome. The result would cover the boundary exercised by those tests.

The deployment notes give another example: a build can pass and a health endpoint respond while a required feature remains disabled in the configuration. Acceptance has to exercise that feature on the intended version and environment, with its preconditions. The merge-gate article explains the difference between a passing check and authority to deliver the change.

This connects with my October 30 DevSecOps compilé with Bruno Soulez, debriefing Tugdual Grall’s original episode, Why automation alone will not save security. I would ask the same question of the harness: checking generated code and limiting the agent’s access serve different purposes.

The scripts and connected tools around the model also need scrutiny: what they can execute, which credentials they receive and which resources they can reach. My guide to Claude Code’s execution surfaces develops that side of the setup. Adding verification tools also means deciding what authority to give those tools.

AgentDojo, by Debenedetti and colleagues, separates these measurements. In this evaluation environment published in 2024, malicious instructions appear in tool responses. The authors separately measure ordinary task success, task success under attack and execution of the attacker’s goal. The benchmark evaluates tool-using assistants rather than our coding harness. It offers a way to test access and unwanted effects alongside the intended behavior.

Building a harness the team can use

Bruno brings up an onboarding approach in which developers build their own coding-agent harness to understand what it does. He asks: “Does that seem to you to be becoming something of a necessity in our profession?”

I’m interested in what the developer learns by building it. I replied: “To monitor something, by definition, you have to know what to monitor.”

Someone can use a prepared workflow without knowing how to build every part of it. Creating and maintaining the workflow asks for a different level of understanding: what can fail, where the checks belong and what the tool cannot establish. A developer with deeper expertise in a particular language or testing method can improve controls that I would not know to design.

Bruno asks whether developer-experience teams will grow or everyone will have to maintain these tools. I do not know how large those teams will be. I expect technical specialists and people who understand the business to work on different parts of the system.

Bruno puts it this way: “We don’t build the code; we build what makes it possible to build the code.”

I recognize some of my previous management work here: give contributors a frame to work within, make expectations explicit and investigate where it fails. I returned to development, but the people operating the system still own what it produces.

The shift became concrete when a product teammate said they were spending time explaining changes they could make themselves. My personal harness became something the team depended on. We had to decide what context they needed, what they could decide and who would handle failures they could not diagnose.

Product knowledge also enters through that colleague. They can correct a business definition that I would misunderstand, even if they do not maintain the technical checks. A generated implementation and its tests can agree with a mistaken specification and deliver the wrong behavior. Better generation does not resolve what the requirement should mean.

I describe spending much of my time on the harness during this period. That is an account of my workload, without time records or a staffing study behind it. Someone has to maintain the shared production process while others use it; my example does not tell another organization what percentage of its team to allocate. The team instruction-system article covers the configuration and distribution mechanics.

That includes configuration choices I forget. In the episode, I describe pinning an older model version, then wondering why I did not share everyone else’s enthusiasm about the new results. Weeks later, I noticed my own configuration. Despite my fear of missing out, I had missed something I had configured myself. Someone needs to follow these changes and bring useful updates to the team without making everyone track each release.

The roles around coding agents

Bruno pushes the factory analogy further: “Does that mean you see harness engineering becoming the future of software engineering?”

I describe possible directions rather than a single replacement job, as in my map of context engineering responsibilities. Some engineers may specialize in the tools that produce and verify software. Others may work closer to product decisions, where understanding the business matters more as implementation becomes easier to obtain. Starting a new application and changing a legacy system also ask for different knowledge.

The conversation then moves into whether programming languages themselves could change. Bruno brings us back to an obstacle that also exists in today’s specifications: natural language leaves room for interpretation. As he points out, “A word won’t necessarily have the same meaning from one person to another.”

Working on an education product, with people who care about language, makes this familiar. An instruction can sound precise to its author while leaving a product decision unresolved. Confidence in the generated code does not settle that disagreement. Someone still has to define the intended behavior and check that the implementation expresses it.

We speculate about future languages and ways to express logic in the recording. That remains speculation; I have no demonstrated replacement for programming languages to present here. What I can describe today is where I put my attention and what I still need to understand about the product to decide.

I said, “My ego isn’t in the code.” That explains my willingness to move away from writing each implementation myself. It does not make expertise disappear. Architecture, business meaning, verification and the consequences of a failure still need someone who can judge them. This is the point in the conversation where Bruno turns from the shape of the profession to the practical question of reviewing its output.

The code I no longer inspect, and Bruno’s objections

Bruno asks: “So, for you, is the idea of the harness to make it so we no longer need to review the code?”

His full question goes further: could we ask an agent to generate code and push it directly to production because we no longer need to inspect it? My answer describes an existing practice, with conditions that matter as much as the claim.

I replied: “Right now, we push code to production that I don’t review, because I move my attention elsewhere. But let’s be clear: I didn’t do that on day one.”

I had reached that practice after roughly eighteen months working on AI-assisted development, with repeated corrections and mistakes along the way. I describe paying attention to design, architecture, file organization and the controls around implementation. I also say that I remain vigilant around payment work and changes that make me uneasy. I no longer systematically open every generated diff to inspect it line by line.

Bruno agrees with part of the argument, then names its boundary. Familiar web applications can leave implementation choices open when the required behavior is satisfied. In embedded systems, constraints on memory, processing and safety can make those choices part of the requirement itself.

Bruno pushes back: “There are contexts where the choice of implementation can have more or less harmful consequences.”

Working on a web product does not give every change in the repository the same consequences. Updating a public link, changing an account permission and altering payment behavior need separate judgments.

The distinction also applies inside an interface. A visual adjustment may be easy to reverse, while a form can collect sensitive information, initiate a charge or change an authorization decision. In the recording, I use a broken form as an example of something that can be corrected quickly on the web. That example needs its effects examined: a quick redeployment does not reverse every action a form may already have triggered.

I also make a broader argument that developers who still inspect everything may not have invested enough in their harness. That diagnosis goes beyond what my example establishes. A team can have substantial verification and still require detailed human review because of the consequences of its changes or gaps in its evidence. My account cannot tell that team why it retains the check.

Bruno describes the progression as gradually granting autonomy within sufficiently safe scopes. I would make that transfer explicit: which judgment moves, which controls cover it, what happens if those controls cannot run, and who accepts the remaining uncertainty. Confidence in an operator, evidence about a candidate and permission to deploy concern different decisions. The episode does not supply a controlled before-and-after comparison of escaped defects. My UVAL protocol and its review policy describe the understanding I retain on sensitive paths alongside delegated routine review.

Matt Pocock offers a useful starting point in his AI coding workflow walkthrough, at 1:11:06 on AI Engineer. Looking at a service and its tests, he explains that he would review the tests first to check that they test reasonable things, then inspect the code. His approach retains human review and puts attention on what the tests check, a question I retain as I delegate more.

Fixing the bug and the harness

Bruno’s next question follows directly from that transfer of attention. If a production defect gets through, should I repair the factory before asking it to fix the output? He asks: “Or do you start by fixing the problem before fixing what caused the problem?”

I answered: “Broadly speaking, you have to do both.”

The order depends on the consequences of waiting. In staging, where I can afford to investigate, I describe starting with the harness: what went wrong, which context was missing, whether a skill or check needs to change. In production, I correct the problem first when the service needs restoring. If the correction is obvious, I may write it myself. Investigating the process should not delay a necessary recovery.

Afterward, the defect still deserves a diagnosis. The agent may have received a stale fact, missed an instruction, lacked a tool or followed an incomplete requirement. A test may have checked the wrong behavior. Adding the same warning to every future task treats those causes as interchangeable.

I would retain the requirement, failing revision, relevant context and observed result before changing the shared setup. A mechanical invariant may belong in a test or linter; an unresolved product decision still needs a person who can decide it. An automated explanation can help investigate, while a maintainer needs to check it against the trace.

Our internal code-review notes add a constraint: evidence belongs to a particular version of the code. After a fix, a review of the previous revision does not establish the state of the new one. Before reusing a result, check that its inputs still match the claimed scope, including configuration and tool versions.

I would keep a confirmed defect separate from an inability to establish a result. An assertion that exposes incorrect behavior can identify a correction. An unavailable reviewer, incomplete test report or result for the wrong revision leaves the conclusion unknown. The person deciding what can proceed needs to see the distinction and obtain the missing evidence or make a narrower decision under explicit authority.

I also propose something I have not evaluated in a before-and-after study: after repairing the harness, replay the original implementation task. I compare it to my children learning spelling. Telling them which word contains the error makes the correction easier than asking them to write the sentence again.

I described the proposal this way: “You could almost have it replay the whole implementation, rather than just the correction.”

A guided correction tests recovery with a hint. A fresh implementation tests whether the revised setup avoids the known failure under the chosen conditions. Once I tune instructions against that task, it becomes a regression case. Reserved tasks and valid alternatives are needed to investigate whether the change helps more broadly. The guide’s section on freezing the reference cases and decision rule details how to keep those evaluation criteria separate from the candidate being improved.

The article on turning recurring review comments into tested controls develops that evaluation method. The distinction matters here because recording a repair, reproducing a failure and demonstrating an improvement require different observations. A convincing explanation of what went wrong is still something to test.

Better models and the full cost of shipping

Near the end, Bruno challenges the direction of all this tooling: “Are we in a phase where we add more and more tools to our harnesses, but will eventually remove more and more because our LLMs will be able to work without us having to control them?”

I expect some technical controls to become unnecessary or change as capabilities improve. I do not expect a general model to arrive knowing the product we have chosen to build.

I explained why that matters: “Especially when we’re innovating, when we’re a tech company making a new product: by definition, it doesn’t know that product.”

Business context still has to come from somewhere. Delivery authority also remains a decision for the people operating the system. A stronger model gives us a reason to requalify the affected behavior after an upgrade, without proving that removing a control preserves the behavior we need.

We discuss another unresolved choice: using a larger model with more reasoning, or a smaller one that may require more intervention. A slower response can lead to fewer corrections. A fast response can create additional human interruptions and CI attempts. The episode contains the question and my intended measurement approach, without a matched experiment that establishes a winner. My article on pricing an accepted task rather than a token details the accounting behind that comparison.

The Tokenomics preprint, by Salim and colleagues, gives an example of that breakdown. Across 30 tasks run with ChatDev and a GPT-5 reasoning model, the agent-to-agent code-review stage accounts for an average of 59.4% of tokens. That result is limited to this framework and model. Token types have different prices, so the proportion cannot be transferred to our bill. It gives a reason to inspect consumption after the first generation.

The contribution and cost reports I revisited help define what to measure. A trace shows an operation, without establishing who designed, reviewed and verified the entire change. A bot finding needs confirmation. Missing attribution can come from missing instrumentation. Commit counts and generated lines do not supply the missing quality comparison.

Quantity	Comparison boundary
Elapsed delivery time	Includes waiting, review and recovery; adding parallel job durations does not produce it
Compute and model usage	Includes failed, cancelled and repeated attempts, with configurations recorded
Billed amount	Needs the matching billing period and allocation; a credit purchase is not a record of consumption
Human work	Includes investigation, correction and control maintenance, even when fewer lines receive manual inspection

A lower cost per CI job can coexist with a higher total if the workflow runs more jobs or needs more attempts. Reduced consumption inside a fixed plan can free capacity while leaving the bill unchanged. I would report that capacity separately from a cash saving and keep unallocated charges visible. These are distinctions from historical cost-analysis notes, without importing their amounts or claiming a new saving.

The verification-bottleneck article defines the stage-by-stage measurement, and the bounded review-loop guide supplies the stopping and evidence contract. To investigate the question locally, I would choose one class of changes, keep acceptance criteria stable and retain the failed attempts and human interventions. The comparison also needs a decision in advance about which missing evidence or escaped failure would end the trial.

METR’s randomized study published in 2025 shows why I would want that measurement. Across 246 tasks completed by 16 experienced developers in familiar repositories, access to AI tools increased completion time by 19%, although participants believed they had saved time. They primarily used Cursor Pro and Claude 3.5/3.7. This result concerns those tools and that early-2025 setting; it does not settle the question for our harness or current models. My sense of working better is insufficient to claim a measured gain.

Bruno ends with the podcast’s usual question: spaces or tabs? My answer is shorter: spaces. For the rest, I’d like to hear from people who have reduced a specific code-review step: what convinced them, and what made them keep, restore or narrow that decision afterward.

YSNK

(You should now know)

• Bruno’s questions move from context and tools to the people responsible for building and operating them
• My reduced line-by-line review is a scoped account of practice, with Bruno’s objections and the consequences of each change still relevant
• A control’s presence, loading, execution and tested effect need their own evidence
• Correcting a defect, replaying its task and evaluating reserved cases answer different questions
• Comparing models requires accepted work, failed attempts, CI and human intervention under the same criteria