5 ways to demonstrate progress with AI agents
A test that passes for the wrong reason leads to a practical way to assess how a team scopes, verifies, explains, fixes and hands over an agent’s work.
Florian Bruniaux 9 min read The test is green. It was supposed to check that a signup form rejects an invalid email address. The service sent the address to the database anyway. The database timed out, and the test treated that error as the rejection it expected.
This is an example I put together for training. In a team review, I’d start here when deciding what responsibility to hand over next. Before widening the scope, I’d look at how the person spotted the false positive and verified the fix.
TL;DR
| What to look at | What to look for |
|---|---|
| The result | The check distinguishes the expected error from an outage, and the valid case passes |
| Progress | The person scopes, verifies, explains, fixes and hands over work within a known scope |
| The next responsibility | A different task to try, with an agreed level of help and review |
Make the right test fail
assert.rejects(() => register('adresse-invalide')) accepts any rejection. A database outage is enough to make it pass. In the exercise, the agent spots the weakness, then proposes a fix that returns the same error for every address. The invalid case passes with a more precise assertion. The valid case fails.
A fake repository that records calls to save lets us check the contract. Here, makeRegister builds the service with that repository.
const saved = []
const register = makeRegister({
async save(email) { saved.push(email) },
})
await assert.rejects(() => register('adresse-invalide'), {
code: 'EMAIL_INVALIDE',
})
assert.deepEqual(saved, [])
await register('valide@example.test')
assert.deepEqual(saved, ['valide@example.test'])I’d rerun the check on the version that skips validation. With the fake repository, the service accepts the invalid address and the test fails. With a database set up to simulate a timeout, the database error fails to match EMAIL_INVALIDE. The version that rejects every address fails on the valid case. These 2 counterexamples give reviewers something concrete to check in the diff. A sound diagnosis and a fix that still needs checking, in From Code to Merge
Look for 5 kinds of evidence in the same change
1. Scope
Before starting the agent, the developer sets the rule. The service must reject an invalid address before any write and still save a valid address. The change is limited to the service and its tests. If the error contract is unclear, the developer checks with the person who owns it. The ticket or versioned instructions give reviewers a place to find that decision.
2. Verify
The developer makes the test fail on the faulty version and reruns the check on the corrected diff. They keep the command and the revision they tested. That gives reviewers a record of the behavior checked and the code it ran against. The fake repository covers the call to save in this exercise. If the change also touches persistence, check how the real database behaves.
3. Explain
I’d ask for a prediction before changing the case. What should happen if the database goes down with a valid address? The address should pass validation, and the write can then fail. The answer shows whether the person can locate both decisions in the code. The UVAL protocol goes into how to check understanding and transfer it to another task.
4. Fix
The agent’s diff gives the developer a lead. They reproduce the rejection of a valid address, isolate the condition that matches too broadly, and put the validation back where it belongs. They rerun both cases on the fix. If the team keeps finding the same defect in reviews, it can turn that pattern into a shared check. Turn recurring review comments into tested controls
5. Hand over
A colleague starts from the specified revision with the command, the test data and the expected results. Watch where they get stuck. If they have to ask which error to inject or what order to run the tests in, the handover needs that detail. The next attempt can use a different field on the form, with a different rule and effects to check.
Choose the next responsibility
Use these scopes to choose an exercise that fits the career ladder your team already uses. A person might design tests for a familiar service and need close review on a new flow. Recording the help they received makes that difference visible, whether it came from the agent or a colleague.
Fill in the rubric during review
Take a real change and use one row per capability you observed. Here is a filled row for the signup scenario.
| Capability | Observed behavior | Evidence to review | Scope and help received | Limit | Next attempt |
|---|---|---|---|---|---|
| Verify | Made the check fail on the faulty version and verified the valid case after the fix | Diff revision, command and output from both runs | Signup with a fake repository. The agent flagged the broad assertion. The lead helped distinguish the errors | Real persistence needs an integration test | Choose a check for a different form, then ask a colleague to rerun it |
At the end of the review, choose a responsibility to try on the next change and agree on the level of help. That’s also the starting point for the advanced training course, where the exercise is repeated on one of the team’s own tasks.
YSNK
(You should now know)
- A generic rejection test can pass on a database error
- Checking both an invalid address and a valid one catches a fix that rejects everything
- Look for the 5 kinds of evidence in decisions, checks and handovers
- The tested revision, scope and help received give the observation context
- Try the next responsibility on a different task
Go Further in the Claude Code Guide
Practical resources selected to help you take the next step.