1/6 · Opposite AI results: what context can explain

TL;DR

I think context explains part of the gap between Anthropic’s 67% more merged PRs and METR’s 19% longer tasks, as one factor among others, and these two studies cannot say how much. Context belongs inside the evaluated model-and-harness pair, so I expect context engineering to be absorbed by harness engineering as a discipline. Within two years I still expect “context engineer” to exist as a job title, because the discipline will still need an owner.

Two observations	Anthropic: more merged PRs per engineer per day. METR: longer task completion time
Evidence limit	Different populations, tasks and units; the studies do not isolate context as the explanation
Long context	Performance can vary with model, task, input length and information position
The shift	Prompt design sits inside the wider work of selecting information, tools and task state
Practical target	Preserve required evidence, exclude irrelevant detail, then verify quality and cost

Anthropic reported 67% more pull requests merged per engineer per day after Claude Code adoption. In a separate METR randomized trial, 16 experienced open-source developers working on 246 tasks took 19% longer with the early-2025 AI tools allowed in the experiment.

These are different measurements in different settings. Merged PR throughput is not task duration, and an internal observation is not a randomized comparison. Neither study establishes that the participants used the same model, stack or context system. They do not show that METR participants started from an empty project context each morning.

My hypothesis is that the way a team assembles context explains part of such differences, as one factor among others. Testing that explanation requires a comparison that holds the model and task conditions stable while changing the context setup. These two studies do not supply that experiment.

---

What context rot actually is

Chroma Research’s July 2025 study tested 18 models while varying input length on controlled tasks. It found that longer inputs can reduce performance even when the necessary information remains present. The size and shape of the effect vary with the model and task; there is no single accuracy curve that describes every session.

The important distinction is between accepting an input and reliably using it. A larger advertised window tells you what fits. It does not, on its own, tell you whether a model will answer your particular question correctly.

Liu et al.’s Lost in the Middle also found sensitivity to the position of relevant information in multi-document question answering and key-value retrieval. Those results support testing placement and length. They do not establish a universal token threshold, a fixed reliable zone, or an architectural limit that no model improvement can change.

---

Capacity and effective use are different measurements

For a useful evaluation, name the model, the task, the input lengths and the acceptance criterion. Test enough examples to see whether the result holds beyond one successful answer. A task-specific effective window can then describe where that criterion still holds.

This does not prove that a managed 128K window beats an unmanaged 1M window on most real work. A larger window may be necessary when the evidence itself is large; context selection still needs its own evaluation.

The practical question is what information this task requires and how the model uses it, rather than which advertised capacity looks largest.

---

Prompt engineering versus context engineering

I use context engineering here to mean the design of the information, instructions, tools and state available to a model while it works. Prompt engineering is part of that job: it formulates the request and its constraints. Context engineering also addresses what the request relies on.

Dimension	Prompt engineering	Context engineering
Scope	One request	A session or a system
Lifespan	Can be reused or revised	Assembled for tasks and maintained across interactions
Effort	Design and test the request	Design and test what the request relies on
Artifact	A request and constraints	Selected information, tools, instructions and state

Prompt engineering is scribbling a good note to a contractor. Context engineering covers the onboarding, style guide, architecture docs, and team norms that make the contractor understand the project before reading a single note. One is a sentence. The other is a building. Prompt engineering is a subset of context engineering, and teams stuck on output quality often craft better sentences when they need to construct the building. A more capable model in the same underbuilt context can actively make results worse. Guillaume Laforge described a “mansplaining” effect where a sufficiently capable model ignores injected context and generates from its training priors instead, producing confident but contextually wrong answers (ep 361). The full discipline, its budget, hierarchy, and lifecycle, sits in the guide’s context engineering reference.

---

Keep sufficient evidence, remove irrelevant detail

A useful context must preserve what the task needs: requirements, constraints, relevant code, uncertainty and evidence. Removing those details can make the result worse. Keeping every available document can also make the task harder to follow. The goal is sufficient context for this task, not the smallest possible input.

Quality and cost therefore need separate observations. Check the result against the task’s acceptance criteria. Compare costs using the same billing surface and workload; a local token reduction is not automatically a smaller subscription bill or a proportional reduction in total task cost.

The rest of this series follows both questions: how to maintain and verify useful configuration, and how to measure what token-reduction tools change. Less context is an intervention to test, not a guarantee of fewer hallucinations.

---

Three checks before adding tools

Three things are worth doing before you touch a single tool.

• Record the model, task and context conditions when a session begins producing inconsistent or off-spec output.
• Try removing irrelevant material while preserving the evidence needed to verify the task; compare the result with the original setup.
• Separate output quality, local token counts, account limits and billed cost before choosing an optimization.

The next article goes after the quality pressure directly: a maturity model for context configuration, why a CLAUDE.md that works on day one starts lying about your project three months later, and the verification loop that catches it. The third maps the tooling: the map of token-reduction mechanisms, and the tools behind each.

From the field, via IFTTD episode transcripts: Guillaume Laforge, ep 361 on model capability vs context grounding; Quentin Adam, ep 341 on multi-model orchestration by context-window size.

If you’ve watched the same model give your team and another team opposite results, I’m curious what you think the variable was. Tell me where this matches what you’ve seen and where it doesn’t.

---

Context belongs inside the evaluated model-and-harness pair. The harness guide connects it to tools, permissions, recovery and acceptance. Compare those changes on the same tasks and budgets before assigning a gain to context alone.

YSNK

(You should now know)

• Anthropic’s merged-PR observation and METR’s task-duration experiment have different units and populations
• Those studies do not isolate context architecture as the cause of their different results
• Long-context evaluations need a named model, task, input conditions and acceptance criterion
• Capacity, information position and reliable use are separate questions; no universal reliable-zone threshold follows from the studies cited here
• Keep required evidence, remove irrelevant detail and measure the result before claiming an improvement