All four times, the tests were green

01

The same failure in four shapes

Only after writing all four did the pattern become visible. None of them was a story about fixing a bug. The code largely behaved as written, and both the unit and regression suites were passing.

The problem sat in a different layer. We believed this code guaranteed something, and that guarantee did not actually hold. All four times, what revealed it was not a test but the result of actually running the thing.

Because the four are different kinds of gap, walking through them one at a time makes it clearer what a test proves and what it does not.

02

One: the limit we set was not the real limit

File attachments on agent tasks had an application-level size limit, and that limit was well tested. But a public request passes through a proxy before it reaches the application. Cut off at 100MB at the edge, it fails before our validation or progress reporting ever begins.

Tests only see inside our own code. A request that dies before reaching that code is outside their field of view. Nothing was wrong with the code; we had mistaken the boundary of the system for the boundary of the codebase.

It surfaced outside the code as well. Only after sending a 130MB file to the real service address and confirming the stored result hashed identically to the original was the whole path proven.

03

Two: a fallback had quietly become the normal path

While building OS-level isolation for external MCP servers, we made it run the original command with a warning wherever the tool was unavailable. Not taking the service down for a missing sandbox is a reasonable call in itself.

That call was undone by the fact that the tool we picked is Linux-only and the production host is macOS. A fallback meant for an exceptional case was firing on every request. The code existed, the tests passed, and actual isolation was zero.

What the tests verified was that the gate opened and closed according to its conditions. That the gate was permanently closed in production was never a subject of verification. This is worse than a bug, because everything behaved as designed and so nothing felt wrong.

04

Three: instrumentation attached to a path nothing takes

We added instrumentation to measure how often tool results get truncated, on the principle of measuring before fixing. The instrumentation itself worked exactly right. The tools used by sandbox tasks simply do not go through the execution path it was attached to.

So the number was zero. Two tool-result steps existed with not one recording attempt. Zero reads both as nothing happened and as we saw nothing, and the dashboard had no way to tell those two apart.

This one differs slightly from the previous two. What was wrong here was not the guarantee but the observation, and when the observation is wrong every judgement built on top of it is wrong with it.

05

Four: a mechanical rewrite left everything outside its pattern

Migrating the inference backend moved more than a hundred call sites at once with a codemod. Every reference that went through the configuration object was caught. Places reading the environment variable directly had a different shape and stayed put.

The leftover code compiles and the tests pass. Code that calls the old backend's proprietary route and assumes its response shape can sit there quietly, and until that route is actually taken there is no signal at all.

A bulk rewrite is powerful, but it does not tell you what it missed. Going looking for the leftovers that same evening was the most important step in that migration.

06

All four surfaced the same way

Line up how they were found and the pattern is stark. We sent a 130MB file to the real service, checked which operating system the production host runs, drove one chat-to-task run through a browser, and hand-searched for references the codemod had left behind.

All of it is cheap. None of it was a multi-day investigation; each was running the thing once for real. And in all four cases nobody knew until that one run happened.

This is not an argument that tests are useless. In all four cases the tests did their job: they blocked regressions, made refactoring safe, and spoke up when a rule broke. But a test proves only what we decided to verify, and what was wrong every time was that definition itself.

07

What changed as a result

Three habits settled in over these four cases.

First, no more silent fallbacks. Isolated servers are now listed in the log, and the ones that are not say so in their configuration. A state you cannot see is a state you cannot notice is wrong.

Second, record not knowing as data. When attributing execution steps to plan nodes, anything unattributable is left empty rather than guessed. Invent a value and the judgement built on that metric is contaminated the moment it is used.

Third, anything left in place gets a reason and a deadline written down. The identifier pinned by a database constraint and the version pinning deferred despite known risk are both recorded that way in their commits. That is why one was cleared thirteen days later and why we still know the other is outstanding.

08

What this retrospective does not catch

Those three habits are also reactive. All four were learned after the fact, and there is no guarantee the next gap takes one of these three shapes.

Things are still open. The upload protocol still does not take a file checksum at the completion step, so byte identity is not proven by the protocol itself. Two MCP servers still run unisolated because they depend on host-installed binaries, and package version pinning is deferred over the risk of everything breaking at once.

These are written down so that we do not claim to have solved it. That was the common thread across all four: the problem was never where we knew we were ignorant. It was where we believed we knew.