OpenMake
01
There was a plan, and there was execution, but no link
Agent tasks always kept two records: the plan the model wrote, and the execution log of what actually happened. The problem was that neither referenced the other. There was no way to ask how many tool calls the second plan step consumed, or how far execution had drifted from the plan.
The roadmap names the Execution Graph as the next step — a DAG whose nodes carry their own dependencies, required permissions, cost caps, retry policy, and completion criteria. But designing those nodes means knowing what to attach to them, and that cannot be known without measurement. So before building the graph, we built somewhere to attach things to.
02
First, record what each turn intended to do
In 1.21.0 each turn began persisting which tool it intended to call as part of its step. The same release made exponential-backoff retries on transient LLM errors, and the demotion that strips approval-gated tools after repeated unanswered approvals, persist as steps of their own.
The intent is simple: count how often retries and demotions actually fire straight from the database instead of digging through logs. It stockpiles the numbers that will justify a per-node retry policy later.
03
When you do not know, leave it empty
In 1.22.0, migration 088 added a nullable plan step index to the execution step table. When a step is recorded, the index of the in-progress plan step is stamped onto it deterministically. The model's judgement never enters this path.
The most important choice was what not to do. If no step is in progress, the column is left empty rather than guessed at by picking the first unfinished step. The whole point of the column is to measure how well plan and execution line up, and inventing values you do not have makes the measurement meaningless.
- Attributed: assistant responses, plan steps, tool results, and retries
- Deliberately left empty: closing turns, approval demotions, user steering, and code diffs — management events that do not belong to any one step
- Existing rows are unaffected, and running the migration repeatedly produces the same result
04
The first measurement was 45%, and the cause was not the code
Measured retrospectively against the accumulated data, strict attribution coverage came in around 45%. Slightly under half of all steps could say which plan step produced them.
Breaking down the cause showed it was not a code problem. In 40% of cases the model had updated the plan without marking anything as in progress at all. Even with an explicit instruction to do so, we reproduced a live run where the mark was dropped on the second step.
That created a fork: push harder on the prompt, or correct it in code. Touching the prompt risks shifting unrelated behaviour, so that decision was deferred until forward-looking data exists.
There was a side benefit. The distribution of tool calls per node came out with a median of four and a maximum of 26. That distribution is what a per-node cost cap will eventually be argued from.
05
Correct it deterministically instead of persuading the model
Increment three leaves the prompt alone and fills the gap in code. Right after a plan is created, and after any step completes or becomes blocked, if nothing is in progress the first not-started step is promoted. It is a deterministic rule assuming linear execution, and again there is no model judgement in it.
What the model does say is still respected. If a step is already in progress nothing happens, and a step the model explicitly moved back to not-started is not promoted again. The correction can be switched off with an environment variable, defaults to on, and six tests hold the rule in place.
06
Instrumentation you cannot see does not get checked
Finally, the task detail timeline shows a badge for the plan step each entry belongs to. The backend was untouched — the column was already in the step response, so the interface only had to read it, and one key was added across four locales.
Surfacing the measurement was not about appearances. You notice something is wrong when you can see it. And right afterwards, we did.
07
The instrumentation had its own blind spot
In the same spirit we added instrumentation measuring how often tool results get truncated — measure before fixing. Then, while checking the chat-to-PDF delegation path in a browser, it turned out that for sandbox tasks this instrumentation recorded nothing at all.
The cause was the code path. Sandbox tools do not go through the turn executor; they take a separate execution route, and the counter had been attached where those tools never pass. In the measured run there were two tool-result steps and zero recording attempts.
In 1.22.1 the counter moved to the point where results are actually capped, and the magic number sitting there was folded onto the existing environment variable. Measure first, fix second only holds when the measurement is in the right place.
08
What comes next
What exists now is the instrumentation attributing steps to plan nodes, a deterministic correction for when the progress mark is missing, and a surface that shows the result. What does not exist yet is a durable DAG whose nodes own their dependencies, required permissions, cost caps, retry policy, and completion criteria.
The next decision will be argued from numbers: how far 45% climbs in forward-looking data now that the correction is on, and where the tail of the per-node tool-call distribution cuts off. Those are the first candidates for the cost cap and completion criteria a node will carry.
Source evidence