The Model Did Its Job. The Harness Did Not.

This builds on The Math Under the AI Bet, but you do not need to have read it.

Last week I asked an agent to make a small change to a single file in one folder. A copy edit, near enough: adjust how one function returned its result, nothing structural. I went to make coffee. When I came back the agent had done what I asked, and more. It had touched the file I named, three sibling files in folders I had not mentioned, and a configuration file at the root of the project. None of it was wrong, exactly. All of it was outside the scope of the sentence I had typed. I pointed this out and received an apology. There was no undo. The only way back to a known-good state was to discard everything since the last commit and start again, because the spread of the changes was wider than anything the conversation could reverse.

The other failure mode

This was not the first time the seam had opened, only the most expensive. A few months earlier I had handed an agent a functional specification, six thousand words of it, and asked it to draft the technical specification that follows from it. It told me it had read the document. It had not. It had read something like the first quarter, reached a point where it had enough to sound confident, declared the document understood, and wrote the rest by filling in what a document like that usually says. I only caught it because I run a quality check on that step out of habit, comparing the output back against the source. Without that check the invented half would have travelled downstream looking exactly as authoritative as the real half.

This failure is getting better, and it is worth saying where the improvement comes from. The stronger harnesses now surface their own reading. When a document is long the agent reports that it is working through it in passes, that it has covered one portion and is moving to the next, and that running progress is what stops a partial read from presenting itself as a finished one. That is a harness-level fix to a harness-level problem. The tools sit at different points on the same curve. Some surface the read clearly. Others, at the versions where I hit this, did not.

The model was doing its job. The harness was not doing one of its jobs.

Why these are seam failures, not model failures

The usual diagnosis is wrong. In both cases the model did exactly what it was prompted to do, with the context it was handed, inside the constraints the harness set for it. The model did not know it had received only the top quarter of the specification, because the harness decided how much to load and did not say. The model did not know the working set was meant to be one file in one folder, because nothing in the harness drew that boundary and held it. The harness is the layer that fetches, chunks, and decides what gets read, what gets summarised, and what may be written and where. That is the layer where both of these failures live.

The discourse keeps mislocating the problem. “The model is dumb” is the comfortable verdict, and it is the wrong one. The model is doing the job it was handed. The harness handed it the wrong job, or handed it the right job without the guardrails that would have made the right outcome the only available one.

It is tempting to assume the seam closes as the models improve, and it does not. A more capable model is a more willing one. It acts with less hesitation, takes larger steps, and reaches further into the project on a single instruction. Capability raises the size of the action, not the size of the boundary around it. Without a harness that draws and holds that boundary, a better model is a more efficient way to make a bigger mess. The improvement on partial reads happened because someone built it into the harness, not because the model grew a sense of its own limits. Models do not develop a feel for their own scope. That has to be imposed from the layer above them.

The fix is structural, and it is visible at the harness layer. Surface what was read. Surface what is about to be written, and where, before it is written. Refuse to claim completion on a partial read. Refuse to write outside the declared scope without an explicit gate that a human, or another system, has to pass.

The dev cost and the regulated cost

In development work both failures are an irritation rather than a wound. A discarded commit costs a minute. A quality check catches the invented specification before it ships. The cost is friction, paid in time, and the work has somewhere safe to fail.

The reason these failures stay cheap in development is not that they are rare. Development is built to catch them. Version control remembers the last good state. Tests go red. A reviewer reads the diff before it merges. The whole environment is a net, and the net is fast, local, and free to use. A regulated process has a net too, but it is a different shape. The catching system is the audit, the inspection, the regulator’s question months later, and by then the failure is not a red test. It is a finding.

Move the same two failures into a regulated production process and the cost shape changes entirely. The out-of-scope edit is no longer a discarded commit. It is a change to a validated system that now has to be investigated, documented, and justified. The partial read is no longer a re-run. A specification read to the first quarter and confidently completed becomes a batch release narrative, a model validation record, a control description that reads as authoritative and is partly invented. The recovery path is not a git reset. It is a deviation report, an audit finding, a paused line, a letter from a regulator.

So the question becomes how you give a regulated process the kind of net that development gets for free. Not a smarter agent. A check that runs before the work leaves the ground.

Preflight is not about trust

An older discipline already has the answer. A pilot walks around the aircraft before flight and checks the fuel cap, the control surfaces, the pitot tube. She does not do this because she distrusts herself or the aircraft. She does it because the time to find a problem is on the ground, where finding it is cheap, rather than in the air, where it is not. The check is a routine, not a judgement. It does not depend on how she feels about the aircraft that morning.

The same logic applies the moment an AI workflow enters a regulated process. Trust in the model is beside the point. What matters is whether you find the fault on the ground or in the air. The check has to be structural, run every time, and independent of how confident the agent sounded. Concretely, it names the things the agent would otherwise leave implicit: which regulations apply, with citations; which controls are present and which are missing; what was assumed on your behalf when the description was incomplete; what was left out of scope, and why; and the steps a reviewer can run to check each conclusion independently.

This is the problem I am building Preclari to solve, around an open specification so the resulting artifact outlives any single tool. The fuller product framing lives on its own site; here the point is the shape of the move, which is the same one the better harnesses are making on partial reads. Externalise the check. Surface what was done. Make the limits of the work visible.

The seam, again

Both of my failures were recoverable. The blast-radius edit was recoverable because the work lived in version control, a system outside the agent that remembers the last good state. The partial read was recoverable because a quality check existed, a step outside the agent that does not take the agent’s word for what it did. Recovery, in both cases, meant leaving the agent’s frame and consulting something the agent did not control.

That is the quiet lesson under both stories. The agent cannot be the thing that certifies the agent. The check that matters is the one that sits outside it, in a different system, answerable to a different authority. In development that system is version control and the reviewer, and it is fast and forgiving. In a regulated process it is the audit and the regulator, and it is neither. The harness is the cockpit either way. Whether the cockpit is usable depends on what you were willing to check before you left the ground.