EssayMay 5, 2026
The Best Get Worse
Essay 2 of 5, Decision-Native 2040 series. Draft 1, 2026-04-23.
In 2013, Povyakalo and colleagues reanalysed the CADET II study, a UK trial of computer-aided detection in mammography.1 Fifty radiologists each read one hundred and eighty screening cases, once with CAD help and once without. The within-subject design gave a clean measurement of what CAD actually did to each reader.
The headline result was unsettling.
CAD helped the less accurate readers. It helped them notice subtle cancers they would otherwise have missed. For a certain kind of radiologist, mostly more junior, CAD produced a measurable lift.
It hurt the best readers. The most accurate radiologists in the sample performed worse with CAD than without it. On the subtle cases that matter most, CAD actively dragged their performance down.
The benefit curve wasn’t flat. It wasn’t rising. It flipped.
This is the essay about what that does to an organisation over twenty years.
The cognitive/motor split
The Povyakalo finding is not isolated. In 2014, Casner and colleagues ran a within-subject experiment on sixteen currently qualified B747-400 pilots — seven captains and nine first officers.2 The pilots flew routine and non-routine scenarios in a full-motion simulator, with the level of cockpit automation varied systematically.
Motor skills held up. Stick-and-rudder, instrument scan, the physical act of flying the aircraft under degraded conditions. Even pilots who had not hand-flown an approach in months performed at or near baseline.
Cognitive skills did not. The captains lost the ability to track their own position without the map display. They lost speed at choosing the next navigational step. They lost the ability to recognise system failures from indirect cues. The physical skill was retained through some form of motor memory. The cognitive scaffolding was not.
This is the atrophy that matters. Not the thing you practice visibly. The thing you do in the background, without noticing.
Three and a half minutes
If you want to see what the cognitive atrophy looks like at the moment it matters, the record is already written.
At 02:10:05 on the night of 1 June 2009, over the South Atlantic, the pitot tubes of Air France Flight 447 iced over. Three instruments that measure airspeed became unreliable. Over the next three and a half minutes, an Airbus A330 and its 228 occupants fell twelve kilometres into the ocean.
The BEA final report documents the sequence in forensic detail.3 Read it closely and it becomes a textbook of cognitive atrophy under automation failure.
Step one. The autopilot disconnected. The aircraft handed control back to the crew, which is what it is supposed to do when the sensors it depends on disagree.
Step two. The flight control law reconfigured from Normal to Alternate 2. This is what Airbus calls a modal transition. Under Normal Law, the aircraft protects the flight envelope. Under Alternate Law, many of those protections are removed. The pilot flying now has authority he did not have thirty seconds earlier, and responsibilities he did not have either.
Step three. The pilots had three independent airspeed indicators. One was now telling lies, consistent with the pitot icing. The question the system cannot answer is which one. The pilot had to reconstruct the aircraft’s state from a now-conflicting instrument panel, while also flying the plane.
Step four. The stall warning sounded. The aircraft was pitched up at an unusual angle. A correct recovery was to push the stick forward, increase airspeed, recover lift.
Step five. The pilot pulled back. The response was wrong.
Step six. Here is where the algorithm reveals itself as the third character in the cockpit. When the aircraft’s calibrated airspeed fell below 60 knots, the stall warning stopped. The design objective was to prevent spurious warnings at low ground speeds, where the aircraft is not in a flight regime where stall matters. At 38,000 feet, deep in an aerodynamic stall with the angle of attack past 40 degrees, the same algorithm silenced the warning because the CAS was below threshold.
When the pilot briefly pushed the nose down, CAS rose back above 60, and the stall warning fired again. The pilots appear to have interpreted the second warning as confirmation that the nose-down input was the wrong response. They pulled back. The warning silenced again.
The target was prevent spurious warnings. The consequence was a system that silenced itself at the moment it was most needed. This is the broken-objective algorithm, and it is not an exotic case. It is a textbook example. The objective was chosen by a human engineer making a reasonable decision with limited data; the consequence of that decision was cognitively invisible until a specific configuration of failure made it lethal.
The BEA report also documented that the pilot flying had not been trained for manual handling at cruise altitude. Air France’s stall-recovery training was limited to low altitude. The specific cognitive skill of reconstructing the aircraft’s state at 38,000 feet under degraded instruments was a skill he had never been asked to practice.
The cockpit of AF447 is the Povyakalo result playing out under time pressure, with 228 lives attached.
From the cockpit to the boardroom
A boardroom in 2040 is not going to kill anyone in three and a half minutes. The failure mode is different in kind, not in shape.
The decision-native company has routed its routine calls through systems. Most calls, most days, the systems perform. Some fraction of decisions require the humans in the loop to engage, but only when the system flags low confidence, or the stakes are high, or the outcome will be audited.
The first thing to atrophy is not the ability to make a hard call. It is the cognitive scaffolding that would let someone notice a hard call the system has not flagged. That is the thing radiologists lost when given CAD. That is the thing Casner’s pilots lost when given autopilot. That is the thing a board loses when it reads the quarterly pack the system prepared.
The second thing to atrophy is the ability to reconstruct state from disagreement. When two decision systems or two internal teams present recommendations that diverge, the muscle that decides which one is closer to the truth is the muscle that gets used least in a well-oiled decision-native company. Used less, it atrophies faster.
The third thing, which is the one the tail event reveals, is the ability to build from first principles when the automation is wrong. This is not a Tuesday skill. It is a once-a-year skill. And like any once-a-year skill, it disappears if it is not practiced deliberately.
Why outcomes look fine for a long time
Routine quarters do not expose atrophy. Results stay acceptable. Revenue hits. Risk reduces. No one dies.
The atrophy is a tail asset. The inventory of judgement is drawn down only when the system fails, and the system fails rarely enough that the drawdown is invisible over five, maybe ten years. The board at the end of that period is not the board that designed the system. The people who remember how a specific kind of call used to be made have retired. The institutional pattern-recognition is gone, and no one notices until the pattern matters.
By then, the company is effectively running on whatever its systems can reconstruct from the past. That is a fine way to run a company in a steady state. It is a terrible way to survive a discontinuity.
The unison problem
The worst feature of organisational atrophy is that peer review does not catch it.
Every competitor in a given industry is using the same automation stack, roughly the same data, and roughly the same processes to interpret what the automation tells them. The cognitive scaffolding of the whole industry is atrophying together. The drift is unison, not divergence.
A peer who challenges a call is not drawing on independent instincts. They are drawing on the same atrophied instincts, filtered through the same systems. The review feels adversarial and produces concurrence.
When Fenton and colleagues looked at community-wide CAD deployment across 429,345 mammograms, specificity fell from 90.2% to 87.2% after CAD introduction, biopsy rates rose by nearly twenty percent, and cancer detection did not improve.4 Every radiologist in that dataset was working within the same automation stack. Every radiologist drifted in the same direction. The peer review was intact. The direction was wrong.
Credit-scoring shows the same pattern outside medicine. Every US consumer lender is downstream of the same three bureau feeds and a small number of FICO-derived models; when the underlying instinct drifts, every lender’s risk pricing drifts in the same direction at the same time. Peer review across competitors confirms each underwriter’s call because each underwriter is reading the same atrophied instinct, filtered through the same systems.
This is why atrophy is so hard to see from inside: the social proof runs in the wrong direction. Everyone agrees, because everyone has lost the same thing.
The evidence gap
No published randomised controlled trial withdraws AI from working practitioners after months of use and measures what happens to their judgement over time. That experiment has not been run. The Agarwal NBER paper from 2023, which is the best designed study on radiologist-plus-AI collaboration, is a single-session experiment.5 It can tell us about one-shot reliance. It cannot tell us about atrophy over years.
We are running it anyway, at scale, on every decision-native practitioner alive. There is no control group. The aviation evidence is the warning. The CAD evidence is the warning. The readout is a tail event that has not happened yet.
It feels like efficiency
Atrophy does not announce itself.
It feels like a team getting a lot done with less friction. It feels like the smartest people in the room agreeing quickly. It feels like the quarterly pack having fewer open questions than it used to. It feels, most of all, like the absence of friction where friction used to be.
Friction is the phenotype of judgement being used. Its absence is the phenotype of judgement being replaced.
Until the system drops, and then it feels like three and a half minutes over the South Atlantic, trying to decide which of three speed indicators is telling the truth.
Sources
1. Andrey A. Povyakalo, Eugenio Alberdi, Lorenzo Strigini, and Peter Ayton, “How to Discriminate between Computer-Aided and Computer-Hindered Decisions: A Case Study in Mammography,” Medical Decision Making 33(1): 98–107 (2013). Reanalysis of CADET II; 50 readers × 180 cases, within-subject.
Share feedback
No public comments. If you want to share a reaction or correction, send it here.