Why AI Copilots Produce Articulate Mediocrity

The Forecast That Looked Right

I’ve spent 25 years in supply chain rooms where real money moves. A lot of that time in crop protection: European agrochemical companies planning production runs, logistics capacity, and inventory positions for seasonal product lines where timing is everything.

Here’s a scenario that plays out in planning meetings now. A European crop protection company is forecasting Q2 demand for a herbicide product line. The planning team asks an AI copilot for a demand forecast. The copilot has access to everything: five years of shipment history, regional weather forecasts, commodity price indices, acreage reports, regulatory filings. Complete information.

The copilot returns a confident, structured response: “+12% year-over-year based on favorable weather patterns in Northern Europe, increased planted acreage, and commodity price trends supporting farmer investment in crop protection.” It cites real factors. It sounds like what a competent analyst would present. The planning team nods and moves to the next agenda item.

The answer is wrong. The copilot had all the data. It’s wrong because it cannot fit a non-linear interaction model between spring rainfall timing and regional order patterns against this company’s specific shipment history.

Here’s what a statistical model like XGBoost¹ finds when trained on the same data. When March rainfall in Northern Europe exceeds a specific threshold, farmers delay herbicide purchases by three to four weeks because they can’t get into waterlogged fields to spray. Total volume doesn’t change much: +8% instead of +12%. But the demand shape changes radically. Instead of a gradual ramp through April and May, orders compress into a sharp spike in the second and third weeks of May.

That spike means you need 40% more logistics capacity in a three-week window. Warehouse staging areas adequate for a gradual ramp become a bottleneck. Working capital tied up in finished goods peaks three weeks earlier than the “+12% gradual growth” plan assumed.

The copilot had the rainfall data. It had the historical order patterns. It cannot detect that threshold, quantify the lag, or model the demand shape change. Ask it directly and it will tell you “weather can affect the timing of farmer purchases.” That’s pattern-matching against language about agriculture, not computation against your data.

Nobody catches it. The forecast looks rigorous. The real exposure is invisible.

What Copilots Actually Do

Let me be precise about what happens when you ask an AI copilot a business question.

Large language models are prediction engines. They compress statistical patterns from billions of tokens of human text and reproduce plausible continuations on demand. When a copilot forecasts “+12% year-over-year based on favorable weather patterns,” it hasn’t calculated anything. It has predicted what text would plausibly follow your question, based on patterns from millions of similar conversations about demand planning.

This is the architecture working as designed. The model knows what good advice sounds like. It doesn’t know if it’s good advice for you.

This distinction matters because it’s invisible. A wrong number in a spreadsheet is obviously a wrong number. A fluent paragraph of wrong advice looks identical to a fluent paragraph of right advice.

Where Copilots Genuinely Help

Before I go further, let me be honest about what language models do well. This isn’t an anti-AI argument. It’s an argument about fit.

Language models excel at communication tasks: summarizing long documents, translating between languages, drafting emails, explaining technical concepts to non-technical audiences, extracting structured information from unstructured text. These are tasks where prediction is the right capability. The model isn’t deciding anything. It’s transforming information from one form to another.

If you ask a copilot to summarize your Q3 supply review deck, it will do a good job. If you ask it to draft a supplier communication about a delivery delay, it will save you twenty minutes. These are real, valuable applications.

The problem starts when you ask it to decide, or when its output looks enough like a decision that you treat it as one.

Three Approaches to the Same Decision

The difference becomes concrete when you compare how different approaches handle the same problems:

Read the middle column carefully. Every response sounds more rigorous than gut-feel. None of them are computed from your data. The copilot has no model of your warehouse capacity, your working capital limits, or your supplier lead times. It generated text that sounds like it considered these things.

The right column is different in kind, not degree. Those are deterministic computations against your actual constraints, with an LLM translating the results into language you can act on.

Copilots Have No Model of Your Constraints

When a copilot suggests “increase safety stock by 20%,” it doesn’t know this ties up €2M in working capital. It doesn’t know your credit facility covenant limits working capital to €15M. It doesn’t know your warehouse is at 94% capacity.

The copilot made a prediction that sounds like good supply chain advice. The consequences are invisible to it.

Better prompting won’t fix this. You can feed it context in the prompt, but the model still has no representation of your constraints: no way to reason about interactions between capacity, capital, lead times, and service levels simultaneously. It will incorporate your context the way a good writer incorporates background material: as flavor for more plausible-sounding text, not as variables in a computation.

The Confidence Problem

A human analyst hedges when they’re uncertain. “The March rainfall numbers are unusual this year and I’m not confident about the May timing, but based on what I can see…” That hedging is information. It tells you where to probe.

A copilot expresses equal confidence whether it’s right or wrong. The most dangerous recommendations, where the model is furthest from reality, are delivered with the same fluent tone as trivial ones. There’s no signal for uncertainty because the model has no representation of what it doesn’t know.

This creates a specific failure mode in organizations: the most consequential decisions, where you most need caution, are exactly the ones where the copilot sounds most assured.

Same Question, Different Answer

Ask a copilot the same question twice. You’ll get different answers.

Not wildly different. Both will sound reasonable. Both will be articulate. But they won’t be identical, because the model samples from a probability distribution. Each response is a plausible completion, not a deterministic calculation.

Same inputs, different outputs. That’s prediction. Same inputs, same outputs. That’s computation.

When the CFO asks why you made a particular call, “the AI gave me a different answer this time” is not a defensible response. Enterprise decisions require computation.

Prompt Chains Make This Worse, Not Better

The current wave of enterprise AI solutions layers sophistication on top of this foundation. Agent frameworks. Knowledge retrieval. Prompt chains orchestrating multi-step reasoning. The computation layer is still a language model predicting text.

When you chain prompts, each step takes the previous step’s output as input. If step two hallucinated a number, step five builds on it. By the final recommendation, it’s impossible to trace which parts were computed from your data and which were generated from training patterns.

The output looks more rigorous than a single copilot response. Multiple steps. Structured reasoning. Citations. But the architecture is the same: prediction layered on prediction. Each layer sounds more confident. None of them are computing anything.

This isn’t a quality control problem you can solve with better prompts or more retrieval. It’s architectural. The system has no separation between computation and generation.

The Decision Laundering Problem

There’s a subtler dynamic that nobody in enterprise AI talks about.

Executives face career risk with every significant decision. When you commit €50M to a capacity expansion, your name is on it. When you kill a product line, it’s traceable to you. The higher the stakes, the more exposure.

Copilots offer something seductive: the appearance of analytical rigor without the accountability of an auditable process. You asked the AI. The AI recommended it. The recommendation sounded sophisticated. You followed it.

This isn’t a technology problem. It’s a human incentive problem that technology enables. The copilot becomes a mechanism for decision laundering: transforming gut-feel into something that looks like analysis, without the computational substance that would make it actual analysis.

The organization thinks it’s making data-driven decisions. It’s making copilot-validated decisions, which is a fundamentally different thing.

No Audit Trail Exists Because No Auditable Process Occurred

When you accept a copilot’s recommendation, what gets recorded? The recommendation. Maybe the prompt. The reasoning path? Gone. The constraints considered? None were. The alternatives weighed? The model doesn’t weigh alternatives. It generates one plausible response.

Six months later, someone asks: “Why did we decide this?”

No audit trail exists because no auditable process occurred. The copilot produced text that sounded like a decision. The actual decision happened in the user’s head, informed by a confident-sounding prediction.

The EU AI Act will require transparency about how high-risk AI decisions are reached. But regulation aside, accountability matters for simpler reasons. When decisions go wrong, you need to understand why. When they go right, you need to replicate them. A copilot gives you neither.

Organizational Amnesia as a Feature

Copilots have no memory of what they told you yesterday. No record of what you accepted or rejected. No understanding of what your organization has learned over cycles of decisions.

Each interaction is isolated. Knowledge cannot compound. The same questions get re-asked. The same mistakes get re-made. The organization learns nothing from its decisions because the system retains nothing.

This isn’t a missing feature. It’s a consequence of the architecture. Prediction engines don’t need memory because each prediction is independent. But decision systems need memory because decisions build on each other. This quarter’s demand plan should reflect what you learned from last quarter’s forecast errors.

A copilot is a brilliant colleague with amnesia. What you need is organizational memory that compounds.

What Decision Infrastructure Actually Looks Like

The alternative isn’t “no AI.” It’s different AI for different tasks, with clear architectural separation between what computes and what communicates.

Forecasting uses statistical and machine learning models: algorithms that analyze your historical data and external signals. These are deterministic. Same inputs, same outputs. The math is auditable.

Constraint modeling explicitly represents your business limits. Capacity, lead times, working capital, regulatory requirements. The system knows what’s feasible, not just what sounds reasonable.

Scenario simulation computationally analyzes alternatives. Monte Carlo methods running millions of scenarios against your constraint model. Consequences calculated, not predicted.

Communication is where language models belong. Translating computational outputs into business language. Helping users explore reasoning. Answering questions about underlying data. Summarizing complex trade-offs.

The language model never decides. It presents what computational engines have determined. The boundary between computation and communication is explicit and auditable.

The Test

Next time a copilot gives you a recommendation, ask four questions:

1. Can I trace how this was derived from my data?
2. Does it account for my actual business constraints: not generic ones, mine?
3. Will I get the same answer if I ask again tomorrow?
4. When this decision is questioned in six months, can I reconstruct the reasoning?

If the answer to any of these is no, you don’t have a decision system. You have a text generator that sounds like one.

The difference matters when consequences do.