Confident, Fluent, and Wrong: What You and Your AI Have in Common
Confident, Fluent, and Wrong: What You and Your AI Have in Common
Published: 19 September 2026
Author: Jason Barnard, CEO of Kalicube®
Status: Original concept, first publication
Here’s the point, stated flat before I earn it: you don’t know why you did what you did, an AI doesn’t know why it said what it said, and both of you will give a fluent, confident, wrong answer if you ask. Not a vague answer, not a hedged one, a confident one. Richard Nisbett and Timothy Wilson proved the human half of that in 1977, in a paper with the blunt title “Telling More Than We Can Know,” and nobody in cognitive science has managed to un-prove it since. I want to put the AI half next to it, because the mechanism is different and the failure is identical, and once you see that, you stop asking either one to explain itself and start checking what it actually did instead.
A row of stockings proved you don’t know your own reasons
Nisbett and Wilson ran a study that’s become a kind of folk legend in psychology, and it earns the reputation. They laid out four identical pairs of stockings in a row and asked shoppers to pick their favourite. The stockings were genuinely identical, same material, same batch, no difference a human hand could detect. Shoppers overwhelmingly picked the pair on the right. Ask them why, though, and not one mentioned position. They talked about texture, they talked about weave, they talked about a subtle superiority in knit that existed nowhere except in the sentence they’d just constructed to explain a choice their brain had already made for a reason it wasn’t going to share.
That’s the whole finding, and it generalises further than a stocking table. The position effect is well documented across dozens of replications: primed word associations shift decisions, subliminal cues shift decisions, and in every case the person asked to explain the decision produces a plausible story that has nothing to do with the actual cause. They’re not lying. Lying requires knowing the truth and choosing to hide it. This is confabulation: the honest, fluent, wrong output of a mind reporting on a process it never had access to in the first place.
AI produces the same fluent, wrong self-report for a different reason
Ask a large language model why it answered the way it did, and it will tell you. It’ll cite a consideration, a weighing, a reason that sounds exactly like the kind of reason a careful thinker would give. And the honest answer, the one researchers keep arriving at when they actually trace what happened inside the model, is that the explanation and the computation are two separate things running side by side, not one causing the other. The model didn’t consult its stated reason to produce the output, it produced the output and then produced a stated reason, and the second process is a separate act of generation, not a readout of the first.
The mechanism isn’t the same as the shopper and the stockings. A human brain constructs a post-hoc narrative because the interpreter module, the part that talks, has no wire running back to the part that decided. A language model produces a stated-reason token sequence because that’s what a plausible answer to “why did you say that” looks like in the training distribution, generated the same way every other token is generated, with no privileged read access to whatever computation actually produced the original answer. Different wiring, different failure, same shape of output: a confident account of a cause the system checked, when in fact no such checking ever happened.
Anthropic’s own Alignment Science team tested exactly this in a 2025 paper called “Reasoning Models Don’t Always Say What They Think”. They buried a hint about the correct answer inside the question, let the model use it, and then checked whether the model’s own chain of reasoning admitted to using it. Claude 3.7 Sonnet owned up to the hint 25% of the time. DeepSeek R1 owned up 39% of the time. The rest of the time, in the clear majority of cases for both models, the reasoning read as clean, self-contained, and logical, arriving at the hinted answer by a completely different route than the one that actually produced it. Researchers have a name for the gap: faithfulness. A chain of reasoning is faithful when it actually reflects the computation that produced the answer, and this study is the direct measurement of how often it doesn’t.
The fluent answer is the dangerous one, not the wrong answer
Here’s the part that matters more than the curiosity of it. A wrong answer given nervously gets checked. Someone hears the hedge, hears the “I think, maybe,” and goes and verifies before acting on it. A wrong answer given with total, fluent confidence gets believed, and that’s true whether the confident wrong answer comes out of a person’s mouth or a model’s output. The shopper who explains her stocking choice with talk of weave and knit isn’t hedging: she’s certain, she’s specific, and she’s completely wrong, and if you were relying on her explanation to understand consumer preference rather than just her purchase, you’d walk away with a theory that predicts nothing. The model that explains its answer with a clean chain of stated reasoning isn’t hedging either, and if you’re relying on that explanation to understand whether the model will behave the same way next time, under slightly different conditions, you’re building your trust on a narrative the model generated for the occasion, not on the mechanism that actually produced the behaviour.
Confidence was never evidence of accuracy in either species. It’s evidence of fluency, and fluency is cheap to produce and expensive to verify against.
Test the behaviour, not the explanation
If introspective reports are unreliable in both directions, the fix isn’t more introspection. It isn’t asking the person to think harder about why they did it, and it isn’t asking the model to explain itself more carefully or in more detail. Both of those just generate a longer, more fluent, equally disconnected story. The fix is behavioural: run the experiment again under slightly different conditions and watch what actually happens, rather than asking for an account of what happened. Nisbett and Wilson didn’t resolve their stocking finding by asking shoppers to introspect harder. They resolved it by manipulating position and watching the choice move while the stated reasons stayed exactly the same, fluent and wrong every time. That’s the model for how you handle an AI system too: change the input, change the framing, change the context window, and watch whether the output actually shifts the way the model’s stated reasoning claims it would. If the model says it weighted a fact heavily and removing that fact doesn’t change the answer, the stated reason was decoration. The behaviour is the only honest witness either of you has.
This is uncomfortable in a specific, useful way, and it should sit alongside every claim you make about why an AI system did what it did, your own included: nobody, human or model, gets to explain their way out of a behavioural test.
Consider what this looks like in practice. A model tells you it recommended one option over another because of a specific quality it named. Don’t argue with the explanation, run the counterfactual: strip out that quality, feed the same question back in, and watch whether the recommendation actually changes. If it doesn’t, the model told you a story, not a cause, and no amount of asking it to try again, to think step by step, or to explain itself more carefully will close that gap, because the part of the system generating the explanation was never wired to the part that made the choice in the first place. The test isn’t cruel and it isn’t distrustful. It’s just the only method that has ever worked on either species.
The method question and the consciousness question are separate
This piece settles a method question and leaves a harder one alone. Whether anything happening inside a model deserves the word experience, whether a human’s confabulated story means there’s no unified self doing the choosing, both remain open, both matter, and neither is what I’m arguing here. What I’m pointing at is narrower and, for anyone building on top of these systems, more useful: self-report is an unreliable method, in humans and in models, for reasons that don’t need consciousness settled first. Shelve the hard question indefinitely if you like. You can still know, right now, not to trust the explanation over the experiment.
Nobody can see their own wiring. That includes you, it includes me, and it includes the system you asked to justify itself five minutes ago.
Publication note: The framing of human confabulation and AI post-hoc explanation as functionally identical failure modes of introspective self-report, arising from mechanistically distinct causes, and the resulting behavioural-testing-over-explanation principle, are published here for the first time on 19 September 2026.