The term suggests a malfunction, something breaking that could be repaired. It is more accurate to say the system is doing precisely what it was built to do, and that the behaviour we object to is the same behaviour we rely on, pointed at a question it has no way to answer.
What the machinery actually does
A language model predicts a probability distribution over the next token given everything before it, then selects from that distribution and repeats. There is no separate stage where a claim is checked, because there is no representation of a claim, only of what tends to follow what.
This has an important implication. The model has no store of facts to consult. Information absorbed during training exists as patterns distributed across the weights, entangled with everything else, and there is no lookup and no flag distinguishing something learned reliably from something reconstructed on the spot.
The training corpus is vastly larger than the parameters holding it, so the encoding is necessarily lossy. Frequently repeated facts are represented robustly. Facts appearing once or twice are represented weakly, and what emerges at generation time is a plausible reconstruction rather than a retrieved record.
Why it invents rather than declines
The obvious question is why the model does not simply state that it does not know, and the answer is that nothing in the mechanism produces that behaviour by default.
At every step the model emits the most probable continuation. When it knows something well, the most probable continuation is correct. When it does not, there is still a most probable continuation, and it is a fluent, well formed, entirely wrong sentence. Ignorance does not present itself as a gap that could be reported. It presents as a flatter distribution over equally plausible options.
Subsequent training on human preferences can push in the wrong direction here. Responses that answer are generally preferred to responses that hedge, so a system optimised on that signal learns that confident answers are rewarded. Modern training deliberately counteracts this, with partial success, which is why models decline more readily than they used to and still fabricate under pressure.
Where fabrication concentrates
It is not evenly distributed. It clusters in a recognisable shape, which is content where the format is highly predictable and the specific value is arbitrary.
References and citations are the clearest case. The model has seen thousands of citations and knows exactly how one is constructed, including plausible author names, a credible journal, a sensible year, and a well formed identifier. Every structural element is right and the thing itself does not exist. The same applies to legal cases, standards numbers, function names in a library, statistics, dates, and quotations.
The common thread is that the surrounding pattern is learnable and the particular fact is not derivable from it. That is exactly the situation in which a next token predictor performs confidently and wrongly, and it explains why fabricated output so often looks more polished than genuine output.
Confidence in words is not confidence in the model
A distinction worth holding is that the model does carry an internal measure of uncertainty, in the shape of the probability distribution at each step, and that this measure does not reliably appear in the text it produces.
The phrasing of a response, whether it hedges or asserts, is itself generated text. It reflects the register of similar answers in training rather than any inspection of how certain the model actually was. A response can be worded with complete assurance while the underlying distribution was nearly flat, and the reader has no way to tell from the words.
This is why the surface confidence of an answer carries no information about its reliability, and why systems that need a usable uncertainty signal have to derive it from the probabilities themselves rather than from what the model says about its own certainty.
What reduces it
Supplying the facts is the only approach that addresses the cause. If the relevant material is placed in the context, the model is reading rather than recalling, and accuracy improves substantially. This is the entire argument for retrieval based systems.
It is a reduction rather than a cure. A model given passages can still misread them, combine them incorrectly, or supplement them with recalled material that was never in the retrieved text, and it will do all of this in the same fluent register.
Requiring a source alongside every claim helps, because it makes verification cheap rather than because it makes the model honest. Constraining output to a schema helps for structured extraction. Asking for the source rather than the answer, then reading it, is the strongest available approach where correctness genuinely matters.
What does not help
Instructing the model not to fabricate has little effect, because it is not choosing to. Asking whether it is certain produces an assessment generated by the same process that produced the original claim, so a confident wrong answer is followed by confident agreement that it was correct.
Asking the same question repeatedly and comparing answers is slightly more useful, since genuine knowledge tends to be stable while fabrication varies, but it is a weak signal and it costs several calls to obtain.
When it matters
The severity depends entirely on what happens next, and it is worth being deliberate about the distinction rather than treating all output as equally risky.
| Condition | Low risk | High risk |
|---|---|---|
| What happens to the output | Reworked before use | Acted on directly |
| Reader's expertise | Would notice an error | Cannot detect one |
| Consequence of being wrong | Reversible | Irreversible |
The useful design question is not how to eliminate the behaviour, since that is not currently available. It is whether the system can tell the difference between the two columns, and whether it behaves differently in each.
Note: fluency and accuracy are produced by the same process and are not correlated in the output. A confident, well structured, precisely detailed answer is not evidence of correctness, and the details that make it convincing are the ones most likely to be invented.