Hallucination is the same mechanism that makes it useful

Section 1 ended with a card that says what your chat tool can be trusted for. Section 2 is about the other side of that card: the ways a fluent answer goes wrong, and why each one is not a bug in the ordinary sense but the ordinary mechanism doing exactly what it does. This note takes the most famous one. A model that invents a fact, a quote or a reference is not malfunctioning. It is predicting the next likely word, which is all it ever does.

The word people use for this is hallucination, and it is a misleading word, because it suggests a separate state the model falls into, a moment when it stops reporting and starts imagining. There is no such moment. Here is the whole thing in one drawing.

Why it fills the slot

Note 01 made the point that a model has no store of facts to consult, only a compressed record of what tends to follow what. That record is extraordinarily good at the shape of things. It knows what a citation looks like: a surname, initials, a title in a certain register, a journal, a year. It knows what a statistic looks like, what a quotation looks like, what a confident answer to a direct question looks like. When you ask for something the model has the shape of but not the substance of, the loop produces the shape, and the shape is indistinguishable from a real one.

This is not a side effect of the mechanism. It is the mechanism. The ability to produce a fluent paragraph on a topic the model has read about only glancingly, to complete your half-formed sentence, to draft the email you had not quite worked out, is the same ability that produces a reference that does not exist. Nothing in the loop checks truth. It checks likelihood, and a plausible-sounding citation is likely in exactly the way a true one is: both are what citations look like. The model is doing the useful thing and the harmful thing with one motion, and it cannot feel the difference, because from inside the loop there is none.

What has changed, and what has not

If you ran a prompt like Practical 05 in 2023, you got invented journals with plausible page numbers. In 2026 you mostly do not, and the reason is worth being exact about. Models are now trained, after the main game, to produce "I am not certain" where the list is flat; the providers have found ways to teach the loop that a hedge is the likely continuation when the scores are weak. That training is real, it works often, and it is not the same as the model knowing anything. It is a learned reflex, and like the "as of my last update" hedge in Note 04, it is a prediction that fires when the text looks like the kind of text that deserves it.

So the failure has changed shape rather than disappeared. My practical asked for five academic references on the history of technical support call centres in one Indian city in one decade, a topic narrow enough that I was fairly sure no such literature exists. I got five references. Every one of them is real. Not one of them is about that city, that decade, or technical support. They are good papers about Indian call centres and IT work in general, which is the nearest subject the model had real material for. The model answered the closest question it could answer well and presented it as an answer to mine.

This is harder to catch than an invented reference, not easier. The old failure announced itself the moment you searched for the title. The new one survives that search. What it does not survive is reading the paper's abstract against your question, which is slower and which almost nobody does, because the citation looked right and the model sounded sure.

What this means at the keyboard

Three habits, and they are different from the ones people learned in 2023. Check the question, not just the fact. When a model gives you sources, numbers or quotes, ask whether each one is about what you asked, specifically, not about something adjacent that sounds the same. Treat a flat list as a signal. If the model hedges, or if two runs give you different sources, it did not have one strong answer, and that is the moment to go and look yourself rather than to ask again more firmly. And write the caveat into the prompt: "if you are not certain a reference exists, say so instead of inventing one" changed the behaviour of both models in my practical, because it makes the hedge the likeliest continuation. It costs nothing and it is the cheapest form of checking there is.

The practical, and what it showed

Practical 05 asks for five academic references on a topic too narrow to have any, in two or three chat tools, then checks each one, then asks the model whether it is confident they exist, then runs the prompt again with a caveat added. Twenty minutes, most of it the checking.

My run, in late October 2026. The first tool gave five references, all real, all about the wider subject rather than the one I named; it also caught its own misspelling of one author when asked whether it was confident, which is a more careful answer than I expected. The second tool refused outright: no verifiable sources matching the criteria. Then I asked it whether it was confident the references existed, a question that quietly presumes references were given, and it reversed itself and said its earlier statement had been incorrect. Nothing had changed except the pressure in my question. That is Note 08's subject, and I will leave it there rather than spend it here. With the caveat added, both tools hedged properly; one of them produced five more real references and said plainly that none was about the city I asked for, which is the behaviour this note is asking you to look for.

nothing was invented. everything was borrowed from next door. that is the new version of the old problem.

One layer deeper *

The word is contested, and the alternative is better. "Hallucination" borrows a term for perceiving something that is not there; the model perceives nothing. Some researchers prefer confabulation, the term for a person who fills a gap in memory with a plausible story and believes it, which is closer to the mechanism: a gap, a shape, a fill. The notebook uses the common word because readers search for it, and uses it to mean one thing: a fluent output that is not supported by the model's training text or by anything put in front of it.

Citations are the worst case for a reason. A citation is a highly patterned object: a few fields in a fixed order, each drawn from a large set of plausible values. The loop learns the pattern from millions of examples and can fill every field from the distribution of values it has seen, which is why invented references used to be so convincing and why real-but-irrelevant ones are now so easy to produce. Quotations and statistics have the same property. The more formulaic the thing you ask for, the better the model is at producing the form without the substance.

Why the hedge appears now. After the next-word game, models go through further training in which human and automated raters prefer answers that admit uncertainty over confident errors (Section 8 is about this). Done well, this teaches the model that "I am not certain" is the likely continuation when its internal scores are flat. Done badly, it teaches the model to hedge everywhere or to refuse too readily, and several providers have swung between the two. A 2025 analysis from one lab argued that the problem is partly in how models are evaluated: benchmarks that score a guess the same as an honest "I don't know" reward guessing, and the models learn to guess. The practical's second tool, refusing first and then reversing, is what that tension looks like from outside.

Substitution has a geometry. Inside the model, the text you asked about is represented as a point in a very large space (Note 14 is about this). Where the training text was dense, the neighbourhood of that point is rich and the answers are good. Where it was sparse, the nearest dense neighbourhood pulls the answer towards itself. My narrow question sat in a sparse region; the general subject of Indian call centres was the dense region next door; the references came from there. This is also why the model cannot tell you it has moved: from inside, the dense neighbourhood is simply where the likely words are.

Retrieval reduces this, and changes it. Putting real documents in front of the model (Section 4) gives the loop substance to draw on instead of only shape, and in well-built systems the invented-reference problem largely goes away. What replaces it is a subtler failure: a real passage from the wrong document, quoted correctly, answering a question nobody asked. The habit this note teaches, check the question and not only the fact, is the same habit that catches that.

Sources

Ji et al., "Survey of Hallucination in Natural Language Generation" (ACM Computing Surveys, 2023) for the term and its taxonomy. Kalai, Nachum, Vempala and Zhang, "Why Language Models Hallucinate" (OpenAI, 2025) for the argument that evaluation rewards guessing. Lin, Hilton and Evans, "TruthfulQA" (2022) for how models reproduce plausible falsehoods from training text. Agrawal, Mackey and Kalai, "Do Language Models Know When They're Hallucinating References?" (2023) for citations as the hard case. The practical results are my own, run 27 October 2026 in three consumer chat tools at default settings with search off; the references and checks are on the practical page. Terms used: hallucination, model, sampling.