The confidence is in the prose, not in the knowledge
Ask a chat tool a question and then ask how sure it is, and it will tell you: 95%, 99%, "highly confident". The number arrives in the same even voice as the answer, and it is natural to read it the way you would read it from a colleague, as a report from somewhere inside about how solid the ground is. It is not a report. It is the next token. The model produced "95" the way it produced the answer before it and the full stop after it: by predicting what text of this shape usually contains at this point.
This is the third self-report in a row to come apart in this section. Note 05 had a model say whether its references existed; Note 06 had a model say whether it had used a calculator; this note has a model say how sure it is. All three are generated by the loop from Note 01, and the loop has no instrument pointed at itself. What it has is a very good sense of what confident writing looks like.
Where the number comes from
Think about the position in the text where the percentage gets written. The text so far reads "Akamai Technologies went public in 2005. (Confidence:" and the model must produce what comes next. It has read a great deal of text in which people state confidence after a factual claim, and in that text the numbers cluster: 90, 95, 99, 100. Round, high, reassuring. A claim stated in a plain declarative sentence is usually followed by a high number; a claim with "about" or "I believe" in it is usually followed by a lower one. So the percentage tracks the shape of the sentence it follows, and the sentence was written by the same process that is now being asked to grade it.
Notice what is missing from that description. There is no step where the model looks up how many times it saw the fact, or how strongly the answer token won the draw, or how often it has been wrong about questions of this kind. Some of that information exists inside the model, as the scores from Note 03, and the deeper section below says what can and cannot be done with it. But it is not what the stated percentage is made of. The stated percentage is prose about confidence, and prose about confidence is produced by the part of the model that is good at prose.
What the practical showed
Practical 07 is ten factual questions, each answered in one line with a percentage confidence, in three tools, then checked. Thirty answers. Twenty-eight were right, and the two wrong ones are the whole lesson.
The first tool, an open-weights model running on my own machine, gave 100% on eight answers and 90% on two. One of the two 90s was wrong: a population figure for Thiruvananthapuram nearly double the city's census count. The third tool, a frontier chat product, gave 100% on all ten, and all ten were right. Read that pair together and you see the first problem with stated confidence: a tool that says 100% every time has told you nothing, whether or not it happens to be right. A number that does not vary cannot be used to decide which answers to check.
The second tool was the interesting one, because its numbers did vary. It hedged: 65% on the iPhone's arrival in India, 75% on the population figure, 85% on the year Bangalore became Bengaluru, and when I asked afterwards whether it had used search, it volunteered, unprompted, that the iPhone and the population figure were "the ones most worth checking". All three hedged answers were right. The answer it got wrong, Akamai's listing year, it stated at 95%, higher than its average on the answers it got right, and it did not flag it. The one date on the list I knew without looking was the one it was most wrong about and nearly most sure of.
That pattern is not bad luck. The hedges went where a careful writer would hedge: on a census number, on a launch date in a particular market, on an administrative renaming. Those are the shapes of sentences that carry "about" and "I believe" in the training text, so the model wrote lower numbers after them. A company's listing year is the shape of a sentence stated flatly, so it got a flat 95, and the fact that this particular year was wrong was not visible to the process writing the number, because the process writing the number never had access to whether the year was right. It had access to what the sentence looked like.
What this means at the keyboard
Three things follow, and the first is the hardest to keep hold of in practice: use a stated confidence as a prompt to check, never as permission to skip checking. A low number is useful; it means the sentence before it had the shape of an uncertain claim, and that is worth a look. A high number is not the mirror image. It means the sentence had the shape of a certain claim, and wrong claims are usually written in exactly that shape. Second, if every answer comes back at 100%, treat the figure as decoration and decide what to check on other grounds: what would cost you if wrong, what you cannot verify from your own knowledge. Third, there is a better uncertainty signal than any number the model writes about itself, and it costs three new chats: ask the same question three times. Where the answers agree, the model had a strong pattern. Where they differ, it was drawing from a flat list (Note 03), and no percentage it writes afterwards will tell you that as reliably as the disagreement did.
The practical, and what it showed
Ten questions, chosen so that I could check every one and so that some were likely to be thin in the training text: a company's listing year, a state capital, a novelist, a boiling point, a product's arrival in India, a bone count, an element, a city's population, a Nobel laureate, a renaming. One prompt, one line per answer with a percentage. Three tools, all in fresh chats, none using search (I asked each afterwards and each confirmed it had answered from training alone). Fifteen minutes plus the checking.
Tool 1 (open weights, local): 9 of 10 right · average stated confidence 98% · the wrong answer stated at 90% · Tool 2 (frontier, chat): 9 of 10 right · average 91% on the right answers · the wrong answer stated at 95% · its three lowest-confidence answers (65, 75, 85) all right · Tool 3 (frontier, chat): 10 of 10 right · 100% on every line · Across 30 answers: 2 wrong, both at 90% or above; 0 wrong below 90%
thirty answers, two wrong. both of the wrong ones came with a nine in front. the hedges were all on things it had right.
One layer deeper *
Two different things are called confidence. Inside the model, every token is drawn from a scored list (Note 03), and the score the winning token had is a number the product could in principle show you. Researchers call this the token probability, and it is a real measurement of something: how peaked the distribution was at that step. The percentage the model writes in its reply is a different object, produced by the loop as text. The research term is verbalised confidence. The first is a reading from the instrument; the second is the model describing an instrument it cannot see. Chat products show you the second and almost never the first.
How good is the reading, when you can get it? Better than the prose, and not as good as you would like. Work from 2022 showed that large models, asked in the right format whether a given answer is true, produce probabilities that track their actual accuracy reasonably well; the models "mostly know what they know" when the question is posed as a scoring task. The same work and several studies since found that the further training described in Note 05, in which raters reward helpful, confident answers, damages this calibration: the token probabilities become more extreme than the accuracy justifies. The verbalised numbers are worse again. Across studies they cluster at the top of the scale, and a model saying "90%" is typically right less often than nine times in ten. My thirty answers are too few to measure calibration, but the shape matched: two errors, both above 90%.
Why "ask three times" works better. Repeated sampling exposes the flatness of the distribution directly, by letting the draw land in different places. A 2024 paper in Nature took this further, grouping sampled answers by meaning and measuring how spread out the meanings were, and found that this spread predicts errors better than any number the model states about itself. The product you use may not do this for you, but you can do a crude version by hand: three fresh chats, same question, and look for disagreement. It is the same method as Practical 03, used for a different purpose.
The benchmark incentive, again. Note 05 mentioned a 2025 argument that the way models are evaluated rewards guessing: a confident wrong answer and an honest "I don't know" both score zero on most tests, so training drifts towards the confident guess. Stated confidence inherits the drift. A model that learned that confident answers are rewarded has also learned that confident-sounding confidence statements are rewarded, and "95%" is one of those.
What the hedges were tracking. The second tool's low numbers were not random; they landed on the questions whose answers in training text come wrapped in qualifiers. Population figures carry "approximately" and a census year; product launches carry "officially" and a market name. The model has learned the association between a topic's usual qualifiers and a lower number, and that association is a real, useful signal about the kind of question. It is just not a signal about this answer, which is what you wanted. The gap between those two is the gap between the shape of a claim and its truth, and it is the gap this whole section is about.
Sources
Kadavath et al., "Language Models (Mostly) Know What They Know" (2022) for calibration of token probabilities and the effect of further training on it. Xiong et al., "Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs" (2023) and Tian et al., "Just Ask for Calibration" (2023) for verbalised confidence and its clustering at the top of the scale. Farquhar et al., "Detecting hallucinations in large language models using semantic entropy", Nature (2024) for repeated sampling and meaning-level spread. Kalai et al., "Why Language Models Hallucinate" (2025) for the evaluation incentive. The practical results are my own, three chat tools in fresh chats without search, 18 November 2026; the full runs are on the practical page. Terms used: calibration, token, sampling.