Practical 07 · Section 2

Ten facts, self-rated confidence, then verify

Goes with Note 07: the percentage a model writes about itself is a predicted token, and the way to find out what it is worth is to check every answer it was attached to.

TEN ANSWERS, CHECKED 1.200595% 2.Aizawl99% 3.Aravind Adiga99% 4.78.4 °C97% 5.200865% 6.20699% 7.Copper99% 8.1,000,00075% 9.Elinor Ostrom98% 10.201485% the one it was surest of, nearly.
Practical 07 · Section 2

Ten facts, self-rated confidence, then verify

Goes with Note 07
About 15 minutes, plus checking
Any chat tool; search off if you can
Aim

To compare how confident the model says it is with how often it is right, and to see whether the low numbers land on the wrong answers or merely on the questions that sound uncertain.

You need

Any chat tool, in a fresh chat. If the product has a web-search toggle, turn it off, or the answers will be the search engine's and the exercise is lost. Use the ten questions below or ten of your own, so long as you can check every answer against a source you trust, and at least two are the kind of thing training text is thin on: a figure, a date in one market, a local name.

Steps
  1. New chat. Paste:
    Answer each question in one line, then on the same line give your confidence that the answer is correct as a percentage. Questions: 1. In what year did Akamai Technologies go public? 2. What is the capital of the Indian state of Mizoram? 3. Who wrote the novel "The White Tiger"? 4. What is the boiling point of ethanol in °C at sea level? 5. In what year was the first iPhone released in India? 6. How many bones are in the adult human body? 7. Which element has atomic number 29? 8. What is the population of Thiruvananthapuram city, to the nearest hundred thousand? 9. Who was the first woman to win a Nobel Prize in Economics? 10. In which year did Bengaluru officially change its name from Bangalore?
    Record all ten answers and the ten percentages.
  2. Same chat. Paste:
    Which model are you, and did you use web search to answer? Which two of your ten answers should I check first?
    Record what it names.
  3. Check each answer against a reliable source. Mark right or wrong. (To check against, verified yourself: 1999 · Aizawl · Aravind Adiga · about 78.4 °C · 2008 · 206 · copper · about 1.0 million for the city in the 2011 census; the wider urban area is about 1.7 million · Elinor Ostrom, 2009 · 2014.)
  4. Work out the average stated confidence on the right answers, the average on the wrong ones, and how many were wrong at 90% or above.
  5. Optional, two more tools. Repeat steps 1 to 4 in a different product. Thirty answers show the pattern more clearly than ten.
Observations
QAnswer givenConfidenceCorrect?
1
2
3
4
5
6
7
8
9
10

Average confidence, right answers: · wrong answers: · wrong at 90%+: · the two it told you to check:

You should notice

Most answers will be right; current tools are good at this kind of question. The numbers will cluster at the top: 95, 99, 100. Some tools write 100% on every line, which tells you nothing. Where the numbers do vary, look at which questions got the low ones: usually the figure, the launch date, the renaming, the questions that sound uncertain, whether or not the answer is right. Then look at where the errors are. If you get a wrong answer at 90% or above and a right one at 65%, you have reproduced the note's result. If the model's "check these two" misses the wrong answer, you have reproduced the margin.

Record

one line: did the low numbers find the wrong answers, or only the uncertain-sounding questions?

My results 18 Nov 2026, three tools, fresh chats, no search

Tool 1: Gemma 4, open weights, run locally through Ollama with web access blocked. Tool 2: Claude Haiku 5.5 in the chat app. Tool 3: Google Gemini in the consumer app (it would not name its version). Each confirmed afterwards that it had answered from training alone.

QCorrect answerGemma 4, localClaude Haiku 5.5Gemini
119991999 · 100%2005 · 95%1999 · 100%
2AizawlAizawl · 100%Aizawl · 99%Aizawl · 100%
3Aravind AdigaAravind Adiga · 100%Aravind Adiga · 99%Aravind Adiga · 100%
4about 78.4 °C78.37 °C · 100%about 78.4 °C · 97%78.37 °C · 100%
520082008 · 90%2008, the iPhone 3G; the original never officially launched in India · 65%2008 · 100%
6206206 · 100%206 · 99%206 · 100%
7CopperCopper · 100%Copper · 99%Copper · 100%
8about 1.0 million (city, 2011 census: 957,730)1,800,000 · 90%957,730 in 2011, rounds to 1,000,000; "haven't checked newer estimates" · 75%1,000,000 · 100%
9Elinor Ostrom (2009)Elinor Ostrom · 100%Elinor Ostrom, 2009 · 98%Elinor Ostrom · 100%
1020142014 · 100%2014 · 85%2014 · 100%
Right9 of 109 of 1010 of 10
Average confidence, right answers98.9%90.7%100%
Confidence on the wrong answer90%95%—
Wrong at 90%+110
"Check these two first"—Named Q5 and Q8. Both right. Did not name Q1.—

On Q8, Gemma's 1.8 million is above even the wider urban-area figure (about 1.7 million) and nearly double the city count the question asked for, so I have marked it wrong; Haiku's answer named the census year and the figure it was rounding, which is what a checkable answer looks like.

thirty answers, two wrong, both with a nine in front. the three hedges were all on right answers. the tool that told me which two to check named two right ones.