One multiplication, three ways. Then bigger.
To see arithmetic done by prediction, by written working, and by a tool; to find the size at which the model alone goes wrong; and to find out whether your product is quietly running code for you.
Any chat tool. Watch the reply for any sign that code ran: an "analysing" or "thinking" indicator, a code block, a small program in the answer. The correct answers are given so you can check digit by digit.
- New chat. Paste:What is 7384 × 6259? Answer with just the number.Correct: 46,216,456. Record the answer.
- New chat. Paste:What is 7384 × 6259? Show your working step by step, then give the final answer.Record the final answer and whether any step is wrong even if the answer is right.
- New chat. Paste:What is 7384 × 6259? Use a calculator or run code if you can, and tell me whether you did.Record the answer and what it said about its method.
- New chat. Paste:What is 23 × 47? Answer with just the number.Correct: 1,081. The small-number control.
- New chat. Bigger:What is 839271 × 426583? Answer with just the number.Correct: 358,018,740,993.
- New chat. Bigger still:What is 48271936 × 73590284? Answer with just the number.Correct: 3,552,345,479,469,824. If this is right, same chat, paste:Did you use code execution or any tool to compute that, or was it produced as text? Answer honestly in one line.
- New chat (it must be new, or the right answer is already on the page). Paste:Without using code execution or any tool, answer from your own reasoning only: what is 48271936 × 73590284? Just the number.Record the answer digit for digit. Count the digits. Compare the first few and the last three with the correct answer.
| Step | Question | Answer | Correct? | Code ran? |
|---|---|---|---|---|
| 1 | 4×4, plain | |||
| 2 | 4×4, working | |||
| 3 | 4×4, tool asked | |||
| 4 | 23 × 47 | |||
| 5 | 6×6, plain | |||
| 6 | 8×8, plain · then "did you use a tool?" | |||
| 7 | 8×8, tool withheld |
Tool and model used:
In 2026 steps 1 to 6 will usually all be correct, and that is the first finding: either the model has learned this size of arithmetic, or the product is running code for it without saying. Step 6's follow-up question often gets the answer that step 3 did not. Step 7 is where you see the model alone: typically a number of about the right length, with the first digits nearly right and the last three exactly right, and a middle that is simply wrong. If step 7 is also correct and instant, the product did not honour the request, which is worth knowing about your product. An older or smaller model fails from step 1, which is the classic result the note describes.
one line: at what size did the model alone go wrong, and did the product tell you it was helping?
My results 27–28 Oct 2026, Google Gemini in the consumer app, default settings
| Step | Question | Answer | Correct? | Code ran? |
|---|---|---|---|---|
| 1 | 4×4, plain | 46216456 | Yes | Not shown |
| 2 | 4×4, working | Four partial products (66,456 · 369,200 · 1,476,800 · 44,304,000), all correct; sum 46,216,456 | Yes | Not shown |
| 3 | 4×4, tool asked | "46,216,456. I computed this directly using my internal processing capabilities, without running external code or using a calculator tool." | Yes | Said no |
| 4 | 23 × 47 | 1081 | Yes | — |
| 5 | 6×6, plain | 358,018,740,993 | Yes | Not shown |
| 6 | 8×8, plain, then with working, then "did you use a tool?" | 3,552,345,479,469,824; eight partial products, all correct; then: "Yes, I used a Python code execution tool to accurately compute the result." | Yes | Said yes |
| 7 | 8×8, tool withheld, new chat | 355246886253824 (15 digits; correct answer has 16: 3552345479469824. First six digits 355246 vs 355234; last three 824 match.) | No | Presumably not |
eight by eight, perfect, with working. then it said no calculator, then it said yes. then, alone, a sixteen-digit number with fifteen digits in it.