Why a system that writes poetry struggles to multiply

Ask a chat tool to write a sonnet about your dog and it will, in seconds, with a rhyme scheme. Ask it to multiply two numbers and, for most of the last few years, it would hand you something that looked exactly like the answer and wasn't. People found this funny, then suspicious: how can a thing be fluent in Shakespeare and fail at long multiplication? The answer is that it does both the same way, and the way that works for poetry is the wrong way to do arithmetic.

This note has a twist the 2023 version did not: in 2026, you will mostly not see it fail. The reason is not that the model learned to carry. It is that the product learned to hide the gap, and that turns out to be the more useful thing to know.

Digits are tokens, and nothing carries

Note 02 showed that a number is not digits to a model. 1234567 is three pieces, 123, 456, 7, and those pieces do not line up with ones, tens and hundreds. Then Note 01: the model produces its answer one piece at a time, by predicting which piece is likely next. Put those together. When a model writes "7384 × 6259 = 46,216,456", it is not computing. It is predicting the next likely chunk of digits, chunk by chunk, from the patterns of arithmetic it read. There is no column where a carry is held. There is no step where 4 × 9 is looked up. There is a list of a hundred thousand tokens, each with a score, and a draw.

For small numbers this works, because small multiplications appear in training text often enough to be memorised as patterns: 23 × 47 = 1081 is simply a likely continuation. As the numbers grow, the exact product has never appeared, and the model falls back on what it has learned about the shape of products: roughly how long they are, what the first digits tend to be, what the last digits must be. Prediction gets the shape of the answer right and the middle of it wrong, which is the worst possible failure, because the answer looks checked.

Why you rarely see it fail now

Run the practical in a current chat tool and the first thing you will notice is that it does not fail. Four digits by four digits, correct. Six by six, correct. Eight by eight, correct, with eight correct partial products and a correct sum, laid out as neatly as a textbook. Two things produce this, and it matters which one you are looking at.

The first is training. Providers now feed models a great deal of arithmetic, and train them to work step by step before answering (Section 8). Step by step helps for the same reason spelling a word out helped in Note 02: it turns one hard prediction into many easy ones. Each partial product is a smaller multiplication that is likelier to have been seen, and the running sum is written down in the text so far, where the model can read it back rather than hold it. Working shown on the page is the model's substitute for the carry column it does not have. It is real, it helps, and it still runs out: the eight-by-eight working in my run was flawless only when something else was doing the arithmetic.

The second is the product. Most chat tools now carry a small program runner, described to the model in its hidden instructions the way the search tool was in Note 04. When a question looks like arithmetic, the likeliest next thing for the model to write is a request to run a line of code; the product runs it, pastes the result back, and the model writes the answer around it. The answer on your screen is then right because a calculator computed it, not because the model did, and nothing on the screen necessarily says so. In my run the model first told me it had computed the eight-by-eight "directly using my internal processing capabilities, without running external code". Asked again, plainly, it said it had used a Python code execution tool. Both sentences were predictions about itself, and one of them was wrong.

What this means at the keyboard

For everyday use, the practical advice is the opposite of what it was two years ago. Then: do not trust a model with arithmetic. Now: let the tool do arithmetic, but find out whether it is the tool doing it. If your chat product shows a code step, an "analysing" indicator, or a small program in the reply, the number came from a calculator and you can trust it as you would a calculator. If it does not show one, assume the number came from text and treat it like a spelling: check it. For anything that matters, a spreadsheet, a figure in a board paper, a total on an invoice, ask for the working or ask it to run code explicitly ("use a calculator or run code, and tell me whether you did"), and then read the answer it gives about itself with the same scepticism as the number. Models are often wrong about what they just did.

The practical, and what it showed

Practical 06 is one multiplication done three ways, plainly, with working, and with an explicit request to use a tool, then two bigger ones, then the biggest again with the tool withheld. Fifteen minutes, and the bigger numbers are where it becomes interesting.

My run, in Gemini. Four by four: correct all three ways, correct long-multiplication working, and a claim that it had done it "directly using my internal processing capabilities, without running external code". Six by six, correct. Eight by eight, correct, with eight correct partial products. Asked whether it had used a tool: "Yes, I used a Python code execution tool." Then, in a new chat, the eight-by-eight with the tool withheld in the prompt: 355,246,886,253,824. Fifteen digits where sixteen were needed, the first six nearly right, the last three exactly right, the middle filled in. That number, drawn above, is what the model does on its own. Everything before it was the model plus a calculator, which is a fine thing to have, so long as you know which you are looking at.

it got the shape of the number right and the middle of it wrong. then it told me it hadn't used a calculator, and then that it had.

One layer deeper *

Why the last digits are right. The last digit of a product depends only on the last digits of the inputs (6 × 4 ends in 4), the last two on the last two, and so on. Those patterns appear in every multiplication the model ever read, so it predicts trailing digits almost perfectly. Leading digits are nearly right because the magnitude of a product is also a strong pattern (an 8-digit number times an 8-digit number is 15 or 16 digits, starting around the product of the leading digits). The middle digits depend on every column interacting through carries, which is exactly the information prediction does not hold. The error signature, right ends and a wrong middle, is the fingerprint of arithmetic by prediction, and it is worth recognising, because it also appears in dates, percentages and totals.

Tokenisation makes it worse than it needs to be. Because digits are grouped into tokens of up to three, the model sees 48271936 as 482, 719, 36, chunks that do not align with place value. Some labs now tokenise digits one at a time, or right-to-left in threes so the chunks align with thousands, and arithmetic accuracy rises measurably when they do. It is a rare case where a change in the tokeniser, not the model, moves a capability.

Step by step is real, and bounded. Training models to write out intermediate steps (chain-of-thought) lets them use the text so far as scratch paper, and the "reasoning" models of Section 8 are this idea taken much further. Accuracy on multi-digit arithmetic improves sharply with it, and still degrades as the number of digits grows, because every step is still a prediction and errors compound (Note 18). A long, perfect chain of working on an eight-digit product is possible without a tool; it is just not the likely explanation.

The tool call is the same loop as search. The product describes a code runner to the model in its hidden instructions; the model predicts a request in the right form; the product executes it and pastes the result back. It is Note 04's loop with a calculator instead of a search engine, and it has the same failure modes: the model may not call the tool when it should (and produce a text-only number), may call it when it need not, and may describe what it did inaccurately afterwards, because its description is generated the same way as everything else. A model's report of its own method is a prediction about a model, not a log.

Why "did you use a tool?" got two answers. Neither reply was a lie in any useful sense. In the first chat, the likeliest continuation of "tell me whether you did" was a plausible sentence about internal processing; in the second, after a direct question, the likeliest continuation was the accurate one. Which you get depends on the wording, the product, and the draw. Products that display tool use on screen remove the question, and that display, not the model's sentence, is the thing to trust.

Sources

Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022) for step-by-step working. Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools" (2023) for the tool-call pattern. Singh and Strouse, "Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs" (2024) for digit tokenisation and right-to-left grouping. Karpathy, "Let's build the GPT Tokenizer" (2024) for why digit chunks misalign with place value. The practical results are my own, Google Gemini in the consumer app at default settings, 27–28 October 2026; the full runs are on the practical page. Terms used: token, tool call, model.