Why the same question gives different answers: temperature and sampling

Ask a chat tool the same question twice and you will often get two different answers. People read a lot into that. It must be moody. It must be learning between the two. It must not really know. None of those is what is happening, and the real reason is one small dial that Note 01 left turned away from you.

Note 01 ended each turn of the loop with a list: a score for every possible next word. It then said "one is picked" and moved on. This note is about the picking. Given a list of scored words, which one gets written? That choice is called sampling, and temperature is the dial that controls how adventurous it is.

The dial reshapes the list before the draw

Think of the scores as a lottery. Every word on the list holds some tickets; the top word holds the most. One ticket is drawn, and that word is written. Temperature changes how the tickets are handed out before the draw.

At temperature 0 the top word gets every ticket, so the draw is not a draw at all: the highest-scoring word wins, every time, at every turn of the loop. The whole answer becomes repeatable. Ask twice, get the same words twice. As the dial goes up, tickets are spread towards the lower words. The top word still usually wins, but not always, and each time it does not, the answer takes a different path and everything after it changes too. That is the entire mechanism behind "the same question gives different answers". Nothing was learned, nothing was moody. A different ticket came out.

Not every question is a lottery

Here is the part that took me a practical to see. The dial only matters when the list has more than one strong candidate. Some questions have that shape; some do not.

"Name a bookshop in Pune" has hundreds of good answers, so the list is flat: many words with similar scores. Turn the dial up and you get five different names in five runs. "What does a context window mean?" has one right answer that everyone phrases roughly the same way, so the list is peaked: at each turn one word is far ahead. Turn the dial up and the meaning barely moves; only the odd word wobbles, "amount" for "length", "processing or generating" for "generating".

So the title of this note is slightly wrong, and usefully so. The same question does not always give different answers. It gives different answers when it has several to choose from. A chat tool giving you two different answers to a factual question is telling you something: the list was flat, which means the model did not have one strong candidate, which is worth knowing before you trust either.

What this means at the keyboard

You cannot see the dial in a chat tool, but it is there, set by the product, usually somewhere around the middle. That is a deliberate choice: a little spread makes answers feel natural and less repetitive, and it is why two colleagues asking the same thing get two paragraphs that differ in wording. In a playground or through an API you can set it yourself, and then the rule of thumb is simple. Low when you need the same output every time: extracting fields from a document, classifying support tickets, writing code, anything you will test. High when you want spread: names, ideas, first drafts to react against, anything where five different answers is the point.

One warning, because the wrong lesson is easy to take. Temperature 0 is not "more accurate". It is more repeatable. If the top-scoring word is wrong, temperature 0 gives you that wrong word every time, with perfect consistency. Repeatable and right are different properties, and the notebook keeps them apart.

The practical, and what it showed

Practical 03 needs a place where you can set the dial, which any of the free playgrounds gives you; there is a fallback for a plain chat tool. Two prompts, one flat and one peaked, five runs each at temperature 0 and at 1. Twenty short runs, about twenty minutes. My results are the drawing above; the full table is on the practical page. What I did not expect was how clean the temperature-0 runs were: five identical names, five identical sentences, character for character. The "one layer deeper" section says why that is not guaranteed.

the dial doesn't change what the model thinks. it changes how often it says its second thought.

One layer deeper *

Where the word comes from. The raw scores the model produces are turned into probabilities by an operation called softmax, and before that step each score is divided by the temperature. Divide by a number below 1 and the gaps between scores stretch, so the top word pulls away; divide by a number above 1 and the gaps shrink, so the list flattens. At exactly 0 the division breaks down and tools switch to simply taking the maximum, which is called greedy decoding. The name is borrowed from physics, where the same formula describes how much a system at a given temperature explores states other than its lowest-energy one.

Temperature is not the only dial. Most tools also cut the list before drawing. Top-k keeps only the k highest words; top-p (nucleus sampling) keeps the smallest set of words whose shares add up to p, say 90%, so a flat list keeps many candidates and a peaked list keeps few. These exist because at high temperature the long tail of a hundred thousand rare words starts winning tickets, and text degenerates into nonsense. Cutting the tail keeps the spread without the nonsense.

Why temperature 0 is not a guarantee. My ten runs came back identical, and usually they do, but "usually" is the honest word. The arithmetic inside a model runs on hardware that adds numbers in slightly different orders depending on what else is being processed at the same time, and two scores that are nearly tied can swap places. Some model designs route different requests through different internal pathways, which adds more of this. If you build something that depends on identical output, test for it rather than assume it, and keep a record of the exact model version, because the version changes underneath you (Note 24).

"Creativity" is the wrong word. Products sometimes label the dial creative versus precise. It is worth being exact about what is and is not happening: the model's ranking of words is the same at every setting. Temperature changes only how often a lower-ranked word is chosen. It cannot produce an idea the model had not already scored; it can only let a lower-scored one through. That is spread, not imagination, and treating it as imagination leads people to turn the dial up for tasks that needed it down.

What Note 01's results now mean. The five different continuations in Note 01 were the default temperature of a chat tool at work on a flat list. Run the same practical at temperature 0 and you would get five identical continuations, which is a good way to convince yourself that the variation was the dial, not the model.

Sources

Holtzman et al., "The Curious Case of Neural Text Degeneration" (2020) for nucleus sampling and why high-temperature text degenerates. Ackley, Hinton and Sejnowski, "A Learning Algorithm for Boltzmann Machines" (1985) for the temperature formula's origin in this field. Karpathy, "Intro to Large Language Models" (2023). The practical results are my own, Claude Haiku 4.5 in a playground with the temperature set by hand, 8 September 2026. Terms used: model, token, context window, temperature.