# Same prompt, different answer > Run a toy model on the page ten times at two temperatures, count how often the answers agree, and see why one good run proves nothing and what check a workflow built on a model needs. Page: https://schubergphilis.github.io/ai-training/concepts/same-prompt-twice/ License: CC BY-SA 4.0, https://creativecommons.org/licenses/by-sa/4.0/ In the lesson on how a model works you saw the widget score five candidate tokens and pick one, and you saw that the pick is where the randomness is. In this lesson we run that pick ten times in a row and count. The toy model answers one fixed question, and we run it at two temperatures, then with a small wobble in the scores, and then with a check applied to every answer. By the end you can say what one good answer proves about a prompt, and what a process built on a model needs instead. You need only this page. The toy model is a widget further down, and it runs in your browser. It draws new random numbers on every press, so your runs differ from the ones this page shows. Each set of runs shown on the page is one recorded set from the same toy model. ## Run it at temperature 0 The toy model holds five candidate answers to "What is the capital of Australia?" with made-up scores, in the same spirit as the widget in the lesson on how a model works. Four of the candidates name Canberra in different wordings, and the fifth, with the lowest score, says Sydney. Each run turns the scores into probabilities at the temperature you set, draws one answer, and lists it. The summary line counts how many runs gave the same text as the first run. At temperature 0 the probabilities collapse onto the top score. Before you press the button, predict the summary line for ten runs. #### Checkpoint: Predict the summary at temperature 0 You set the toy model to temperature 0, leave both boxes unchecked, and press the button for ten runs. What does the summary line say? - 10 of 10 runs give the same answer as run 1 - 9 of 10 runs give the same answer as run 1 - A count between 3 and 6, different on every press - 1 of 10 runs give the same answer as run 1 Now try it. Set the temperature to 0, leave both boxes unchecked, and press the button. Every line reads `Canberra`, because with all the probability on one candidate the draw has only one place to land. The summary line is the same on every press: ```text 10 of 10 runs give the same answer as run 1 ``` ## Raise the temperature Now raise the temperature. Move the slider in the widget above to 1.0 and press the button again. The scores are unchanged, and so is the order of the candidates. The candidates below the top one keep more of the probability. Here is one recorded set of ten runs at 1.0: ```text What is the capital of Australia? (temperature 1.0, 10 runs) 1 Canberra 2 Canberra. 3 Canberra 4 The capital of Australia is Canberra. 5 Canberra 6 Canberra. 7 It's Canberra. 8 Canberra 9 Sydney 10 The capital of Australia is Canberra. 4 of 10 runs give the same answer as run 1 ``` Ten runs, five different texts, and one of them wrong. This is **non-determinism**: the same prompt gives different answers on different runs, because each answer is a draw from the probabilities and the draw can land anywhere the probability is above zero (AI capabilities and limitations, Claude Academy). Nothing about the prompt changed between run 1 and run 9. Run 9 says Sydney because Sydney had a few percent of the probability and, on the ninth draw, the number came up. Press the button a few more times and tally how often the ten runs agree with run 1. The count moves. The exact value is the toy's, but the picture is the real one, and it is the picture behind the exercise in the lesson on how a model works, where the opening sentence of a book came out differently in two fresh chats. ## Variation at the lowest temperature Temperature 0 gave ten identical answers, so it is tempting to decide that a workflow only has to set the temperature to 0 and the variation is gone. On a real model it isn't. The scores the toy holds as fixed numbers come, in a real model, from billions of multiplications done on hardware that rounds. The order of those multiplications changes with the other requests served alongside yours, and rounding in a different order gives a score that differs in the last decimal places. When two candidates are close, that last decimal is enough to swap which one is on top, and the greedy pick follows the swap. The box "Shake the scores before each pick" in the widget imitates that wobble. Before each pick it shakes every score a little. The shake is far larger than the rounding wobble in a real model, so ten runs are enough to see the effect. Set the temperature back to 0, check the box, and press the button a few times. Here is one recorded set: ```text What is the capital of Australia? (temperature 0.0, 10 runs) 1 Canberra 2 Canberra 3 Canberra 4 Canberra 5 Canberra. 6 Canberra 7 Canberra 8 Canberra 9 Canberra 10 Canberra 9 of 10 runs give the same answer as run 1 ``` In this set one run in ten differs from the rest, at the lowest temperature there is. In the toy the difference is a full stop. In a real model the swapped token is somewhere in the middle of a long answer, and every token after it is predicted from a slightly different prefix, so two temperature-0 answers can share their first sentence and part ways after it. The Messages API reference says the same about the real thing: even at a temperature of 0 the results aren't fully deterministic (Messages, Claude Platform documentation). On some current models you can't change the setting at all. The same reference marks temperature as deprecated: Claude models newer than Opus 4.6 refuse a request that asks for any temperature except the default of 1.0 (Messages, Claude Platform documentation). Put the two results together. At temperature 1 a run can pick a wrong answer that has a few percent of the probability. At temperature 0 a run can still differ from the run before it, and some models don't offer temperature 0. So one run tells you which answer came up once. It tells you nothing about how often that answer comes up. A prompt that gave the right answer once has been tested zero times for what a workflow needs from it: a usable answer every time it runs. #### Pitfall: Reliable after one run A team writes a prompt that classifies incoming support tickets into one of four queues. Someone pastes in a ticket, the model returns the right queue, and the prompt goes into the ticket system that afternoon as "tested". In the first week a fraction of tickets come back with a fifth queue name the model invented and a further fraction with a sentence of explanation where the system expected one word. Nobody changed the prompt. Rule: one run is a sample of size one, and a prompt is only as reliable as the check that runs after it, on every run. #### Checkpoint: One good run A colleague ran their invoice prompt once, got the right total, and wants to put it in the payment process today. What do you tell them? - Yes. The prompt gave the right total, and the model is deterministic enough for a fixed email. - Yes, if the temperature is set to 0, because then every run gives the same answer and the process never sees a different total. - Not yet. Run it on a set of emails with known totals, count the answers that are right and in the right form, and add a check that rejects an answer that fails either. - Not yet. Ask the model whether it is confident in the total, and only pay when it says yes. A model usually knows when it is unsure. ## The check a workflow needs A person reading the ten recorded runs at temperature 1.0 sees at once that nine of them are right. A program comparing them with the first answer sees four matches and six failures, and a program comparing them with the full sentence sees two matches and eight failures. Neither count is the one the workflow cares about. The workflow needs a check that accepts the variation that doesn't matter and rejects the variation that does. For this question the check is short: does the answer name Canberra? Any wording that names it passes, and the Sydney answer fails. A check this loose is fine here because a person reads the answer, and a person reads "It's Canberra." and "Canberra" as the same fact. When a program reads the answer, as in the ticket system of the pitfall, the wording is part of what the check has to pin down. The box "Check each answer" in the widget applies the check to every run and adds a second summary line. Set the temperature to 1.0, uncheck "Shake the scores before each pick", check "Check each answer", and press the button. Here is the check applied to the recorded set from temperature 1.0: ```text What is the capital of Australia? (temperature 1.0, 10 runs) 1 pass Canberra 2 pass Canberra. 3 pass Canberra 4 pass The capital of Australia is Canberra. 5 pass Canberra 6 pass Canberra. 7 pass It's Canberra. 8 pass Canberra 9 FAIL Sydney 10 pass The capital of Australia is Canberra. 4 of 10 runs give the same answer as run 1 9 of 10 runs pass the check: the answer names Canberra ``` The summary lines answer different questions. The first asks whether the text is identical, and a model gives a "no" to that on a regular basis without being wrong. The second asks whether the answer does the job, and it is the count a workflow can act on: nine items go through and one is held for a person. The check lives outside the model, in code, and it runs on every answer, so a change in the model's variation shows up as a change in the pass count instead of as a surprise. What the check looks at depends on the job. A classification prompt checks that the answer is one of the allowed labels and nothing else, and an extraction prompt checks that the field parses as a number or a date. The checker in [the lesson on structured output](https://schubergphilis.github.io/ai-training/concepts/structured-output/) is a check of this kind for action items. A program can run each of them a hundred times without anyone reading a hundred answers. #### Checkpoint: Which check does the workflow need? The ticket workflow from the pitfall needs a check. Which one does the job? - Compare each answer with the answer the prompt gave in testing, and reject any that differ. - Strip whitespace and case, and accept the answer when it is exactly one of the four queue names. - Set the temperature to 0 and skip the check, because the answers no longer vary. - Accept any answer that contains one of the four queue names somewhere in the text. ## Exercise With "Check each answer" checked in the widget, press the button three times at temperature 0.3 and three times at temperature 1.5. The temperature scale is the toy's own, and a real model's scale differs. Each press shows both counts, how many runs agree with run 1 and how many pass the check, so note both. Put the twelve numbers in a small table. Then write the check a workflow that used this model would need, in one or two sentences, and say which wording differences it accepts and which answers it rejects. Ten minutes is enough. Doing this once with numbers in front of you is what makes you ask "how often?" the next time someone shows you one good answer. A good result is a table where the agreement count falls as the temperature rises while the pass count changes little, and a check that names the fact to test for and the wording it lets through. Which of the two counts would you have reported after one run, and what would it have told you? Stretch: Take a stricter check that passes only the bare word Canberra. Apply it by hand to the recorded ten runs at temperature 1.0 above, and decide whether the stricter check helped the workflow or hurt it. ## Recap 1. Each answer is one draw from the probabilities the model assigns, so the same prompt gives different answers on different runs. This is non-determinism, and it comes from how the answer is produced. A better prompt narrows the variation and doesn't end it. 2. Temperature 0 puts the probability on the top candidate and removes most of the variation. It doesn't remove it all, because a real model computes its scores in batches with other requests, the rounding follows the order of the arithmetic in the batch, and a near tie can swap. Some current models refuse any temperature except their default, so a workflow can't count on the setting. 3. One good run is a sample of size one. It shows that an answer can come up, and says nothing about how often it does or what a bad run looks like. 4. A process built on a model needs a check that runs on every answer, accepts the variation that doesn't matter, and rejects the answers that don't do the job. Its pass count, over many runs, is the reliability of the prompt. ## References - AI capabilities and limitations, Claude Academy, https://academy.claude.com/courses/ai-capabilities-and-limitations - Messages, Claude Platform documentation, https://platform.claude.com/docs/en/api/messages