Why does the same LLM give a different answer when I ask twice?

Because generation is sampled. At each step the model picks from a probability distribution over the next token, and temperature controls how far from the top pick it will wander. Two runs take two different paths.

Does temperature 0 make an LLM deterministic?

Mostly, not entirely. Batched inference, floating point math on GPUs, and provider-side changes can still produce small differences between runs.

Why does one model know about an event and another does not?

Every model has a training cutoff. Anything after that date is missing unless the model has a tool like web search. Two models with different cutoffs disagree about recent facts.

Can the same model ID return different answers from different providers?

Yes. Routers can serve one model ID from several providers with different quantization, batching, and default settings. The label matches, the weights or serving stack may not.

Why do LLMs give different answers to the same question?

Brad Lowenstein · 2026-09-11

Large language models give different answers to the same question because generation is sampled, not looked up. Each model was trained on different data with a different cutoff, then tuned by different people toward different preferences, then wrapped in a different system prompt, then served with a different reasoning budget by a provider you may not have picked. Any one of those six things changes the output. Usually several of them do at once.

I built fiddle_llm because I got tired of explaining this in meetings. Now I just show people. Here are the six reasons, in the order they surprised me.

Why does the same model answer differently twice?

This is the one that gets people. Same model, same prompt, two answers. The reason is sampling. An LLM does not produce a sentence. It produces a probability distribution over the next token, and something has to pick one. Temperature is the knob. At temperature 0 you take the top token every time. At 1 you sample proportionally. Most chat APIs default somewhere around 0.7 to 1.0, so the model wanders.

Temperature 0 is not a guarantee either. GPU floating point math is not perfectly associative, requests get batched with other people's requests, and providers change things under you. It is close to deterministic. It is not deterministic.

Why does one model know something the other does not?

Training cutoffs. Every model stops reading the internet on some date. Ask about anything after that date and you get either a confident guess or a polite "I don't have information about that." Two models with cutoffs six months apart will disagree on recent events, recent library versions, and recent prices. Neither is lying. One is just older.

The data before the cutoff differs too. Different crawls, different filtering, different licensed corpora. The Mesopotamia currency question on the home page is a nice example. Every model agrees it was silver and barley by weight. They disagree on how much detail to give and whether to mention the shekel. That is a data and tuning difference, not a knowledge gap.

What does post-training change?

The base model out of pretraining is a text predictor. It becomes a chat assistant through post-training: supervised fine-tuning and reinforcement learning from human feedback, plus whatever each lab does that they do not publish. This is where personality comes from. One lab's raters preferred bullet points. Another's preferred a direct sentence then a caveat. Another's preferred no caveat at all.

Ask "Is a hot dog a sandwich? One sentence." across three models and you get three different hedging strategies. Same facts, different manners. That is post-training.

How much do system prompts and safety tuning matter?

More than people think. When you call a model through an API you usually get the raw model with no system prompt. When you use a consumer chat app you get the same model behind a few thousand words of instructions about tone, formatting, and what to refuse. The API answer and the app answer to the same question can look like two different products. They kind of are.

Safety tuning is baked in deeper. Some models refuse borderline requests. Some answer with a disclaimer. Some just answer. Compare a medical or legal question across three models and the refusal behavior is often the biggest diff on the page.

Why do reasoning models answer differently?

Newer models can "think" before they answer, spending tokens on hidden scratch work. That changes the answer and it changes the bill. In my runs, a flash-class model with reasoning on spent about 500 hidden tokens to produce a one-sentence answer that a non-reasoning model produced in 28 tokens. The reasoning answer was a little more careful. It also cost 35 times more and took three times as long.

The token budget matters too. Cap output at 1,024 tokens and a reasoning model may run out before it says anything. Cap at 4,096 and it finishes. Same model, same prompt, different cap, different answer. fiddle_llm shows a "cut off" badge when this happens so you do not blame the model.

Does it matter which provider served the model?

Yes, and this is the one nobody tells you. A router like Concentrate or OpenRouter takes a model ID and picks a provider. The same open-weight model can be served by several providers with different quantization, different batching, different default parameters. The label says the same thing. The bytes coming back do not have to. fiddle_llm shows a "via" badge with the provider that actually served the call, because I got burned by this and wanted the receipt.

So which answer is right?

Often more than one. For factual questions, check the facts. For everything else, the honest answer is that you are looking at six sources of variation stacked on top of each other, and the right move is to look at several outputs instead of trusting one.

Try it. Go to fiddlellm.com, leave the default prompt in the box, hit Run, and turn on the diff. Then hit Run again.