How to compare LLM outputs

Brad Lowenstein · 2026-09-11 · updated 2026-09-12

To compare LLM outputs, send one prompt to several models at once, read the answers in columns, and diff them against one model you pick as the base. That's the whole method. fiddle_llm does it in a browser tab with no account and no signup. The rest of this post is what to look for once the columns are on screen, because the columns alone will fool you.

How to compare LLM outputs in five steps

  1. Write one neutral prompt. Ask it the way a user would, without hinting at the answer you want.
  2. Pick three or more models from different labs. One Claude, one GPT, one Gemini or open-weight model is a good spread.
  3. Run them in parallel. Same prompt, same moment, so latency and cost are measured under the same conditions.
  4. Diff against a base. Pick the model you use today. Green is what the other model added, red is what it dropped.
  5. Run it again. Outputs are sampled. Two or three runs before you conclude anything.

Two minutes, start to finish. Everything below is about doing those five steps well.

What should I actually look at?

I go in this order, every time.

  1. Facts. Are the claims true? Three models agreeing is not evidence. Three models trained on the same Wikipedia page will agree on the same mistake.
  2. Hedging. Count the qualifiers. "Generally," "many people consider," "it depends." Some models hedge by default. Some never do. Neither is right, but you should know which one you are talking to.
  3. Length. Ask for one sentence and see who gives you one. In my runs, one model answered a one-sentence question in 28 tokens. Another used 56 for the same job. That ratio holds across prompts and it shows up on your bill.
  4. Structure. Bullets or prose. Bold or plain. Headers on a two-paragraph answer. This is post-training taste, and it is the thing you will fight with the system prompt later.
  5. Refusals. Anything near medical, legal, or security gets a different treatment per model. The refusal is often the biggest diff on the page.
  6. Latency and cost. The badges under each answer. A model that is a little better and four times slower is a different product.

How do word diffs help?

Reading three 200-word answers and spotting the differences is slow and you will miss things. A word-level diff makes them jump out. Green is what this model added relative to the base. Red is what it dropped. Two answers that look the same at a glance turn out to differ in exactly one clause, and that clause is usually the interesting part. The claim one model made and another quietly left out.

The diff is also great for catching hedging. Turn it on and the qualifiers light up green in the cautious model's column.

How do I pick the base model?

Pick the one you are currently using in production, or the one you are trying to replace. The diff then reads as "what changes if I switch." If you have no incumbent, pick the shortest answer. Diffing against the shortest one shows you what everyone else added, which is easier to read than what everyone else removed.

What mistakes make the comparison meaningless?

I have made all of these.

Do I need an API key?

Not to start. Every IP gets a handful of free requests a day. That is enough to run the default prompt against three models a few times, which is enough to see the point.

After that, bring your own key. Pick a router in the toolbar and paste the key. Supported today: Concentrate, OpenRouter, Google Gemini, Groq, AWS Bedrock (with a region), and Azure OpenAI (with a resource name). The key lives in your browser's localStorage and is forwarded only to the router you picked. It is never stored on the server. Bedrock uses Bedrock API keys, not IAM access keys, and I will not add a field for IAM access keys.

A prompt to start comparing LLM outputs with

"What is the currency in Mesopotamia?" It is in the box when you open the site. It has a clear factual answer, room for detail, and a nice trap about coins. Run it, diff it, run it again. Then go read what I got when I did.