How to compare LLM outputs side by side (free, no signup)
The fastest way to compare LLM outputs is to send one prompt to several models at once, read the answers in columns, and diff them against one model you pick as the base. That is the whole method. fiddle_llm does it in a browser tab with no account. This post is about what to look for once the columns are on screen, because the columns alone will fool you.
What should I actually look at?
I go in this order, every time.
- Facts. Are the claims true? Three models agreeing is not evidence. Three models trained on the same Wikipedia page will agree on the same mistake.
- Hedging. Count the qualifiers. "Generally," "many people consider," "it depends." Some models hedge by default. Some never do. Neither is right, but you should know which one you are talking to.
- Length. Ask for one sentence and see who gives you one. In my runs, one model answered a one-sentence question in 28 tokens. Another used 56 for the same job. That ratio holds across prompts and it shows up on your bill.
- Structure. Bullets or prose. Bold or plain. Headers on a two-paragraph answer. This is post-training taste, and it is the thing you will fight with the system prompt later.
- Refusals. Anything near medical, legal, or security gets a different treatment per model. The refusal is often the biggest diff on the page.
- Latency and cost. The badges under each answer. A model that is a little better and four times slower is a different product.
How do word diffs help?
Reading three 200-word answers and spotting the differences is slow and you will miss things. A word-level diff makes them jump out. Green is what this model added relative to the base. Red is what it dropped. Two answers that look the same at a glance turn out to differ in exactly one clause, and that clause is usually the interesting part. The claim one model made and another quietly left out.
The diff is also great for catching hedging. Turn it on and the qualifiers light up green in the cautious model's column.
How do I pick the base model?
Pick the one you are currently using in production, or the one you are trying to replace. The diff then reads as "what changes if I switch." If you have no incumbent, pick the shortest answer. Diffing against the shortest one shows you what everyone else added, which is easier to read than what everyone else removed.
What mistakes make the comparison meaningless?
I have made all of these.
- n=1. Outputs are sampled. Run it three times. If the model flips between runs, that is your finding.
- Leading prompts. "Explain why X is better than Y" gets you an essay on why X is better than Y from every model. Ask the neutral version.
- Reasoning vs non-reasoning at different budgets. A reasoning model with a 1,024 token cap may run out before it answers. A non-reasoning model never hits the cap. Compare them at the same cap and watch for the "cut off" badge.
- Ignoring the provider. The same model ID can be served by different providers through a router. Check the "via" badge before you conclude anything about "the model."
- Comparing an app to an API. The chat app has a system prompt. The API does not. Compare API to API.
Do I need an API key?
Not to start. Every IP gets a handful of free requests a day. That is enough to run the default prompt against three models a few times, which is enough to see the point.
After that, bring your own key. Pick a router in the toolbar and paste the key. Supported today: Concentrate, OpenRouter, Google Gemini, Groq, AWS Bedrock (with a region), and Azure OpenAI (with a resource name). The key lives in your browser's localStorage and is forwarded only to the router you picked. It is never stored on the server. Bedrock uses Bedrock API keys, not IAM access keys, and I will not add a field for IAM access keys.
A prompt to start with
"What is the currency in Mesopotamia?" It is in the box when you open the site. It has a clear factual answer, room for detail, and a nice trap about coins. Run it, diff it, run it again. Then go read what I got when I did.