Guide

What this is, how to use it, and what it won't do.

What is fiddle_llm?

A scratchpad for comparing LLM outputs side by side. Type one prompt. It goes to several models at once. The answers come back in columns, and the diff shows you which words changed between them.

That's the whole product. There's no benchmark, no leaderboard, no score. I built it because reading about model differences is boring and watching them happen isn't. Run the same question against three models, or one model twice, and you'll want to know why the answers moved.

Zero retention. No accounts, no history, no saved runs. The server keeps a per-IP counter for the free tier and forgets everything else the moment the response goes back to your browser. Answers and keys are never written down on my side. The request log keeps the first 20 characters of each prompt for 30 days so I can tell real traffic from bots, and nothing more. The router you pick has its own policy, and that part is out of my hands. One exception on the free tier: Claude Fable 5 and 5.1 have no zero-retention provider yet, so runs of those two go through a key without it, and the panel says so with a "no ZDR" badge. Share links work the same way: the prompt and answers are compressed into the URL, so the link is the storage. Send it to someone and they see exactly what you saw. Nothing is stored on my server.

Zero memory turns out to be the useful part. Every call here is a raw LLM call: one user message going straight to the model. Compare that with a chat app or a coding agent, where a harness wraps your words in a system prompt, tool definitions, and whatever it remembers about you from last week. Those layers are great for getting work done. They also make it impossible to tell what the model itself would have said. Here, nothing sits between you and the model.

Why do LLMs give different answers to the same question?

• Sampling. Models pick the next token with a little randomness, so the same model on the same prompt drifts run to run.
• Training data. Each model read a different pile of text and stopped reading on a different date.
• Post-training. Humans rated answers and the model learned their taste. That's why one lab's model bullets everything and another's writes paragraphs.
• Hidden system prompts. Providers wrap your prompt in instructions you never see. Refusals and length come from there as much as from the model.
• Reasoning modes. Some models think before they answer. You pay in tokens and seconds. Sometimes the answer gets better. Sometimes it just gets longer.

The long version, with real examples: Why LLMs give different answers.

How to compare LLM outputs

Short version below. The full method, with the mistakes to avoid: how to compare LLM outputs.

1. Type a prompt. One is loaded already, so you can hit Run and see something in ten seconds.
2. Pick models. The chips under the prompt are what will run. Click a chip to drop it. Type in the add box to pull from the router's catalog. Ids are whatever the router calls them, like claude-sonnet-5 or gpt-6-astra.
3. Hit Run. All models are called at once and every panel streams in as its model writes, so you watch them answer side by side in real time.
4. Read. One panel per model, output in monospace, stats in the header.

The diff is where it gets interesting. Tick Diff vs and pick a base model. The base panel gets a blue border, and every other panel is rewritten as a word diff against it: green for words that model added, red for words it dropped. It's a plain word diff. Two answers that mean the same thing in different words light up like a Christmas tree, and that's fine. You're looking at how each model talks, not just what it concludes.

Tick Web search and every model in the run gets the same search tool. The router runs the search, not me, so nothing changes about what my server keeps. Each panel shows what the model searched for and which pages it cited. A model that answers without searching gets a "no search" badge, and that's a result too. Searches cost real money, so the box is off by default and stays off until you tick it.

Got a result worth arguing about? Hit Share. The link carries the prompt and every answer, diff included, so the other person sees exactly what you saw and can hit Run to get their own.

What the badges mean

Free tier and bring your own key

You get a few free requests per IP address per day. Those run through Concentrate on my key. When they're gone you'll see a message saying so.

To keep going, pick a router from the dropdown in the top bar and paste your own key:

• Concentrate
• OpenRouter
• Gemini
• Groq
• AWS Bedrock, plus a region like us-east-1
• Azure OpenAI, plus your resource name

Your key lives in your browser's localStorage. Each request carries it to the router you picked and nowhere else. The server passes it through and doesn't keep it. There are no accounts because there's nothing to keep.

For Bedrock, use a Bedrock API key rather than an IAM access key. Pasting long-lived AWS credentials into a web form is a bad habit, and this form won't help you build it.

Limits

• 4096 output tokens per answer.
• 8000 characters of prompt.
• One user message. No system prompt and no follow-ups.
• No history. Close the tab and it's gone.

Run the same prompt twice and you'll often get two different answers. That's the whole reason this exists.

Prompts worth trying

Want a model added, or found a bug? Contact me.