Guide
What fiddle_llm is
fiddle_llm is a scratchpad. You type one prompt, it goes to several LLMs at once, and the answers come back side by side. Flip on the diff and you see exactly which words changed between models.
That's the whole product. It is a thin router UI. It is not a benchmark and it is not a leaderboard. I built it because reading about model differences is boring and watching them happen is not. Run the same question against three models, or the same model twice, and you'll see it. Then you start wondering why. That wondering is the point.
Zero retention, across all usage. No accounts, no history, no saved runs. Your prompt goes out, the answers come back, and the moment you close the tab they're gone. The server keeps a per-IP counter for the free tier and nothing else. Not the prompt, not the answer, not your key. The only exception is whatever retention policy the router you pick applies on its own side.
Zero memory is a feature. Every call here is a raw LLM call: one user message, no system prompt, no chat history, no tools, no memory of what you asked five minutes ago. That's different from using a model through ChatGPT or Claude or a coding agent, where a harness wraps your words in instructions, settings, and remembered context you never see. Those layers are useful for getting work done. They're terrible for understanding the model underneath. Here you're talking to the model with nothing in between.
Why the answers differ
Short version. Five things push the output around:
- Sampling. Models pick the next token with some randomness. Same prompt, same model, different run, different words.
- Training data and cutoffs. Each model read a different pile of text and stopped reading on a different date.
- Post-training. Humans rated answers and the model learned their taste. Different labs, different taste. That's why one model bullets everything and another writes paragraphs.
- System prompts and safety tuning. The provider wraps your prompt in instructions you never see. Those instructions shape tone, refusals, and length.
- Reasoning modes. Some models think before they answer. You pay for it in tokens and latency. Sometimes the answer gets better. Sometimes it just gets longer.
Longer version with examples: Why LLMs give different answers.
How to use it
- Type a prompt in the box. There's one loaded by default so you can just hit Run and see something.
- Pick models. The chips under the prompt are what will run. Click a chip to remove it. Type in the add box to pick from the router's catalog. Model ids are whatever the router calls them, like
claude-sonnet-5orgpt-6-astra. - Hit Run. Every model is called in parallel. Panels fill in as each answer lands.
- Read. One panel per model, output in monospace, stats in the header.
The diff. Tick Diff vs and pick a base model. The base panel gets a blue border. Every other panel is rewritten as a word diff against it: green for words that model added, red for words it dropped. It is a plain word diff, not a semantic one. Two answers that mean the same thing in different words light up like a Christmas tree. That's fine. You're looking at how they say it, not just what they say.
Reading the badges
| Badge | Meaning |
|---|---|
| latency | Wall time for the call. Green under 2s, yellow under 8s, red above. |
| tokens in / out | Prompt tokens and completion tokens as reported by the provider. |
| reasoning | Shown only when the model spent tokens thinking before answering. Those count toward output and cost. |
| tok/s | Completion tokens divided by latency. A rough throughput number, nothing more. |
| cost | What the call cost, when the router reports it. |
| via <provider> | Which provider actually served the call. Routers pick, and it's not always who you'd guess. |
| cut off | The model hit the 4096 output-token cap and stopped mid-thought. |
Free tier and bring your own key
You get a few free requests per IP address per day. Those go through Concentrate on my key. When they run out you'll see a message saying so.
To keep going, pick a router from the dropdown in the top bar and paste your own key:
- Concentrate
- OpenRouter
- Gemini
- Groq
- AWS Bedrock, plus a region like
us-east-1 - Azure OpenAI, plus your resource name
Your key lives in your browser's localStorage and nowhere else. Each request forwards it to the router you picked, and only that router. The server never stores it. There are no accounts because there's nothing to keep.
For Bedrock, use a Bedrock API key. Not an IAM access key. Pasting long-lived AWS credentials into a web form is a bad habit and this form won't help you build it.
Limits
- 4096 max output tokens per answer.
- 8000 character cap on the prompt.
- One user message. No system prompt, no conversation, no follow-ups.
- No streaming. Panels fill in when each answer finishes.
- No history. Nothing is saved. Close the tab and it's gone.
Run the same prompt twice and you may get different answers. That is not a bug. That is the thing I'm trying to show you.
Prompts worth trying
- What is the currency in Mesopotamia? Watch who answers the question as asked and who corrects the premise.
- Is a hot dog a sandwich? One sentence. Opinion questions expose post-training taste fast.
- Write a haiku about a database migration. Syllable counting is harder for models than you'd think.
- Write a Python function that returns the second largest number in a list. Compare edge case handling and how much explanation you get unasked.
- Who won the most recent World Cup? Anything near a training cutoff shows you who knows what, and who admits they don't.
Want a model added, or found a bug? Contact me.