OpenRouter's Auto-Routing Can Quietly Swap How Your Model Behaves

OpenRouter's pitch is simple: call one API for a model name, and it routes your request to whichever backend provider is cheapest or fastest at that moment, falling back automatically if one is down. For a lot of prototyping, that's a genuine convenience. Developer Mohamed Moustafa ran the numbers on what that convenience actually costs, and Simon Willison flagged the results — the short version is that 'same model' does not mean 'same behavior.'

The same model ID, six different personalities

Moustafa's full writeup tested models including DeepSeek V4 Flash, GLM 5.3 Flash, Qwen 3.5 122B, and MiniMax M3 across the providers OpenRouter routes to. The gaps weren't cosmetic:

• Tool-calling accuracy swings hard. On a TAU-Bench-style tool-use test, DeepSeek V4 Flash scored 81% on the first-party endpoint versus 58-70% on some third-party providers serving the identical weights.
• Vision support is inconsistent even when advertised. DeepInfra's Qwen endpoint misread a letter K as an R and called red blue; Venice and Together returned 'no image provided' errors on MiniMax despite listing image support.
• `reasoning_effort` gets ignored. DigitalOcean, GMI-Cloud, Mancer, and Venice showed flat reasoning-token counts whether the setting was low, high, or max.
• Quantization labels don't track quality. Providers declaring fp4 sometimes landed in the middle of the fp8 pack, and some of the strongest performers declared no quantization at all.
• Some failures are silent. Responses came back as HTTP 200 with content: null and no tool call — a request that looks successful to anything not checking payload contents.
• Per-provider quirks compound. SiliconFlow rejects an empty reasoning_content field that Baidu and Alibaba accept fine, so error handling has to be written per provider, not per model.

Why this matters if you're shipping agents, not just chat

None of this shows up in a quick demo. It shows up in production, under load, when OpenRouter's router quietly shifts traffic to whichever provider is cheapest right now. For agentic pipelines — the kind this publication cares about, whether that's a trading system parsing market data with a vision-capable model or a coding agent chaining tool calls — a silent drop in tool-call accuracy or a null response that looks like a 200 OK is exactly the failure mode that's hardest to catch in review and easiest to miss until it costs you a bad trade or a broken deploy. 'Model routing' sounds like an infrastructure detail. In practice it's a behavior contract, and OpenRouter's default auto mode doesn't guarantee one.

What to actually do about it

The fix isn't to abandon OpenRouter — it's to stop trusting the default. Moustafa's mitigations are concrete and apply to anyone wiring an LLM into a system that has to be right, not just fluent:

• Use OpenRouter's provider.only option and the /endpoints API to pin specific, benchmarked providers instead of leaving routing on auto.
• Benchmark against your actual workload (tool use, vision, long context) rather than trusting a provider's quantization label.
• Test from your production environment, not your laptop — rate limits can differ by IP and infra, not just by account.
• Add client-side detection for null-content, no-tool-call responses, since these return as HTTP 200 and won't trip a normal error handler.
• Write provider-specific edge-case handling (e.g. empty reasoning blocks) into your retry logic rather than assuming one contract fits every backend.

If you're teaching a team to build with LLMs in the loop — which is most of what 'vibecoding' actually is once you're past the demo — this is a good case study in reading the fine print of your own dependencies before you ship, not after something breaks.