What a Cartoon Pelican Reveals About Picking Your Next Coding Model

Since 2025, Simon Willison has run the same odd little test on almost every notable LLM release: ask it to generate an SVG of a pelican riding a bicycle. He calls it his 'deeply unscientific benchmark,' and it's produced well over a hundred posts by now. It's a joke on the surface. It's also become one of the more useful longitudinal datasets in the field, precisely because the prompt never changes — only the model does.

His latest post puts early access to GPT-6 Astra through the same drill, then lines the results up in a grid against three GPT-5.6 models — Sol, Terra, and Luna — each run at low, medium, high, xhigh, and max reasoning (Astra has no reasoning=none option).

The result: quality jumps, but not evenly

Willison's read is blunt: Astra's pelicans are much better than anything Sol produced, to the point that even Astra's low-reasoning output beats his favorite high-effort Sol result. But the improvement isn't uniform across reasoning levels — he notes Astra still doesn't *reliably* draw the bicycle's legs on both sides of the frame unless it's run at max reasoning. Quality scales with reasoning effort even within a single, clearly stronger model.

That's the part worth sitting with if you build on top of these models rather than just admire the drawings: a 'better model' isn't one setting. It's a curve, and where you land on that curve is a deliberate choice, not a default.

The economics are less obvious than the sticker price

Astra lists at roughly double Sol's per-token price — about $10/$50 versus $5/$30 per million input/output tokens, going by the numbers in Willison's post. Sticker price alone would suggest Astra costs twice as much to run. It doesn't, in practice, because Astra needed noticeably fewer tokens to finish the same task. Per-token pricing without a per-task token count is a meaningless comparison — the kind of thing that's easy to get wrong when you're picking a model for an agentic pipeline off a pricing page instead of a real workload.

A tokenizer clue hiding in plain sight

One more detail from the grid: Astra and Luna both consumed exactly 16 input tokens for the identical prompt, while Sol and Terra used 26. Willison flags it as a hint that Astra and Luna may share a tokenizer or underlying architecture that Sol and Terra don't. Nobody outside the lab knows the model family tree for certain — but token counts on an identical prompt are a rare, cheap-to-check signal for guessing at it.

Why a silly benchmark is a legitimate vibecoding habit

None of this replaces a real eval suite for a production system. But the discipline underneath the joke — one fixed, cheap, reproducible prompt, run against every new model on day one, at every reasoning setting, with token counts recorded — is exactly the kind of quick smoke test worth having before you wire a new model into an agentic coding workflow. It surfaces the two things that actually bite teams in production: reasoning-effort/quality tradeoffs, and the gap between advertised and effective cost.

• Test a new model at more than one reasoning/effort setting before defaulting to 'high' everywhere — cost and latency compound fast in agent loops.
• Compare cost per completed task, not cost per token — a pricier model that uses fewer tokens can be cheaper in practice.
• Keep the same probe prompt across model versions so results are actually comparable over time, instead of anecdotal.