Qwen3.8 27B Adds in Words: 97% on Short Numbers, 6% on Long Ones
Simon Willison reran a GPT-4o arithmetic experiment on a local Qwen3.8 27B. Without reasoning, accuracy collapses as numbers get longer, and turning reasoning on nearly fixes it.
By TRAGenX Desk
Colin Fraser posted on Bluesky about an experiment he ran more than two years ago with GPT-4o. The task was to compute a sum and return the answer in words, across increasingly large numbers. Simon Willison reran the idea on local hardware, a DGX Spark, so that every variable is under the experimenter's control.
What was tested
The model was Qwen3.8-27B-Q4_K_M.gguf, a 4-bit quantized build. Operands ranged from 1 to 13 digits. The first run used 30 randomly selected pairs for each digit-length combination, which is 30 × 13 × 13 = 5,070 problems. The run was driven from a Codex Remote session, and the full report with reasoning traces is in a public gist.
What happened without reasoning
- Overall numeric accuracy was 23.57% (1,195 of 5,070).
- Accuracy was 97.04% for one- to three-digit operands and 6.44% for ten- to thirteen-digit operands.
- Format compliance was 96.17%, so the model nearly always produced a well-formed answer. It was usually just the wrong one.
That last point matters most for production work. The failures are not malformed outputs that a schema validator would catch. They are fluent, confident, well-formed answers that happen to be wrong.
What changed with reasoning
A follow-up with medium reasoning enabled used one pair per digit combination, so n = 169. The model got 167 of 169 right. The sample is much smaller than the first run, so treat it as a strong signal and not a precise accuracy figure. Even so, the gap between the two configurations is large.
Why this matters for builders
Arithmetic is a clean stand-in for a wider class of tasks, where the model must carry exact state across many steps. If you run an LLM in a trading, payments or credit workflow, a few habits follow from this result.
- Evaluate by input size, not only on average. A blended score of 23.57% hides a near-perfect regime and a near-useless one.
- Test each configuration separately. The same weights behaved very differently with and without reasoning.
- Keep exact math in code. Let the model decide which calculation to run and have deterministic code do the sum.
- Verify outputs independently. Because the answers look fine, only a recomputed check will catch the wrong ones.
One caveat: this is a single quantized model on a single task, reported by one experimenter. It says nothing about how other models or quantization levels would score, and the source does not claim that.
FAQ
Frequently asked questions
- How accurate was Qwen3.8 27B at adding numbers and answering in words?
- In Simon Willison's run of 5,070 problems without reasoning, the quantized model scored 23.57% overall. It reached 97.04% on one- to three-digit operands and 6.44% on ten- to thirteen-digit operands.
- Does enabling reasoning fix the problem?
- In a smaller follow-up of 169 problems with medium reasoning, the model got 167 right. The sample is small, so it shows the direction of the effect more than an exact rate.
- Should I let an LLM do arithmetic in a financial workflow?
- Not for anything that must be exact. Have the model call a deterministic calculation and independently recheck the result, because wrong answers from these models can still look well-formed.
Sources
- Qwen3.8 27B addition in words — Simon Willison
- Colin Fraser's Bluesky post on the original GPT-4o experiment — Bluesky
- Full report with reasoning traces — GitHub Gist