Benchmark release · July 2026
Obvious mistakes are now a product tradeoff.
ObviousBench measures whether models avoid simple, visible mistakes across repeated attempts—and what that reliability costs.
Large language models can now do things that would have sounded absurd a few years ago. They can summarize thousands of pages in seconds, write simple applications in minutes, and plan entire businesses in hours.
And yet, the same class of system can show up in a flagship product and get basic arithmetic wrong, make obvious contradictions, or even fail to spell its own name. These are the kinds of mistakes that can cause users to lose trust in the technology, the product, and the company behind it.
Larger models and deeper reasoning can reduce these failures, but they also add cost and latency. Product teams still need to know what visible mistakes become more likely when they choose a smaller model, cheaper route, or shorter reasoning budget.
ObviousBench is not a smart-versus-dumb ranking. It is a way to compare capability, visible brittleness, and cost before those choices reach users.
What this measures
Plain tasks with objective answers: literal counting, spelling transforms, ordering, negation, formatting, arithmetic, word counting, and simple constraint awareness.
How to read it
Answer pass^3 is not pass@3. It requires all three sampled answers to be correct. Strict pass^3 stays available as a formatting diagnostic.
What it is not
Not a global intelligence ranking, not a shame board, and not a claim that one visible miss makes a model generally bad.
The aggregate thesis
The results form a cost–reliability frontier.
Pass rates should be read alongside cost and reported reasoning effort. The preferred region is the upper-right: higher repeated reliability at lower estimated full-run cost.
A decision table, not a podium
What it costs to cross a reliability bar.
Because the benchmark is intentionally saturatable, the practical view is the cheapest setting in each family that clears a chosen quality threshold.
| # | Model | Effort | Pass^3 | Cost | Tokens | Weights |
|---|
One-model proof
The same model can be brittle or reliable.
The benchmark’s central result is visible inside a single model family. Turning up reasoning does not merely add a few points for GPT‑5.4 nano; it changes the apparent product risk.
GPT‑5.4 nano moves from — answer pass^3 with no explicit reasoning to — at low, — at medium, — at high, and — at xhigh.
The first jumps are enormous. Later settings still improve reliability, but the larger points show the progressively higher run cost and reported reasoning-token use required to buy those gains.
A ceiling by design
Saturation is evidence, not a defect.
While most benchmarks seek to test harder and harder capabilities, ObviousBench aims to be saturatable by top models. It is designed to provide contrast between model sizes and reasoning depths, not to remain unsolved at the frontier or obscure false negatives.
Once several systems solve nearly every item, the useful question changes. Rank matters less than the cost and reasoning budget required to reach the same reliability bar.
—
Secondary stories
Useful cuts through the same result surface.
The same aggregate data tells several product stories: newer rows are not always cheaper reliability improvements, early reasoning models remain surprisingly strong, and open-weight rows can be unusually efficient.
OpenAI history
No-thinking performance has improved unevenly.
The progress of OpenAI's no-thinking models has been uneven, with GPT‑4.1 matching GPT‑5.4 performance at 56% of the cost. GPT‑5.6 Sol improves on GPT‑5.5 from 84.0% to 88.2% at essentially the same measured run cost, while Terra reaches 77.1% at roughly half Sol's cost.
The useful product question is not just whether the newest label is better. It is whether each generation shifts the efficient frontier, or asks teams to pay more for similar visible-risk exposure.
GPT‑5 family
Reasoning settings do not guarantee reasoning use.
GPT‑5 through 5.6 show why ObviousBench is a chart of tradeoffs rather than a straight leaderboard. GPT‑5.2 regressed from GPT‑5, while GPT‑5.6 Terra's high-to-max rows reach 95.8–96.5% for $0.24–$0.27. In this run Terra has a stronger reasoning/cost curve than the more expensive Sol.
At xhigh and max, Sol reported reasoning on only about one-fifth of attempts, and 42 of its 44 wrong attempts across those rows occurred outside the reported-reasoning path. Terra reported reasoning on roughly four-fifths of attempts at the same settings. This is evidence of different compute-routing behavior, not confidence calibration: the benchmark does not collect model confidence, and zero reported tokens do not prove zero internal computation.
Gemini Flash
Gemini Flash improved, then became more expensive.
Gemini 2.5 Flash to 3 Flash improved the reliability and cost efficiency across the board. However, Gemini 3.5 Flash's high pricing regressed the efficient frontier to that established by 2.5 Flash.
The 3.5 Flash low-effort row already reaches the ceiling region, which is a useful signal for product teams: Google’s flash path may need surprisingly little extra reported reasoning to avoid these visible mistakes.
OpenAI reasoning progress
Strong reasoning became dramatically cheaper.
o1 remains strong despite its 2024 vintage, though that comes at more than a 10x price compared to modern GPT‑5.5.
o3 remains incredibly competitive in both reasoning and cost for such an early reasoning model. The most likely explanation was that o3 was priced significantly under its actual compute cost.
Claude Opus Progress
Opus improves, regresses, then recovers unevenly.
In this run, Opus 4.6 was a clear improvement over 4.5 at a similar cost, however Opus 4.7 was a visible regression, failing questions at all reasoning levels. Opus 4.8 recovers some of the ground against Opus 4.6 but at markedly higher prices, potentially due to the new tokenizer.
Claude Sonnet Progress
Sonnet gains reliability, but not in a perfectly monotonic line.
Sonnet 4.5 improves the high-effort frontier over Sonnet 4 in this slice, while Sonnet 4.6 shows a stronger high-effort ceiling with a different cost shape. The no-reasoning rows also move upward, which is the important product story: the base model is getting less brittle before extra compute is applied. Claude Sonnet 4.6 Low reported little reasoning and scored poorly, but that is a telemetry observation rather than evidence of confidence calibration. Sonnet 5 performs similarly to 4.6 but at higher costs due to the tokenizer change.
Claude Haiku Progress
Haiku has made limited progress.
It is interesting to note that Claude Haiku has made little progress in the efficient frontier, with changes from Claude 3 Haiku and 3.5 Haiku almost perfectly increasing performance (but also costs) in lockstep.
Appendix
Data.
The launch story uses curated, defensible cuts. This appendix keeps the complete aggregate surface available for checking alternative questions.