Long-horizon tasks cut full-task success sharply
Google released Android Bench 2.0 on 17 September with a 30-task long-horizon suite intended to represent work that can take an engineer several days or a week. The set covers app creation, app conversion, migrations and new features, and the published results use specialised coding agents rather than evaluating model APIs in isolation. Each task is run five times.
On the current leaderboard, GPT-6 Astra with Codex has a 28.0% full-task pass rate and an 82.2% average completion rate. Claude Fable 5.1 with Claude Code records 22.7% pass and 82.4% completion, while GPT-5.6 Sol with Codex records 19.3% pass and 74.3% completion. Google reports a 13.3% to 42.0% confidence interval around Astra’s pass rate, underscoring the uncertainty that comes with a 30-task set.
Partial progress often stops short of a passing solution
The gap between pass rate and completion rate is the central result. Android Bench gives partial credit for functionality, regressions, explicit requirements and visual fidelity, while a strict pass still requires the task to clear the benchmark’s validation. An agent can therefore implement most of a migration or feature and still fail the full task because tests, validation or build requirements do not all pass.
The public task table makes that visible. Several migrations and app-building tasks show completion in the 90% range while recording zero successful runs out of five. Google says the earlier short-task benchmark reached roughly 91% at the top, so the long-horizon set is measuring a materially different failure surface rather than simply extending the old leaderboard.
The harness is part of the benchmarked system
Google pairs models with provider-specific agents such as Codex, Claude Code and Antigravity SDK. The Android team says harness design can affect outcomes through mechanisms such as prompt caching and tool-window compaction, and plans to publish more cross-agent combinations later. The current table should therefore be read as model-agent system results rather than a clean ranking of model weights alone.
That distinction also applies to cost and latency. The leaderboard lists Astra with Codex at an average 7.9 hours and $375.70 per 30-task benchmark run, Fable 5.1 with Claude Code at 22.2 hours and $492.60, and Sol with Codex at 8.6 hours and $235.80. Those figures depend on the benchmark harness, provider pricing and execution conditions and should not be treated as general productivity estimates.
Useful evidence, with a narrow scope
The story is Confirmed because Google publishes the task set, scoring approach and current leaderboard directly. Hacker News supplied the discovery lead only; none of the quantitative claims here rely on community comments. The benchmark is Android-specific, uses 30 long-horizon tasks and currently compares a limited set of provider-agent pairings.
The most defensible conclusion is narrow: on this Android suite, current coding agents frequently make substantial progress but still fail end-to-end requirements, and the measured outcome changes with the surrounding agent system. Independent reruns with pinned model and agent versions, broader cross-harness comparisons and larger task sets would be needed before generalising the 28% ceiling beyond this benchmark.