Android Bench has been around since March — Google's attempt to answer a question that kept coming up in every Android dev community: which AI coding assistant actually knows Android? Generic SWE-bench numbers don't tell you much when you care about whether the model understands ViewModelStoreOwner, generates correct Jetpack Compose layouts, or handles KSP migration without hallucinating APIs that don't exist.
On July 8, the benchmark got a meaningful upgrade. Google migrated it to the Harbor framework, added latency and cost dimensions to the leaderboard, expanded to open-weight models, and added 8 new contestants. The result is the most practical data I've seen for actually deciding which AI tool to put in your Android workflow — so let me walk through what changed and what the numbers mean for day-to-day development.
What Android Bench Actually Tests
Before looking at the leaderboard numbers, it's worth understanding what "Android tasks" means here, because it shapes how much you should weight any given score for your use case.
The 100 test cases are drawn from real Android development challenges pulled from public GitHub repositories. They're organized across these areas:
- Architecture & Design — clean architecture patterns, repository layer correctness, testability concerns, separation of concerns across layers
- UI Development — Jetpack Compose layouts, state hoisting, adaptive layouts for phones and tablets, Material3 component usage
- Testing — unit tests, instrumented tests, Compose UI tests, handling coroutine test dispatch
- Performance — profiling annotations, avoiding recomposition, identifying startup bottlenecks, Baseline Profile generation
- Security & Privacy — permission handling, data protection patterns, secure storage
- Multi-device support — phones, tablets, foldables, Wear OS — the areas where generic LLMs tend to fall apart hardest
- Modern APIs — latest Android framework features, which requires models to have current Android knowledge, not just general code comprehension
Each task is evaluated with statistical rigor: 10 independent runs per model, average pass rate reported with a confidence interval (p-value < 0.05). So you're not looking at a single lucky run — the CI range column tells you how consistent each model actually is.
Why this matters for real work: when I'm using AI for Android code, my pain points cluster around exactly these areas — Compose state management, Room migrations, Hilt module setup, and multi-device adaptability. These tasks map to that. A high score on this benchmark isn't guaranteed to mean a good experience on your specific codebase, but it's far more predictive than SWE-bench or generic coding challenges.
The Current Leaderboard (July 8)
Here are the top models as of the July 8 update, with all four dimensions the Harbor migration unlocked:
| Model | Score | CI Range | Latency | Cost/run |
|---|---|---|---|---|
| Claude Fable 5 | 84.5% | 79.9–88.8% | 8.0 h | $133.2 |
| GPT 5.5 | 80.2% | 73.5–86.6% | 11.4 h | $138.3 |
| Claude Sonnet 5 | 76.2% | 69.0–82.1% | 12.3 h | $99.9 |
| GPT 5.4 | 74.1% | 66.0–80.9% | 8.4 h | $83.4 |
| Gemini 3.1 Pro Preview | 73.7% | 66.1–80.4% | 10.6 h | $87.4 |
And the open-weight models, newly included in this update:
| Open-Weight Model | Score | Notes |
|---|---|---|
| Qwen 3.6 27B | 45.1% | Best self-hosted option by a margin |
| Gemma 4 31B IT | 36.3% | Google's own open model |
| Gemma 4 26B A4B IT | 25.1% | MoE variant, lower accuracy |
Reading the CI ranges
The confidence interval column matters more than the headline score. GPT 5.5's range spans 13 percentage points (73.5–86.6%) compared to Fable 5's 8.9-point range (79.9–88.8%). That wider range means GPT 5.5 is more variable run to run — you'll get excellent answers on some Android tasks and significantly weaker ones on others. Fable 5 is not only higher on average, it's more consistent, which is arguably more important for a daily development workflow where you need reliable, not occasionally brilliant.
What the Harbor Migration Actually Adds
Before July 8, Android Bench reported just one number: pass rate. The Harbor framework brought the benchmark up to a four-dimension evaluation:
Score is still the primary metric — average percentage of 100 tasks resolved correctly across 10 runs. Nothing changed here.
CI range is new. It tells you how repeatable each result is. A tight range means you can trust the score. A wide range means the model is inconsistent and a high average hides some genuinely bad runs.
Average latency (in hours to complete all 100 tasks × 10 runs) is a proxy for how fast the model thinks. Fable 5 at 8.0h versus Sonnet 5 at 12.3h is a 54% speed difference on the same tasks. For interactive development — autocomplete, inline suggestions, chat-style code generation — faster matters a lot.
Average cost per full benchmark run translates directly to API usage cost at scale. Sonnet 5 at $99.9 versus GPT 5.5 at $138.3 for a meaningfully lower score starts to look like the obvious efficiency choice for teams running AI-heavy CI pipelines or heavy agentic workflows.
Harbor also publishes the full dataset on Harbor Hub, so you can inspect the actual tasks and expected outputs — not just the aggregated numbers. If you want to understand why a model scored the way it did on multi-device tasks or Compose UI generation, the raw data is there.
The Open-Weight Gap Is Real — and Matters for On-Device AI
The gap between the best closed model (84.5%) and the best open-weight model (45.1%) is 39 percentage points. That's not a rounding error — it's a fundamentally different capability tier. If you're running a model locally via LM Studio, Ollama, or on-device via AICore, you need to know this going in.
For on-device inference — which I wrote about in detail when covering Gemma 4 and the ML Kit GenAI Prompt API — Gemma 4 31B scores 36.3%. That's still meaningful for well-scoped tasks: code explanation, simple snippet generation, answering questions about a local codebase. But asking it to write a full clean architecture repository layer, handle a Hilt module setup with edge cases, or generate a correct Compose Pager ViewModel with Lifecycle 2.11 bindings — you're going to get plausible-looking but often broken output. The benchmark gives you a calibration point for how much to trust versus verify.
The practical read: on-device models are useful for privacy-sensitive contexts where you can't send code off device, or for low-cost background tasks where verification is cheap. For complex Android-specific architecture decisions, the score gap justifies using a cloud API model.
How to Actually Use This for Your Workflow
I've been using AI assistants heavily across Nodat and Musist — mostly for boilerplate generation, architecture reviews, and exploring unfamiliar APIs. Here's how I map the leaderboard to practical decisions:
For primary development (AI assistant in your IDE)
Fable 5 or GPT 5.5 if raw capability is the priority. Sonnet 5 if you want to save ~30% on cost without a dramatic accuracy drop. In practice, the Sonnet 5 vs GPT 5.5 comparison (76.2% vs 80.2%, similar latency at 12h vs 11h, but $99 vs $138) makes Sonnet 5 the better efficiency pick unless you're hitting the ceiling on very hard Android tasks.
For automated code review or AI-assisted CI
Cost per run matters more than single-query latency. The $99 vs $138 delta compounds across hundreds of CI runs. Sonnet 5 is a reasonable default. If you're running agentic tasks that chain dozens of tool calls (AppFunctions-style workflows, automated test generation), model latency also starts to matter — GPT 5.4 at $83 and 8.4h is actually worth a look here.
For privacy-sensitive or air-gapped environments
Qwen 3.6 27B at 45.1% is your best self-hosted option. Treat it like a capable junior engineer: it can do real work on well-specified tasks but needs your review on anything touching Android-specific patterns, lifecycle management, or multi-device support. Don't let it write Hilt modules unsupervised.
Watching the benchmark over time
The July 8 update archived older Claude and OpenAI model generations. The leaderboard will keep refreshing as new models ship. The Harbor framework makes it straightforward to add new models via community contribution, so the dataset should stay current as the model landscape keeps moving fast. I'd treat the leaderboard as a quarterly recalibration point rather than a one-time lookup.
One thing the benchmark doesn't capture: context window quality. Some of my highest-value AI usage is asking a model to reason about an entire module — 3–5 files at once — and suggest architectural improvements. The benchmark tasks are scoped to individual problems, not full-module comprehension. Fable 5 and Sonnet 5 both have large context windows and perform well on that kind of multi-file reasoning in my experience, but the benchmark number alone doesn't prove it.
How to Contribute Your Own Tasks
Google opened Android Bench to the community as part of this update. You can submit your own custom Android development tasks to the Harbor Hub dataset and share benchmark evaluations. If you've got a real-world Android problem that trips up current AI tools — a specific Compose state management edge case, a tricky multi-module Hilt setup, a KMP migration scenario — it's worth contributing. Better benchmark coverage means more trustworthy scores for everyone.
The full dataset, methodology, model archive, and changelog are at developer.android.com/bench.
The Bigger Picture
What I appreciate about Android Bench is that it's not trying to claim any model is good at coding in general. It's asking the specific question Android developers actually care about: does this model know the ecosystem? Does it understand Compose, Jetpack, KSP, coroutines, multi-device layouts — the things that distinguish Android work from generic Java or Kotlin?
The Harbor migration makes the evaluation methodology transparent and reproducible. Anyone can run the benchmark themselves on their own setup or on newer model versions before they hit the official leaderboard. That kind of openness is how the numbers stay trustworthy as the AI landscape moves fast.
For my own workflow, the July 8 update changes my default recommendation: Fable 5 for capability-first work, Sonnet 5 for cost-conscious regular usage, Qwen 3.6 27B when you need to keep code local. That's a more concrete answer than I could have given six months ago, and it came from an Android-specific benchmark rather than extrapolating from generic code generation scores.
No comments yet. Be the first to leave one!