JetBrains officially launched the Kotlin Benchmark this month — a public leaderboard that evaluates AI coding agents on 105 real engineering tasks pulled from production Kotlin codebases. Not toy puzzles, not HackerRank questions. Real, merged pull requests from active open-source repos, with the agent's output judged by whether it actually passes the existing regression tests.
The timing is right. At this point every Android developer is fielding some version of the question: which AI assistant is actually worth trusting for Kotlin work? Claude, Copilot, Cursor, Junie — they all claim to handle Kotlin well. Now there's a public dataset and a verifiable methodology to push back on those claims with numbers.
Why This Benchmark Is Different
Most AI coding benchmarks measure something that turns out not to predict much: code completion accuracy on synthetic snippets, or performance on human-eval problems where the "right" answer is a few lines of pure algorithmic logic with no project context. Those numbers are easy to game and easy to misread.
The Kotlin Benchmark takes a different approach, borrowed from the SWE-bench methodology originally developed for evaluating agents on Python codebases. The dataset is built entirely from real, merged GitHub pull requests. Each task gives the agent a natural-language issue description and the state of the repository at the time the issue was filed. The agent has to produce a patch. The patch is then run against the project's hidden regression tests — the same tests that the human developer wrote alongside their fix. There is no LLM judge deciding if the output "looks right." It either passes the tests or it doesn't.
JetBrains built this on the open-source Multi-SWE-bench infrastructure, extended with first-class JVM support including Docker images with layered-cache strategies for reproducible builds. The full dataset and harness are public at github.com/Kotlin/kotlin-swe-bench, which means any team can reproduce scores independently instead of taking the vendor's word for it.
The Eight Repositories Behind the Tasks
The 105 tasks are sourced from eight repositories selected for popularity and contributor activity. The distribution matters because it tells you what kind of Kotlin work is actually being evaluated:
- ktlint/ktlint — 43 tasks. The Kotlin linter that every serious Android project runs in CI. Tasks here tend to involve rule implementation, AST traversal, and edge cases in language construct detection.
- detekt/detekt — 28 tasks. Static analysis for Kotlin. Similar flavor to ktlint but with a more plugin-heavy architecture and a broader set of rule categories.
- oss-review-toolkit/ort — 12 tasks. A large, complex Kotlin codebase for license and dependency management. ORT is deeply idiomatic Kotlin — sealed classes, coroutines, extension functions everywhere — so tasks here have teeth.
- Hannah-Sten/TeXiFy-IDEA — 8 tasks. A LaTeX IDE plugin for IntelliJ. Plugin architecture means lots of PSI manipulation and IntelliJ Platform APIs, which tests whether agents understand framework-specific Kotlin patterns.
- ankidroid/Anki-Android — 6 tasks. A real Android app. This is the most directly relevant bucket for Android developers, and it's the only one where you're evaluating the agent on actual app code — Activities, ViewModels, Fragments, Android-specific APIs.
- Kotlin/dataframe — 5 tasks. The Kotlin DataFrame library for data manipulation. Exercises Kotlin's type-safe builder DSLs and extension function heavy APIs.
- square/okhttp — 2 tasks. The networking library that's inside nearly every Android app that ever shipped. Two tasks is a thin sample but the codebase quality is high.
- GradleUp/shadow — 1 task. A Gradle plugin for creating fat/shadow JARs. One task, Gradle-specific.
The honest read on this task distribution: it's heavily weighted toward Kotlin tooling (ktlint + detekt = 71 of 105 tasks). The Android-specific signal comes almost entirely from the 6 Anki-Android tasks. That's a limitation JetBrains explicitly acknowledges — Android and KMP coverage are on the roadmap for the next iteration. For now, the benchmark measures Kotlin language and tooling proficiency more than Android app development capability.
What "resolved" actually means: a task is counted as resolved only when the generated patch passes all of the project's required regression tests. Partial credit, approximate fixes, or "close but the test still fails" count as zero. This is the same bar a code reviewer at any of these projects would apply.
The Leaderboard Results
Here are the top five results from the leaderboard at kotlinlang.org/benchmark as of the benchmark's public launch this month. These runs were conducted between March and June 2026 during the pre-launch period:
| Setup | Resolved | Rate | Avg Tokens | Latency |
|---|---|---|---|---|
| Claude Code + Opus 4.7 (xhigh) | 90 / 105 | 85.71% | 10.59M | 10h 37m |
| Junie + Opus 4.7 (max) | 86 / 105 | 81.90% | 19.98M | 19h 58m |
| Codex + GPT 5.5 (xhigh) | 86 / 105 | 81.90% | 38.78M | 8h 30m |
| Claude Code + Opus 4.6 (max) | 84 / 105 | 80.00% | 13.93M | 12h 39m |
| Codex + GPT 5.3 Codex (xhigh) | 82 / 105 | 78.10% | 8.56M | 8h 01m |
The top five are within about 8 percentage points of each other, which on 105 tasks is a difference of roughly 8 resolved issues. These agents are all in a meaningful cluster, not a clean hierarchy. The more interesting story is in the second and third columns: token usage and latency vary wildly at similar resolution rates.
The Efficiency Dimension Is Where It Gets Interesting
Junie and Codex both hit 81.90% — but Codex (GPT 5.5) used 38.78M tokens compared to Junie's 19.98M. Nearly double the token consumption for an identical pass rate. If you're running these agents at scale in CI or for automated code review, that difference matters directly in cost.
On the other end, Codex (GPT 5.3) resolved 82 tasks using only 8.56M tokens and finished in about 8 hours total. That's a substantially more efficient run than the top result — it's missing 8 resolved tasks compared to the leader, but using roughly 80% fewer tokens to do it. Whether that tradeoff makes sense depends entirely on what you're using the agent for.
Claude Code + Opus 4.7 sits in what looks like a reasonable efficiency spot: the best resolution rate (85.71%), moderate token usage (10.59M), and mid-range latency (10h 37m). Notably, the Opus 4.7 result is a meaningful step up from the Opus 4.6 result (80.00% → 85.71%) while barely moving on token consumption (13.93M → 10.59M). That's a model generation improvement that shows up cleanly in this kind of benchmark.
The latency numbers are total wall time across all 105 tasks, not per-task latency. When people see "10h 37m," they sometimes read that as "this agent takes 10 hours to fix a bug." That's not what it means. It's the cumulative time to run the full benchmark suite. Individual task latency varies widely and isn't broken out in the current leaderboard — that's flagged as a planned future metric.
What This Means for Android Development Right Now
The honest answer is: partially applicable, with important caveats.
The 6 Anki-Android tasks are real Android app code — Kotlin, Android APIs, the works. But 6 out of 105 is a thin signal. Most of the benchmark's weight is on tooling code that exercises the Kotlin language deeply but doesn't reflect the patterns that dominate Android app development: Activity lifecycle management, ViewModel state, Compose recomposition logic, Room schema migrations, WorkManager task chaining, Hilt injection graphs.
That said, there's still useful signal here. Every Android project uses ktlint and detekt in CI. If an agent struggles on the ktlint tasks, it's going to struggle when you ask it to write a custom ktlint rule or debug why a rule is firing incorrectly. The ORT tasks test Kotlin idioms — sealed classes, coroutines, complex generics — that you'll find in any well-written Android codebase. The okhttp tasks, thin as the sample is, directly overlap with networking code most apps contain.
The more meaningful implication is what's coming. JetBrains has explicitly stated that the next iteration will expand into Android and KMP domains. When that lands, the benchmark will be much more directly applicable to app development work. And because the methodology is sound — real code, real tests, no LLM judge — those future results will actually mean something.
How to use this when evaluating tools for your team
A few practical reads from these numbers:
- For automated code review and refactoring runs — where you're running the agent across many files at once and cost per token adds up — the token efficiency column matters as much as the resolution rate. An agent that resolves 4% fewer issues but costs half as much to run may be the better operational choice.
- For interactive development assistance — where you're in a feedback loop with the agent, correcting and iterating — leaderboard resolution rate is a rougher proxy. How the agent explains its reasoning and handles ambiguous requirements matters more than batch pass rate.
- For CI-integrated automated fixes — where you want the highest possible automated resolution with human review as the safety net — the top resolution rate is the number to optimize for, and Claude Code + Opus 4.7 currently holds it.
The benchmark dataset and harness are publicly available. If your team has a Kotlin codebase, you can add your own tasks to the evaluation — particularly useful if you want Android-specific signal before the official Android expansion lands. The Multi-SWE-bench infrastructure handles the Docker reproducibility and test evaluation machinery.
The Meta-Point: This Is What Good Benchmarking Looks Like
For context, we now have two credible, methodology-transparent Kotlin/Android coding benchmarks running publicly: Android Bench (Google's, focused on Android development tasks with the Harbor framework) and the Kotlin Benchmark (JetBrains', focused on Kotlin software engineering tasks at the repository level). Neither is perfect, and they measure somewhat different things. But having two independent, publicly verifiable benchmarks with reproducible methodology is genuinely useful — it means vendor claims about AI coding tool performance now have something to be checked against.
Five years ago, evaluating whether an AI tool would actually help with my Android code meant using it on real code and forming a subjective opinion. That's still necessary, but now there's a quantitative layer underneath it. The Kotlin Benchmark specifically has the property that you can trust its numbers — not because JetBrains says so, but because the dataset is public, the harness is public, and the scoring is binary test pass/fail with no interpretation layer in between.
I'll be watching the Android/KMP expansion closely. The current benchmark is a useful signal. When app-level tasks land in the dataset, the results will be directly actionable for choosing the right AI tools for a production Android project — something the current leaderboard can only gesture toward.
The leaderboard is live at kotlinlang.org/benchmark. The dataset is at github.com/Kotlin/kotlin-swe-bench. Worth bookmarking both — these will update as newer models run the benchmark and as the task set expands.
No comments yet. Be the first to leave one!