Benchmark 1.5.0 Beta: UiAutomator 2.4, Safer AOT Defaults, and In-Process Trace Merging

Benchmark 1.5.0-beta01 landed on July 29 alongside the rest of the large AndroidX drop that week. If you track the benchmark library the way most developers track media or compose — only when something breaks — you might have missed that the alpha cycle leading up to this beta was quietly substantial. Seven alphas over eight months, each fixing a real paper cut or unlocking something that wasn't possible before.

I've been using Macrobenchmark since the 1.2 days, and it's been foundational for tracking the startup wins I documented in the 60% cold start reduction post. The 1.5 cycle is the most consequential upgrade since the library graduated to stable. Here's the full picture across every meaningful change.

1. UiAutomator 2.4 Changes How You Write Benchmark Tests

The most immediately visible change from the alpha01 (December 2025) work is that MacrobenchmarkScope now extends UiAutomatorTestScope. In practice, this means you can call UiAutomator 2.4's modern onElement { } APIs directly inside your measureRepeated blocks without going through device.findObject() boilerplate.

Before 1.5.0, a typical interaction block to tap the compose button in Nodat's quick-note flow looked like this:

benchmarkRule.measureRepeated(
    packageName = "com.nodat.app",
    metrics = listOf(FrameTimingMetric()),
    iterations = 10,
    setupBlock = {
        pressHome()
        startActivityAndWait()
    }
) {
    val fab = device.findObject(By.res("com.nodat.app:id/fab_compose"))
    fab.click()
    device.waitForIdle()
}

With 1.5.0 and UiAutomator 2.4's scope integration:

benchmarkRule.measureRepeated(
    packageName = "com.nodat.app",
    metrics = listOf(FrameTimingMetric()),
    iterations = 10,
    setupBlock = {
        pressHome()
        startActivityAndWait()
    }
) {
    onElement { hasContentDescription("New note") }.click()
    waitForIdle()
}

The ergonomic difference is real. onElement { } with a semantic predicate is more resilient to resource ID changes — content descriptions and roles survive refactors better than resource IDs, and the lambda style composes better with hasTestTag() for Compose hierarchies. The startActivity wait behavior also changed to wait for a new window rather than a fixed idle state, which makes cold-start benchmarks more reliable on slower emulators and lower-end CI devices.

Black Hole APIs stabilized

Also in alpha01: the BlackHole APIs in androidx.benchmark are now stable. If you're writing microbenchmarks for pure computation — sorting, parsing, transform functions — BlackHole.consume() is your escape hatch for preventing dead code elimination from skewing results. Stable means you can depend on it without the experimental suppression annotation.

2. requireAot = true by Default — The Pitfall It Prevents

This change (alpha07, July 1) is the one I wish had happened in 1.2. The benchmark Gradle plugin now sets androidx.benchmark.requireAot to true by default when using AGP 8.4+. This means the benchmark run will fail fast if the measured app isn't AOT-compiled — rather than silently running against a JIT-warm process and reporting optimistic numbers that disappear in production.

The subtle gotcha this prevents: if you run your benchmark immediately after building without explicitly forcing compilation, ART might be running the first few iterations against interpreted or JIT-compiled code. The numbers look stable across iterations — JIT warms up fast — but they're systematically 15-30% better than what a real cold-launched production install would show. It's not a benchmark bug you'd catch by looking at variance; the numbers are just quietly wrong.

The same alpha also updated the TraceProcessor implementation to Perfetto v56.0 and extended the Perfetto kill timeout from 5 seconds to 30 seconds. The longer timeout matters on CI runners — particularly the more constrained ones where Perfetto capture sessions would occasionally fail with a cryptic kill error that had nothing to do with the test itself.

3. TraceProcessor Moves to benchmark-common

Also in alpha07, TraceProcessor.runServer and its extension APIs moved from androidx.benchmark:benchmark-macro to androidx.benchmark:benchmark-common, and into the androidx.benchmark package rather than the macro-only subpackage.

The reason this matters: benchmark-common is the dependency that microbenchmark modules pull in. Previously, if you wanted to analyze a Perfetto trace in a microbenchmark test — say, to verify that a specific trace section appears a certain number of times during a tight computation loop — you had to pull in the heavier benchmark-macro artifact just to get TraceProcessor. Now you don't.

// In a microbenchmark test (benchmark-common dependency only)
@Test
fun countCacheHits() = benchmarkRule.measureRepeated {
    runWithTimingDisabled {
        TraceProcessor.runServer {
            val query = """
                SELECT count(*) as hits
                FROM slice
                WHERE name = 'CacheStrategy#hit'
            """.trimIndent()
            val result = loadPerfettoTrace(perfettoTrace) {
                query(query).toList()
            }
            check(result.first()["hits"] as Long > 0) {
                "Expected cache hits but found none"
            }
        }
    }
}

It's a dependency hygiene improvement more than a new capability, but if you've been keeping macro and micro benchmark modules separate to control build graph size, this matters.

4. KMP Baseline Profile Support (alpha02)

Alpha02 (January 2026) added partial Baseline Profile generation support for Kotlin Multiplatform Library modules. This is specifically for KMP libraries that compile to Android — your shared network or data layer in a KMP project can now produce and consume Baseline Profiles using the baselineProfile extension on the KMP variant.

// In a KMP library module
plugins {
    id("org.jetbrains.kotlin.multiplatform")
    id("com.android.kotlin.multiplatform.library")
    id("androidx.baselineprofile.consumer")
}

baselineProfile {
    variants {
        androidMain {
            from(project(":benchmark"))
        }
    }
}

For pure Android apps this doesn't change anything — Baseline Profile generation worked fine before. But if you're maintaining a KMP library consumed by an Android app, you can now generate profiles that actually improve startup and scroll jank for the shared code path, not just the Android-specific code around it.

5. In-Process Trace Merging with Tracing 2.0 (beta01)

The beta01 feature is the most architecturally interesting one, and it closes a gap that's existed since Jetpack Tracing 2.0 shipped. The new capability: PerfettoCapture and PerfettoTraceRule can now capture in-process Tracing 2.0 trace events and merge them into the same Perfetto trace file that Macrobenchmark produces.

Here's what this unlocks. Before beta01, if you instrumented your code with Tracing 2.0's Tracer.trace() calls — custom slice events, counter tracks, metadata-tagged events — those traces lived in separate files from your Macrobenchmark output. Correlating them required manual tooling: open the macro trace in Perfetto UI, open the in-process trace separately, try to line them up by timestamp. For complex startup paths with multiple initialization phases, this was genuinely painful.

// Define your in-process trace events (Tracing 2.0)
class DatabaseInitializer(private val tracer: Tracer) {

    fun initialize() {
        tracer.trace(category = "startup", name = "db-init") {
            tracer.trace(category = "startup", name = "db-schema-migrate") {
                runMigrations()
            }
            tracer.trace(category = "startup", name = "db-warm-cache") {
                prewarmQueryCache()
            }
        }
    }
}

// In your Macrobenchmark test — beta01
@get:Rule
val traceRule = PerfettoTraceRule(
    enableAppTagging = true,
    // merges in-process Tracing 2.0 events into the macro trace
)

@Test
fun startupWithDbInitTrace() {
    benchmarkRule.measureRepeated(
        packageName = "com.nodat.app",
        metrics = listOf(StartupTimingMetric()),
        compilationMode = CompilationMode.Full(),
        startupMode = StartupMode.COLD,
        iterations = 5,
    ) {
        startActivityAndWait()
    }
}

With beta01, the PerfettoCapture merge target is the same Macrobenchmark output file. Open one trace in Perfetto UI and you see both your system-level startup timeline (activity launch, binder calls, frame timing) and your application-level custom slices (db-init, schema-migrate, cache-warm) on the same timeline. Correlation becomes trivial — scroll horizontally, see what your code was doing at the exact moment the first frame drew.

ArtMetric and the AGP 9.1 Repackaging Fix

Two alphas (alpha04 and alpha05) fixed ArtMetric to handle top-level packages, specifically in projects using AGP 9.1+ or R8's -repackageclasses option. This matters for anyone measuring class-load counts as part of startup benchmarks — artClassLoad* measurements would silently report zero for classes in repackaged namespaces, making it look like your startup wasn't loading anything when in fact R8 had just moved everything to a flat package structure.

If you're on AGP 9.1+ and use ArtMetric, upgrade to at least alpha04 before trusting your class-load numbers.

Upgrading and Migration

Bump the Benchmark BOM to 1.5.0-beta01:

// libs.versions.toml
[versions]
benchmark = "1.5.0-beta01"

[libraries]
benchmark-macro-junit4 = { module = "androidx.benchmark:benchmark-macro-junit4", version.ref = "benchmark" }
benchmark-junit4 = { module = "androidx.benchmark:benchmark-junit4", version.ref = "benchmark" }

// build.gradle.kts (benchmark module)
dependencies {
    androidTestImplementation(libs.benchmark.macro.junit4)
}

// Baseline Profile plugin
plugins {
    id("androidx.benchmark") version "1.5.0-beta01"
}

If you were previously setting androidx.benchmark.requireAot = false in testInstrumentationRunnerArguments to avoid the compilation check, you can remove it — false was already the default before 1.5, so it was a no-op. Now that the default flips to true with AGP 8.4+, decide whether you want to enforce it (recommended) or explicitly opt out.

If you use TraceProcessor.runServer in a microbenchmark module, move the import from the benchmark-macro package to benchmark-common. The API itself didn't change, just the artifact and package it lives in. The old location in benchmark-macro still works but will likely be deprecated in a subsequent release.

For UiAutomator migration: nothing breaks. Your existing device.findObject() calls still compile and run. The onElement { } APIs are additive — migrate incrementally as you touch tests, or run a codemod if you have a large suite.

The Bigger Picture

Benchmark 1.5 is the first release where the library visibly coordinates with two other in-progress Jetpack efforts at once: UiAutomator 2.4's modernized API surface and Tracing 2.0's in-process event model. That's a sign the toolchain is maturing past isolated library improvements toward a coherent measurement story — write custom trace events in production code, instrument user interactions with semantic selectors, merge everything into a single Perfetto file, analyze it in Studio's Performance Analyzer or the Perfetto UI.

The safer requireAot default is the most impactful change for teams that weren't already explicitly setting it. If your benchmark numbers have ever seemed better than production cold-start reality, this was probably part of why. With 1.5, the library enforces the measurement discipline that was always implied but never guaranteed.

For a library that tends to be stable-and-boring between major releases, this cycle delivered genuine capability upgrades. Worth the upgrade — beta01 is solid enough to run in CI; the API churn risk from here to stable is low.