ML Kit GenAI Gets @Generable Structured Output and Gemini Nano Speech Recognition

Google published a four-part "Build intelligent Android apps" series on July 22 alongside the Jetpacker open-source travel app — a reference implementation that uses ML Kit GenAI APIs end to end. The series itself is worth reading, but what I want to dig into are two specific additions to the ML Kit GenAI stack that haven't received their own spotlight: a KSP-based structured output API using @Generable annotations, and a brand-new genai-speech-recognition library (1.0.0-alpha1) that routes through Gemini Nano on supported devices. Both are genuinely new patterns that go well beyond what the earlier ML Kit GenAI release covered.

If you read my earlier post on Gemma 4 and the ML Kit GenAI Prompt API, you know the basics — Generation.getClient(), ModelPreference.FULL/FAST, hybrid inference. What I'm covering here is the layer on top: how to reliably extract structured data from model output without parsing JSON yourself, and how to get high-quality real-time transcription without leaving the device.

The Structured Output Problem

Here's a pattern you've probably written at least once: prompt an LLM, get back a string, then try to parse JSON out of it. It works until it doesn't. The model wraps the JSON in markdown code fences, adds an explanation paragraph, or silently drops a field you expected to be there. You end up adding retry logic and a fragile regex to clean the output, then wondering why your feature feels flaky in testing but never on your own device.

The @Generable approach inverts this. Instead of telling the model to "output valid JSON", you define a Kotlin data class, annotate it, and let a KSP-generated schema compiler handle the contract between your type system and the model's output. The model is constrained to produce output that maps to your class — you get a typed Kotlin object back, not a raw string.

Setup

// build.gradle.kts (app)
plugins {
    id("com.google.devtools.ksp")
}

dependencies {
    implementation("com.google.mlkit:genai-prompt:1.0.0-beta3")
    ksp("com.google.mlkit:genai-schema-compiler:1.0.0-alpha1")
}

The genai-schema-compiler is a KSP plugin — it processes your annotated classes at compile time and generates the schema the model constrains its output against. No reflection, no runtime parsing overhead.

Defining a Generable data class

@Generable("Information extracted from an expense receipt")
data class ParsedReceipt(
    @Guide("Generated title under 6 words, based on the restaurant or activity name.")
    val title: String,

    @Guide("Total amount. Look for values at the bottom and words like 'total' or 'balance due'.")
    val amount: Double,

    @Guide("Type of expense", enumValues = ["travel", "food", "shopping", "entertainment", "other"])
    val category: String,
)

Two annotations do the work. @Generable on the class provides a high-level description of what this type represents — the model uses this as context for how to populate it. @Guide on each field gives per-field instructions: what to look for, what format to use, what the valid enum values are. The @Guide on category effectively creates a closed vocabulary without you having to write a prompt-level instruction like "only output one of these five strings".

This is the key advantage over prompt engineering alone. Constraints live in code, next to the types they constrain, version-controlled and refactorable. When you rename a field, the schema updates automatically on the next build.

Making the request

val config = generationConfig {
    modelConfig = modelConfig {
        releaseStage = ModelReleaseStage.PREVIEW
        preference = ModelPreference.FULL
    }
}
val model = Generation.getClient(config)

val prompt = "Determine if this image is a receipt. " +
    "If NOT a receipt, output the text 'NOT_A_RECEIPT'. " +
    "Otherwise, parse the receipt information."

val baseRequest = generateContentRequest(ImagePart(bitmap), TextPart(prompt)) {}
val typedRequest = generateTypedContentRequest(baseRequest, ParsedReceipt::class)

val response = model.generateContent(typedRequest)
val parsedReceipt: ParsedReceipt? = response.candidates.firstOrNull()?.response

generateTypedContentRequest() wraps the base request and injects the compiled schema. The response comes back as ParsedReceipt? — a null-safe Kotlin object. No JSON decoding, no field mapping, no try/catch around a JSONObject.

The multimodal angle matters here. ImagePart(bitmap) passes the receipt image directly. Gemini Nano 4's multimodal capability handles the visual extraction — reading amounts from a photo, recognizing merchant names from a logo — without you needing a separate OCR pass first. The Jetpacker reference app's expense tracker feature is built exactly this way: photograph a receipt, get a structured ParsedReceipt object back, no cloud call involved.

Gemini Nano Speech Recognition

The second new library is genai-speech-recognition:1.0.0-alpha1. On the surface this looks like another on-device speech API, and Android already has several of those. What's different here is the Advanced mode, which routes recognition through Gemini Nano itself rather than a traditional ASR model.

Two modes

The library exposes two modes through SpeechRecognizerOptions.Mode:

• MODE_BASIC — traditional on-device speech recognition, requires API level 31+, works on a wide device range
• MODE_ADVANCED — Gemini Nano handles transcription, currently available on Pixel 10 and newer devices; broader language coverage, better accuracy on accented speech and domain-specific vocabulary

The API surface is identical between modes — you switch the enum value, not the code structure. That makes the fallback pattern clean: you can try MODE_ADVANCED, catch the availability exception, and fall back to MODE_BASIC on unsupported devices without restructuring your flow.

Dependency and streaming API

// build.gradle.kts
implementation("com.google.mlkit:genai-speech-recognition:1.0.0-alpha1")

val options = speechRecognizerOptions {
    locale = Locale.US
    preferredMode = SpeechRecognizerOptions.Mode.MODE_ADVANCED
}
val recognizer: SpeechRecognizer = SpeechRecognition.getClient(options)

suspend fun transcribeFromMic(recognizer: SpeechRecognizer): String {
    var transcription = ""
    val request = speechRecognizerRequest {
        audioSource = AudioSource.fromMic()
    }
    recognizer.startRecognition(request).collect { response ->
        when (response) {
            is SpeechRecognizerResponse.PartialTextResponse ->
                updateLiveUI(response.text)
            is SpeechRecognizerResponse.FinalTextResponse ->
                transcription = response.text
        }
    }
    return transcription
}

startRecognition() returns a Flow. PartialTextResponse carries interim results as the model transcribes in real time — useful for displaying live captions or giving the user visual feedback that something is happening. FinalTextResponse is the committed output once the model determines the utterance is complete. The whole thing integrates naturally with coroutines and collectAsState() in Compose.

Prompt Engineering for Performance: The 13-Second Problem

One of the more honest technical details in the Jetpacker blog series is that their initial on-device inference took 13 seconds for receipt parsing. That's clearly not shippable for a feature users expect to be instant. They got it under 2 seconds through iterative prompt optimization — which is worth understanding concretely, because the same principles apply to any structured output task.

The bottlenecks in on-device LLM inference are largely on the input side, not the generation side. More tokens in = more processing before the model produces the first output token. Three things tend to bloat prompts without adding precision:

• Verbose "please" language — models don't respond to politeness; every extra word is compute
• Restating the schema in the prompt — when you use @Generable, the schema is already injected; repeating it in the prompt text doubles the constraint cost
• Ambiguous negative instructions — "if this is not a receipt, output NOT_A_RECEIPT" gives the model a cheap short-circuit path; without it, the model reasons through the full extraction even on clearly non-receipt images

The pattern that worked: give the model a short decision gate first ("is this a receipt?"), then the extraction instruction. Short prompts that front-load the decision reduce both latency and the likelihood of the model producing an unexpected output shape.

For Nodat and Musist I've run into the same dynamic with cloud inference — prompt tokens are billed, and verbose prompts that feel "safer" actually hurt both cost and latency. The constraint is tighter on-device because there's no horizontal scaling to absorb inefficiency, but the optimization principle is identical.

When to Use This Stack vs. Firebase AI Logic

The Jetpacker architecture makes the on-device vs. cloud decision explicitly, per feature, and it's worth understanding the reasoning rather than defaulting to one pattern everywhere.

On-device ML Kit GenAI is the right call when:

• The data is privacy-sensitive (receipts, voice, health metrics) and should never leave the device
• The task doesn't require world knowledge (receipt parsing, text summarization, categorization from a known vocabulary)
• Offline capability matters for your use case
• You want to eliminate cloud inference cost entirely for features that can handle it

Firebase AI Logic stays in scope when:

• The task requires grounding in real-world information (Jetpacker's Place Q&A with Google Maps grounding)
• You need to reach all Android devices, not just the ones with Gemini Nano
• The generation task is too complex or long for an on-device model

The hybrid inference path — PREFER_ON_DEVICE with cloud fallback — covers the middle: features where on-device is ideal but you don't want to gate the feature on device capability. Firebase AI Logic's hybrid inference handles the routing transparently, so the app code stays the same regardless of which inference path fires.

What This Means for the Average Android App

Before this ML Kit GenAI release, "on-device AI" on Android mostly meant Core ML Kit features — barcode scanning, face detection, translation — plus the ML Kit GenAI Prompt API for freeform text generation. The @Generable addition changes the category: structured data extraction from unstructured input (images, text, audio) is now a first-class on-device primitive.

Think about the features this enables without a backend call:

• Receipt and document parsing (expense trackers, receipt capture, document digitization)
• Voice-to-structured-action (voice notes categorized into tasks, dictation with automatic field extraction)
• Image classification into custom taxonomies beyond what the pre-built ML Kit models handle
• Any flow where you prompt the user for text and want structured output — feedback forms, intake questionnaires, diagnostic inputs

The genai-speech-recognition library in Advanced mode is particularly interesting for apps that already use voice. If you're shipping an app with voice commands or dictation and currently routing through Android's standard speech recognition, MODE_ADVANCED on Pixel 10+ gives you Gemini Nano-quality transcription with the same API surface. The streaming Flow approach maps cleanly onto Compose's state model, and the PartialTextResponse / FinalTextResponse split gives you the data you need for real-time UI updates without polling.

The Reference App

The Jetpacker source is available at github.com/android/ai-samples/tree/main/jetpacker. It's worth cloning even if you only browse it — the architecture shows exactly how Google structures the on-device/cloud decision boundary, how @Generable data classes are organized alongside the features that use them, and how the AppFunctions integration ties the app's capabilities into Android's intelligence system. For anyone building features in this space, it's the most complete end-to-end sample Google has published for the current ML Kit GenAI stack.