Gemma 4 and the ML Kit GenAI Prompt API: Building Real On-Device AI in Android Apps

On-device AI on Android has always felt like a demo-first, ship-second proposition. You'd see the keynote slide, get excited, integrate something experimental, and then spend the next release cycle chasing renamed APIs. Gemma 4 and the production path Google has built around it is a different situation. The model is open, the API is shipping, and there's a clear upgrade path as hardware catches up. This is the moment where "later" becomes "now."

I've been watching this space closely because it directly affects the kind of apps I build. Nodat's note-taking flows, Musist's metadata enrichment — both have features I've wanted to power with local inference rather than cloud round-trips. The latency, the offline reliability, the cost model — all of it points toward on-device. The question was always whether the platform was ready. With Gemma 4, it mostly is.

The Architecture You Need to Understand First

Before you write a single line of code, understand the stack. Google has built a tiered system and the terminology is easy to confuse:

AICore is the Android system service that manages model distribution, download, execution, and hardware acceleration on-device. It surfaces on devices with specialized AI accelerators from Google, MediaTek, and Qualcomm. Critically, it also provides a CPU fallback on devices without dedicated hardware — slower, not representative of production performance, but enough to develop and test against.

The AICore Developer Preview is a program that lets you prototype specifically with Gemma 4 E2B and E4B on AICore-enabled devices before those models are broadly rolled out. The access path is a sign-up form; models push directly to test devices.

The ML Kit GenAI Prompt API is the production-facing API, built on top of AICore, that your shipping app code should target. This is where the stable contract lives. The AICore Developer Preview is a prototyping ramp; the ML Kit Prompt API is where you land.

The migration story matters: code you write against Gemma 4 via the ML Kit API today will automatically work on Gemini Nano 4-enabled devices when those ship later in 2026. Gemma 4 is the foundation for next-generation Gemini Nano. Build against the API now, get the future performance improvement for free when hardware lands.

Gemma 4: E2B vs E4B — Pick Your Trade-Off

Google ships two Gemma 4 model variants for on-device use:

• E4B (Full/FULL preference): the more capable model, designed for complex reasoning, nuanced generation, and tasks where quality matters more than raw speed. Think: summarizing a long document, generating structured content, multi-step reasoning over context.
• E2B (Fast/FAST preference): three times faster than E4B for lower latency tasks. Text autocomplete, quick rewrites, real-time proofreading — anything where the user is waiting and 300ms vs 100ms is a felt difference.

The models support 140+ languages and handle text, images, and audio — proper multimodal, not a text-only API with vision bolted on. That said, for most apps I'd start with text and add vision when there's a specific user-facing feature that justifies the added model weight.

The ML Kit GenAI Prompt API — The Code That Ships

The Prompt API is the entry point for most use cases. During the AICore Developer Preview phase, you select the model via ModelReleaseTrack.PREVIEW and choose your model tier with ModelPreference.FULL (E4B) or ModelPreference.FAST (E2B). The checkStatus() call is non-negotiable — you need to handle the case where the model hasn't downloaded yet or the device doesn't support it:

// Add to your build.gradle:
// implementation("com.google.mlkit:genai-prompt:16.0.0-beta1")

import com.google.mlkit.genai.prompt.GenerativeModel
import com.google.mlkit.genai.prompt.generationConfig
import com.google.mlkit.genai.prompt.ModelConfig
import com.google.mlkit.genai.prompt.ModelReleaseTrack
import com.google.mlkit.genai.prompt.ModelPreference
import com.google.mlkit.genai.common.FeatureStatus

// Configure for Gemma 4 E4B (full reasoning)
val config = generationConfig {
    modelConfig = ModelConfig {
        releaseTrack = ModelReleaseTrack.PREVIEW
        preference = ModelPreference.FULL  // E4B
    }
}

val model = GenerativeModel.getClient(config)

// Always check status before generating
val status = model.checkStatus()
if (status == FeatureStatus.AVAILABLE) {
    val response = model.generateContent("Summarize this in two sentences: $userText")
    showResult(response.text)
} else if (status == FeatureStatus.DOWNLOADABLE) {
    model.download()  // trigger download, show a "preparing" state
} else {
    // Device doesn't support on-device inference — fall back to cloud
    fallbackToCloud(userText)
}

Swap ModelPreference.FULL to ModelPreference.FAST and you get E2B — same code, different model. For streaming responses where you want to show text as it generates, the API also exposes a generateContentStream() variant that returns a Flow<GenerateContentResponse>.

The higher-level ML Kit GenAI APIs

If your use case is standard enough, ML Kit wraps common inference patterns into dedicated APIs so you don't manage prompt engineering yourself:

• Summarization API — article or conversation → bulleted summary
• Proofreading API — correct short chat messages
• Rewriting API — change tone or style of a message
• Image Description API — generate a short description from an image
• Speech Recognition API — transcribe audio to text on-device

For Musist, the Image Description API is interesting — I've been doing cover art metadata extraction via a third-party OCR library. Doing that on-device with a multimodal model that actually understands context would be a meaningful quality upgrade, and it keeps user media off any server. For Nodat's quick-capture notes, the Summarization API is an obvious candidate for the "distill this note" feature I've wanted to ship for months.

Hybrid Inference: The Pragmatic Production Pattern

Pure on-device is the ideal. Practical apps need a fallback. Firebase AI Logic has added a Hybrid Inference API that handles routing between on-device and cloud models, and it's the pattern I'd actually recommend for most production apps rather than writing your own fallback logic:

import com.google.firebase.ai.FirebaseAI
import com.google.firebase.ai.InferenceBackend
import com.google.firebase.ai.InferenceMode

// PREFER_ON_DEVICE: tries on-device first, falls back to cloud if unavailable
// PREFER_CLOUD: cloud first, on-device as fallback
// ONLY_ON_DEVICE: fail if on-device isn't available
// ONLY_CLOUD: always cloud

val ai = FirebaseAI.getInstance(InferenceBackend.GOOGLE_AI)
val model = ai.generativeModel(
    modelName = "gemini-2.0-flash",
    inferenceMode = InferenceMode.PREFER_ON_DEVICE
)

// The API handles the routing — your prompt code doesn't change
val result = model.generateContent("Summarize: $text")

The PREFER_ON_DEVICE mode is what you want for most features: run locally on supported devices, cloud when not. The model name you pass is used as the cloud fallback target; on-device, the system uses whatever local model is available. This means your feature works everywhere without you writing device capability checks.

Prefix Caching: The Performance Win You Should Know About

On-device inference is meaningfully faster than a cloud round-trip for short completions, but there's still startup overhead when a model needs to process a long system prompt or a shared context block every time. Prefix caching addresses this by storing the intermediate LLM state of recurring prompt segments so the model doesn't reprocess them on every call.

The practical implication: if you have a fixed system instruction (e.g., "You are a proofreader. Correct grammar only, preserve the author's voice.") that appears in every call, prefix caching lets the model skip reprocessing that prefix. On-device, where you can't amortize warm-up across thousands of concurrent users the way a server can, this matters more than it does in cloud inference.

// Prefix caching is configured at the model level
val config = generationConfig {
    modelConfig = ModelConfig {
        releaseTrack = ModelReleaseTrack.PREVIEW
        preference = ModelPreference.FULL
    }
    // Cached prefix — processed once, reused across multiple generateContent() calls
    systemInstruction = content("You are a concise technical editor. Improve clarity. Never change code.")
}

val model = GenerativeModel.getClient(config)

// These calls reuse the cached system instruction processing
val result1 = model.generateContent(userNote1)
val result2 = model.generateContent(userNote2)

LiteRT-LM: When You Need Your Own Fine-Tuned Model

If Gemma 4 out of the box doesn't fit your domain — say, you've fine-tuned a smaller language model on your app's specific vocabulary or task structure — LiteRT-LM (formerly TFLite) now supports bringing your own small language models to Android. This is the "BYOM" path: export your fine-tuned model, convert it for LiteRT, and run it through the same on-device infrastructure.

I don't have a concrete use case for this in my current apps, but I can see it being relevant for apps with specialized domains (medical terminology, legal text, a specific language dialect) where a general model's latency vs. accuracy trade-off doesn't work out.

What About AppFunctions and the ADK?

Two more experimental features from Google I/O '26 that are worth tracking:

AppFunctions (experimental preview) lets your app expose capabilities as on-device MCP (Model Context Protocol) server functions — essentially making your app's features callable by system agents and other AI-powered apps. Think: an agent asks "save this note to Nodat" and your app handles it without the user opening it. The Jetpack library is available, but this is early-stage.

ADK for Android (first experimental version) provides orchestration for building multi-agent workflows that span on-device and cloud models — managing context, sessions, and agent coordination. If you're building anything where one model's output feeds another model's input, ADK removes a lot of plumbing. Not something to build production features on yet, but the direction is clear.

My Take: When Does On-Device AI Actually Make Sense?

After working through all of this, here's how I'd actually decide:

Build on-device when: the feature involves user-generated content the user shouldn't want leaving their device (journal entries, private notes, chat messages), the UX requires sub-200ms responses (inline autocomplete, real-time suggestions), or your user base is network-constrained (spotty connectivity, data costs matter).

Keep it cloud when: the task is computationally heavy enough that even E4B would struggle (long-form generation, complex reasoning over large documents), you need the absolute latest model capabilities, or the feature is used infrequently enough that download overhead would be felt.

Use hybrid when: you want on-device as the fast path and cloud as a quality guarantee. This is where most features actually land in practice — and the Firebase AI Logic Hybrid Inference API makes it a three-line decision rather than a defensive if-else tree across your entire codebase.

The 140+ million devices already running AICore-capable hardware means on-device isn't a "future" feature for a niche subset of your users. For many apps, it's already the majority path. The question isn't whether to build on-device — it's which features to start with. For me, that's Nodat's summarize-my-notes flow, and it's shipping as soon as I validate latency on a couple of target devices.