An Offline AI Assistant That Drives Your Other Android Apps

Posted on November 13, 2026

Category: Technology

Tags: android, on-device, llm, ai-assistant, offline, intents, deep-links, media-session, flutter, kotlin, llama-cpp, privacy

Views: 21

An Offline AI Assistant That Drives Your Other Android Apps

There are several good free apps that run a language model on an Android phone. You download a model, you chat with it, and nothing leaves the device. That part is solved.

What those apps mostly do not do is touch the rest of the phone. Ask one to start a podcast and it will tell you, politely, that it cannot play audio. Yet that is exactly where an assistant that works without a connection earns its keep: on a plane, on a trail, in a kitchen with wet hands.

MobileAI is built around that idea. It runs the model on the phone, and it drives the apps that are already installed. This post is about what that looks like in practice, how it works with nothing but standard Android plumbing, and one podcast app that ignored every request until it was woken up the right way.

Here is the whole thing in 49 seconds, recorded in airplane mode:

If the player does not load, watch it on YouTube.

What it can do without a connection

Turn on airplane mode and the following still work, typed or spoken:

The pattern behind the list: the phone already has apps that are good at their jobs. The assistant's job is to understand the request and hand it to the right one.

A few situations where that matters more than raw model quality:

Apps that live online stay online, of course. "Search YouTube for bread recipes" opens YouTube's results, and YouTube needs a connection to fill them in.

How a request becomes an action

Every request takes one of two paths.

The keyword path. Most commands have a fixed shape: "play X on Y", "skip N seconds", "set a timer for N minutes". A small set of patterns catches those directly and turns them into an action with no model involved. On a mid-range phone this takes 40 to 140 milliseconds from tapping send to the action firing.

The model path. Anything that does not match a pattern goes to the on-device model with a short description of the available actions, and the model answers with a tool call. The output is grammar-constrained, so it can only name an action that exists. With a 2-billion-parameter model this takes a few seconds; "search YouTube for bread recipes" went through this path in 3.4 seconds when I tested it today.

Either way, the result is an action such as podcasts_play with a query, and the action is carried out with ordinary Android mechanisms.

Standard intents and deep links

There is no private integration with any app. Everything goes through interfaces Android publishes for exactly this purpose.

Common intents. Android defines intents that any app can register for. A clock app handles SET_TIMER and SET_ALARM; an app with search handles SEARCH with a query; a media app can declare MEDIA_PLAY_FROM_SEARCH. MobileAI looks at which installed apps declare these, so an app you add on its Apps screen gets "search" and "play" actions without any per-app code.

Deep links. Some actions are URIs that a particular app understands:

A JSON registry. The apps and their actions live in JSON files, not in code. An entry names the app, the phrases that should trigger an action, and the intent to send — action, package, data URI and extras. Adding a deep link for another app is a matter of adding an entry; the prompt for the model, the grammar that constrains it and the keyword patterns are all generated from the same file.

{
  "name": "navigate",
  "description": "start Google Maps navigation; query = destination",
  "params": ["query"],
  "run": {"intent": {
    "action": "android.intent.action.VIEW",
    "data": "google.navigation:q={query_url}",
    "package": "com.google.android.apps.maps"
  }}
}

Two guardrails sit on top. Only apps installed from a trusted store are targeted, so a sideloaded app cannot register for "play" and receive your queries. And a phone number only goes to the dialer you chose in Settings, never to another app that registered for tel: links.

The podcast app that would not play

Playback is the one place where an intent is not enough. Sending MEDIA_PLAY_FROM_SEARCH to the podcast app I use opens the app and shows its home screen; nothing plays. The request that actually plays is a call on the app's media session: playFromSearch(query) on its TransportControls. MobileAI finds the session through MediaSessionManager.getActiveSessions, which is why it asks for notification access — Android ties access to other apps' sessions to that permission.

That worked well, until it did not. Some mornings "Play [show] on [podcast app]" answered "[podcast app] didn't start a media session." Launching the app first and retrying for eight seconds did not help either: the app's screen came up, but no session ever appeared.

The pattern, once I saw it, was simple. The app creates its media session only once something has played. Open it fresh and there is a full UI and no session; play an episode for a second, pause it, and from then on playFromSearch works. With nothing to talk to, a search request had nowhere to go.

What did not work

The textbook route to an app that is not playing is its MediaBrowserService, the interface Android Auto uses to list and play content. The app publishes one. Connecting to it from MobileAI failed immediately: the service refused the connection, most likely because it only answers clients it recognizes, such as Android Auto and Wear OS. That is the service's call to make, and it closes that route.

An older version of the app had solved the problem differently, by looking up episode IDs through the podcast service's unofficial web API and opening an episode with an internal deep link. It played from a cold start, but it sent requests to a third-party server from an app that promises to work offline, through an interface that was never offered for this. I removed it before the first release, and it was not coming back.

What did

A media app has one more door that Android guarantees: its media button receiver, the component that reacts to the play key on a Bluetooth headset. Sending it a PLAY key starts the app's playback service, and the playback service creates the session.

private fun wake(pkg: String) {
    for (a in listOf(KeyEvent.ACTION_DOWN, KeyEvent.ACTION_UP)) {
        val key = KeyEvent(a, KeyEvent.KEYCODE_MEDIA_PLAY)
        sendBroadcast(
            Intent(Intent.ACTION_MEDIA_BUTTON)
                .setPackage(pkg)
                .putExtra(Intent.EXTRA_KEY_EVENT, key)
        )
    }
}

The key goes to that one app only (setPackage), not to whatever played last. The play request now runs in three steps:

  1. Try playFromSearch on the app's session.
  2. If there is no session, send the PLAY key and retry every half second for up to three seconds.
  3. If there is still none, launch the app, send the key again, and retry for up to eight seconds.

On the test phone, the session appeared 1.9 seconds after the key. From a cold start — the app's process not running at all — "Play [show] on [podcast app]" now starts the requested show in 2.0 to 2.2 seconds, in the background, without the app ever coming to the front. The fix uses no network and no permission the app did not already have.

Showing it without lying about it

Demonstrating an offline assistant has a practical problem: offline maps only route from where the phone is. For the demo video, the phone needed to be in Barcelona and was not.

The answer was a simulated GPS position, supplied by a small service that exists only in debug builds. Android lets one app at a time act as the "mock location app", chosen in Developer options; that service feeds a fixed position once a second, and Google Maps routes from it inside the downloaded Barcelona area — 3.3 km, 13 minutes, in airplane mode. The clip carries a "Location simulated" label, because it is. The video above shows the result; the summary scene is also labelled, because it runs at 2x speed.

Where this goes next

The useful part of an on-device model is not that it can chat without a server. It is that it can understand a loose request and turn it into a precise one for an app that already does the job well. Most of that is available today through intents and deep links that apps publish for anyone to use. The work is in the details: which apps listen, which ones need waking up, and which doors are closed on purpose.

MobileAI is on Google Play.

References

Disclaimer: This blog post was created with assistance from Claude, an AI developed by Anthropic, under my direct supervision and guidance to ensure accuracy and alignment with my vision for the content.