Mobile agents on Android: from Appium automation to a mobile agent client
I have put agents on Android phones in several ways: an LLM agent that drives Android apps through Appium, an agent TUI built on mobile-mcp, a prototype that embedded an agent runtime in an Android app, and a signed Android client for a running Hermes agent, installed and working on a real phone. With Piphone on iPhone, this is the second platform for the computer-use environments I build.
Phones are where most people actually use software, and they are a harder target for agents than a browser: small screens, gestures, app sandboxes and permission prompts.
What I've built
- An agent TUI on top of mobile-mcp Private
- Hermes Agent Mobile: a signed Android client for a running Hermes agent over Tailscale Public
- An agent runtime embedded in an Android app, then a pivot to a thin client Private
- Android-Use: an LLM agent driving Android apps Public
- An Android agent combining screen vision with Appium, driven by a smolagents code agent Private
- Agent Smith: Android automation with a smolagents code agent and Appium Private
What the work taught
- Keep the agent where it's strong. The embedded runtime worked, but the better product was a thin client: the agent runs on a server, and the phone app is a careful window onto it.
- Fix mobile problems in the app, not the server. Keyboard, scroll and resume bugs belong in the Android shell, so every user gets the fix without patching their agent.
- Write the failures down. Each project keeps dated checkpoints of what broke and what was reverted, which is also what makes later environments reproducible.
What you can buy
Computer-use environments on Android, built to your spec: tasks on real apps, graders that read UI state through the accessibility layer, and limits on side effects. Paired with Piphone for iPhone, the same task can be run on both platforms.
FAQ
How do AI agents control Android apps?
Usually through an automation layer such as Appium or a mobile MCP server, which exposes the screen's UI elements and lets the agent tap, type and swipe on them by reference.
Can you build environments for Android and iPhone?
Yes. I build computer-use tasks and graders on both, using the accessibility layer on each platform so graders check real UI state.
License it, or have me build an environment in the same style.