Most computer-use agents are trained and tested in a browser or a desktop VM. Piphone puts one on a real iPhone: you plug the phone into a Linux machine, give the agent a task in plain language, and it works through the apps while you watch or leave it running.
How it works
- A supervisor in Go (
iphone-wdad) watches the USB connection, runs WebDriverAgent on the phone over a userspace tunnel, and serves it only on owner-only Unix sockets. No jailbreak, no root and no TCP port. - The agent's phone tools (a Pi extension in TypeScript) read the screen of the app a session is allowed to use, act on elements by reference, take screenshots for inspection only, and sleep between events while watching.
- A shell plugin (QML for Omarchy's bar) starts sessions, shows them live, and lets you follow up, stop or review them.
- Small C tools handle diagnostics and the live MJPEG view.
Why this shape is useful for training and evaluation
The properties that make Piphone safe to leave running are the same ones that make a good environment:
- Every action targets a labelled element. The action space is the accessibility tree, so a grader can check state from the same tree instead of guessing from pixels.
- Sessions are app-scoped. A task only sees the apps granted to it, which bounds what the agent can touch.
- Side effects are gated. Send, post, follow and message go through a guard with per-hour and per-session caps, so an autonomous run can't spam.
- Sessions are long and resumable. Tasks can run with the panel closed, take follow-ups and be reviewed afterwards.
What you can buy
Computer-use environments on real devices, built to your spec:
- tasks written against real apps, scoped to the apps they need;
- graders that read the accessibility tree for the end state you define;
- side-effect limits, so runs are safe on real accounts or test accounts;
- delivered as HUD tasksets with their environment, or as a repository with a runner.
Facts
| Item | Detail |
|---|---|
| Public plugin | Piphone-plugin-for-Omarchy, MIT licence |
| Languages | Go (supervisor), TypeScript (agent tools), QML (shell plugin), C (diagnostics) |
| Phone control | WebDriverAgent over USB, signed with a free Apple ID through xtool |
| Default model | DeepSeek V4.1 Flash through OpenRouter; any model Pi supports |
FAQ
What is a computer-use agent?
An AI agent that operates software through its interface the way a person does: it reads the screen, then taps, types and swipes. Piphone does this on a real iPhone, reading the accessibility tree rather than guessing from pixels.
Does Piphone need a jailbroken iPhone?
No. It runs a signed WebDriverAgent on a normal iPhone over USB. A free Apple ID works, but its signing profile expires every seven days and has to be renewed.
Does the agent tap screen coordinates?
No. Every action targets a labelled element from the phone's accessibility tree. Screenshots exist for inspection only.
Can you build computer-use environments like this for us?
Yes. I build tasks and graders on real devices to your spec, with app scopes and side-effect limits, delivered as HUD tasksets or as a repository with a runner.
Is Piphone open source?
The Omarchy plugin is public under the MIT licence. The environments I build for clients are delivered under the agreement we sign.