---
title: Computer-use agent on a real iPhone: Piphone, driven from Linux
url: https://www.consultingai.dev/work/iphone-computer-use-agent/
original: https://www.consultingai.dev/work/iphone-computer-use-agent/
---


# Computer-use agent on a real iPhone: Piphone, driven from Linux

Environments built to order · by [Shreenivas Sangoram](/) · updated 28 Oct 2026

Short answer

Piphone lets an agent run plain-language tasks on a real iPhone plugged into a Linux machine. It acts through the phone's accessibility tree, never blind screen taps, and posting is gated by ask or autopilot limits. The plugin is public under MIT. What I sell is computer-use environments in the same style, on real devices, with tasks and graders built to your spec.

Most computer-use agents are trained and tested in a browser or a desktop VM. Piphone puts one on a real iPhone: you plug the phone into a Linux machine, give the agent a task in plain language, and it works through the apps while you watch or leave it running.

Piphone: an agent opens X on a real iPhone, reads the screen and acts on labelled elements (14 s, from the public repo)

## How it works

- **A supervisor in Go** (`iphone-wdad`) watches the USB connection, runs WebDriverAgent on the phone over a userspace tunnel, and serves it only on owner-only Unix sockets. No jailbreak, no root and no TCP port.
- **The agent's phone tools** (a Pi extension in TypeScript) read the screen of the app a session is allowed to use, act on elements by reference, take screenshots for inspection only, and sleep between events while watching.
- **A shell plugin** (QML for Omarchy's bar) starts sessions, shows them live, and lets you follow up, stop or review them.
- **Small C tools** handle diagnostics and the live MJPEG view.

## Why this shape is useful for training and evaluation

The properties that make Piphone safe to leave running are the same ones that make a good environment:

- **Every action targets a labelled element.** The action space is the accessibility tree, so a grader can check state from the same tree instead of guessing from pixels.
- **Sessions are app-scoped.** A task only sees the apps granted to it, which bounds what the agent can touch.
- **Side effects are gated.** Send, post, follow and message go through a guard with per-hour and per-session caps, so an autonomous run can't spam.
- **Sessions are long and resumable.** Tasks can run with the panel closed, take follow-ups and be reviewed afterwards.

## What you can buy

Computer-use environments on real devices, built to your spec:

- tasks written against real apps, scoped to the apps they need;
- graders that read the accessibility tree for the end state you define;
- side-effect limits, so runs are safe on real accounts or test accounts;
- delivered as HUD tasksets with their environment, or as a repository with a runner.

## Facts

Item | Detail
Public plugin | Piphone-plugin-for-Omarchy , MIT licence
Languages | Go (supervisor), TypeScript (agent tools), QML (shell plugin), C (diagnostics)
Phone control | WebDriverAgent over USB, signed with a free Apple ID through xtool
Default model | DeepSeek V4.1 Flash through OpenRouter; any model Pi supports

## FAQ

### What is a computer-use agent?

An AI agent that operates software through its interface the way a person does: it reads the screen, then taps, types and swipes. Piphone does this on a real iPhone, reading the accessibility tree rather than guessing from pixels.

### Does Piphone need a jailbroken iPhone?

No. It runs a signed WebDriverAgent on a normal iPhone over USB. A free Apple ID works, but its signing profile expires every seven days and has to be renewed.

### Does the agent tap screen coordinates?

No. Every action targets a labelled element from the phone's accessibility tree. Screenshots exist for inspection only.

### Can you build computer-use environments like this for us?

Yes. I build tasks and graders on real devices to your spec, with app scopes and side-effect limits, delivered as HUD tasksets or as a repository with a runner.

### Is Piphone open source?

The Omarchy plugin is public under the MIT licence. The environments I build for clients are delivered under the agreement we sign.

Want this, or something like it?

License it, or have me build an environment in the same style.

Email me →