Niches · Desktop & mobile app control
Hand a screen-reading agent an open-ended task on a computer
No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.
What counts as done
The goal's end state is visible on screen or in files, the agent stops and asks before irreversible or credentialled steps, and every action is logged with its screenshot.
Measured, for our own tests: a 20-step office task completes in under 15 minutes. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: set up this app for me
, do this on my computer
, install and configure the VPN client
Does the job, with a limit 2
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| AutoCua D Ashish Yadav and contributors (ComputerUseAgent on GitHub) | Takes a goal in words on macOS, Windows or Linux, saves the chat with screens, and sends no telemetry. But it clicks the affirmative on macOS permission dialogs, lets the model answer UAC prompts, documents no VM setup, and acc-23 found Telegram control open to any chat until 4 November. source |
| Mage-VL-Cowork C Microsoft | 66.0 per cent on 361 OSWorld-Verified tasks in 50 steps (Microsoft's two-run average), and the card mandates a disposable sandbox. The reference harness asks before nothing (the card tells deployers to add confirmation), loading needs trust_remote_code and BF16 on one GPU, and acc-41 found no FP8 build. source |
Couldn't check 2
The docs were closed to us, or silent. These carry no mark either way.
| Listing | Why |
|---|---|
| Automation Anywhere D Automation Anywhere, Inc. | The pages read don't describe an open-ended screen agent; AI Agent Studio is listed as an API group but wasn't read. |
| UiPath B UiPath Inc. | The MCP and agent pages read don't describe an open-ended screen agent an outside agent can hand a goal; UiPath's Studio agent functions weren't read. |
Doesn't do this job 6
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Aura Autopilot C TribeVR Inc. (named in Aura's terms of service and privacy policy), trading as Ramen | Plays games in Unreal, Unity, Godot or Roblox editors through engine plugins; it doesn't operate other desktop apps. source |
| AutoHotkey C AutoHotkey Foundation LLC | No model or screen-reading agent; it runs scripts an agent writes. source |
| AutoIt D AutoIt Consulting Ltd | No model or screen-reading agent; it runs scripts an agent writes. source |
| Beam (beam.win) F Noats, Inc. | Beam shares a Mac window with a person at another Mac; llms.txt lists no API, MCP server, CLI or scripting interface an agent could call. source |
| Robocorp B Sema4.ai (full legal name not confirmed) | Libraries for scripted automation; no goal-driven screen agent or vision model. source |
| Robot Framework BB Robot Framework ry (Robot Framework Foundation), business ID 2754775-1, Helsinki | No model or goal-driven agent; it runs keyword suites. source |
3 listings in the category were added after this reading and aren't marked yet.
What stops agents finishing
- remote-control channel open to anyone who finds it
- model weights need remote code or a full-precision GPU
- headless or virtual display detected by the app
Around it
After: human-in-the-loop.screen-handoff, human-in-the-loop.approve-action.
Where this comes from
Weight 12 of 100 in Desktop & mobile app control: 7 of 45 accounts (AutoCua, Mage-VL-Cowork, Pig and Cua threads); demand is clear, completed outcomes are rare in the accounts
The same as data: /api/v1/use-cases/desktop-automation.screen-agent-task.json, or the MCP tool get_use_case with id desktop-automation.screen-agent-task. Other jobs in this category: /api/v1/niches?category=desktop-automation.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.