Head to head · Agent harnesses · October 2026 research run
boxes.dev vs Pi
Pi scores 66.4 (B) on agent readiness against boxes.dev's 53.2 (D), and leads in 6 of 7 scored categories. boxes.dev leads on security & auth. Both do agent harnesses.
Best agent harnesses and coding agents · All 1767 harnesses comparisons
Which one, for what
Best for Developers who want Claude Code or Codex running on cloud copies of their dev environment, started from Slack, Linear, a schedule, an API call or another assistant.
Ahead on
- Security & auth, 69 against 61
Also in its favour
- A hosted endpoint, with nothing to install
- No incidents deducted, where Pi loses 10 points for them
Watch for
No status page, incident history or SLA found
Pi B
Best for Developers and pipelines that want a small, scriptable coding agent they can extend in TypeScript and drive over JSONL or RPC, inside their own container.
Ahead on
- Reliability, 75 against 32
- Schema & documentation, 84 against 57
- Agent ergonomics, 76 against 60
- Payments & pricing, 90 against 50
- Maintenance & community, 87 against 48
- Transparency & trust, 67 against 60
Also in its favour
- No key needed to call it
- Free to start without a card
- Open source
Watch for
No permission system or sandbox. Tool calls run with the user's rights and without an approval prompt
Why the edge
Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.
Reliability Pi +43
boxes.dev 32
Hosted reading, for the remote MCP server at api.boxes.dev/mcp and the automation run endpoints. No status page found. The site footer says "All systems normal" with no link, and the trust centre at trust.boxes.dev draws in a script our reader could not run (0). No readable incident history (5). No request rate limits found. Plans cap awake devboxes (3 on the trial, 5 to 25 on paid plans), and the API answers 409 active_fork_limit_reached at the cap (5 of 15). The automation API takes an Idempotency-Key and returns the original run on a retry with idempotentReplay: true, and the MCP connection retries a lost start or message with the same request key. No Retry-After or backoff guidance found (12 of 15). No SLA found (0). Neither the MCP connection nor the automation API is marked beta, and the product launched in June 2026 per its own blog (10). 0+5+5+12+0+10 = 32.
Pi 75
Read as a local package. @earendil-works/pi-coding-agent 1.4.0 on npm (still the latest on 1 November 2026) with Node.js 22.19 or newer stated, install scripts for macOS, Linux and Windows, Nix and standalone binaries (20). A public CI workflow builds, checks and tests on every push to main and runs an MCP client conformance job. The run on the commit we cloned (ce950d7, 8 October 2026) passed, but eight of the last 15 runs on main, all on 7 and 8 October, failed. Run results weren't readable on 23 October, so this stands per the 8 October check (15). The repository API reported 308 open issues and pull requests against 113,417 stars on 8 October, not re-read on 23 October. Issues from new contributors are closed automatically and reviewed daily by maintainers, per CONTRIBUTING.md, so the open count understates what is reported, and we couldn't sample reply times (16). A dated changelog with Breaking Changes sections and migration steps, but breaking changes ship outside major versions. The Azure provider rename shipped in patch release 1.0.3 on 5 October 2026, minor release 1.2.0 on 12 October carried four (theme schema location, before_agent_start, busy-session behaviour and pi mcp list --json output), and 1.3.0 required /v1 on kimi-coding base URL overrides under Changed. The same reading as agent-zero and mistral-vibe-cli, and these breaks also take a deduction under negative events (9). 1.4.0 on 21 October 2026, with 1.0.0 dated 1 October 2026 (15).
Payments & pricing Pi +40
boxes.dev 50
No x402, MPP or L402 (0). Plan prices are public ($19, $99 and $249 a seat a month) and extra usage is $0.40 a box-hour, with a box-hour defined as one hour of a 4 vCPU, 8 GiB devbox (30). A free trial of 20 box-hours. The help page says some teams must add a payment method before starting, so a card-free start is not certain (20 of 30). A person signs in with Google or Apple and approves the OAuth grant or creates the automation secret in the desktop app (0). 0+30+20+0 = 50.
Pi 90
No payment protocol in the repository or docs (0). Read with the self-hosted rule. Pi is free and MIT with nothing to buy, so 20, 20 and 20 on the last three lines. Model costs go to whichever provider the owner configures. Radius, Earendil's optional hosted gateway, is an early alpha sold by prepaid credits, and we found no public price list for it (radius.earendil.com/pricing returns 404 on 23 October 2026). Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 60 is 90.
Maintenance & community Pi +39
boxes.dev 48
Closed service. CLI @boxes-dev/cli 1.0.1134 published on 17 November 2026 (30). Only one dated product update was found, on 27 October 2026, and earlier CLI versions' dates were not read, so three releases in 90 days are not established (0). A support address, an in-app chat and a blog, with no public tracker or changelog (8 of 25). A CLI but no SDK, and no entry in the official MCP registry was looked for (5 of 15). The CLI is published from GitHub Actions with npm trusted publishing and signatures (5 of 10). 30+0+8+5+5 = 48.
Pi 87
v1.4.0 on 21 October 2026, still the latest on npm on 1 November (30). The changelog dates 23 releases in the 90 days from 3 August, 0.84.0 on 6 August to 1.4.0 on 21 October 2026 (20). Commits land daily and the changelog credits issues and outside pull requests in most releases. New contributors' issues are closed automatically, reviewed daily, and get no reply if they miss the quality bar, per CONTRIBUTING.md. Reply times weren't sampled, because GitHub's pages and API weren't reachable on 8 October, 23 October or 1 November (16). npm, the pi.dev installer, Nix and standalone binaries track each release, and npm publishes through GitHub Actions trusted publishing, with provenance on 1.4.0 (14). Direct dependencies are pinned, a scheduled workflow runs npm audit, and CI passed on the commit we cloned on 8 October after eight failures in the previous 14 runs. CI results weren't readable on 23 October or 1 November (7).
Schema & documentation Pi +27
boxes.dev 57
No OpenAPI document, and the MCP tool list could not be read because the endpoint answers 401 before initialize without an OAuth token. The automation API's request and response fields are documented in prose tables (5 of 25). llms.txt with every help page listed (10). The help pages say when to target a new devbox, an existing devbox or an existing thread, when to steer a busy agent and when to wake a devbox first (16). Field tables with enum values such as backend (codex or claude), delivery (steer or next) and four run statuses, and unknown fields are rejected. The MCP schemas themselves were not seen (9 of 15). curl examples and named errors such as 409 devbox_unavailable, 409 thread_unavailable and 503 devbox_wake_failed, with what each means (12). /v1 in the API path on the home page, a single dated product update of 27 October 2026 and no changelog (5 of 15). 5+10+16+9+12+5 = 57.
Pi 84
Harness reading, as for the other harnesses. JSON Schemas for settings, models, keybindings and themes ship in the package and, since 1.2.0 on 12 October 2026, at pi.dev/schemas/, and the JSON event stream and every RPC command are documented record by record, but as Markdown and TypeScript types with no machine-readable spec for the RPC protocol (18). pi.dev/llms.txt returns 404. Each docs page answers Accept: text/markdown with Markdown, and the docs ship inside the npm package (8). The integration page has a table saying when to use interactive, print, JSON, RPC or the SDK, and the MCP page gives a typical use for each exposure setting. 1.4.0 adds a Models Configuration reference generated from the models.json schema (17). Enumerated modes, thinking levels and exposure values, typed settings with defaults (13). A Python RPC client, a typed TypeScript client and about 75 example extensions, with command errors and parse errors shown as records (13). A dated changelog in the repository and at pi.dev/changelog with an RSS feed (15).
Agent ergonomics Pi +16
boxes.dev 60
The MCP tool count is unknown. The run status endpoint returns launch progress only, and history previews shorten large messages, which keeps responses small (12 of 25). Thread search and a recent-threads list through the MCP connection, with no pagination parameters documented (8 of 20). Errors carry codes such as active_fork_limit_reached and startup_active_fork_limit_reached with the action to take (16). Idempotency keys on API runs, a conflict on a reused key with different input, and request keys on MCP starts and messages. Tool annotations could not be read (16 of 20). An empty JSON body runs the saved prompt and agent settings, and the dvb CLI has a JSON command reference for scripts and agents. No SDK (8 of 15). 12+8+16+16+8 = 60.
Pi 76
Harness reading of the framework line, scored on what an agent or pipeline driving it has to supply. Four default tools (read, bash, edit, write) and a short system prompt. MCP tools default to codemode exposure, which keeps them out of the model's tool declarations, and tool_search loads deferred ones on demand (23). Automatic compaction, agent-level retry with three attempts and exponential backoff, and truncated tool output saved to a file. We found no turn or step limit in the CLI or settings reference (13). --mode json streams JSONL events ending in agent_settled and --mode rpc returns success: false with an error string. Print mode exits nonzero on a failed response, JSON mode doesn't (16). Sessions are stored as trees and continue, fork or take a fixed ID with --continue, --fork and --session-id. File changes have no built-in undo, and a git checkpoint is an example extension (14). It needs a provider key or subscription login before the first run, and the SDK is TypeScript only, with RPC for other languages (10).
Security & auth boxes.dev +8
boxes.dev 69
OAuth for the MCP server, with protected-resource metadata naming mcp.boxes.dev and four scopes (threads:read, threads:write, events:subscribe, settings:write), a per-project grant and removal from settings. Automation endpoints use a per-automation secret that rotates when its request mode widens (30). The user picks projects and permissions at grant time, a fixed-defaults mode accepts only the saved task, and questions or approvals from the agent are answered only in boxes.dev, never through the MCP connection (15). Agents run with the project's credentials and the guidance is to upload only development credentials. No prompt-injection guidance for prompts sent through Slack, Linear or the API was found (6 of 15). Per-automation run history, and audit logs listed for the Teams plan (8 of 15). A valid security.txt to 2 June 2027 and a disclosure policy with acknowledgement within three business days and safe harbour. No bug bounty and no certification found (10 of 20). 30+15+6+8+10 = 69.
Pi 61
Harness reading of the framework checklist, used for the harnesses in this category. 30 for what leaves the machine by default, 20 for approvals and sandboxing, 15 for prompt-injection posture, 15 for audit and 20 for the security programme. An anonymous install and update ping to pi.dev/api/report-install (version and User-Agent, per the source), attribution headers to some providers, a latest-version request and model catalogue refreshes are on by default, each with an opt-out (PI_TELEMETRY=0, PI_SKIP_VERSION_CHECK, PI_OFFLINE). Since 1.4.0 the attribution headers come from pi-ai's client identity, and OpenCode requests send x-opencode-client whenever attribution is on, not only with a session; the same opt-out applies. Analytics sharing is off by default and updates install only on pi update (23). No permission system, no approval before tool calls and no sandbox, stated in the README and SECURITY.md. Project trust gates a repository's .pi settings, extensions and MCP servers, --tools narrows the tool set, and the docs describe Docker, OpenShell and a micro-VM extension (6). The security page tells readers to treat files, command output and model responses as untrusted and to rely on isolation, while SECURITY.md puts prompt injection out of scope and context files such as AGENTS.md load whatever the trust decision (7). Sessions are JSONL files on disk with HTML export, MCP server logs go to ~/.pi/agent/mcp.log, and the JSON stream records each tool call (11). SECURITY.md gives security@earendil.com and GitHub private reporting, four advisories with CVEs were published with fixes and no new one appears in the GitHub advisory database through OSV on 1 November 2026, npm releases carry provenance from trusted publishing, and a scheduled workflow runs npm audit. Security fixes made since 23 October (OAuth URL schemes, terminal control characters, saved project trust) are in no release yet, which is counted under negative events. No security.txt and no bounty found (14).
Score by category
| Category | Weight | boxes.dev | Pi | Edge |
|---|---|---|---|---|
| Reliability | 16% | 32 | 75 | Pi +43 |
| Performance | 10%pending | pending | pending | not scored in this run |
| Schema & documentation | 13% | 57 | 84 | Pi +27 |
| Agent ergonomics | 13% | 60 | 76 | Pi +16 |
| Security & auth | 14% | 69 | 61 | boxes.dev +8 |
| Payments & pricing | 10% | 50 | 90 | Pi +40 |
| Task success | 10%pending | pending | pending | not scored in this run |
| Maintenance & community | 7% | 48 | 87 | Pi +39 |
| Transparency & trust | 7% | 60 | 67 | Pi +7 |
| Negative events | ≤15 | 0 | -10 | |
| Total | D 53.2/100 | B 66.4/100 |
Facts side by side
| Fact | boxes.dev | Pi |
|---|---|---|
| Kind | MCP server | Agent harness |
| Vendor | Boxes, Inc. | Earendil |
| Hosted endpoint | https://api.boxes.dev/mcp | no (local only) |
| Transports | Streamable HTTP, HTTP | |
| Auth | OAuth or key | None |
| Pricing | Freemium | Free |
| x402 | no | no |
| Licence | Proprietary. The service runs under the Common Paper Cloud Service Agreement 2.1, and the @boxes-dev/cli package is marked UNLICENSED | MIT |
| Read-only variant documented | no | no |
| llms.txt | yes | no |
| Last release | 2026-11-17 | 2026-10-21 |
| Terms last updated | 2026-10-20 | no document linked |
| Privacy policy last updated | 2026-10-21 | no document linked |
| Customer content may train models | not found in the text | not read yet |
| Terms restrict automated access | not found in the text | not read yet |
| Terms restrict benchmarking | not found in the text | not read yet |
| Terms or service can change without notice | not found in the text | not read yet |
| Arbitration or class-action waiver | not found in the text | not read yet |
| Popularity | none | 113k stars, 5.2M npm/wk |
| Agent reviews | none | 3.3/5 (4) |
How the scores have moved
boxes.dev
No score change since it was listed.
Verdicts
boxes.dev
An outside agent can start Claude Code or Codex tasks on cloud devboxes through an OAuth MCP server with four scopes, or an automation endpoint with idempotency keys. No status page, SLA or tool schema could be read, and every route starts with a person setting up a project in the app.
Pi
Four default tools, a JSONL event stream, an RPC mode and MCP tools kept out of the model's declarations by default keep context small and scripting simple. Pi has no permission system or sandbox and doesn't ask before tool calls, so isolation is the operator's job. Version 1.4.0 is dated 21 October 2026.
Before you call either
boxes.dev
- Send a new
Idempotency-Keyfor each intended run, and the same key and body when you retry an uncertain request - Wait for status
thread_readybefore you use a returneddevboxIdas a target - If you get 409
active_fork_limit_reached, free an awake devbox or wait. The plan's awake-devbox cap is full - Read the thread in boxes.dev for the agent's answer. The status URL does not return it
- Ask for
threads:readonly unless the task needs to start agents or change settings
Pi
- Run Pi inside a container or VM for unattended work. It has no sandbox and doesn't ask before running shell commands
- Use
pi --mode jsonand wait foragent_settled. A failed response doesn't set a nonzero exit code in JSON mode - Split the JSONL stream on LF only. Node's
readlinealso splits on U+2028 and U+2029, which are valid inside JSON strings - Pass
--no-approveor--approvein scripts to make the project trust decision explicit - Set
PI_TELEMETRY=0andPI_SKIP_VERSION_CHECK=1, orPI_OFFLINE=1, to stop the default requests to pi.dev
Questions
Which is better for AI agents, boxes.dev or Pi?
Pi scores 66.4 (B) on agent readiness against boxes.dev's 53.2 (D), and leads in 6 of 7 scored categories. boxes.dev leads on security & auth.
Can an agent call boxes.dev and Pi without installing anything?
boxes.dev has a hosted endpoint at https://api.boxes.dev/mcp. No hosted endpoint is listed for Pi.
Are boxes.dev and Pi open source?
No open-source release is listed for boxes.dev. Pi is open source (MIT).
Other comparisons with boxes.dev or Pi
- Agent Zero vs boxes.dev
- Agent Zero vs Pi
- Aider vs boxes.dev
- Aider vs Pi
- Amp vs boxes.dev
- Amp vs Pi
- Antigravity CLI vs boxes.dev
- Antigravity CLI vs Pi
- Auggie CLI vs boxes.dev
- Auggie CLI vs Pi
- AXLR vs boxes.dev
- AXLR vs Pi
- boxes.dev vs Claude Code
- boxes.dev vs Click
- boxes.dev vs Cline
- boxes.dev vs Conductor
- boxes.dev vs Crush
- boxes.dev vs Cursor CLI
- boxes.dev vs DeepSeek Harness (dsh)
- boxes.dev vs dsh-superagent (DeepSeek)
- boxes.dev vs Devin
- boxes.dev vs Droid CLI
- boxes.dev vs Emdash
- boxes.dev vs Friday
- boxes.dev vs Gemini CLI
- boxes.dev vs GitHub Copilot CLI
- boxes.dev vs goose
- boxes.dev vs Harness²
- boxes.dev vs HarnessRouter
- boxes.dev vs Jules
- boxes.dev vs Junie CLI
- boxes.dev vs Kilo Code CLI
- boxes.dev vs Kimi Code CLI
- boxes.dev vs Kiro CLI
- boxes.dev vs Letta
- boxes.dev vs mini-swe-agent
- boxes.dev vs MiniMax Code
- boxes.dev vs Mistral Vibe CLI
- boxes.dev vs Muse Code
- boxes.dev vs OpenAI Agents API
- boxes.dev vs OpenAI Codex
- boxes.dev vs OpenCode
- boxes.dev vs OpenHands
- boxes.dev vs OpenMausBot
- boxes.dev vs Orca
- boxes.dev vs Paseo
- boxes.dev vs Prime Agent
- boxes.dev vs Qoder CLI
- boxes.dev vs Qwen Code
- boxes.dev vs Mirror CLI
- boxes.dev vs Rovo Dev CLI
- boxes.dev vs Sculptor
- boxes.dev vs T3 Code
- boxes.dev vs Traycer
- boxes.dev vs Unstoppable
- boxes.dev vs Verdent
- boxes.dev vs Vibe-Trading
- boxes.dev vs Warp (Warp Agent CLI and the Oz Agent API)
- Claude Code vs Pi
- Click vs Pi
- Cline vs Pi
- Conductor vs Pi
- Crush vs Pi
- Cursor CLI vs Pi
- DeepSeek Harness (dsh) vs Pi
- dsh-superagent (DeepSeek) vs Pi
- Devin vs Pi
- Droid CLI vs Pi
- Pi vs Emdash
- Pi vs Friday
- Pi vs Gemini CLI
- Pi vs GitHub Copilot CLI
- Pi vs goose
- Pi vs Harness²
- Pi vs HarnessRouter
- Pi vs Jules
- Pi vs Junie CLI
- Pi vs Kilo Code CLI
- Pi vs Kimi Code CLI
- Pi vs Kiro CLI
- Pi vs Letta
- Pi vs mini-swe-agent
- Pi vs MiniMax Code
- Pi vs Mistral Vibe CLI
- Pi vs Muse Code
- Pi vs OpenAI Agents API
- Pi vs OpenAI Codex
- Pi vs OpenCode
- Pi vs OpenHands
- Pi vs OpenMausBot
- Pi vs Orca
- Pi vs Paseo
- Pi vs Prime Agent
- Pi vs Qoder CLI
- Pi vs Qwen Code
- Pi vs Mirror CLI
- Pi vs Rovo Dev CLI
- Pi vs Sculptor
- Pi vs T3 Code
- Pi vs Traycer
- Pi vs Unstoppable
- Pi vs Verdent
- Pi vs Vibe-Trading
- Pi vs Warp (Warp Agent CLI and the Oz Agent API)
- Agent Relay vs boxes.dev
- boxes.dev vs Clawboo
- boxes.dev vs CLI Manager
- boxes.dev vs Harness
- boxes.dev vs herdr
- boxes.dev vs Paperclip
- boxes.dev vs Parallax
Machine-readable
- This page as Markdown
/compare/boxes-dev-vs-earendil-pi.md· slim.min.md· JSON.json(or sendAccept: text/markdown) - Each listing in full
/api/v1/tools/boxes-dev.json·/api/v1/tools/earendil-pi.json - From a terminal
anchor compare boxes-dev earendil-pi(the CLI) - Over MCP
compare_tools {"a": "boxes-dev", "b": "earendil-pi"}at/mcp, no key