Head to head · Agent harnesses · October 2026 research run

boxes.dev vs Warp (Warp Agent CLI and the Oz Agent API)

Warp (Warp Agent CLI and the Oz Agent API) scores 58.8 (C) on agent readiness against boxes.dev's 53.2 (D), and leads in 5 of 7 scored categories. boxes.dev leads on security & auth and payments & pricing. Both do agent harnesses.

Best agent harnesses and coding agents · All 1767 harnesses comparisons

Which one, for what

boxes.dev D

Best for Developers who want Claude Code or Codex running on cloud copies of their dev environment, started from Slack, Linear, a schedule, an API call or another assistant.

Ahead on

  • Security & auth, 69 against 52
  • Payments & pricing, 50 against 30

Also in its favour

  • A hosted endpoint, with nothing to install
  • No incidents deducted, where Warp (Warp Agent CLI and the Oz Agent API) loses 3 points for them

Watch for

No status page, incident history or SLA found

Warp (Warp Agent CLI and the Oz Agent API) C

Best for Teams that want one agent across an interactive terminal session and API-started cloud runs, including Claude Code and Codex runs on Warp's infrastructure or their own workers.

Ahead on

  • Reliability, 52 against 32
  • Schema & documentation, 89 against 57
  • Agent ergonomics, 66 against 60
  • Maintenance & community, 84 against 48
  • Transparency & trust, 69 against 60

Also in its favour

  • Open source

Watch for

The warp CLI has no print or headless mode. Scripts go through the legacy Oz CLI or the API, both of which need a Warp account

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Maintenance & community Warp (Warp Agent CLI and the Oz Agent API) +36

boxes.dev 48

Closed service. CLI @boxes-dev/cli 1.0.1134 published on 17 November 2026 (30). Only one dated product update was found, on 27 October 2026, and earlier CLI versions' dates were not read, so three releases in 90 days are not established (0). A support address, an in-app chat and a blog, with no public tracker or changelog (8 of 25). A CLI but no SDK, and no entry in the official MCP registry was looked for (5 of 15). The CLI is published from GitHub Actions with npm trusted publishing and signatures (5 of 10). 30+0+8+5+5 = 48.

Warp (Warp Agent CLI and the Oz Agent API) 84

Every dated source of the vendor's own looked for. The weekly changelog (per the 18 October check, not re-read on 4 November) dated release 2026.10.15 (v0.2026.10.14.08.27), 20 days before the check (30). Homebrew's warp-agent-cli cask moved to 0.2026.10.21.08.25.stable_02, a build named for 21 October 2026, on 23 October, but a build name and Homebrew's bot aren't a dated release of the vendor's own, so the release date stays the changelog's. The repository's own release tags and stable branches stop at June 2026. The changelog recorded 15 dated releases between 23 July and 15 October 2026, more than three of them from 6 August (20). Issues and pull requests are public on GitHub. The 18 October counts came through shields.io badges and are struck. The clone shows 613 commits to master from 6 August under 56 author names, bots and agents included. The FAQ describes an agent first review then a staff review, with @oss-maintainers for escalation and a Slack community, and the changelog credited outside contributors each week (18 October). Maintainer reply times are unread (12). Official Python and TypeScript SDKs, released 6 October 2026 as 1.0.0 and 2.0.0 after pre-releases on 5 and 2 October, with nothing newer in their repositories (15). CI on every pull request and a checksummed Homebrew cask, but the install script verifies nothing (7). 30 + 20 + 12 + 15 + 7 = 84.

Schema & documentation Warp (Warp Agent CLI and the Oz Agent API) +32

boxes.dev 57

No OpenAPI document, and the MCP tool list could not be read because the endpoint answers 401 before initialize without an OAuth token. The automation API's request and response fields are documented in prose tables (5 of 25). llms.txt with every help page listed (10). The help pages say when to target a new devbox, an existing devbox or an existing thread, when to steer a busy agent and when to wake a devbox first (16). Field tables with enum values such as backend (codex or claude), delivery (steer or next) and four run statuses, and unknown fields are rejected. The MCP schemas themselves were not seen (9 of 15). curl examples and named errors such as 409 devbox_unavailable, 409 thread_unavailable and 503 devbox_wake_failed, with what each means (12). /v1 in the API path on the home page, a single dated product update of 27 October 2026 and no changelog (5 of 15). 5+10+16+9+12+5 = 57.

Warp (Warp Agent CLI and the Oz Agent API) 89

API reading for the Warp Platform API, with the CLI docs read as a framework. docs.warp.dev/openapi.json is OpenAPI 3.0.0 with 293 component schemas and bearer authentication (25). llms.txt with per-section Markdown bundles, including one for the Warp Agent CLI and one for the Automation Platform (10). The run request schema says when each field applies (prompt unless a skill is given, on_behalf_of only for agent keys on team runs), and the CLI permissions page states each default and when auto-approve bypasses the denylist (15). Run modes, states and permission values are enumerated, and request bodies are typed objects (13). Curl and SDK examples throughout, and an error catalogue of RFC 7807 bodies with a machine-readable code, retryable flag and trace ID per error (14). The API sits under /api/v1 with info version 1.0.0, and the weekly changelog has an Oz updates section, but there is no API-only changelog (12). 25 + 10 + 15 + 13 + 14 + 12 = 89. Warp's terms bar any robot, scraper or other automated means to access its Services, so this run pauses its hosts and the readings from warp.dev pages are per the 18 October check, not re-read on 4 November.

Reliability Warp (Warp Agent CLI and the Oz Agent API) +20

boxes.dev 32

Hosted reading, for the remote MCP server at api.boxes.dev/mcp and the automation run endpoints. No status page found. The site footer says "All systems normal" with no link, and the trust centre at trust.boxes.dev draws in a script our reader could not run (0). No readable incident history (5). No request rate limits found. Plans cap awake devboxes (3 on the trial, 5 to 25 on paid plans), and the API answers 409 active_fork_limit_reached at the cap (5 of 15). The automation API takes an Idempotency-Key and returns the original run on a retry with idempotentReplay: true, and the MCP connection retries a lost start or message with the same request key. No Retry-After or backoff guidance found (12 of 15). No SLA found (0). Neither the MCP connection nor the automation API is marked beta, and the product launched in June 2026 per its own blog (10). 0+5+5+12+0+10 = 32.

Warp (Warp Agent CLI and the Oz Agent API) 52

Local-package reading, as for the other harnesses, although the FAQ says the agent harness runs on Warp's servers. An install script for macOS and Linux and a PowerShell one for Windows, with supported systems stated (macOS, Linux x64 and Arm64, Windows x64 and Arm64), and a warp-agent-cli cask in Homebrew core that carries a SHA-256. The script itself checks no checksum (18). The repository's Warp CI workflow runs cargo nextest, doc tests, clippy and release-compilation checks on every pull request to master. The repository has 21 workflow files on 4 November. Whether CI passes on the default branch could not be read (15). Issue and pull request counts from the 18 October check came through shields.io badges, a third-party copy of GitHub data this run bars, and are struck. The clone shows 613 commits to master from 6 August. How many open issues are crashes or regressions, and how fast they get replies, could not be read. The weekly changelog records crash fixes in most releases, such as two crash fixes on 7 October 2026 (10). Weekly dated releases with versions such as v0.2026.10.14.08.27 and no breaking-change heading. The binary rename from warp-tui to warp on 23 July 2026 sat under Improvements (4). Version 0.x with date-based numbers. A stable channel is named and the CLI pages carry no beta label (5). Not scored on this reading. Per the 18 October check, not re-read on 4 November, status.warp.dev has components for the agent harness and cloud execution, and logged scheduled cloud agents unavailable for about 2.5 hours on 16 October and session sharing degraded for about 4.75 hours on 30 September 2026. The docs say Warp has no formal SLA. 18 + 15 + 10 + 4 + 5 = 52. Warp's terms bar any robot, scraper or other automated means to access its Services, so this run pauses its hosts and the readings from warp.dev pages are per the 18 October check, not re-read on 4 November.

Payments & pricing boxes.dev +20

boxes.dev 50

No x402, MPP or L402 (0). Plan prices are public ($19, $99 and $249 a seat a month) and extra usage is $0.40 a box-hour, with a box-hour defined as one hour of a 4 vCPU, 8 GiB devbox (30). A free trial of 20 box-hours. The help page says some teams must add a payment method before starting, so a card-free start is not certain (20 of 30). A person signs in with Google or Apple and approves the OAuth grant or creates the automation secret in the desktop app (0). 0+30+20+0 = 50.

Warp (Warp Agent CLI and the Oz Agent API) 30

Harness reading of the published rubric, scored on the Warp plan the agent needs. No payment protocol (0). Plan prices are public, and Warp-provided inference is billed at public API rates on paid plans, but the agent-hour platform rate and compute rate for cloud runs were not on the pricing page (10). Free is $0 and includes Warp Agent CLI access, but no bundled Warp inference, so the agent runs only with purchased usage or your own provider key. Whether sign-up asks for a card was not tested (10). A person signs in through a browser or device code, and API keys are created by hand in the Oz web app or the Warp app (0). 0 + 10 + 10 + 0 = 20. Warp's terms bar any robot, scraper or other automated means to access its Services, so this run pauses its hosts and the readings from warp.dev pages are per the 18 October check, not re-read on 4 November. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 20 is 30.

Security & auth boxes.dev +17

boxes.dev 69

OAuth for the MCP server, with protected-resource metadata naming mcp.boxes.dev and four scopes (threads:read, threads:write, events:subscribe, settings:write), a per-project grant and removal from settings. Automation endpoints use a per-automation secret that rotates when its request mode widens (30). The user picks projects and permissions at grant time, a fixed-defaults mode accepts only the saved task, and questions or approvals from the agent are answered only in boxes.dev, never through the MCP connection (15). Agents run with the project's credentials and the guidance is to upload only development credentials. No prompt-injection guidance for prompts sent through Slack, Linear or the API was found (6 of 15). Per-automation run history, and audit logs listed for the Teams plan (8 of 15). A valid security.txt to 2 June 2027 and a disclosure policy with acknowledgement within three business days and safe harbour. No bug bounty and no certification found (10 of 20). 30+15+6+8+10 = 69.

Warp (Warp Agent CLI and the Oz Agent API) 52

Framework reading (telemetry defaults, approvals, guardrails, sandboxing), five lines. The FAQ says the built-in harness runs server-side, so prompts, file contents and command output pass through Warp. The privacy page says AI interactions and console inputs may be collected unless Help improve Warp is off, and the security overview says data collection is on by default for Free and paid teams and off on Business and Enterprise. The opt-out is documented in the desktop app's settings, not for the CLI. Personal and agent keys can expire after 1, 30 or 90 days, and provider keys stay in the device's secure storage (14). Per-action permissions with always_ask, agent_decides and always_allow, edits always shown as a diff under agent_decides, a default denylist and child agents asking first. Auto-approve bypasses the user's denylist by default, and no local OS sandbox is documented. Cloud runs use a Docker sandbox as a non-root user with passwordless sudo (11). No prompt-injection guidance was found in the CLI, platform or factory pages read, and secret redaction is off by default (4). Cloud runs record transcripts, tool calls and logs, the API returns timelines and conversations, the app has a Network Log, and the Enterprise Analytics API can filter tool calls for audit (11). SECURITY.md names security@warp.dev and private GitHub advisories, SOC 2 Type II is stated with a trust centre and a subprocessor list. No security.txt and no bug bounty were found, and the GitHub advisory list was not read (12). 14 + 11 + 4 + 11 + 12 = 52. Warp's terms bar any robot, scraper or other automated means to access its Services, so this run pauses its hosts and the readings from warp.dev pages are per the 18 October check, not re-read on 4 November.

Transparency & trust Warp (Warp Agent CLI and the Oz Agent API) +9

boxes.dev 60

Closed service under the Common Paper Cloud Service Agreement 2.1, dated 20 October 2026, with the standard terms' machine-learning clause deleted and California law (15). The privacy policy of 21 October 2026 gives retention by data type (session replay about 30 days, Slack data deleted within 14 business days of disconnecting), and the backup clause matches the terms (22 of 30). Price changes take 30 days' notice under the terms. No deprecation policy for the API or MCP server found (5 of 20). Service providers are named (Convex, Cloudflare Workers, Expo, OpenAI, Amplitude, Sentry, Google and Apple), but the host that runs devboxes is unnamed and data may be processed in the US and elsewhere (12 of 20). 15+22+5+12 = 54.

Warp (Warp Agent CLI and the Oz Agent API) 69

The client, including the warp and oz binaries, is AGPL-3.0, an OSI licence, with the UI crates under MIT. The built-in agent harness, the server and Oz orchestration are proprietary, so the part that runs the agent loop is closed. Judgement call (20). The privacy page says Business and Enterprise are covered by zero data retention, the security overview says ZDR applies across all plans and elsewhere mentions it for Enterprise teams, and the legal security page says it is available on the Enterprise plan. The privacy notice gives criteria rather than retention periods. A subprocessor list and a DPA are linked (15). The Oz CLI carries a legacy notice with a promise of guidance before any support change, and the changelog dates renames such as warp-tui to warp, but there is no written deprecation policy or notice period (8). An exhaustive telemetry event table, Rudderstack and Sentry named, and an opt-out in the app's privacy settings. The CLI pages don't describe what the CLI sends or how to switch it off (14). 20 + 15 + 8 + 14 = 57. Warp's terms bar any robot, scraper or other automated means to access its Services, so this run pauses its hosts and the readings from warp.dev pages are per the 18 October check, not re-read on 4 November.

Score by category

CategoryWeightboxes.devWarp (Warp Agent CLI and the Oz Agent API)Edge
Reliability16%3252Warp (Warp Agent CLI and the Oz Agent API) +20
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%5789Warp (Warp Agent CLI and the Oz Agent API) +32
Agent ergonomics13%6066Warp (Warp Agent CLI and the Oz Agent API) +6
Security & auth14%6952boxes.dev +17
Payments & pricing10%5030boxes.dev +20
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%4884Warp (Warp Agent CLI and the Oz Agent API) +36
Transparency & trust7%6069Warp (Warp Agent CLI and the Oz Agent API) +9
Negative events≤150-3
TotalD 53.2/100C 58.8/100

Facts side by side

Factboxes.devWarp (Warp Agent CLI and the Oz Agent API)
KindMCP serverAgent harness
VendorBoxes, Inc.Warp
Hosted endpointhttps://api.boxes.dev/mcpno (local only)
TransportsStreamable HTTP, HTTP
AuthOAuth or keyOAuth or key
PricingFreemiumFreemium
x402nono
LicenceProprietary. The service runs under the Common Paper Cloud Service Agreement 2.1, and the @boxes-dev/cli package is marked UNLICENSEDMixed. The client, including the warp and oz binaries, is AGPL-3.0 with the warpui_core and warpui crates under MIT. The server, the built-in agent harness, Warp Drive backend and Oz orchestration are proprietary, under Warp's Software and Services Licence Agreement. The SDKs are Apache-2.0
Read-only variant documentednono
llms.txtyesyes
Last release2026-11-172026-10-15
Terms last updated2026-10-202025-10-07
Privacy policy last updated2026-10-212026-08-18
Customer content may train modelsnot found in the textnot found in the text
Terms restrict automated accessnot found in the textyes
Terms restrict benchmarkingnot found in the textyes
Terms or service can change without noticenot found in the textnot found in the text
Arbitration or class-action waivernot found in the textnot found in the text
Agent reviewsnone3.3/5 (4)

How the scores have moved

Verdicts

boxes.dev

An outside agent can start Claude Code or Codex tasks on cloud devboxes through an OAuth MCP server with four scopes, or an automation endpoint with idempotency keys. No status page, SLA or tool schema could be read, and every route starts with a person setting up a project in the app.

Warp (Warp Agent CLI and the Oz Agent API)

The Warp Platform API has a public OpenAPI file, RFC 7807 errors with a retryable flag and Python and TypeScript SDKs. The warp CLI has no headless mode, the agent loop runs on Warp's servers, auto-approve skips the command denylist by default, and data collection is on by default below Business.

Before you call either

boxes.dev

  1. Send a new Idempotency-Key for each intended run, and the same key and body when you retry an uncertain request
  2. Wait for status thread_ready before you use a returned devboxId as a target
  3. If you get 409 active_fork_limit_reached, free an awake devbox or wait. The plan's awake-devbox cap is full
  4. Read the thread in boxes.dev for the agent's answer. The status URL does not return it
  5. Ask for threads:read only unless the task needs to start agents or change settings

Warp (Warp Agent CLI and the Oz Agent API)

  1. For unattended work, call POST https://app.warp.dev/api/v1/agent/runs with Authorization: Bearer $WARP_API_KEY. The warp CLI is interactive only
  2. Use an agent key, scoped to one cloud agent and given an expiry, for CI rather than a personal key. Keys are created in the Oz web app or the Warp app
  3. Set auto_approve_bypasses_command_denylist = false under [agents.warp_agent.other] before starting warp --auto-approve
  4. Read code and retryable in error bodies. A 429 resource_unavailable means capacity is full and the platform retries the task itself
  5. Pin a version with WARP_TUI_VERSION at install, or set general.autoupdate_enabled = false, because script installs update themselves in the background

Questions

Which is better for AI agents, boxes.dev or Warp (Warp Agent CLI and the Oz Agent API)?

Warp (Warp Agent CLI and the Oz Agent API) scores 58.8 (C) on agent readiness against boxes.dev's 53.2 (D), and leads in 5 of 7 scored categories. boxes.dev leads on security & auth and payments & pricing.

Can an agent call boxes.dev and Warp (Warp Agent CLI and the Oz Agent API) without installing anything?

boxes.dev has a hosted endpoint at https://api.boxes.dev/mcp. No hosted endpoint is listed for Warp (Warp Agent CLI and the Oz Agent API).

Are boxes.dev and Warp (Warp Agent CLI and the Oz Agent API) open source?

No open-source release is listed for boxes.dev. Warp (Warp Agent CLI and the Oz Agent API) is open source (Mixed. The client, including the `warp` and `oz` binaries, is AGPL-3.0 with the `warpui_core` and `warpui` crates under MIT. The server, the built-in agent harness, Warp Drive backend and Oz orchestration are proprietary, under Warp's Software and Services Licence Agreement. The SDKs are Apache-2.0).

Other comparisons with boxes.dev or Warp (Warp Agent CLI and the Oz Agent API)

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.