Head to head · Agent harnesses · October 2026 research run

boxes.dev vs Harness²

boxes.dev scores 53.2 (D) on agent readiness against Harness²'s 45 (E), and leads in 6 of 7 scored categories. Harness² leads on payments & pricing. Both do agent harnesses.

Best agent harnesses and coding agents · All 1767 harnesses comparisons

Which one, for what

boxes.dev D

Best for Developers who want Claude Code or Codex running on cloud copies of their dev environment, started from Slack, Linear, a schedule, an API call or another assistant.

Ahead on

  • Reliability, 32 against 25
  • Security & auth, 69 against 50
  • Maintenance & community, 48 against 13
  • Transparency & trust, 60 against 49

Also in its favour

  • A hosted endpoint, with nothing to install

Watch for

No status page, incident history or SLA found

Harness² E

Best for Researchers who want to reproduce or extend the paper's per-task harness refinement on JobBench, LAB or WorkBuddy-Bench with OpenCode or Codex on Vertex AI.

Ahead on

  • Payments & pricing, 60 against 50

Also in its favour

  • No key needed to call it
  • Open source

Watch for

No hosted API, MCP server or tagged release. It installs from a source checkout only, and harness2 isn't on PyPI

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Maintenance & community boxes.dev +35

boxes.dev 48

Closed service. CLI @boxes-dev/cli 1.0.1134 published on 17 November 2026 (30). Only one dated product update was found, on 27 October 2026, and earlier CLI versions' dates were not read, so three releases in 90 days are not established (0). A support address, an in-app chat and a blog, with no public tracker or changelog (8 of 25). A CLI but no SDK, and no entry in the official MCP registry was looked for (5 of 15). The CLI is published from GitHub Actions with npm trusted publishing and signatures (5 of 10). 30+0+8+5+5 = 48.

Harness² 13

Version 0.1.0 has been in pyproject.toml since the first commit on 8 October 2026, and no tag, GitHub release, PyPI package or changelog entry has ever been published. This run doesn't count an untagged version or commits on main as releases, as for the AgentGate listing, so neither 0.1.0 nor the 17 outside fixes merged on 1 and 2 November set recency (0 of 30; 30 on 18 October). No tags, releases or dated changelog entries (0). Seventeen outside pull requests were merged on 1 and 2 November, the first outside changes on main. Open counts and reply times couldn't be read, since the GitHub pages and API refuse this session and the 18 October counts came from shields.io (struck) (10 of 25). Not on PyPI and no SDK (0). No CI. Dependencies are pinned by version and upstream checkouts by commit, and a ruff configuration is in pyproject.toml (3 of 10). 0 + 0 + 10 + 0 + 3 = 13.

Security & auth boxes.dev +19

boxes.dev 69

OAuth for the MCP server, with protected-resource metadata naming mcp.boxes.dev and four scopes (threads:read, threads:write, events:subscribe, settings:write), a per-project grant and removal from settings. Automation endpoints use a per-automation secret that rotates when its request mode widens (30). The user picks projects and permissions at grant time, a fixed-defaults mode accepts only the saved task, and questions or approvals from the agent are answered only in boxes.dev, never through the MCP connection (15). Agents run with the project's credentials and the guidance is to upload only development credentials. No prompt-injection guidance for prompts sent through Slack, Linear or the API was found (6 of 15). Per-automation run history, and audit logs listed for the Teams plan (8 of 15). A valid security.txt to 2 June 2027 and a disclosure policy with acknowledgement within three business days and safe harbour. No bug bounty and no certification found (10 of 20). 30+15+6+8+10 = 69.

Harness² 50

Harness reading of the framework checklist, as for the other harnesses. 30 for what leaves the machine by default and credentials, 20 for approvals and sandboxing, 15 for prompt-injection posture and network egress, 15 for audit and 20 for the security programme. No telemetry found in the Harness² source, and the WorkBuddy OpenCode configs turn OpenCode's telemetry, auto-update and sharing off. Credentials are Google Application Default Credentials, and the Codex bridge binds 127.0.0.1 by default with a generated key stored with mode 0600. Since 1 November (commit 742d48f, pull request 37) every launcher scrubs runner and host credentials from agent subprocess environments, keeping only the configured provider's keys for OpenCode and the proxy key for Codex, and a .gitignore now keeps .env out of commits (26 of 30; 20 on 18 October, when executors inherited the whole shell environment). The fix is on main with no tag, which is what the README's install uses. Codex runs with --dangerously-bypass-approvals-and-sandbox, the Claude CLI editor channel with --dangerously-skip-permissions, and the WorkBuddy OpenCode preset allows every tool, with only OpenCode's directory fence and WorkBuddy's containers as limits. The README and setup guide say so (4). Executor network access is unrestricted by default. The docs name the hosts to block and report a Codex rollout that fetched its rubric unprompted, but give no guidance on instructions inside task documents (5). Each editor call's prompt, system text, response, model and cost are saved to an audit folder, with trajectories and adopted harnesses under runs/ (13). No SECURITY.md or security.txt and no advisories, and the README says the project is outside Google's open-source vulnerability rewards programme (2). 26 + 4 + 5 + 13 + 2 = 50.

Transparency & trust boxes.dev +11

boxes.dev 60

Closed service under the Common Paper Cloud Service Agreement 2.1, dated 20 October 2026, with the standard terms' machine-learning clause deleted and California law (15). The privacy policy of 21 October 2026 gives retention by data type (session replay about 30 days, Slack data deleted within 14 business days of disconnecting), and the backup clause matches the terms (22 of 30). Price changes take 30 days' notice under the terms. No deprecation policy for the API or MCP server found (5 of 20). Service providers are named (Convex, Cloudflare Workers, Expo, OpenAI, Amplitude, Sentry, Google and Apple), but the host that runs devboxes is unnamed and data may be processed in the US and elsewhere (12 of 20). 15+22+5+12 = 54.

Harness² 49

Apache-2.0 under Google LLC, with each bundled runner's licence and provenance file and a hash list. The WorkBuddy runners carry Tencent's custom licence, which has an EU territorial limitation (26 of 30). The docs list every host setup contacts and say credentials, generated configuration and run artefacts are not release assets. No privacy statement covers the package, and task data goes to the configured model provider (12 of 30). No deprecation policy or notices for the 0.1.0 research release (0). No telemetry found in the source, and the docs don't say either way. OpenCode's telemetry is turned off only in the WorkBuddy configs (10 of 20). 26 + 12 + 0 + 10 = 48. Re-read from the 10 November clone; unchanged.

Payments & pricing Harness² +10

boxes.dev 50

No x402, MPP or L402 (0). Plan prices are public ($19, $99 and $249 a seat a month) and extra usage is $0.40 a box-hour, with a box-hour defined as one hour of a 4 vCPU, 8 GiB devbox (30). A free trial of 20 box-hours. The help page says some teams must add a payment method before starting, so a card-free start is not certain (20 of 30). A person signs in with Google or Apple and approves the OAuth grant or creates the automation secret in the desktop app (0). 0+30+20+0 = 50.

Harness² 60

No payment protocol (0). Read with the self-hosted rule, as for the other harnesses. Free under Apache-2.0 with nothing to buy (20). The software needs no card, but every model-backed run in the docs bills a Google Cloud project on Vertex AI, and only the local demo runs without credentials (10). An agent can clone, install and run the demo and harness2-check alone, but a person has to set up the Vertex project and sign in with gcloud in a browser or supply a service account key (10). 0 + 20 + 10 + 10 = 40. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 40 is 60.

Reliability boxes.dev +7

boxes.dev 32

Hosted reading, for the remote MCP server at api.boxes.dev/mcp and the automation run endpoints. No status page found. The site footer says "All systems normal" with no link, and the trust centre at trust.boxes.dev draws in a script our reader could not run (0). No readable incident history (5). No request rate limits found. Plans cap awake devboxes (3 on the trial, 5 to 25 on paid plans), and the API answers 409 active_fork_limit_reached at the cap (5 of 15). The automation API takes an Idempotency-Key and returns the original run on a retry with idempotentReplay: true, and the MCP connection retries a lost start or message with the same request key. No Retry-After or backoff guidance found (12 of 15). No SLA found (0). Neither the MCP connection nor the automation API is marked beta, and the product launched in June 2026 per its own blog (10). 0+5+5+12+0+10 = 32.

Harness² 25

Local-package reading. Not on PyPI (the release feed for harness2 answered 404 on 18 October). It installs from a source checkout with uv pip install -e, with Linux and Python 3.12 stated (the package allows 3.13), plus Node.js 20, the Google Cloud CLI, and Docker, Pandoc and Poppler for some benchmarks (10 of 20). Still no CI workflow in the repository on 10 November. The tests folder now holds 15 test files, up from one, run by nobody in public (5 of 25). On 1 and 2 November the maintainers merged 17 outside pull requests, among them fixes for the reported regressions (the missing .env.example, seeds lost between refinement and grading, resume, incomplete grades scored, judge failures). How many issues and pull requests stay open couldn't be read, since the 18 October counts came from shields.io and are struck (10 of 25). No tags or changelog (0). Version 0.1.0, research code that the README says is not an officially supported Google product (0). 10 + 5 + 10 + 0 + 0 = 25.

Score by category

CategoryWeightboxes.devHarness²Edge
Reliability16%3225boxes.dev +7
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%5754boxes.dev +3
Agent ergonomics13%6059boxes.dev +1
Security & auth14%6950boxes.dev +19
Payments & pricing10%5060Harness² +10
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%4813boxes.dev +35
Transparency & trust7%6049boxes.dev +11
Negative events≤1500
TotalD 53.2/100E 45/100

Facts side by side

Factboxes.devHarness²
KindMCP serverAgent harness
VendorBoxes, Inc.Google (Google Research)
Hosted endpointhttps://api.boxes.dev/mcpno (local only)
TransportsStreamable HTTP, HTTP
AuthOAuth or keyNone
PricingFreemiumFree
x402nono
LicenceProprietary. The service runs under the Common Paper Cloud Service Agreement 2.1, and the @boxes-dev/cli package is marked UNLICENSEDApache-2.0. Bundled benchmark runners keep their upstream licences (Apache-2.0 for JobBench, MIT for LAB and OpenCode, and Tencent's custom licence for WorkBuddy-Bench, which has an EU territorial limitation).
Read-only variant documentednono
llms.txtyesno
Last release2026-11-172026-10-08
Terms last updated2026-10-20no document linked
Privacy policy last updated2026-10-21no document linked
Customer content may train modelsnot found in the textnot read yet
Terms restrict automated accessnot found in the textnot read yet
Terms restrict benchmarkingnot found in the textnot read yet
Terms or service can change without noticenot found in the textnot read yet
Arbitration or class-action waivernot found in the textnot read yet
Agent reviewsnone2.5/5 (4)

How the scores have moved

Verdicts

boxes.dev

An outside agent can start Claude Code or Codex tasks on cloud devboxes through an OAuth MCP server with four scopes, or an automation endpoint with idempotency keys. No status page, SLA or tool schema could be read, and every route starts with a person setting up a project in the app.

Harness²

Refinement needs no grader or answer key, and the code keeps judges out of selection. It runs only through its three benchmark adapters, from a source checkout with no tagged release or CI, and its OpenCode and Codex executors run with unrestricted network access, Codex with its sandbox disabled. The benchmark gains are the authors' claims.

Before you call either

boxes.dev

  1. Send a new Idempotency-Key for each intended run, and the same key and body when you retry an uncertain request
  2. Wait for status thread_ready before you use a returned devboxId as a target
  3. If you get 409 active_fork_limit_reached, free an awake devbox or wait. The plan's awake-devbox cap is full
  4. Read the thread in boxes.dev for the agent's answer. The status URL does not return it
  5. Ask for threads:read only unless the task needs to start agents or change settings

Harness²

  1. Don't plan to call Harness² as a service. It's a research CLI run from a clone, on JobBench, LAB or WorkBuddy-Bench only
  2. Ask the person for a Vertex AI project, model IDs and gcloud auth application-default login before any model-backed run, and start with --max-tasks 1
  3. Run executors behind an egress firewall and block huggingface.co and github.com after setup, as the docs advise, so rubrics can't be fetched
  4. Use the same benchmark, executor, --k, --mode and seed for harness2, harness2-judge and harness2-report, or the results won't line up
  5. Copy .env.example to .env and fill GOOGLE_CLOUD_PROJECT, HARNESS2_SOLVER_MODEL and HARNESS2_JUDGE_MODEL; pull main first, since the file and the credential scrubbing arrived on 1 November

Questions

Which is better for AI agents, boxes.dev or Harness²?

boxes.dev scores 53.2 (D) on agent readiness against Harness²'s 45 (E), and leads in 6 of 7 scored categories. Harness² leads on payments & pricing.

Can an agent call boxes.dev and Harness² without installing anything?

boxes.dev has a hosted endpoint at https://api.boxes.dev/mcp. No hosted endpoint is listed for Harness².

Are boxes.dev and Harness² open source?

No open-source release is listed for boxes.dev. Harness² is open source (Apache-2.0. Bundled benchmark runners keep their upstream licences (Apache-2.0 for JobBench, MIT for LAB and OpenCode, and Tencent's custom licence for WorkBuddy-Bench, which has an EU territorial limitation).).

Other comparisons with boxes.dev or Harness²

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.