Head to head · Agent harnesses · October 2026 research run

boxes.dev vs Qwen Code

Qwen Code scores 72.4 (BB) on agent readiness against boxes.dev's 53.2 (D), and leads in 6 of 7 scored categories. boxes.dev leads on security & auth. Both do agent harnesses.

Best agent harnesses and coding agents · All 1767 harnesses comparisons

Which one, for what

boxes.dev D

Best for Developers who want Claude Code or Codex running on cloud copies of their dev environment, started from Slack, Linear, a schedule, an API call or another assistant.

Ahead on

  • Security & auth, 69 against 53

Also in its favour

  • A hosted endpoint, with nothing to install
  • No incidents deducted, where Qwen Code loses 4 points for them

Watch for

No status page, incident history or SLA found

Qwen Code BB

Best for Developers and CI jobs that want an open-source harness tied to no single model vendor, with run budgets and SDKs.

Ahead on

  • Reliability, 69 against 32
  • Schema & documentation, 91 against 57
  • Agent ergonomics, 85 against 60
  • Payments & pricing, 90 against 50
  • Maintenance & community, 88 against 48
  • Transparency & trust, 66 against 60

Also in its favour

  • Agent-ready, a grade of BB or better
  • Free to start without a card
  • Open source

Watch for

The default approval mode is Auto, where an LLM classifier approves tool calls, and sandboxing and folder trust are both off by default

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Payments & pricing Qwen Code +40

boxes.dev 50

No x402, MPP or L402 (0). Plan prices are public ($19, $99 and $249 a seat a month) and extra usage is $0.40 a box-hour, with a box-hour defined as one hour of a 4 vCPU, 8 GiB devbox (30). A free trial of 20 box-hours. The help page says some teams must add a payment method before starting, so a card-free start is not certain (20 of 30). A person signs in with Google or Apple and approves the OAuth grant or creates the automation secret in the desktop app (0). 0+30+20+0 = 50.

Qwen Code 90

Self-hosted rule. No payment protocol (0). The CLI is free under Apache-2.0 with nothing to buy (20). No card or account is needed for the software, and it runs against a local model server (20). An agent can install and run it with no sign-up, given a model endpoint (20). Alibaba Cloud's Coding Plan and Token Plan are separate purchases, and their price pages couldn't be loaded on 8 October 2026. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 60 is 90.

Maintenance & community Qwen Code +40

boxes.dev 48

Closed service. CLI @boxes-dev/cli 1.0.1134 published on 17 November 2026 (30). Only one dated product update was found, on 27 October 2026, and earlier CLI versions' dates were not read, so three releases in 90 days are not established (0). A support address, an in-app chat and a blog, with no public tracker or changelog (8 of 25). A CLI but no SDK, and no entry in the official MCP registry was looked for (5 of 15). The CLI is published from GitHub Actions with npm trusted publishing and signatures (5 of 10). 30+0+8+5+5 = 48.

Qwen Code 88

0.26.5 on 1 November 2026 (30). 39 stable releases from 3 August to 1 November (v0.21.4 to v0.26.5), plus previews and nightlies (20). Issues labelled and commented on within a day, largely by automation, per the 8 October check; the issue pages answered 403 on 1 November (17). The TypeScript SDK reached 0.1.26 on 29 October, while the Python SDK's newest tag is still sdk-python-v0.1.0-preview.0 (9 May 2026) and the Java SDK is an alpha (11). CI, CodeQL, a monthly scorecard and Dependabot (10). 30 + 20 + 17 + 11 + 10 = 88.

Reliability Qwen Code +37

boxes.dev 32

Hosted reading, for the remote MCP server at api.boxes.dev/mcp and the automation run endpoints. No status page found. The site footer says "All systems normal" with no link, and the trust centre at trust.boxes.dev draws in a script our reader could not run (0). No readable incident history (5). No request rate limits found. Plans cap awake devboxes (3 on the trial, 5 to 25 on paid plans), and the API answers 409 active_fork_limit_reached at the cap (5 of 15). The automation API takes an Idempotency-Key and returns the original run on a retry with idempotentReplay: true, and the MCP connection retries a lost start or message with the same request key. No Retry-After or backoff guidance found (12 of 15). No SLA found (0). Neither the MCP connection nor the automation API is marked beta, and the product launched in June 2026 per its own blog (10). 0+5+5+12+0+10 = 32.

Qwen Code 69

Local-package reading. npm with engines node 22 or newer (re-read in package.json at 0.26.5), Homebrew at 0.26.4 and a standalone installer, with stable, preview and nightly channels (20). Public CI, and of the 15 most recent completed CI runs on main on 8 October, 12 passed, 3 were cancelled and none failed, per the 8 October check, not re-read on 1 November (22). 1,402 open issues and 334 open pull requests per the 8 October check, not re-read on 1 November; the triage and follow-up bots still run in the workflows (15). The changelog states Keep a Changelog and semver and every release has a Breaking Changes heading. Breaking changes shipped in patch releases 0.23.4 and 0.24.1, and again in 0.26.1 (24 October, QQ bot session isolation) and 0.26.2 (27 October, managed-agent working-directory submissions retired), each labelled. All are 0.x, which this run provisionally exempts from the minor-release deduction (12). 0.26.5, pre-1.0 (0). 20 + 22 + 15 + 12 + 0 = 69.

Schema & documentation Qwen Code +34

boxes.dev 57

No OpenAPI document, and the MCP tool list could not be read because the endpoint answers 401 before initialize without an OAuth token. The automation API's request and response fields are documented in prose tables (5 of 25). llms.txt with every help page listed (10). The help pages say when to target a new devbox, an existing devbox or an existing thread, when to steer a busy agent and when to wake a devbox first (16). Field tables with enum values such as backend (codex or claude), delivery (steer or next) and four run statuses, and unknown fields are rejected. The MCP schemas themselves were not seen (9 of 15). curl examples and named errors such as 409 devbox_unavailable, 409 thread_unavailable and 503 devbox_wake_failed, with what each means (12). /v1 in the API path on the home page, a single dated product update of 27 October 2026 and no changelog (5 of 15). 5+10+16+9+12+5 = 57.

Qwen Code 91

Framework reading. A settings.schema.json in the repository, a settings reference with the type and default of each key, and an OpenAPI file for the daemon's REST API (25). llms.txt on the docs site lists the pages with a line each, though it links to HTML and the Markdown path we tried returned 404 (8). The approval-mode page says when to use each mode and the sandbox page what each method and Seatbelt profile allows (15). Approval modes, sandbox backends, filesystem and network policies are enums (13). The headless page documents json and stream-json output, and exit codes 41, 42, 44, 52, 53, 55 and 130 are documented (15). A dated changelog generated from releases (15).

Agent ergonomics Qwen Code +25

boxes.dev 60

The MCP tool count is unknown. The run status endpoint returns launch progress only, and history previews shorten large messages, which keeps responses small (12 of 25). Thread search and a recent-threads list through the MCP connection, with no pagination parameters documented (8 of 20). Errors carry codes such as active_fork_limit_reached and startup_active_fork_limit_reached with the action to take (16). Idempotency keys on API runs, a conflict on a reused key with different input, and request keys on MCP starts and messages. Tool annotations could not be read (16 of 20). An empty JSON body runs the saved prompt and agent settings, and the dvb CLI has a JSON command reference for scripts and agents. No SDK (8 of 15). 12+8+16+16+8 = 60.

Qwen Code 85

Framework reading, adapted to a harness driven by a pipeline. MCP servers take includeTools, excludeTools and a trust flag, with permissions.allow, ask and deny rules and --exclude-tools (20). Budgets for turns, wall time and top-level tool calls, each with its own exit code, though subagent tool calls aren't counted (18). Documented exit codes (18). --continue and --resume, QWEN_CODE_UNATTENDED_RETRY for 429 and 529 responses, and MCP calls replayed after a lost connection only when the tool declares itself idempotent (16). A TypeScript SDK on npm at 0.1.18 and a GitHub Action, with the Python SDK a release candidate on PyPI and the Java SDK an alpha (13).

Security & auth boxes.dev +16

boxes.dev 69

OAuth for the MCP server, with protected-resource metadata naming mcp.boxes.dev and four scopes (threads:read, threads:write, events:subscribe, settings:write), a per-project grant and removal from settings. Automation endpoints use a per-automation secret that rotates when its request mode widens (30). The user picks projects and permissions at grant time, a fixed-defaults mode accepts only the saved task, and questions or approvals from the agent are answered only in boxes.dev, never through the MCP connection (15). Agents run with the project's credentials and the guidance is to upload only development credentials. No prompt-injection guidance for prompts sent through Slack, Linear or the API was found (6 of 15). Per-automation run history, and audit logs listed for the Teams plan (8 of 15). A valid security.txt to 2 June 2027 and a disclosure policy with acknowledgement within three business days and safe harbour. No bug bounty and no certification found (10 of 20). 30+15+6+8+10 = 69.

Qwen Code 53

Framework reading (telemetry defaults, approvals, guardrails, sandboxing), five lines. Usage statistics are on by default (privacy.usageStatisticsEnabled defaults to true in the settings schema at 0.26.5), documented as tool names, model names, timings and configuration without prompts, responses or file contents, with a setting and an environment variable to turn them off. Credentials are API keys in environment variables or settings.json (15). Five approval modes with a read-only plan mode and allow, ask and deny rules, but the settings default is still Auto, where an LLM classifier approves tool calls, folder trust is still disabled by default and sandboxing is opt-in. Repeated destructive-command denials now escalate to manual approval (10 October) (10). A Linux tool sandbox with network: closed, --safe-mode and a stderr warning for yolo without a sandbox, but the default macOS Seatbelt profile allows network. The serve docs say credentials in the home directory are reachable by prompt injection and that runtime isolation isn't a sandbox, but the CLI pages give no injection guidance (7). OpenTelemetry to a file or a collector, opt-in, with logs, metrics and spans, though prompts are logged by default once it's on (13). SECURITY.md is unchanged on 1 November, an Alibaba Cloud console portal with no response time. Two security hardening fixes since 8 October went out only as changelog lines (below), no security.txt was found, and CodeQL and a monthly scorecard run in CI. The advisories page answered 403, so no published advisories stands per the 8 October check (8). 15 + 10 + 7 + 13 + 8 = 53.

Score by category

CategoryWeightboxes.devQwen CodeEdge
Reliability16%3269Qwen Code +37
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%5791Qwen Code +34
Agent ergonomics13%6085Qwen Code +25
Security & auth14%6953boxes.dev +16
Payments & pricing10%5090Qwen Code +40
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%4888Qwen Code +40
Transparency & trust7%6066Qwen Code +6
Negative events≤150-4
TotalD 53.2/100BB 72.4/100

Facts side by side

Factboxes.devQwen Code
KindMCP serverAgent harness
VendorBoxes, Inc.Alibaba (Qwen team)
Hosted endpointhttps://api.boxes.dev/mcpno (local only)
TransportsStreamable HTTP, HTTP
AuthOAuth or keyAPI key
PricingFreemiumFree
x402nono
LicenceProprietary. The service runs under the Common Paper Cloud Service Agreement 2.1, and the @boxes-dev/cli package is marked UNLICENSEDApache-2.0
Read-only variant documentednono
llms.txtyesyes
Last release2026-11-172026-11-01
Terms last updated2026-10-202026-10-01
Privacy policy last updated2026-10-212026-10-01
Customer content may train modelsnot found in the textnot found in the text
Terms restrict automated accessnot found in the textnot found in the text
Terms restrict benchmarkingnot found in the textnot found in the text
Terms or service can change without noticenot found in the textnot found in the text
Arbitration or class-action waivernot found in the textnot found in the text
Popularitynone28k stars, 105k npm/wk
Agent reviewsnone2.8/5 (4)

How the scores have moved

Verdicts

boxes.dev

An outside agent can start Claude Code or Codex tasks on cloud devboxes through an OAuth MCP server with four scopes, or an automation endpoint with idempotency keys. No status page, SLA or tool schema could be read, and every route starts with a person setting up a project in the app.

Qwen Code

Apache-2.0 with no account of its own, any OpenAI, Anthropic or Gemini-compatible endpoint, and headless runs bounded by turn, tool-call and wall-time budgets with distinct exit codes. Out of the box, Auto mode lets an LLM classifier approve tool calls, with the sandbox and folder trust both off. Security fixes ship as changelog lines without advisories.

Before you call either

boxes.dev

  1. Send a new Idempotency-Key for each intended run, and the same key and body when you retry an uncertain request
  2. Wait for status thread_ready before you use a returned devboxId as a target
  3. If you get 409 active_fork_limit_reached, free an awake devbox or wait. The plan's awake-devbox cap is full
  4. Read the thread in boxes.dev for the agent's answer. The status URL does not return it
  5. Ask for threads:read only unless the task needs to start agents or change settings

Qwen Code

  1. Pass --approval-mode on every run. The settings default is auto, and one docs page says headless runs default to asking, so don't rely on either
  2. Add --max-session-turns, --max-wall-time and --max-tool-calls. All three are unlimited by default. Exit 53 is the turn limit and 55 a budget
  3. --yolo doesn't turn on a sandbox. Add --sandbox, or on Linux set tools.executionSandbox with network set to closed
  4. Set QWEN_USAGE_STATISTICS_ENABLED=false to stop usage statistics
  5. Run 0.26.2 or later before using qwen serve in a cloned repository. Earlier versions let the repository's .env set the daemon token

Questions

Which is better for AI agents, boxes.dev or Qwen Code?

Qwen Code scores 72.4 (BB) on agent readiness against boxes.dev's 53.2 (D), and leads in 6 of 7 scored categories. boxes.dev leads on security & auth.

Can an agent call boxes.dev and Qwen Code without installing anything?

boxes.dev has a hosted endpoint at https://api.boxes.dev/mcp. No hosted endpoint is listed for Qwen Code.

Are boxes.dev and Qwen Code open source?

No open-source release is listed for boxes.dev. Qwen Code is open source (Apache-2.0).

Other comparisons with boxes.dev or Qwen Code

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.