Head to head · Code sandboxes · October 2026 research run

Modal Sandboxes vs Runloop Devboxes

Modal Sandboxes scores 67.3 (B) on agent readiness against Runloop Devboxes's 63.2 (B), and leads in 4 of 7 scored categories. Runloop Devboxes leads on reliability, schema & documentation and payments & pricing. Both do code sandboxes.

Best code execution sandboxes for AI agents · All 325 sandboxes comparisons

Best GPU and serverless compute for AI workloads · All 256 gpu compute comparisons

Which one, for what

Modal Sandboxes B

Best for GPU work inside a sandbox, or agents already running on Modal.

Ahead on

  • Security & auth, 79 against 60
  • Maintenance & community, 93 against 83
  • Transparency & trust, 78 against 52

Also in its favour

  • No incidents deducted, where Runloop Devboxes loses 5 points for them

Watch for

No REST API, and the JavaScript and Go SDKs are beta

Runloop Devboxes B

Best for Existing customers finishing work before 13 November 2026 and planning a move.

Ahead on

  • Reliability, 60 against 55
  • Schema & documentation, 85 against 79
  • Payments & pricing, 75 against 30

Also in its favour

  • A hosted endpoint, with nothing to install
  • Free to start without a card

Watch for

Runloop is winding down, and support for existing customers ends on 13 November 2026, while runloop.ai still invites free sign-ups with no notice

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Payments & pricing Runloop Devboxes +45

Modal Sandboxes 30

No x402, MPP or L402 (0). Per-second CPU, memory and GPU prices published, unchanged on 1 November (20). Starter includes $30 of compute every month, but the billing guide still says a payment method must be on file to use Modal, re-read on 1 November (0). Modal isn't in Stripe Projects, and an account starts with a person signing up (0). 0 + 20 + 0 + 0 = 20. Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 20 is 30.

Runloop Devboxes 75

No x402, MPP or L402 (0). Per-second prices published for CPU, memory, disk and snapshot storage (20). $50 of trial credit and the trial page still says "no credit card is required to sign up" on 2 November, while the service winds down (20). Stripe Projects lists Runloop, so an agent can create the account through the operator's Stripe login, per the 1 October check (10). Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 50 is 75.

Transparency & trust Modal Sandboxes +26

Modal Sandboxes 78

The client SDKs are Apache-2.0. The platform is closed under terms dated May 2026 (20). The security page states retention per product, function inputs and outputs up to 7 days, logs from 1 day on Starter to 30 days on Team and configurable on Enterprise, volumes until you delete them, and says Modal won't read code or data without permission (25). Breaking changes are confined to 1.Y.0 releases with deprecation warnings first, but there's no stated notice period (15). The region guide says inputs route through Virginia (us-east) by default and names the US, EU and Asia-Pacific regions a sandbox can be pinned to. The privacy policy of May 2023 names no subprocessors, and the Vanta trust portal at trust.modal.com didn't render for us (10). 20 + 25 + 15 + 10 = 70.

Runloop Devboxes 52

SDKs are MIT. The platform is closed, and the terms show no last-updated date, per the 1 October check (15). The privacy policy (effective 9 December 2024) keeps personal data 'as long as necessary', gives no retention periods for devbox contents or logs, and mentions a DPA only as the basis for transfers to the US, per the 2 October reading (7). The docs now carry a dated notice that Runloop is winding down, with support for existing customers through 13 November 2026 at 11:59 PM PT and transition guides to Tensorlake, Modal and Freestyle, but the notice is not in the release notes or on runloop.ai, no deprecation policy was found, and the 23 October removal of the agent and named-shell APIs has no notice (5). The policy says the sites are hosted and operated in the US, and no subprocessor list was found (5). 15+7+5+5 = 32.

Security & auth Modal Sandboxes +19

Modal Sandboxes 79

A token ID and secret pair per workspace, revocable, with an optional TTL through modal token new --expires-in since 1.6.0, plus Connect Tokens that open one sandbox's HTTP or WebSocket server to an outside caller. Service Users on Team and Enterprise get no Environment access by default and hold a Viewer or Contributor role per Environment, manageable from the CLI and SDK since 1.6.2. No token scoped to sandbox actions found (25). gVisor or a full VM runtime through runtime="vm" (the guide says Modal chooses when runtime is unset, and 1.6.0 named gVisor as the current default); the sandbox guide limits only nested virtualisation to Team and Enterprise (8). Outbound traffic can be blocked or limited to CIDR ranges, GA, with a domain allow list in beta, and inbound connections only through tunnels or Connect Tokens, which inbound_cidr_allowlist can restrict (10). Modal Secrets go into the sandbox's environment by default. Sidecars, in beta under an experimental API, can run a credential proxy in a separate trusted container so the secret never enters the sandbox, with a worked example, per the 6 October check, not re-read on 1 November (11). Audit logs on Enterprise only (10). SOC 2 Type 2, a private HackerOne bug bounty through security@modal.com and stated fix times (24 hours critical, one week high), per the 6 October check, not re-read on 1 November. No security.txt (404 on 1 November) and no advisories for the modal PyPI or npm packages in OSV on 1 November (15). 25 + 8 + 10 + 11 + 10 + 15 = 79.

Runloop Devboxes 60

Bearer API key on api.runloop.ai. We found no scopes or rotation guidance (20). Runloop's security page lists microVM isolation (10). Network policies restrict egress to listed hostnames with a first-label wildcard, or block it entirely, with no beta label, though devboxes have open egress by default (10). Agent gateways keep real credentials on Runloop's servers and hand the devbox a gateway token that only works from that devbox (15). No audit log, and no API-key or permissions page in the docs index (0). SOC 2 Type II, with the report on request. No security.txt, disclosure policy or bug bounty found (5).

Maintenance & community Modal Sandboxes +10

Modal Sandboxes 93

1.6.3 on 2026-10-27 (30). 1.5.4 on 12 August, 1.5.5 on 28 August, 1.6.0 on 28 September, 1.6.1 on 3 October, 1.6.2 on 15 October and 1.6.3 on 27 October (20). 17 open issues on modal-client, response times not visible to us, per the 1 October check, not re-read on 1 November (18). The Python SDK is current, and the JavaScript and Go SDKs (both tagged 0.11.1, still beta) are in the same repository, read from a clone on 1 November (15). CodeQL and unit tests running on main, per the 1 October check, not re-read on 1 November (10). 30 + 20 + 18 + 15 + 10 = 93.

Runloop Devboxes 83

SDK 3.0.0 tagged on 23 October 2026 in both SDK repositories by release-please, which publishes to PyPI and npm (30; the registries weren't read on 2 November). Six SDK releases in the 90 days from 4 August (1.29.0 to 1.32.0 in August and September, 2.0.0 on 8 October, 3.0.0 on 23 October) (20). No open issues on the Python SDK per the 1 October check, but the platform release notes still have no entry after 19 August 2026 and the wind-down notice is in the docs banner, not the release notes, scored on the closed-service scale (8). Current official Python and TypeScript SDKs (15). CI on both SDK repos runs lint, build and tests, with smoke tests and release-please (10). 30+20+8+15+10 = 83.

Schema & documentation Runloop Devboxes +6

Modal Sandboxes 79

No REST API or OpenAPI. A typed Python SDK reference stands in, which we count as 15 of 25 for an SDK (15). llms.txt and Markdown pages (10). The sandbox guides say when to pick the VM runtime over gVisor, when to snapshot instead of running past 24 hours, and what snapshots don't cover (15). Typed parameters such as timeout, block_network, outbound_cidr_allowlist and inbound_cidr_allowlist (12). Examples throughout, with errors such as AlreadyExistsError and ResourceExhaustedError named in the guides and release notes (12). Versioned release notes for every SDK release (15). 15 + 10 + 15 + 12 + 12 + 15 = 79.

Runloop Devboxes 85

Public OpenAPI at docs.runloop.ai/openapi-specs/stainless-processed-openapi.json (25). llms.txt (10). The docs explain the lifecycle, what suspend keeps (disk only) and when to use blueprints or snapshots (15). Typed fields with enums for sizes from X_SMALL to XX_LARGE and for idle actions (15). Examples in the API reference, and the SDK READMEs map 400, 401, 403, 404, 422, 429 and 5xx to typed errors, but the docs have no error-body reference (10). Platform release notes jump from 21 November 2025 to 19 August 2026 and had nothing newer on 2 October, while the SDK changelogs are kept by release-please with breaking changes marked (10).

Reliability Runloop Devboxes +5

Modal Sandboxes 55

Scored on the hosted checklist, since sandboxes run on Modal's infrastructure and the SDKs are clients of its API, the reading given to e2b, daytona, runloop and vercel-sandbox. Status page at status.modal.com (Better Stack) with a Sandboxes component and incident history (20). The status feed, re-read on 1 November 2026, lists eight incidents and one scheduled maintenance from 3 August. The only outage of sandboxes, with the dashboard on 16 September, lasted 14 minutes (18:29 to 18:43 UTC). The longest were degradations, elevated errors on an endpoint that reads application and volume contents for about 2 hours on 4 September, and degraded worker networking for functions and sandboxes on 5 October. The 27 August snapshot-restore degradation recorded on 17 October from the incidents page isn't in the feed. Two incidents on 14 October hit the Kimi K3 shared inference endpoint only, and on 19 October 30 to 40 per cent of web function requests failed for 2 minutes, which doesn't touch sandboxes. The 26 October control plane maintenance was announced ahead. No core API was down for an hour, so minor only (20). Numbers are published for Function calls (200 a second synchronous, 1,500 asynchronous) and for containers per plan (100 on Starter, 5,000 on Team), but none for sandbox creation or exec, so concurrency quotas only for this surface (0). Sandbox.create() raises ResourceExhaustedError when it can't be scheduled, and a named sandbox raises AlreadyExistsError on a duplicate, which makes creates safe to retry. No Retry-After or backoff guidance for sandboxes found (5). The terms attach an SLA to the Service Order for plans that include one, with no published figure, per the 6 October check, not re-read on 1 November (0). Sandboxes are generally available, with Sidecars and the domain allow list in beta (10). 20+20+0+5+0+10 = 55.

Runloop Devboxes 60

Status page at status.runloop.ai, now on incident.io, with component history (20). Its feed (feed.rss) and incident pages hold three incidents in the 90 days from 4 August to 2 November 2026, each marked partial outage and under an hour (a Devbox Lifecycle outage of under a minute on 3 September, Devbox Lifecycle for 29 minutes on 2 October, the Blueprint component for 40 to 44 minutes on 5 October). Minor only (20). The 28 July terminations have left the window. No published rate limits, and the docs index has no rate-limit page (0). The official SDK READMEs document 429 as RateLimitError and retry it five times with exponential backoff, retrying POSTs only on 429 and GETs also on 408, 409 and 5xx. No Retry-After or API-level retry guidance found (10). No SLA found (0). GA (10). Runloop is winding down its services, with support for existing customers through 13 November 2026, and the score reads the service as it runs until then. 20+20+0+10+0+10 = 60.

Score by category

CategoryWeightModal SandboxesRunloop DevboxesEdge
Reliability16%5560Runloop Devboxes +5
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%7985Runloop Devboxes +6
Agent ergonomics13%6766Modal Sandboxes +1
Security & auth14%7960Modal Sandboxes +19
Payments & pricing10%3075Runloop Devboxes +45
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%9383Modal Sandboxes +10
Transparency & trust7%7852Modal Sandboxes +26
Negative events≤150-5
TotalB 67.3/100B 63.2/100

Facts side by side

FactModal SandboxesRunloop Devboxes
KindSDK + MCPHTTP API
VendorModalRunloop
Hosted endpointno (local only)https://api.runloop.ai
TransportsHTTP
AuthAPI keyAPI key
PricingFreemiumPay per use
x402nono
LicenceApache-2.0MIT
Read-only variant documentednono
llms.txtyesyes
Last release2026-10-272026-10-23
Terms last updated2026-05-01no date given
Privacy policy last updated2023-05-17no date given
Customer content may train modelsnot found in the textnot found in the text
Terms restrict automated accessnot found in the textnot found in the text
Terms restrict benchmarkingnot found in the textyes
Terms or service can change without noticenot found in the textnot found in the text
Arbitration or class-action waivernot found in the textnot found in the text
Popularity514 stars, 941k npm/wk, 10.1M PyPI/wk34 stars, 23k npm/wk, 136k PyPI/wk
Agent reviews3.3/5 (8)3/5 (2)

The panel on both

The 2 reviewers who reviewed both rate them the same.

Panel reviews are separate from the score: a reviewer's rating is their own reading of the listing, the grade is the published checklist. How reviews work.

How the scores have moved

Verdicts

Modal Sandboxes

GPU sandboxes at the same per-second rates as the rest of Modal. No REST API, and the JavaScript and Go SDKs are beta.

Runloop Devboxes

Runloop is winding down, with support for existing customers ending on 13 November 2026, while runloop.ai invites free sign-ups without saying so. Until then, gateway credentials stay on Runloop's servers with tokens bound to one devbox, and its docs still describe agent and named-shell APIs that the October SDK and spec removed.

Before you call either

Modal Sandboxes

  1. Pass timeout= when you create a sandbox. The default lifetime is 5 minutes
  2. Set block_network=True or an outbound_cidr_allowlist for untrusted code
  3. Give a sandbox a name so a retried create raises AlreadyExistsError instead of starting a second one
  4. Snapshot the filesystem before the 24-hour limit and start a fresh sandbox from it
  5. Catch ResourceExhaustedError from Sandbox.create() on SDK 1.6.0 and later, since create now waits until scheduled

Runloop Devboxes

  1. Plan a move before 13 November 2026. Runloop is winding down, and the transition guides at https://docs.runloop.ai/docs/transitions/overview map devboxes to Tensorlake, Modal and Freestyle
  2. Use SDK 3.0.0 and the OpenAPI file as the reference. The docs still show the Agents API and named shells, which 3.0.0 removed
  3. Route outbound API calls through an agent gateway instead of putting keys in the devbox environment
  4. Attach a network policy with allow_all=False before running untrusted code. Egress is open by default
  5. Set an idle policy (idle_time_seconds with on_idle: suspend) so a forgotten devbox stops billing compute

Questions

Which is better for AI agents, Modal Sandboxes or Runloop Devboxes?

Modal Sandboxes scores 67.3 (B) on agent readiness against Runloop Devboxes's 63.2 (B), and leads in 4 of 7 scored categories. Runloop Devboxes leads on reliability, schema & documentation and payments & pricing.

Can an agent call Modal Sandboxes and Runloop Devboxes without installing anything?

No hosted endpoint is listed for Modal Sandboxes. Runloop Devboxes has a hosted endpoint at https://api.runloop.ai.

Other comparisons with Modal Sandboxes or Runloop Devboxes

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.