Write a review

Head to head · Multi-agent orchestration · October 2026 research run

CLI Manager vs Devin

CLI Manager scores 65.8 (B) on agent readiness against Devin's 55.7 (C), and leads in 4 of 7 scored categories. Devin leads on security & auth and transparency & trust. Both do multi-agent orchestration.

Best agent harnesses and coding agents · All 243 harnesses comparisons

CLI Managersolhun (Jeong Kyeonghun) · Agent harness B65.8/100#329 of 1084 DevinCognition AI, Inc. · Agent harness C55.7/100#694 of 1084

Which one, for what

CLI Manager B

Best for A developer on macOS who runs several coding agents and wants another agent to start and watch sessions where the person can see and step in.

Ahead on

  • Reliability, 68 against 30
  • Schema & documentation, 87 against 80
  • Payments & pricing, 60 against 20

Also in its favour

  • Free to start without a card
  • Open source

Watch for

Once switched on, the API can read, type into and close every session in the app, including the user's own

Devin C

Best for A team that wants to hand whole tasks to a cloud agent and collect pull requests, driven from a pipeline or another agent.

Ahead on

  • Security & auth, 72 against 58
  • Transparency & trust, 69 against 54

Also in its favour

  • A hosted endpoint, with nothing to install

Watch for

www.devinstatus.com lists three critical incidents (22 July, 13 August, 24 September 2026) and six major ones on the cloud agent or web app since 10 July 2026

Score by category

CategoryWeight this runCLI ManagerDevinEdge
Reliability16%206830CLI Manager +38
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%16.28780CLI Manager +7
Agent ergonomics13%16.26362CLI Manager +1
Security & auth14%17.55872Devin +14
Payments & pricing10%12.56020CLI Manager +40
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%8.86263Devin +1
Transparency & trust7%8.85469Devin +15
Negative events≤1500
TotalB 65.8/100C 55.7/100

Facts side by side

FactCLI ManagerDevin
KindAgent harnessAgent harness
Vendorsolhun (Jeong Kyeonghun)Cognition AI, Inc.
Hosted endpointno (local only)https://mcp.devin.ai/mcp
TransportsStreamable HTTPHTTP
AuthAPI keyAPI key
PricingFreeFreemium
x402nono
LicenceMITProprietary service under Cognition's Platform Terms of Service
Tools exposed1213
Read-only variant documentednoyes
llms.txtyesyes
Last release2026-10-132026-10-07
Terms last updated2026-09-142026-06-30
Privacy policy last updated2026-09-142026-03-09
Customer content may train modelsnot found in the textyes, with an opt-out
Terms restrict automated accessnot found in the textnot found in the text
Terms restrict benchmarkingnot found in the textyes
Terms or service can change without noticenot found in the textnot found in the text
Arbitration or class-action waivernot found in the textyes
Popularity15 starsnone

Verdicts

CLI Manager

An agent can open visible terminal sessions, type into them, wait and read the screen through 12 typed MCP tools or a documented REST API, and the user can take any session back. The API reaches every open session once enabled, the app runs only on macOS, and none of the last 35 CI runs on the main branch passed.

Devin

The v3 API is well specified, with a public OpenAPI 3.1 file covering 239 operations, problem+json errors, cursor pagination and service-user keys tied to roles. The status page records three critical and several major incidents on the cloud agent between 22 July and 24 September 2026, and no rate limit figures or retry guidance were found in the reviewed documentation.

Before you call either

CLI Manager

  1. Ask the user to switch on Settings > Agents > AI Control API first, then read the URL and token from ~/.climanager/control-api.json
  2. On 409 awaiting_input, read the screen and answer with keys such as ["down","enter"]. Leave folder-trust and permission choices to the user
  3. Stop using a session on 409 disconnected. The user took it back
  4. Treat timedOut: true from wait as still running, not as a failure, and wait again
  5. Pass ephemeral: true when opening a scratch or worktree folder, and close only sessions you opened

Devin

  1. Use a cog_ service-user key with the Member role. Legacy apk_ keys fail against v3 and the MCP server with 401 or 403
  2. Set max_acu_limit on every session you create. Usage is metered by the work done and has no published unit rate
  3. Session creation is not idempotent in v3. After a timeout, list sessions by tag before creating again
  4. Enterprise keys and personal access tokens must send X-Org-Id to the MCP server. Organisation-scoped keys resolve it automatically
  5. Create scheduled work as an automation. POST to the schedules endpoint returns 403 for migrated organisations since 24 September 2026

Questions

Which is better for AI agents, CLI Manager or Devin?

CLI Manager scores 65.8 (B) on agent readiness against Devin's 55.7 (C), and leads in 4 of 7 scored categories. Devin leads on security & auth and transparency & trust.

Are CLI Manager and Devin open source?

CLI Manager is open source (MIT). No open-source release is listed for Devin.

Other comparisons with CLI Manager or Devin

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.

Get an auditSee a sample report