Crush vs Letta

Crush 52

Harness reading, scored on what an agent or pipeline driving crush run has to supply. About 20 built-in tools, any of which permissions deny hides, MCP servers take per-server disabled_tools, and skills can be turned off or kept out of the model's list. No lazy loading of MCP tools found (17). No turn, time or cost limit on crush run, and the only output controls are --quiet and --verbose (6). Errors come back as plain messages with a non-zero exit, such as the one for an unconfigured provider, with no documented codes and no JSON error record (8). --session and --continue resume a run, and crush session list, show, last, rename and delete take --json (13). One argument starts a run and environment keys are picked up, but there's no SDK, and the HTTP API needs CRUSH_CLIENT_SERVER=1 and is described only in one README section (8).

Letta 67

Harness reading. Toolsets switch with /model, skills load per agent, and subagents run in the background (18 of 25). Context is managed through memory blocks, reflection and dreaming (12 of 20). letta -p --output-format json for scripted runs and an App Server for programmatic control (12 of 20). Conversations resume by agent and conversation id, and memory is git-tracked so changes can be reverted (15 of 20). Runs with no account on the local backend, with one SDK language, TypeScript (10 of 15). Total 67.

Crush 52

Local-package reading. @charmland/crush on npm, a Homebrew tap, Winget, Scoop, AUR, Nix, FreeBSD pkg, Charm's apt and yum repositories and release binaries, with supported platforms named in the README (20). The public build workflow runs go test -race -failfast ./... on Linux, macOS and Windows for every push to main and every pull request, but GitHub pages and the API answer 403 to us, so results are unread (15). The 18 October issue and pull request counts came from shields.io badges, a third-party copy of GitHub data, and are struck. The repository's own refs show 2,574 pull requests to #4224, 272 of them open with a merge ref, and 147 merges into main in the 90 days to 4 November. Crash and regression issues couldn't be read (10). Release notes are generated from commit messages, with no CHANGELOG file and no breaking-change headings found. The legacy JSON config is deprecated in favour of crushrc while still loading (7, held partial). v0.99.0, before 1.0, with no stability statement (0). 20 + 15 + 10 + 7 + 0 = 52.

Letta 58

Local-software lines, read for the harness. @letta-ai/letta-code on npm with Node.js 22.19 or newer, and PyPI wheels for glibc Linux, macOS 14 and Windows x64 with Python 3.9 or newer stated (20). ci.yml, nightly update smoke tests and Telegram live smoke tests exist, but GitHub's API answered with a rate limit, so run results were unread (15 of 25, unchecked). 332 open issues and pull requests, mostly pull requests; recent bug reports such as recall reading the wrong conversation have no reply yet (15 of 25). 398 tags at 0.34.x with release notes on GitHub, no changelog file, and breaking removals such as AgentFile announced in the README (8 of 15). Pre-1.0 (0). Total 58.

Crush 41

Harness reading of the framework checklist, as for the other harnesses. 30 for what leaves the machine by default, 20 for approvals and sandboxing, 15 for prompt-injection posture, 15 for audit and 20 for the security programme. Usage metrics go to PostHog through data.charm.land by default, keyed to a hashed machine ID, aliased to the Hyper user ID after a Hyper login, and including error messages. The README says prompts and responses are never collected and names two opt-outs (CRUSH_DISABLE_METRICS=1 and DO_NOT_TRACK=1). The provider list is fetched from Catwalk at startup unless turned off, the update check only notifies, and provider keys stay local (15). The TUI asks before bash, edit, write, fetch, download and MCP calls, permissions deny hides tools and PreToolUse hooks can block calls. But crush run approves every permission request in its session automatically (internal/app/app.go in v0.99.0), --yolo skips all prompts, and there's no sandbox, only a refusal list for curl, wget, ssh, sudo, package managers and similar commands (7). No prompt-injection guidance found, and a project's .crushrc and $(...) in crush.json run as code at startup, which the README warns about (3). Logs in ./.crush/logs/crush.log with crush logs, sessions in a local database readable as JSON, and crush stats, with no OpenTelemetry export (9). The organisation's SECURITY.md is a disclosure policy with a reporting address (vt100@charm.sh) and no response time. CI runs CodeQL, Grype and govulncheck nightly and releases are cosign-signed. OSV lists no advisory for the npm package or the Go module. No security.txt and no bug bounty. A disclosure route with no stated time is this run's 7 (was 10). 15 + 7 + 3 + 9 + 7 = 41.

Letta 35

Harness reading of the framework checklist, as for the other harnesses. 30 for what leaves the machine by default, 20 for approvals and sandboxing, 15 for injection posture, 15 for audit and 20 for the security programme. Letta Cloud is the default backend, so agent state and conversations go to Letta's servers, and telemetry is on by default (10 of 30). The default permission mode is unrestricted; standard, acceptEdits and strict are opt-in, and cloud sandboxes exist for remote work (4 of 20). No prompt-injection guidance found for fetched pages or channel messages (5 of 15). Conversation search and git history of memory give an operator a record of what the agent saw and changed (10 of 15). SECURITY.md routes reports to support@letta.com and scopes out the retired server; no security.txt, bounty or certification found (6 of 20). Total 35.

Crush 65

Harness reading of the framework line. A JSON Schema for the config at https://charm.land/crush.json (also printed by crush schema), --json on the session commands, and an OpenAPI document that crush server generates from its endpoint list and serves at /v1/docs/openapi.json, which isn't published anywhere we could fetch. crush run output has no typed form (20). No llms.txt (charm.land/llms.txt is a not-found page). The docs are Markdown in the repository and the hooks guide says it was written for humans and agents (5). The README and config docs explain when to allow, deny or hook a tool and warn that configs run as code (12). The schema carries enums and defaults, such as the attribution trailer style, while permissions take bare tool names with no command patterns (11). Many examples for providers, MCP, hooks and local models, with no reference for error messages or exit codes (9). Version tags and GitHub releases with generated notes, which we couldn't open, and no changelog file (8).

Letta 70

Harness reading. docs.letta.com serves llms.txt and a Markdown version of every page, with an applies_to block saying which backend and interface each page covers (10). TypeScript types for the Agent SDK and a documented App Server WebSocket protocol, but no JSON Schema or OpenAPI for the harness (15 of 25). Pages say when to use local or cloud backends, subagents and channels (14 of 20). Permission modes and options are typed in the SDK (12 of 15). Example applications and a remote client reference (11 of 15). Releases most days but no changelog file (8 of 15). Total 70.

Crush 55

No payment protocol (0). Scored on Charm's Hyper provider. Its public provider file at hyper.charm.land/v1/provider lists per-million-token figures for 22 models without a login, but no pricing page confirms them as billed rates, and plan prices sit in a script-rendered app (15). The CLI is free to use and runs with your own provider key or a local model, so nothing needs a card (20). A local model through Ollama, llama.cpp or LM Studio, or an existing provider key in the environment, works with no Charm account, so an agent can install it and run a task without a person signing up (20).

Letta 60

No x402, MPP or L402 (0 of 40). Per-unit API prices are public, $0.10 per active agent a month and $0.00015 a second of tool execution, plus plan prices (20). A $0 plan with bring-your-own keys; no card is mentioned (20). The CLI runs locally without signing in (20). Total 60.