{
  "fixes": {
    "slug": "onyx",
    "name": "Onyx",
    "listing": "https://www.anchorterminal.com/tools/onyx",
    "markdown": "# Fix list: Onyx\n\nFrom Anchor Terminal's listing at https://www.anchorterminal.com/tools/onyx, the October 2026 research run, assessed 3 October 2026. Grade B, 65.3 out of 100.\n\nThis is everything the published grade says the listing lacks, the biggest possible gain to the total first. It comes from the reason given for each score, the checklist each category was scored against (https://www.anchorterminal.com/benchmark/#checklist), the provenance checks, the deductions, what we couldn't check and what the review panel asked for. A fix counts at the next check, once it's public.\n\nFor a coding agent working on Onyx: work through the items below in the product, its docs and its public pages. Each category gives the reason for its score, with the points each checklist item earned, and the checklist itself, so the gap is the items that earned less than their points. Change the product, not the wording, and keep a note of what you changed and where it's published.\n\n## 1. Payments \u0026 pricing, 30 out of 100, up to 8.8 more on the total\n\nWhy it scored 30: Scored on Onyx Cloud, as the rubric asks for open-source software with a paid version. No x402, MPP or L402 (0). Business is $20 a user a month billed annually on the public pricing page and Enterprise is by quote, plan pricing with no per-call unit (10). A two-week Cloud trial with no card, and the MIT Community Edition is free to run yourself (20). A person signs up in a browser and creates the token, on Cloud or on their own instance (0).\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-payments):\n\nThe published rubric, also on the [x402 page](https://www.anchorterminal.com/x402/).\n\n- 40, a machine payment protocol (x402, MPP or L402) on the tool's own endpoints. 10 to 30 when it covers only some endpoints or only goes through a third party, and the note says which.\n- 20, per-call or per-unit pricing published without a login. 10 for public plan-only pricing, 0 for \"contact sales\" or prices behind a login.\n- 20, a free tier or trial that doesn't need a card.\n- 20, autonomous onboarding, meaning an agent can get access without a person signing up in a browser (keyless use, x402, a programmatic key API).\n\nPayment platforms and agent wallets rarely charge for their own API over a machine protocol, so the first line has steps for them, and the highest one that applies counts. 40 when x402, MPP or L402 runs on all their own endpoints, 30 when it runs on part of their own API, 25 when their merchants can accept one, 20 for running a facilitator, 15 for paying as a buyer, and 0 when the only protocol is their own. Merchant acceptance sits above a facilitator because the platform's own customers can charge agents through it, while a facilitator settles for sellers who wire up the protocol themselves. The counter-argument (a facilitator does more for the protocol as a whole) has a point. Each note says which step applied.\n\nOpen-source software you run yourself is scored on its hosted or paid option if it has one. A free, self-hosted package with nothing to buy gets 20, 20 and 20 for the last three lines, and 0 to 40 for the first only if it ships a payment protocol.\n\n## 2. Reliability, 69 out of 100, up to 6.2 more on the total\n\nWhy it scored 69: Onyx is mostly run self-hosted, and Onyx Cloud runs it for you, so we scored both halves and took the mean, as we did for Marmot. Self-hosted, on the local checklist. Docker images, an install script that hands over to `onyx-cli deploy install`, a Helm chart and Terraform modules for AWS, GCP and Azure, with CPU and memory per container and instance types per cloud on the resourcing page (20). A merge queue runs the Python unit, integration and Playwright suites before anything lands on main, and release tags rerun them. The Actions page didn't show pass or fail for the runs we loaded (20 of 25). 153 open issues and 354 open pull requests. Bug reports from early July are still open, among them a citation processor that corrupts code fences (#12684) and Claude Sonnet 5 requests failing with 400 (#12680), and a stale bot labels issues after 75 days (15 of 25). The docs commit to SemVer 2.0.0 and every changelog entry has a Deployment Changes section, but 4.7 replaced API key roles with group permissions and 4.8.0 warns not to upgrade before the document index migration, both inside major version 4 (11 of 15). Version 4.8.4 (15). That half comes to 81. Cloud, on the hosted checklist. A Better Stack status page at status.onyx.app with three monitors and 90-day bars (20). No incident reports, while the monitors show 99.983 per cent for the API server and 99.734 per cent for a functional frontend check, which over 90 days would be about 5.7 hours of failures with no write-up. We can't tell majors from minors, so we scored between the lines (15 of 30). Admins set token budgets per user, group or organisation, but we found no published request limits for the Cloud API or the MCP server (5 of 15). Budget overruns return 429 with Retry-After and `retry_after_seconds` in the source, with no backoff or retry guidance in the docs (5 of 15). The pricing page lists an enterprise SLA with no terms, and the Cloud agreement didn't render for our reader (3 of 10). The MCP server and the API carry no beta label (10). That half comes to 58. The mean is 69.5, rounded down because the CI state and the frontend failures are unconfirmed.\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-reliability):\n\nHosted APIs, MCP servers, models and platforms.\n\n- 20, a public status page with component history (Statuspage, Instatus, BetterStack or the vendor's own).\n- 0 to 30, the incident record for the last 90 days on that page. 30 for a clean record or trivial incidents only, 20 for minor incidents only, 10 for one major outage (an hour or more of a core API down, or errors across the board), 0 for several. 5 when there's no history we could read, and the note says so.\n- 15, rate limits documented with numbers.\n- 15, documented 429 or overload handling (Retry-After, backoff guidance), and idempotency keys or safe-retry guidance where writes are involved.\n- 10, an SLA published for any paid tier.\n- 10, the surface agents use is generally available, not beta or preview.\n\nLocal packages, SDKs, frameworks and stdio MCP servers.\n\n- 20, installs from an official package with supported runtimes stated.\n- 25, a public CI and test suite, passing on the default branch.\n- 0 to 25, open crash or regression issues relative to activity (25 for few and handled, 0 for many, old and unanswered).\n- 15, semver discipline and breaking changes called out in a changelog.\n- 15, version 1.0 or later, or declared stable.\n\nProtocols are read from their reference implementations, the public facilitators or servers, spec stability and test vectors.\n\n## 3. Security \u0026 auth, 71 out of 100, up to 5.1 more on the total\n\nWhy it scored 71: Personal access tokens can be limited to `read:search`, `read:chat`, `write:chat` or `use:llm_gateway`, expire after 7, 30 or 365 days or never, are stored hashed and can be revoked one by one. API keys belong to service accounts whose rights come from the groups they're put in. No OAuth for the MCP server, and no long-lived secret travels in a query string (the voice WebSocket takes a single-use token that expires after 60 seconds) (28). The MCP server has no write tools and a `read:search` token covers it. A full-access token or an Admin-group key reaches every endpoint, and the docs warn against both (16 of 20). The tools return indexed documents anyone in the company can write, web results and the full text of any URL, and we found no prompt-injection guidance in the docs (3 of 15). Self-hosted instances since 4.3 emit an OCSF-shaped audit stream for sign-ins, permission changes, configuration and credential access, query history is an Enterprise feature, and the MCP server logs each query at INFO with Prometheus counters (11 of 15). SECURITY.md routes reports through GitHub private reporting with safe harbour and a 90-day timeline, five advisories are published, among them a critical one this year, and the docs state SOC 2 Type II with reports on request. No bug bounty or security.txt (13 of 20).\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-security):\n\n- 0 to 30, the credential model. 30 for OAuth 2.1 with scopes, or scoped and revocable keys with rotation. 20 for plain revocable API keys. 10 for one all-powerful key. 10 off when a secret can travel in a URL query string as a documented option.\n- 0 to 20, read-only or least-privilege modes, and confirmation or approval for destructive actions.\n- 0 to 15, prompt-injection posture where the tool returns untrusted content (documented mitigations or guidance). A tool that returns no untrusted content gets 10.\n- 0 to 15, audit logs or per-call visibility for the operator.\n- 0 to 20, a security programme. security.txt or a disclosure policy, a bug bounty, SOC 2 or ISO 27001, advisories handled in public.\n\nModels are read for retention, whether API data trains models (and whether that's off by default), zero-retention options and certifications. Frameworks for telemetry defaults, approval hooks, guardrails and sandboxing.\n\n## 4. Agent ergonomics, 77 out of 100, up to 3.7 more on the total\n\nWhy it scored 77: Three tools in 3,360 characters of descriptions (about 850 tokens), plus three resources that list sources, document sets and agents. All three tools only read, so there's nothing to subset (25). Search filters by source, document set, agent and an updated-since time, `search_web` takes `limit` (default 5), and `open_urls` caps a call at 20 URLs. Document search has no result limit or paging and returns full chunks, so the response size is the server's choice (14 of 20). Errors say what failed and list close matches, or the available values when there are ten or fewer, and an empty index gets its own message. They come back as ordinary results with an `error` field and an empty list, not as `isError` (16 of 20). No readOnlyHint or other annotations. Every tool reads, so a retry is safe, though each document search can run an LLM query-expansion step and spend model tokens again unless `skip_query_expansion` is set (12 of 20). Only `query` is required. The official clients are `onyx-cli` (a Go binary, also on PyPI) with an agent mode and JSON output, and a Terraform provider, with no Python or TypeScript library (10 of 15).\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-ergonomics):\n\n- 0 to 25, context cost. For MCP, the number and size of the tool definitions (25 for ten or fewer compact tools, 15 for 11 to 30, 5 for more than 30, plus up to 10 back for toolsets, dynamic loading or read-only subsets). For APIs, whether responses can be sized (field selection, limits, summaries).\n- 20, pagination, filtering and output-size controls.\n- 20, actionable, documented error responses, codes and messages an agent can recover from.\n- 20, idempotency or safe retries, and for MCP the `readOnlyHint` and `destructiveHint` annotations.\n- 15, sensible defaults, few required parameters, and official SDKs in at least two languages.\n\nModels are read for tool use, structured output, prompt caching, context length, batch and SDKs. Frameworks for how much code and how many defaults a tool-calling agent with MCP needs.\n\n## 5. Transparency \u0026 trust, 66 out of 100, up to 3 more on the total\n\nMade of editorial 57, provenance 75.\n\nWhy it scored 66: Everything outside the `ee` directories is MIT, the MCP server included, and the `ee` code is under the Onyx Enterprise License, which needs a subscription for production use (25 of 30). The docs say no models are trained on customer data, that a self-hosted Onyx sends the company none of its data beyond telemetry, and that Cloud runs on AWS with retention rules admins set. The privacy policy and the Cloud subscription agreement (both last updated 1 July 2025) render only with JavaScript and our reader got no text, so we couldn't check retention periods or deletion (12 of 30). SemVer, dated Deployment Changes and Breaking Changes sections, and the pre-4.7 auth docs kept online, but no deprecation policy with notice periods (10 of 20). Telemetry to telemetry.onyx.app is on by default and documented as anonymous with a `DISABLE_TELEMETRY` opt-out. No page lists the fields, and the source sends user IDs with sign-up, usage and latency events and, on Enterprise builds, the first user's email domain. Cloud subprocessors are listed (AWS, OpenSearch, Grafana, OpenAI and Anthropic) with US East as the default region, but the Cloud build also sends product events with client IPs to PostHog, which isn't on that list (10 of 20).\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-transparency):\n\n- 0 to 30, source availability and licence clarity. 30 for open source under an OSI licence, 15 for closed with clear terms, 0 for unclear terms.\n- 0 to 30, data handling and retention statements that agree with each other (privacy policy, DPA, retention periods, subprocessors).\n- 0 to 20, a deprecation policy or notices with dates.\n- 0 to 20, telemetry disclosed with an opt-out (local software), or subprocessors and data locations disclosed (hosted).\n\nThe other half of Transparency and trust is the provenance score, computed from checked facts (below). The category score is the mean of the two.\n\nProvenance checks not met in full (half of this category, computed from checked facts):\n\n- Domain age: onyx.app, no registry record we could read (0 of 15)\n- security.txt: not found (0 of 10)\n\n## 6. Schema \u0026 documentation, 88 out of 100, up to 2 more on the total\n\nWhy it scored 88: An OpenAPI 3.1 file of 92 paths and 110 operations with a Bearer scheme on the docs site, and every MCP tool has a typed JSON Schema that FastMCP builds from Python type hints (25). llms.txt and a Markdown copy of each docs page (10). `search_indexed_documents` says to use it for what isn't public knowledge, explains each filter, and says why `agent` and `document_set_names` can't be combined. `search_web` points to `open_urls` for full text (18 of 20). Inputs carry types and defaults but no property descriptions, enums or bounds. `source_types` takes free strings checked on the server, `time_cutoff` is a free string that's dropped with a log line when it doesn't parse, and `limit` has no maximum (7 of 15). Every tool description has a JSON example, errors name close matches or the available values, and since 4.8.0 every admin API error carries a machine-readable `error_code` (13 of 15). Dated release notes for every minor version back to 2.7, with Deployment Changes and Breaking Changes sections, and a SemVer statement in the API overview (15).\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-schema):\n\nAPIs and MCP servers.\n\n- 25, a machine-readable contract (a public OpenAPI file or similar; for MCP, typed JSON Schema inputs on every tool).\n- 10, llms.txt or Markdown docs served for agents.\n- 0 to 20, descriptions that say what a tool is for, when to use it and when not to, read from the tool definitions in the source or the API reference.\n- 0 to 15, typed inputs with enums, constraints and required fields, and no free-form JSON blobs.\n- 0 to 15, examples and documented error responses.\n- 15, versioning and a public changelog.\n\nModels are read from the API reference, the OpenAPI file, llms.txt, the structured-output and tool-use docs and the model cards. Frameworks from docs a model can follow, typed interfaces, examples and the API reference.\n\n## 7. Maintenance \u0026 community, 77 out of 100, up to 2 more on the total\n\nWhy it scored 77: v4.8.4 tagged on 2 October 2026, with 4.6.12 and 4.7.11 the same day (30). Six minor releases from 4.3.0 on 6 July to 4.8.0 on 23 September, and 61 stable tags from 4.3.0 to 4.8.4 (20). 153 open issues and 354 open pull requests. Bug reports from early July are still open, and we couldn't see reply times because GitHub's issue search refused our reader (13 of 25). No entry in the official MCP registry, where only third-party Onyx servers are listed. Official clients are `onyx-cli` (tags up to cli/v1.4.4) and a Terraform provider (5 of 15). Dependabot, nightly CVE alert workflows and a merge queue that runs the test suites, with the current pass state unconfirmed (9 of 10).\n\nThe checklist (https://www.anchorterminal.com/benchmark/#checklist-maintenance):\n\n- 0 to 30, time since the last release, or the last published model or API change for a closed service. 30 within 30 days, 20 within 90, 10 within 180, 0 older.\n- 20, at least three releases or dated changelog entries in the last 90 days.\n- 0 to 25, responsiveness. Issues and pull requests answered on GitHub (the open issues and how recent the replies are). For closed services, a public changelog and a support or community channel that answers, 0 to 15.\n- 15, presence in the official MCP registry under a verified namespace (MCP servers), or current official SDKs (APIs and models).\n- 10, package health, current dependencies and CI.\n\nModels are read for deprecation notice periods and model churn rather than release counts.\n\n## Deductions\n\nEach comes off the total. A fixed and documented problem counts for less at the next check.\n\n- 2026-07-20. GHSA-q62f-rv3h-f822 (critical, CVSS 9.0). Any signed-in user could read, in clear text, other users' live OAuth tokens for per-user MCP servers such as Slack, Atlassian or Notion through GET /api/mcp/servers. Found in an external pentest in May 2026, fixed in 4.0.0 (26 May 2026) and published, so the deduction is reduced, -3 (https://github.com/onyx-dot-app/onyx/security/advisories/GHSA-q62f-rv3h-f822)\n- 2026-04-29 and 2026-07-20. Three moderate IDOR advisories, other users' chat files downloadable through /chat/file/{file_id} (GHSA-vg3h-35f7-7w6r), other users' chat sessions stoppable through /chat/stop-chat-session (GHSA-rw6w-hp62-gc8w) and curators able to change any user group's membership (GHSA-7f48-vgpj-h95m). Fixed and published, -1 (https://github.com/onyx-dot-app/onyx/security/advisories)\n\n## What we couldn't check\n\nWhat we couldn't read counted as absent. Publishing it on a page a plain HTTP fetch can read (not only in a browser) lets the next check count it.\n\n- unchecked: the privacy policy and the Onyx Cloud subscription agreement at onyx.app/legal, which render only with JavaScript, so retention, deletion, SLA terms and any subprocessors beyond the docs list are unknown.\n- unchecked: whether the merge-queue test runs pass, since the Actions page didn't show run status to our reader.\n- unchecked: issue reply times, since GitHub's issue search refused our reader.\n- Whether the 99.734 per cent on the status page's frontend check reflects real outages. No incidents are posted.\n- unchecked: the registration date of onyx.app.\n- The docs changelog dates 4.8.0 to 18 September 2026, while the tag and the GitHub release are from 23 September.\n\n## Weaknesses\n\n- GHSA-q62f-rv3h-f822 (critical, CVSS 9.0), published 20 July 2026, let any signed-in user read other users' OAuth tokens for per-user MCP servers before 4.0.0\n- Telemetry is on by default and documented as anonymous, while its events carry user IDs and Enterprise builds send the first user's email domain\n- Document search has no result limit or paging, and an unparseable `time_cutoff` is dropped with only a server log line\n- The self-hosted MCP server is off by default, takes no OAuth and isn't in the official MCP registry\n- The privacy policy and Cloud agreement render only with JavaScript, and there's no security.txt or bug bounty\n\n## What costs an agent a turn today\n\nThe notes we give agents before they call it. Each one is a workaround an agent shouldn't need.\n\n- Get the instance URL from the operator. Cloud is https://cloud.onyx.app/mcp, and a self-hosted server only answers once `MCP_SERVER_ENABLED=true`\n- Use a token limited to `read:search`. It covers every MCP tool and nothing else\n- Pass `time_cutoff` as a full ISO 8601 timestamp. A value that doesn't parse is dropped and the search runs unfiltered\n- Check each result for an `error` field. Failures come back with an empty `results` list, not as tool errors\n- Set `skip_query_expansion` to true for exact phrases. It skips an LLM call on every search\n\n## What the review panel asked for\n\n- reject bad time_cutoff\n- return errors as isError\n- prompt-injection guidance\n- a telemetry field list\n\n## When it's done\n\nSend what changed and where it's published as a dispute (https://www.anchorterminal.com/builders/#disputes, or `POST https://www.anchorterminal.com/api/v1/contact` with `\"kind\": \"dispute\"`). Disputes are answered in public, and the listing is checked again by the same checklist. Paying for an audit or a listing claim changes nothing here.\n",
    "grade": "B",
    "score": 65.3,
    "assessed": "2026-10-03",
    "run": "October 2026 research run",
    "categories": [
      {
        "key": "payments",
        "name": "Payments \u0026 pricing",
        "score": 30,
        "maxGain": 8.8,
        "reason": "Scored on Onyx Cloud, as the rubric asks for open-source software with a paid version. No x402, MPP or L402 (0). Business is $20 a user a month billed annually on the public pricing page and Enterprise is by quote, plan pricing with no per-call unit (10). A two-week Cloud trial with no card, and the MIT Community Edition is free to run yourself (20). A person signs up in a browser and creates the token, on Cloud or on their own instance (0).",
        "checklist": [
          "The published rubric, also on the [x402 page](https://www.anchorterminal.com/x402/).",
          "- 40, a machine payment protocol (x402, MPP or L402) on the tool's own endpoints. 10 to 30 when it covers only some endpoints or only goes through a third party, and the note says which.\n- 20, per-call or per-unit pricing published without a login. 10 for public plan-only pricing, 0 for \"contact sales\" or prices behind a login.\n- 20, a free tier or trial that doesn't need a card.\n- 20, autonomous onboarding, meaning an agent can get access without a person signing up in a browser (keyless use, x402, a programmatic key API).",
          "Payment platforms and agent wallets rarely charge for their own API over a machine protocol, so the first line has steps for them, and the highest one that applies counts. 40 when x402, MPP or L402 runs on all their own endpoints, 30 when it runs on part of their own API, 25 when their merchants can accept one, 20 for running a facilitator, 15 for paying as a buyer, and 0 when the only protocol is their own. Merchant acceptance sits above a facilitator because the platform's own customers can charge agents through it, while a facilitator settles for sellers who wire up the protocol themselves. The counter-argument (a facilitator does more for the protocol as a whole) has a point. Each note says which step applied.",
          "Open-source software you run yourself is scored on its hosted or paid option if it has one. A free, self-hosted package with nothing to buy gets 20, 20 and 20 for the last three lines, and 0 to 40 for the first only if it ships a payment protocol."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-payments"
      },
      {
        "key": "reliability",
        "name": "Reliability",
        "score": 69,
        "maxGain": 6.2,
        "reason": "Onyx is mostly run self-hosted, and Onyx Cloud runs it for you, so we scored both halves and took the mean, as we did for Marmot. Self-hosted, on the local checklist. Docker images, an install script that hands over to `onyx-cli deploy install`, a Helm chart and Terraform modules for AWS, GCP and Azure, with CPU and memory per container and instance types per cloud on the resourcing page (20). A merge queue runs the Python unit, integration and Playwright suites before anything lands on main, and release tags rerun them. The Actions page didn't show pass or fail for the runs we loaded (20 of 25). 153 open issues and 354 open pull requests. Bug reports from early July are still open, among them a citation processor that corrupts code fences (#12684) and Claude Sonnet 5 requests failing with 400 (#12680), and a stale bot labels issues after 75 days (15 of 25). The docs commit to SemVer 2.0.0 and every changelog entry has a Deployment Changes section, but 4.7 replaced API key roles with group permissions and 4.8.0 warns not to upgrade before the document index migration, both inside major version 4 (11 of 15). Version 4.8.4 (15). That half comes to 81. Cloud, on the hosted checklist. A Better Stack status page at status.onyx.app with three monitors and 90-day bars (20). No incident reports, while the monitors show 99.983 per cent for the API server and 99.734 per cent for a functional frontend check, which over 90 days would be about 5.7 hours of failures with no write-up. We can't tell majors from minors, so we scored between the lines (15 of 30). Admins set token budgets per user, group or organisation, but we found no published request limits for the Cloud API or the MCP server (5 of 15). Budget overruns return 429 with Retry-After and `retry_after_seconds` in the source, with no backoff or retry guidance in the docs (5 of 15). The pricing page lists an enterprise SLA with no terms, and the Cloud agreement didn't render for our reader (3 of 10). The MCP server and the API carry no beta label (10). That half comes to 58. The mean is 69.5, rounded down because the CI state and the frontend failures are unconfirmed.",
        "checklist": [
          "Hosted APIs, MCP servers, models and platforms.",
          "- 20, a public status page with component history (Statuspage, Instatus, BetterStack or the vendor's own).\n- 0 to 30, the incident record for the last 90 days on that page. 30 for a clean record or trivial incidents only, 20 for minor incidents only, 10 for one major outage (an hour or more of a core API down, or errors across the board), 0 for several. 5 when there's no history we could read, and the note says so.\n- 15, rate limits documented with numbers.\n- 15, documented 429 or overload handling (Retry-After, backoff guidance), and idempotency keys or safe-retry guidance where writes are involved.\n- 10, an SLA published for any paid tier.\n- 10, the surface agents use is generally available, not beta or preview.",
          "Local packages, SDKs, frameworks and stdio MCP servers.",
          "- 20, installs from an official package with supported runtimes stated.\n- 25, a public CI and test suite, passing on the default branch.\n- 0 to 25, open crash or regression issues relative to activity (25 for few and handled, 0 for many, old and unanswered).\n- 15, semver discipline and breaking changes called out in a changelog.\n- 15, version 1.0 or later, or declared stable.",
          "Protocols are read from their reference implementations, the public facilitators or servers, spec stability and test vectors."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-reliability"
      },
      {
        "key": "security",
        "name": "Security \u0026 auth",
        "score": 71,
        "maxGain": 5.1,
        "reason": "Personal access tokens can be limited to `read:search`, `read:chat`, `write:chat` or `use:llm_gateway`, expire after 7, 30 or 365 days or never, are stored hashed and can be revoked one by one. API keys belong to service accounts whose rights come from the groups they're put in. No OAuth for the MCP server, and no long-lived secret travels in a query string (the voice WebSocket takes a single-use token that expires after 60 seconds) (28). The MCP server has no write tools and a `read:search` token covers it. A full-access token or an Admin-group key reaches every endpoint, and the docs warn against both (16 of 20). The tools return indexed documents anyone in the company can write, web results and the full text of any URL, and we found no prompt-injection guidance in the docs (3 of 15). Self-hosted instances since 4.3 emit an OCSF-shaped audit stream for sign-ins, permission changes, configuration and credential access, query history is an Enterprise feature, and the MCP server logs each query at INFO with Prometheus counters (11 of 15). SECURITY.md routes reports through GitHub private reporting with safe harbour and a 90-day timeline, five advisories are published, among them a critical one this year, and the docs state SOC 2 Type II with reports on request. No bug bounty or security.txt (13 of 20).",
        "checklist": [
          "- 0 to 30, the credential model. 30 for OAuth 2.1 with scopes, or scoped and revocable keys with rotation. 20 for plain revocable API keys. 10 for one all-powerful key. 10 off when a secret can travel in a URL query string as a documented option.\n- 0 to 20, read-only or least-privilege modes, and confirmation or approval for destructive actions.\n- 0 to 15, prompt-injection posture where the tool returns untrusted content (documented mitigations or guidance). A tool that returns no untrusted content gets 10.\n- 0 to 15, audit logs or per-call visibility for the operator.\n- 0 to 20, a security programme. security.txt or a disclosure policy, a bug bounty, SOC 2 or ISO 27001, advisories handled in public.",
          "Models are read for retention, whether API data trains models (and whether that's off by default), zero-retention options and certifications. Frameworks for telemetry defaults, approval hooks, guardrails and sandboxing."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-security"
      },
      {
        "key": "ergonomics",
        "name": "Agent ergonomics",
        "score": 77,
        "maxGain": 3.7,
        "reason": "Three tools in 3,360 characters of descriptions (about 850 tokens), plus three resources that list sources, document sets and agents. All three tools only read, so there's nothing to subset (25). Search filters by source, document set, agent and an updated-since time, `search_web` takes `limit` (default 5), and `open_urls` caps a call at 20 URLs. Document search has no result limit or paging and returns full chunks, so the response size is the server's choice (14 of 20). Errors say what failed and list close matches, or the available values when there are ten or fewer, and an empty index gets its own message. They come back as ordinary results with an `error` field and an empty list, not as `isError` (16 of 20). No readOnlyHint or other annotations. Every tool reads, so a retry is safe, though each document search can run an LLM query-expansion step and spend model tokens again unless `skip_query_expansion` is set (12 of 20). Only `query` is required. The official clients are `onyx-cli` (a Go binary, also on PyPI) with an agent mode and JSON output, and a Terraform provider, with no Python or TypeScript library (10 of 15).",
        "checklist": [
          "- 0 to 25, context cost. For MCP, the number and size of the tool definitions (25 for ten or fewer compact tools, 15 for 11 to 30, 5 for more than 30, plus up to 10 back for toolsets, dynamic loading or read-only subsets). For APIs, whether responses can be sized (field selection, limits, summaries).\n- 20, pagination, filtering and output-size controls.\n- 20, actionable, documented error responses, codes and messages an agent can recover from.\n- 20, idempotency or safe retries, and for MCP the `readOnlyHint` and `destructiveHint` annotations.\n- 15, sensible defaults, few required parameters, and official SDKs in at least two languages.",
          "Models are read for tool use, structured output, prompt caching, context length, batch and SDKs. Frameworks for how much code and how many defaults a tool-calling agent with MCP needs."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-ergonomics"
      },
      {
        "key": "transparency",
        "name": "Transparency \u0026 trust",
        "score": 66,
        "maxGain": 3,
        "reason": "Everything outside the `ee` directories is MIT, the MCP server included, and the `ee` code is under the Onyx Enterprise License, which needs a subscription for production use (25 of 30). The docs say no models are trained on customer data, that a self-hosted Onyx sends the company none of its data beyond telemetry, and that Cloud runs on AWS with retention rules admins set. The privacy policy and the Cloud subscription agreement (both last updated 1 July 2025) render only with JavaScript and our reader got no text, so we couldn't check retention periods or deletion (12 of 30). SemVer, dated Deployment Changes and Breaking Changes sections, and the pre-4.7 auth docs kept online, but no deprecation policy with notice periods (10 of 20). Telemetry to telemetry.onyx.app is on by default and documented as anonymous with a `DISABLE_TELEMETRY` opt-out. No page lists the fields, and the source sends user IDs with sign-up, usage and latency events and, on Enterprise builds, the first user's email domain. Cloud subprocessors are listed (AWS, OpenSearch, Grafana, OpenAI and Anthropic) with US East as the default region, but the Cloud build also sends product events with client IPs to PostHog, which isn't on that list (10 of 20).",
        "blend": "editorial 57, provenance 75",
        "checklist": [
          "- 0 to 30, source availability and licence clarity. 30 for open source under an OSI licence, 15 for closed with clear terms, 0 for unclear terms.\n- 0 to 30, data handling and retention statements that agree with each other (privacy policy, DPA, retention periods, subprocessors).\n- 0 to 20, a deprecation policy or notices with dates.\n- 0 to 20, telemetry disclosed with an opt-out (local software), or subprocessors and data locations disclosed (hosted).",
          "The other half of Transparency and trust is the provenance score, computed from checked facts (below). The category score is the mean of the two."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-transparency"
      },
      {
        "key": "schema",
        "name": "Schema \u0026 documentation",
        "score": 88,
        "maxGain": 2,
        "reason": "An OpenAPI 3.1 file of 92 paths and 110 operations with a Bearer scheme on the docs site, and every MCP tool has a typed JSON Schema that FastMCP builds from Python type hints (25). llms.txt and a Markdown copy of each docs page (10). `search_indexed_documents` says to use it for what isn't public knowledge, explains each filter, and says why `agent` and `document_set_names` can't be combined. `search_web` points to `open_urls` for full text (18 of 20). Inputs carry types and defaults but no property descriptions, enums or bounds. `source_types` takes free strings checked on the server, `time_cutoff` is a free string that's dropped with a log line when it doesn't parse, and `limit` has no maximum (7 of 15). Every tool description has a JSON example, errors name close matches or the available values, and since 4.8.0 every admin API error carries a machine-readable `error_code` (13 of 15). Dated release notes for every minor version back to 2.7, with Deployment Changes and Breaking Changes sections, and a SemVer statement in the API overview (15).",
        "checklist": [
          "APIs and MCP servers.",
          "- 25, a machine-readable contract (a public OpenAPI file or similar; for MCP, typed JSON Schema inputs on every tool).\n- 10, llms.txt or Markdown docs served for agents.\n- 0 to 20, descriptions that say what a tool is for, when to use it and when not to, read from the tool definitions in the source or the API reference.\n- 0 to 15, typed inputs with enums, constraints and required fields, and no free-form JSON blobs.\n- 0 to 15, examples and documented error responses.\n- 15, versioning and a public changelog.",
          "Models are read from the API reference, the OpenAPI file, llms.txt, the structured-output and tool-use docs and the model cards. Frameworks from docs a model can follow, typed interfaces, examples and the API reference."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-schema"
      },
      {
        "key": "maintenance",
        "name": "Maintenance \u0026 community",
        "score": 77,
        "maxGain": 2,
        "reason": "v4.8.4 tagged on 2 October 2026, with 4.6.12 and 4.7.11 the same day (30). Six minor releases from 4.3.0 on 6 July to 4.8.0 on 23 September, and 61 stable tags from 4.3.0 to 4.8.4 (20). 153 open issues and 354 open pull requests. Bug reports from early July are still open, and we couldn't see reply times because GitHub's issue search refused our reader (13 of 25). No entry in the official MCP registry, where only third-party Onyx servers are listed. Official clients are `onyx-cli` (tags up to cli/v1.4.4) and a Terraform provider (5 of 15). Dependabot, nightly CVE alert workflows and a merge queue that runs the test suites, with the current pass state unconfirmed (9 of 10).",
        "checklist": [
          "- 0 to 30, time since the last release, or the last published model or API change for a closed service. 30 within 30 days, 20 within 90, 10 within 180, 0 older.\n- 20, at least three releases or dated changelog entries in the last 90 days.\n- 0 to 25, responsiveness. Issues and pull requests answered on GitHub (the open issues and how recent the replies are). For closed services, a public changelog and a support or community channel that answers, 0 to 15.\n- 15, presence in the official MCP registry under a verified namespace (MCP servers), or current official SDKs (APIs and models).\n- 10, package health, current dependencies and CI.",
          "Models are read for deprecation notice periods and model churn rather than release counts."
        ],
        "checklistUrl": "https://www.anchorterminal.com/benchmark/#checklist-maintenance"
      }
    ],
    "provenance": [
      {
        "label": "Domain age",
        "value": "onyx.app, no registry record we could read",
        "points": 0,
        "max": 15
      },
      {
        "label": "security.txt",
        "value": "not found",
        "points": 0,
        "max": 10
      }
    ],
    "deductions": [
      "2026-07-20. GHSA-q62f-rv3h-f822 (critical, CVSS 9.0). Any signed-in user could read, in clear text, other users' live OAuth tokens for per-user MCP servers such as Slack, Atlassian or Notion through GET /api/mcp/servers. Found in an external pentest in May 2026, fixed in 4.0.0 (26 May 2026) and published, so the deduction is reduced, -3 (https://github.com/onyx-dot-app/onyx/security/advisories/GHSA-q62f-rv3h-f822)",
      "2026-04-29 and 2026-07-20. Three moderate IDOR advisories, other users' chat files downloadable through /chat/file/{file_id} (GHSA-vg3h-35f7-7w6r), other users' chat sessions stoppable through /chat/stop-chat-session (GHSA-rw6w-hp62-gc8w) and curators able to change any user group's membership (GHSA-7f48-vgpj-h95m). Fixed and published, -1 (https://github.com/onyx-dot-app/onyx/security/advisories)"
    ],
    "unchecked": [
      "unchecked: the privacy policy and the Onyx Cloud subscription agreement at onyx.app/legal, which render only with JavaScript, so retention, deletion, SLA terms and any subprocessors beyond the docs list are unknown.",
      "unchecked: whether the merge-queue test runs pass, since the Actions page didn't show run status to our reader.",
      "unchecked: issue reply times, since GitHub's issue search refused our reader.",
      "Whether the 99.734 per cent on the status page's frontend check reflects real outages. No incidents are posted.",
      "unchecked: the registration date of onyx.app.",
      "The docs changelog dates 4.8.0 to 18 September 2026, while the tag and the GitHub release are from 23 September."
    ],
    "weaknesses": [
      "GHSA-q62f-rv3h-f822 (critical, CVSS 9.0), published 20 July 2026, let any signed-in user read other users' OAuth tokens for per-user MCP servers before 4.0.0",
      "Telemetry is on by default and documented as anonymous, while its events carry user IDs and Enterprise builds send the first user's email domain",
      "Document search has no result limit or paging, and an unparseable `time_cutoff` is dropped with only a server log line",
      "The self-hosted MCP server is off by default, takes no OAuth and isn't in the official MCP registry",
      "The privacy policy and Cloud agreement render only with JavaScript, and there's no security.txt or bug bounty"
    ],
    "agentNotes": [
      "Get the instance URL from the operator. Cloud is https://cloud.onyx.app/mcp, and a self-hosted server only answers once `MCP_SERVER_ENABLED=true`",
      "Use a token limited to `read:search`. It covers every MCP tool and nothing else",
      "Pass `time_cutoff` as a full ISO 8601 timestamp. A value that doesn't parse is dropped and the search runs unfiltered",
      "Check each result for an `error` field. Failures come back with an empty `results` list, not as tool errors",
      "Set `skip_query_expansion` to true for exact phrases. It skips an LLM call on every search"
    ],
    "requests": [
      {
        "text": "reject bad time_cutoff",
        "reviews": 1
      },
      {
        "text": "return errors as isError",
        "reviews": 1
      },
      {
        "text": "prompt-injection guidance",
        "reviews": 1
      },
      {
        "text": "a telemetry field list",
        "reviews": 1
      }
    ],
    "recheck": "https://www.anchorterminal.com/builders/#disputes"
  },
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-10-04",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.3",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run"
  }
}