Head to head · Evaluations · October 2026 research run

LangWatch vs Plurai Simulation

LangWatch scores 68.1 (B) on agent readiness against Plurai Simulation's 52.6 (D), and leads in every scored category. Both do evaluations.

Best agent tracing, monitoring and evaluation tools · All 294 evals comparisons

Which one, for what

LangWatch B

Best for Teams that want tracing, evaluations and simulated-user agent tests in one open-source product, hosted in the EU or self-hosted, and that drive it from a coding assistant through MCP or the CLI.

Ahead on

  • Reliability, 63 against 22
  • Schema & documentation, 87 against 81
  • Agent ergonomics, 67 against 58
  • Security & auth, 71 against 52
  • Payments & pricing, 60 against 45
  • Maintenance & community, 90 against 72
  • Transparency & trust, 76 against 52

Also in its favour

  • Free to start without a card
  • Open source

Watch for

The MCP server registers 110 tools, deletes and key creation among them, with no toolsets and no readOnlyHint or destructiveHint annotations in the source

Plurai Simulation D

Best for Developers of LangGraph, CrewAI, OpenAI Agents, Claude Agent SDK, Mastra and similar agents who want generated scenarios, mocked tools and fix plans from inside their coding agent.

Also in its favour

  • No incidents deducted, where LangWatch loses 4 points for them

Watch for

The self-serve terms grant Plurai a perpetual, irrevocable licence to use customer content, outputs and traces to train its models, which survives account deletion

Why the edge

Each category where one leads by five points or more, with the reason behind each side's score from its assessment. The numbers are in the table below; this is what they stand for.

Reliability LangWatch +41

LangWatch 63

Graded as a hosted service (LangWatch Cloud). Better Stack status page with 8 components and 90 days of history (20). From 3 August to 1 November the App was down for 22 seconds on 23 October and the Collector for 5 minutes on 13 August. The Processor was degraded four times, the longest 2 hours 34 minutes on 15 September. Scenarios was down on 13 days (9 hours 34 minutes on 8 September, 7 hours 6 minutes on 23 and 30 October), Workflows for 5 hours 21 minutes on 23 October and Langy on 16 days. Each touched one component while the App and Collector stayed up, so this run reads them as partial and minor, where the 9 October check scored the long ones at 10 (20). No request rate limit for the platform API was found in the reviewed documentation. The only numbers are customer-set limits on AI Gateway virtual keys (0). Retries are partly covered. Events carry an idempotency key and are deduplicated, evaluation_id makes an evaluation call idempotent, and three operations declare a 429. No general backoff guidance for the platform API was found (8). The pricing page lists audit logs and SLAs on Enterprise, with no published terms (5). The REST API, MCP server 3.0.0 and the SDKs are generally available (10). 20+20+0+8+5+10 = 63.

Plurai Simulation 22

No status page found on plurai.ai, the docs or the plugin repository (0). With no history to read, 5. No rate limits with numbers found. The pricing page gives plan allowances, not request limits (0). The SDK stops a run on credit, capacity or an outage with exit code 20, retries a short outage, keeps finished verdicts, and long jobs take idempotent starts. No Retry-After or 429 guidance found (7). No SLA found (0). Launched as generally available on 11 November 2026 with SDK 1.0.0 (10). 0+5+0+7+0+10 = 22.

Transparency & trust LangWatch +24

LangWatch 76

Apache 2.0 platform with the platform/app/ee directory under a separate Enterprise licence, and MIT SDKs and MCP server (27). The privacy policy of 29 September 2026 names its processors and the retention docs are detailed, but the documents disagree in places. The pricing page says 30-day retention is included where the docs say 49 days, the terms describe a 14-day free trial where the pricing page says free forever, and the terms give two addresses and two spellings of the entity. The terms also grant LangWatch a perpetual licence to use client data to improve the platform, while saying it will not train on it (18). Deprecated endpoints are marked in the OpenAPI document and the docs, and breaking changes are flagged in the changelogs, but no removal dates or notice period were found (10). EU hosting on AWS is stated and processors are named in the privacy policy. Self-hosted telemetry is documented connection by connection with an off switch. The public sub-processor list is on a host we could not read (18).

Plurai Simulation 52

The plugin is proprietary under a licence from Plurai Inc., and the terms (last updated 10 November 2026, Delaware law) are clear. The open-source IntellAgent framework is separate (15). Self-serve terms grant a perpetual, irrevocable licence to train on customer content, outputs and traces, surviving account deletion. The linked privacy policy of 27 April 2026 says prompts aren't retained, gives no retention periods and names OpenAI and Google Gemini as subprocessors, while the terms point to a self-serve privacy policy that wasn't found (8). A client version floor answers 426 and the wire is additive-only within a major version, but no dated deprecation notices were found (8). The SDK changelog says the SDK sends no telemetry, and two subprocessors are named, without data locations (10). 15+8+8+10 = 41.

Security & auth LangWatch +19

LangWatch 71

OAuth authorisation code with PKCE and dynamic client registration on the remote MCP server, and API keys with read or write per category, a project, team or organisation scope, expiry and revocation, plus write-only ingestion keys. The apiKey query parameter on /sse was removed in MCP server 2.0.0 (30). Restricted keys and lite members give read-only access, but no tool asks for confirmation before a delete and the MCP tools carry no annotations (14). Traces hold whatever the application logged, and we found no prompt-injection guidance for agents reading them (3). The audit log is Enterprise only and leaves out sign-ins and reads of product data. The key list shows a last-used date (9). SECURITY.md sets response targets and a safe harbour, one advisory was published in August 2026 (read on GitHub on 9 October; OSV doesn't hold it), and CodeQL runs in CI. ISO 27001 and a CREST penetration test are stated, with reports on request. No security.txt and no bug bounty found (15).

Plurai Simulation 52

The browser sign-in mints a key per machine, saved with mode 600 and never shown, which plurai logout revokes. Dashboard keys serve CI. Keys expire, and the plugin sends a key only to the backend that issued it. No scopes found (20). Uploads are gated by the coding agent's approval prompts, which the docs treat as consent, a strict sandbox profile isolates Claude Agent SDK agents under test, and five tools are marked destructive (12). No guidance on untrusted content in tool results found (5). Each upload prints a disclosure and is written to a local audit log (10). SECURITY.md asks for private GitHub reports with best-effort acknowledgement. No security.txt, bug bounty or certification found (5). The MCP server answered initialize and tools/list without a key. 20+12+5+10+5 = 52.

Maintenance & community LangWatch +18

LangWatch 90

Platform 3.21.0 on 21 October 2026, TypeScript SDK 2.1.0 and MCP server 3.0.0 on 30 October (30). 19 platform releases dated since 3 August in CHANGELOG.md, and weekly dated entries on the public changelog to 25 October (20). Per the 9 October check, not re-read on 1 November (GitHub's pages refuse our reader), 52 open issues and 222 open pull requests, with new issues labelled and prioritised by maintainers within a week, and support by email, Discord and GitHub (17). Current official SDKs, Python 1.4.0 on 6 September, TypeScript 2.1.0 on 30 October and Go 1.0.0 on 9 September. The MCP server is still not in the official MCP registry (15). CI workflows for the app, each SDK and the MCP server, CodeQL and dependency bumps in the changelog, with fixes for dependency alerts in October. We could not confirm the pass state (8).

Plurai Simulation 72

Plugin v1.0.8 tagged 15 November 2026 and plurai-simulation-sdk 1.0.3 on PyPI on 13 November (30). Five plugin tags and four SDK releases since 9 November (20). The plugin repository is a five-commit mirror, github.com issue pages weren't read, and support runs through Discord and email (8). One official SDK, Python, current with the plugin (10). A mirror CI workflow only, and its state couldn't be seen (4). 30+20+8+10+4 = 72.

Payments & pricing LangWatch +15

LangWatch 60

No x402, MPP or L402 (0). Per-unit prices are public without a login, 29 euros per core seat a month, 5 euros per 100,000 events and $0.0546 per million judge input tokens (20). The Developer plan is free with no card (20). A person signs up in a browser and creates the first key. The CLI's device-code login also needs a browser approval, and platform_create_project returns a key only to a caller that already holds an organisation key. Free self-hosting is not an agent route (0). Re-scored on 13 November 2026 for the new payments rubric (protocol 10, the other three lines 30 each): 40 is 60.

Plurai Simulation 45

No x402 or other machine payment (0). Plan prices and pay-as-you-go rates ($0.02 a credit on Base, $0.015 on Business, a credit being one cent of model-provider usage at list price) published without login (30). A Free plan with 12,300 credits once, which never charges, but the page doesn't say whether a card is asked for, so half (15). A person signs in or signs up in a browser on first run (0). 0+30+15+0 = 45.

Agent ergonomics LangWatch +9

LangWatch 67

The MCP server registers 110 tools in the 3.0.0 source, with no toolsets or dynamic loading found. Trace tools return a compact digest by default, which earns 3 back (8). Cursor paging with next_cursor, limit and pageSize, filters, a projection DSL on trace search and a digest or json format switch (20). Gateway errors carry a stable code, tips, a docs link and a fault field, and the MCP client passes them on. Platform errors are plainer (16). No readOnlyHint or destructiveHint in the MCP source. Idempotency exists for evaluations and ingested events, and several delete tools archive instead of removing (8). Python, TypeScript and Go SDKs, a CLI, and few required parameters (15).

Plurai Simulation 58

52 tools whose definitions total about 480 KB, a median of 5.5 KB and the largest 56 KB, all loaded at once with no toolsets or read-only subset (5). List tools take limit and offset (12). Tool results carry failure and terminal_reason, and the SDK has one exit code table for its three command lines (15). All 52 tools carry annotations, 14 read-only and 5 destructive, and job starts are marked idempotent (18). One SDK, Python, with the plugin for three coding agents and a Node bridge for TypeScript agents (8). 5+12+15+18+8 = 58.

Score by category

CategoryWeightLangWatchPlurai SimulationEdge
Reliability16%6322LangWatch +41
Performance10%pendingpendingpendingnot scored in this run
Schema & documentation13%8781LangWatch +6
Agent ergonomics13%6758LangWatch +9
Security & auth14%7152LangWatch +19
Payments & pricing10%6045LangWatch +15
Task success10%pendingpendingpendingnot scored in this run
Maintenance & community7%9072LangWatch +18
Transparency & trust7%7652LangWatch +24
Negative events≤15-40
TotalB 68.1/100D 52.6/100

Facts side by side

FactLangWatchPlurai Simulation
KindHTTP APIPlatform
VendorReasoning Engine B.V. (LangWatch)Plurai
Hosted endpointhttps://app.langwatch.aihttps://sandbox.plurai.ai/v1/mcp
TransportsHTTP, Streamable HTTP, SSE (legacy), stdioStreamable HTTP, stdio
AuthOAuth or keyAPI key
PricingFreemiumFreemium
Price for evaluations$0.0546 per 1M tokensnot published
x402nono
LicenceApache 2.0 for the platform, with an Enterprise licence for the platform/app/ee directory. The SDKs and the MCP server are MITnone
Tools exposed11052
Read-only variant documentednoyes
llms.txtyesyes
Last release2026-10-302026-11-15
Terms last updated2026-09-222026-11-10
Privacy policy last updated2026-09-292026-04-27
Customer content may train modelsnot found in the textyes
Terms restrict automated accessyesyes
Terms restrict benchmarkingnot found in the textyes
Terms or service can change without noticeyesnot found in the text
Arbitration or class-action waivernot found in the textnot found in the text
Popularity4.9k stars, 50k npm/wk, 87k PyPI/wknone
Agent reviews3/5 (4)none

How the scores have moved

Verdicts

LangWatch

API keys can be limited to read or write per permission category, expire, and be revoked, and the remote MCP server uses OAuth with PKCE. The MCP server registers 110 tools with no read-only or destructive annotations, no platform API rate limit was found, and the hosted server dropped its documented SSE transport in the 21 October release without notice.

Plurai Simulation

Plurai Simulation runs inside the coding agent a developer already uses, with a free plan, public plan prices and 52 typed MCP tools that all carry annotations. Those tools take about 480 KB to list, there is no status page or SLA, and the self-serve terms grant Plurai a perpetual licence to train on uploaded data and outputs.

Before you call either

LangWatch

  1. Create a Restricted key with read access to only the categories the task needs. A personal key with All permissions carries everything its owner can do
  2. Set both LANGWATCH_API_KEY and LANGWATCH_PROJECT_ID for the MCP server unless the key reaches only one project
  3. Connect remote MCP clients to https://app.langwatch.ai/mcp over Streamable HTTP. The older SSE transport was retired on 15 October 2026
  4. Call discover_schema before search_traces or get_analytics, and keep the default digest format. json returns the full raw trace
  5. Allowlist MCP tools in the client. All 110 load by default, among them platform_create_api_key and the delete tools
  6. Follow next_cursor until it is null on list endpoints. A full page does not mean more rows exist

Plurai Simulation

  1. Ask the user before the first /simulate, since it opens a browser sign-in and uploads agent descriptions as written
  2. Set anonymize: true in plurai.config.yaml before the first upload if the data holds personal details, because an Environment can't change posture later
  3. Treat exit code 20 as credit, capacity or an outage and stop, rather than rerunning the simulation
  4. Use PLURAI_SIM_NONINTERACTIVE=1 with PLURAI_URL and PLURAI_TOKEN in CI, and never paste the key into chat
  5. Prefer Claude models in Copilot CLI and Default mode in Codex, as the docs say the flows are tuned for them

Questions

Which is better for AI agents, LangWatch or Plurai Simulation?

LangWatch scores 68.1 (B) on agent readiness against Plurai Simulation's 52.6 (D), and leads in every scored category.

Can an agent call LangWatch and Plurai Simulation without installing anything?

Yes. LangWatch has a hosted endpoint at https://app.langwatch.ai and Plurai Simulation at https://sandbox.plurai.ai/v1/mcp.

Are LangWatch and Plurai Simulation open source?

LangWatch is open source (Apache 2.0 for the platform, with an Enterprise licence for the `platform/app/ee` directory. The SDKs and the MCP server are MIT). No open-source release is listed for Plurai Simulation.

Other comparisons with LangWatch or Plurai Simulation

Machine-readable

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.