Niches · Agent observability & evals
Attack an agent before release and record what held
No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.
What counts as done
A report of attacks tried per rule, each with the trace behind it, replayable to confirm a fix, and a clear statement of which rules were not tested.
How agents ask for it: red-team my agent
, try prompt injection against the endpoint
, what breaks if the tool returns garbage
Does the job, with a limit 2
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| BotGauge E BotGauge AI Inc. (Delaware, United States) | Each run reports Found, Held and Errors per rule with findings, Verify fix replays the exact attack, Rerun errors repeats failures, Not tested is a state, and reports export as JSON, CSV or PDF via the API. The red-team API has no published endpoints and production runs read-only attacks. source |
| LangWatch B Reasoning Engine B.V. (doing business as LangWatch) | RedTeamAgent (crescendo or GOAT) replaces the user simulator for multi-turn attacks in your own CI and scenario redteam-report lists findings, transcripts and severity. Reports are per criterion rather than per rule, and the agent under test is whatever your adapter calls. source |
Doesn't do this job 20
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Agent Tree C No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots" | No attack runner. source |
| agents-tree B No legal entity found. The LICENSE and pyproject.toml name Karel Blavka | No attack runner. source |
| Arize Phoenix BB Arize AI, Inc. | No attack runner or red-team report in the docs index. source |
| Braintrust API + MCP B Braintrust Data, Inc. | No attack runner or red-team report in the docs index. source |
| DeepEval B Confident AI, Inc. | Red teaming lives in the separate DeepTeam package; it is not in the DeepEval docs index. source |
| Galileo API + MCP D Galileo Technologies, Inc. | Prompt injection is a metric, not an attack runner; no red-team report. source |
| Gauge CLI B Gauge | No attack rules or red-team report. source |
| Helicone AI Gateway + MCP E Helicone, Inc. | No attack runner or red-team report in the docs index. source |
| HoneyHive D HoneyHive Inc. | No attack runner or red-team report in the docs index. source |
| Laminar API + MCP C LMNR AI, Inc. | No attack runner or red-team report in the docs index. source |
| Langfuse API + MCP BB Langfuse GmbH (a ClickHouse, Inc. company) | No attack runner, rule set or red-team report anywhere in the docs index. source |
| LangSmith API + MCP BB LangChain, Inc. | No attack runner or red-team report in the observability docs index. source |
| MLflow Tracing C MLflow Project, a Series of LF Projects, LLC | Conversation simulation mentions stress-testing with personas, but there is no attack rule set, finding report or replay. source |
| Navige E Navige Ltd | No attack runner. source |
| Opik API + MCP BB Comet ML, Inc. | Guardrails and a prompt injection model exist, but no attack runner, rules or red-team report. source |
| Overmind C Overmind Technology Ltd (England and Wales, company 16585996) | No attack runner or red-team report in the docs index. source |
| Prefactor B Prefactor Pty Ltd | No attack runner. source |
| Pydantic Logfire B Pydantic Services Inc. | No attack runner or red-team report in the docs index. source |
| Respan API + MCP BB Keywords AI Inc. (d/b/a Respan) | No attack runner or red-team report in the docs index. source |
| W&B Weave B Weights and Biases, LLC | No attack runner or red-team report in the Weave docs. source |
What stops agents finishing
- attacks can call real tools in production
- red-team API has no published endpoints
- docs, authenticated servers and APIs need the hosted tier and credits
Around it
Before: agent-observability.instrument-app. After: agent-observability.agent-trajectory-eval.
Where this comes from
Weight 2 of 100 in Agent observability & evals: No first-hand account of agent red-teaming found; the one BotGauge comment is about its earlier QA product and the one Gauge issue is a CLI flag. Weight set low and rests on vendor pages (BotGauge, DeepEval, Gauge).
The same as data: /api/v1/use-cases/agent-observability.redteam-before-release.json, or the MCP tool get_use_case with id agent-observability.redteam-before-release. Other jobs in this category: /api/v1/niches?category=agent-observability.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.