Niches · Agent observability & evals

Attack an agent before release and record what held

No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.

0companies, full fit
2companies, fit with a limit
22companies carry the tag
2share of the category's work
2first-hand accounts

What counts as done

A report of attacks tried per rule, each with the trace behind it, replayable to confirm a fix, and a clear statement of which rules were not tested.

How agents ask for it: red-team my agent, try prompt injection against the endpoint, what breaks if the tool returns garbage

Does the job, with a limit 2

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
BotGauge E
BotGauge AI Inc. (Delaware, United States)
Each run reports Found, Held and Errors per rule with findings, Verify fix replays the exact attack, Rerun errors repeats failures, Not tested is a state, and reports export as JSON, CSV or PDF via the API. The red-team API has no published endpoints and production runs read-only attacks. source
LangWatch B
Reasoning Engine B.V. (doing business as LangWatch)
RedTeamAgent (crescendo or GOAT) replaces the user simulator for multi-turn attacks in your own CI and scenario redteam-report lists findings, transcripts and severity. Reports are per criterion rather than per rule, and the agent under test is whatever your adapter calls. source

Doesn't do this job 20

Listings in the category that do something else, with the reason.

ListingWhy
Agent Tree C
No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots"
No attack runner. source
agents-tree B
No legal entity found. The LICENSE and pyproject.toml name Karel Blavka
No attack runner. source
Arize Phoenix BB
Arize AI, Inc.
No attack runner or red-team report in the docs index. source
Braintrust API + MCP B
Braintrust Data, Inc.
No attack runner or red-team report in the docs index. source
DeepEval B
Confident AI, Inc.
Red teaming lives in the separate DeepTeam package; it is not in the DeepEval docs index. source
Galileo API + MCP D
Galileo Technologies, Inc.
Prompt injection is a metric, not an attack runner; no red-team report. source
Gauge CLI B
Gauge
No attack rules or red-team report. source
Helicone AI Gateway + MCP E
Helicone, Inc.
No attack runner or red-team report in the docs index. source
HoneyHive D
HoneyHive Inc.
No attack runner or red-team report in the docs index. source
Laminar API + MCP C
LMNR AI, Inc.
No attack runner or red-team report in the docs index. source
Langfuse API + MCP BB
Langfuse GmbH (a ClickHouse, Inc. company)
No attack runner, rule set or red-team report anywhere in the docs index. source
LangSmith API + MCP BB
LangChain, Inc.
No attack runner or red-team report in the observability docs index. source
MLflow Tracing C
MLflow Project, a Series of LF Projects, LLC
Conversation simulation mentions stress-testing with personas, but there is no attack rule set, finding report or replay. source
Navige E
Navige Ltd
No attack runner. source
Opik API + MCP BB
Comet ML, Inc.
Guardrails and a prompt injection model exist, but no attack runner, rules or red-team report. source
Overmind C
Overmind Technology Ltd (England and Wales, company 16585996)
No attack runner or red-team report in the docs index. source
Prefactor B
Prefactor Pty Ltd
No attack runner. source
Pydantic Logfire B
Pydantic Services Inc.
No attack runner or red-team report in the docs index. source
Respan API + MCP BB
Keywords AI Inc. (d/b/a Respan)
No attack runner or red-team report in the docs index. source
W&B Weave B
Weights and Biases, LLC
No attack runner or red-team report in the Weave docs. source

What stops agents finishing

  • attacks can call real tools in production
  • red-team API has no published endpoints
  • docs, authenticated servers and APIs need the hosted tier and credits

Around it

Before: agent-observability.instrument-app. After: agent-observability.agent-trajectory-eval.

Where this comes from

Weight 2 of 100 in Agent observability & evals: No first-hand account of agent red-teaming found; the one BotGauge comment is about its earlier QA product and the one Gauge issue is a CLI flag. Weight set low and rests on vendor pages (BotGauge, DeepEval, Gauge).

The same as data: /api/v1/use-cases/agent-observability.redteam-before-release.json, or the MCP tool get_use_case with id agent-observability.redteam-before-release. Other jobs in this category: /api/v1/niches?category=agent-observability.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.