Niches · Agent observability & evals
Judge whether the agent took the right steps, not only gave the right answer
Two or three companies' docs show a full fit.
What counts as done
A score per trajectory that names the wrong or missing tool call or the turn where the run went off, reproducible on the same trace, with a clear result when the agent answered without calling any tool.
How agents ask for it: did it call the right tool
, score the whole session not each turn
, simulate a user and see if it books the flight
, where did these two runs diverge
Does the job 2
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| LangWatch B Reasoning Engine B.V. (doing business as LangWatch) | Scenario runs a simulated user against the agent in pytest or vitest and a judge passes or fails each criterion with reasoning; suite evaluators read tool_calls.<tool>.input and output or every span, and a required evaluator fails the run. source |
| Opik API + MCP BB Comet ML, Inc. | TrajectoryAccuracy scores a ReAct trajectory with a reason, the agent trajectory guide builds tool-call test datasets, and multi-turn evaluation uses an LLM user simulator; issue 2529 asked for precisely this metric. source |
Does the job, with a limit 10
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Arize Phoenix BB Arize AI, Inc. | Tool Selection and Tool Invocation evaluators score LLM spans and flag hallucinated or unnecessary tools; ATIF trajectories can be imported. No user simulator and no whole-run divergence score. source |
| Braintrust API + MCP B Braintrust Data, Inc. | Trace-scoped scorers get trace.getSpans() and {{thread}} so a judge can read tool calls, and the agents guide covers tool choice and arguments. No built-in trajectory metric and no user simulator. source |
| DeepEval B Confident AI, Inc. | Trajectory-based evals score the ordered chain of tool calls and handoffs, ToolCorrectness can require ordering and arguments, component metrics attach to spans and ConversationSimulator plays a user. ToolCorrectness scored 20 per cent when no tool was called (dev.to) and no divergence metric exists (issue 3019). source |
| Galileo API + MCP D Galileo Technologies, Inc. | Agent Flow validates a session's trajectory against natural-language tests including expected tool or agent calls, and Tool Selection Quality judges tool and argument choice per span. No user simulator. source |
| Gauge CLI B Gauge | Agent Experience judges whether a real coding agent completed a task with your product, with pass criteria, session evidence and recurring-failure grouping. The agent under test is the harness using your product, not your own agent. source |
| HoneyHive D HoneyHive Inc. | The Trajectory view plots spans by step to spot loops and stuck steps, and conversation evaluator templates score multi-turn sessions. No tool-choice metric or user simulator in the pages read. source |
| Laminar API + MCP C LMNR AI, Inc. | Agent evaluations run the real app per trial and a verifier scores the state, trajectory and artefacts, with the trajectory attached as LLM and TOOL spans or imported in ATIF; a tool-call evaluation guide exists. No user simulator. source |
| LangSmith API + MCP BB LangChain, Inc. | Multi-turn online evaluators measure the trajectory of tool calls across a thread, and datasets can hold expected trajectories. No user simulator and no named divergence metric in the pages read. source |
| MLflow Tracing C MLflow Project, a Series of LF Projects, LLC | ToolCallCorrectness and ToolCallEfficiency judge TOOL spans in a trace, scorers can parse the whole trace for trajectory and routing, and ConversationSimulator (experimental, 3.10) plays a user. No divergence-between-runs metric. source |
| Overmind C Overmind Technology Ltd (England and Wales, company 16585996) | Stock checks cover tool selection, tool arguments and goal state, and live trajectory and plan-drift scoring is mentioned with bounded attempts. No user simulator. source |
Couldn't check 1
The docs were closed to us, or silent. These carry no mark either way.
| Listing | Why |
|---|---|
| Pydantic Logfire B Pydantic Services Inc. | pydantic-evals may score trajectories, but its docs were not opened and the Logfire pages read show no trajectory metric or user simulator. |
Doesn't do this job 9
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Agent Tree C No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots" | No scoring. source |
| agents-tree B No legal entity found. The LICENSE and pyproject.toml name Karel Blavka | No scoring. source |
| BotGauge E BotGauge AI Inc. (Delaware, United States) | Built-in judges score spans, traces or sessions against criteria; no trajectory metric or simulator in the pages read. source |
| Helicone AI Gateway + MCP E Helicone, Inc. | No trajectory metric, tool-call judge or simulator in the docs index. source |
| Langfuse API + MCP BB Langfuse GmbH (a ClickHouse, Inc. company) | Evaluators see one observation and do not load sibling or child observations; multi-span evaluation is promised for the new data model. No trajectory metric or user simulator in the docs. source |
| Navige E Navige Ltd | No trajectory scoring. source |
| Prefactor B Prefactor Pty Ltd | No trajectory scoring or simulator. source |
| Respan API + MCP BB Keywords AI Inc. (d/b/a Respan) | No trajectory metric, tool-choice judge or simulator in the docs index. source |
| W&B Weave B Weights and Biases, LLC | Scorers are custom functions; no trajectory metric, tool-choice judge or user simulator appears in the pages read. source |
What stops agents finishing
- metrics score one run against a static expectation only
- no simulation of a user
- tool calls not in the trace so nothing to judge
Around it
Before: agent-observability.debug-failed-run. After: agent-observability.offline-eval-regression.
Where this comes from
Weight 5 of 100 in Agent observability & evals: 6 of 86 accounts: the DeepEval dev.to post (ToolCorrectness at 20 per cent), the Opik trajectory metric request, the DeepEval divergence proposal, the Cekura thread's wrong-tool support agent, the Lucidic thread on session-level evals, Phoenix multi-hop evals; LangWatch Scenario appears only in its staff's posts
The same as data: /api/v1/use-cases/agent-observability.agent-trajectory-eval.json, or the MCP tool get_use_case with id agent-observability.agent-trajectory-eval. Other jobs in this category: /api/v1/niches?category=agent-observability.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.