Niches · Agent observability & evals

Judge whether the agent took the right steps, not only gave the right answer

Two or three companies' docs show a full fit.

2companies, full fit
10companies, fit with a limit
20companies carry the tag
5share of the category's work
6first-hand accounts

What counts as done

A score per trajectory that names the wrong or missing tool call or the turn where the run went off, reproducible on the same trace, with a clear result when the agent answered without calling any tool.

How agents ask for it: did it call the right tool, score the whole session not each turn, simulate a user and see if it books the flight, where did these two runs diverge

Does the job 2

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
LangWatch B
Reasoning Engine B.V. (doing business as LangWatch)
Scenario runs a simulated user against the agent in pytest or vitest and a judge passes or fails each criterion with reasoning; suite evaluators read tool_calls.<tool>.input and output or every span, and a required evaluator fails the run. source
Opik API + MCP BB
Comet ML, Inc.
TrajectoryAccuracy scores a ReAct trajectory with a reason, the agent trajectory guide builds tool-call test datasets, and multi-turn evaluation uses an LLM user simulator; issue 2529 asked for precisely this metric. source

Does the job, with a limit 10

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Arize Phoenix BB
Arize AI, Inc.
Tool Selection and Tool Invocation evaluators score LLM spans and flag hallucinated or unnecessary tools; ATIF trajectories can be imported. No user simulator and no whole-run divergence score. source
Braintrust API + MCP B
Braintrust Data, Inc.
Trace-scoped scorers get trace.getSpans() and {{thread}} so a judge can read tool calls, and the agents guide covers tool choice and arguments. No built-in trajectory metric and no user simulator. source
DeepEval B
Confident AI, Inc.
Trajectory-based evals score the ordered chain of tool calls and handoffs, ToolCorrectness can require ordering and arguments, component metrics attach to spans and ConversationSimulator plays a user. ToolCorrectness scored 20 per cent when no tool was called (dev.to) and no divergence metric exists (issue 3019). source
Galileo API + MCP D
Galileo Technologies, Inc.
Agent Flow validates a session's trajectory against natural-language tests including expected tool or agent calls, and Tool Selection Quality judges tool and argument choice per span. No user simulator. source
Gauge CLI B
Gauge
Agent Experience judges whether a real coding agent completed a task with your product, with pass criteria, session evidence and recurring-failure grouping. The agent under test is the harness using your product, not your own agent. source
HoneyHive D
HoneyHive Inc.
The Trajectory view plots spans by step to spot loops and stuck steps, and conversation evaluator templates score multi-turn sessions. No tool-choice metric or user simulator in the pages read. source
Laminar API + MCP C
LMNR AI, Inc.
Agent evaluations run the real app per trial and a verifier scores the state, trajectory and artefacts, with the trajectory attached as LLM and TOOL spans or imported in ATIF; a tool-call evaluation guide exists. No user simulator. source
LangSmith API + MCP BB
LangChain, Inc.
Multi-turn online evaluators measure the trajectory of tool calls across a thread, and datasets can hold expected trajectories. No user simulator and no named divergence metric in the pages read. source
MLflow Tracing C
MLflow Project, a Series of LF Projects, LLC
ToolCallCorrectness and ToolCallEfficiency judge TOOL spans in a trace, scorers can parse the whole trace for trajectory and routing, and ConversationSimulator (experimental, 3.10) plays a user. No divergence-between-runs metric. source
Overmind C
Overmind Technology Ltd (England and Wales, company 16585996)
Stock checks cover tool selection, tool arguments and goal state, and live trajectory and plan-drift scoring is mentioned with bounded attempts. No user simulator. source

Couldn't check 1

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
Pydantic Logfire B
Pydantic Services Inc.
pydantic-evals may score trajectories, but its docs were not opened and the Logfire pages read show no trajectory metric or user simulator.

Doesn't do this job 9

Listings in the category that do something else, with the reason.

ListingWhy
Agent Tree C
No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots"
No scoring. source
agents-tree B
No legal entity found. The LICENSE and pyproject.toml name Karel Blavka
No scoring. source
BotGauge E
BotGauge AI Inc. (Delaware, United States)
Built-in judges score spans, traces or sessions against criteria; no trajectory metric or simulator in the pages read. source
Helicone AI Gateway + MCP E
Helicone, Inc.
No trajectory metric, tool-call judge or simulator in the docs index. source
Langfuse API + MCP BB
Langfuse GmbH (a ClickHouse, Inc. company)
Evaluators see one observation and do not load sibling or child observations; multi-span evaluation is promised for the new data model. No trajectory metric or user simulator in the docs. source
Navige E
Navige Ltd
No trajectory scoring. source
Prefactor B
Prefactor Pty Ltd
No trajectory scoring or simulator. source
Respan API + MCP BB
Keywords AI Inc. (d/b/a Respan)
No trajectory metric, tool-choice judge or simulator in the docs index. source
W&B Weave B
Weights and Biases, LLC
Scorers are custom functions; no trajectory metric, tool-choice judge or user simulator appears in the pages read. source

What stops agents finishing

  • metrics score one run against a static expectation only
  • no simulation of a user
  • tool calls not in the trace so nothing to judge

Around it

Before: agent-observability.debug-failed-run. After: agent-observability.offline-eval-regression.

Where this comes from

Weight 5 of 100 in Agent observability & evals: 6 of 86 accounts: the DeepEval dev.to post (ToolCorrectness at 20 per cent), the Opik trajectory metric request, the DeepEval divergence proposal, the Cekura thread's wrong-tool support agent, the Lucidic thread on session-level evals, Phoenix multi-hop evals; LangWatch Scenario appears only in its staff's posts

The same as data: /api/v1/use-cases/agent-observability.agent-trajectory-eval.json, or the MCP tool get_use_case with id agent-observability.agent-trajectory-eval. Other jobs in this category: /api/v1/niches?category=agent-observability.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.