# Judge whether the agent took the right steps, not only gave the right answer

> Judge whether the agent took the right steps, not only gave the right answer: 2 companies fully do it, 10 with a limit, 20 carry the tag. Every listing in Agent observability & evals with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/
- Markdown: https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/index.md (~1,800 tokens)
- Slim: https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/index.min.md (~380 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

Two or three companies' docs show a full fit.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 2 | 10 | 20 | 5 | 6 |

## What counts as done

A score per trajectory that names the wrong or missing tool call or the turn where the run went off, reproducible on the same trace, with a clear result when the agent answered without calling any tool.

How agents ask for it: did it call the right tool; score the whole session not each turn; simulate a user and see if it books the flight; where did these two runs diverge

## Does the job

The docs show it meets the test above, and nothing we read says otherwise.

- **LangWatch (Reasoning Engine B.V. (doing business as LangWatch))**: Scenario runs a simulated user against the agent in pytest or vitest and a judge passes or fails each criterion with reasoning; suite evaluators read tool_calls.<tool>.input and output or every span, and a required evaluator fails the run.
- **Opik API + MCP (Comet ML, Inc.)**: TrajectoryAccuracy scores a ReAct trajectory with a reason, the agent trajectory guide builds tool-call test datasets, and multi-turn evaluation uses an LLM user simulator; issue 2529 asked for precisely this metric.

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Arize Phoenix (Arize AI, Inc.)**: Tool Selection and Tool Invocation evaluators score LLM spans and flag hallucinated or unnecessary tools; ATIF trajectories can be imported. No user simulator and no whole-run divergence score.
- **Braintrust API + MCP (Braintrust Data, Inc.)**: Trace-scoped scorers get trace.getSpans() and {{thread}} so a judge can read tool calls, and the agents guide covers tool choice and arguments. No built-in trajectory metric and no user simulator.
- **DeepEval (Confident AI, Inc.)**: Trajectory-based evals score the ordered chain of tool calls and handoffs, ToolCorrectness can require ordering and arguments, component metrics attach to spans and ConversationSimulator plays a user. ToolCorrectness scored 20 per cent when no tool was called (dev.to) and no divergence metric exists (issue 3019).
- **Galileo API + MCP (Galileo Technologies, Inc.)**: Agent Flow validates a session's trajectory against natural-language tests including expected tool or agent calls, and Tool Selection Quality judges tool and argument choice per span. No user simulator.
- **Gauge CLI (Gauge)**: Agent Experience judges whether a real coding agent completed a task with your product, with pass criteria, session evidence and recurring-failure grouping. The agent under test is the harness using your product, not your own agent.
- **HoneyHive (HoneyHive Inc.)**: The Trajectory view plots spans by step to spot loops and stuck steps, and conversation evaluator templates score multi-turn sessions. No tool-choice metric or user simulator in the pages read.
- **Laminar API + MCP (LMNR AI, Inc.)**: Agent evaluations run the real app per trial and a verifier scores the state, trajectory and artefacts, with the trajectory attached as LLM and TOOL spans or imported in ATIF; a tool-call evaluation guide exists. No user simulator.
- **LangSmith API + MCP (LangChain, Inc.)**: Multi-turn online evaluators measure the trajectory of tool calls across a thread, and datasets can hold expected trajectories. No user simulator and no named divergence metric in the pages read.
- **MLflow Tracing (MLflow Project, a Series of LF Projects, LLC)**: ToolCallCorrectness and ToolCallEfficiency judge TOOL spans in a trace, scorers can parse the whole trace for trajectory and routing, and ConversationSimulator (experimental, 3.10) plays a user. No divergence-between-runs metric.
- **Overmind (Overmind Technology Ltd (England and Wales, company 16585996))**: Stock checks cover tool selection, tool arguments and goal state, and live trajectory and plan-drift scoring is mentioned with bounded attempts. No user simulator.

## Couldn't check

The docs were closed to us, or silent. These carry no mark either way.

- **Pydantic Logfire (Pydantic Services Inc.)**: pydantic-evals may score trajectories, but its docs were not opened and the Logfire pages read show no trajectory metric or user simulator.

- not this job: Agent Tree (No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots"): No scoring.
- not this job: agents-tree (No legal entity found. The LICENSE and pyproject.toml name Karel Blavka): No scoring.
- not this job: BotGauge (BotGauge AI Inc. (Delaware, United States)): Built-in judges score spans, traces or sessions against criteria; no trajectory metric or simulator in the pages read.
- not this job: Helicone AI Gateway + MCP (Helicone, Inc.): No trajectory metric, tool-call judge or simulator in the docs index.
- not this job: Langfuse API + MCP (Langfuse GmbH (a ClickHouse, Inc. company)): Evaluators see one observation and do not load sibling or child observations; multi-span evaluation is promised for the new data model. No trajectory metric or user simulator in the docs.
- not this job: Navige (Navige Ltd): No trajectory scoring.
- not this job: Prefactor (Prefactor Pty Ltd): No trajectory scoring or simulator.
- not this job: Respan API + MCP (Keywords AI Inc. (d/b/a Respan)): No trajectory metric, tool-choice judge or simulator in the docs index.
- not this job: W&B Weave (Weights and Biases, LLC): Scorers are custom functions; no trajectory metric, tool-choice judge or user simulator appears in the pages read.

## What stops agents finishing

- metrics score one run against a static expectation only
- no simulation of a user
- tool calls not in the trace so nothing to judge

## Around it

Before: agent-observability.debug-failed-run. After: agent-observability.offline-eval-regression.

## Where this comes from

Weight 5 of 100 in Agent observability & evals: 6 of 86 accounts: the DeepEval dev.to post (ToolCorrectness at 20 per cent), the Opik trajectory metric request, the DeepEval divergence proposal, the Cekura thread's wrong-tool support agent, the Lucidic thread on session-level evals, Phoenix multi-hop evals; LangWatch Scenario appears only in its staff's posts

As data: https://www.anchorterminal.com/api/v1/use-cases/agent-observability.agent-trajectory-eval.json, or the MCP tool get_use_case with id agent-observability.agent-trajectory-eval. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
