{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/",
    "json": "https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/index.min.md"
  },
  "markdown": "Two or three companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 2 | 10 | 20 | 5 | 6 |\n\n## What counts as done\n\nA score per trajectory that names the wrong or missing tool call or the turn where the run went off, reproducible on the same trace, with a clear result when the agent answered without calling any tool.\n\nHow agents ask for it: did it call the right tool; score the whole session not each turn; simulate a user and see if it books the flight; where did these two runs diverge\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **LangWatch (Reasoning Engine B.V. (doing business as LangWatch))**: Scenario runs a simulated user against the agent in pytest or vitest and a judge passes or fails each criterion with reasoning; suite evaluators read tool_calls.\u003ctool\u003e.input and output or every span, and a required evaluator fails the run.\n- **Opik API + MCP (Comet ML, Inc.)**: TrajectoryAccuracy scores a ReAct trajectory with a reason, the agent trajectory guide builds tool-call test datasets, and multi-turn evaluation uses an LLM user simulator; issue 2529 asked for precisely this metric.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Arize Phoenix (Arize AI, Inc.)**: Tool Selection and Tool Invocation evaluators score LLM spans and flag hallucinated or unnecessary tools; ATIF trajectories can be imported. No user simulator and no whole-run divergence score.\n- **Braintrust API + MCP (Braintrust Data, Inc.)**: Trace-scoped scorers get trace.getSpans() and {{thread}} so a judge can read tool calls, and the agents guide covers tool choice and arguments. No built-in trajectory metric and no user simulator.\n- **DeepEval (Confident AI, Inc.)**: Trajectory-based evals score the ordered chain of tool calls and handoffs, ToolCorrectness can require ordering and arguments, component metrics attach to spans and ConversationSimulator plays a user. ToolCorrectness scored 20 per cent when no tool was called (dev.to) and no divergence metric exists (issue 3019).\n- **Galileo API + MCP (Galileo Technologies, Inc.)**: Agent Flow validates a session's trajectory against natural-language tests including expected tool or agent calls, and Tool Selection Quality judges tool and argument choice per span. No user simulator.\n- **Gauge CLI (Gauge)**: Agent Experience judges whether a real coding agent completed a task with your product, with pass criteria, session evidence and recurring-failure grouping. The agent under test is the harness using your product, not your own agent.\n- **HoneyHive (HoneyHive Inc.)**: The Trajectory view plots spans by step to spot loops and stuck steps, and conversation evaluator templates score multi-turn sessions. No tool-choice metric or user simulator in the pages read.\n- **Laminar API + MCP (LMNR AI, Inc.)**: Agent evaluations run the real app per trial and a verifier scores the state, trajectory and artefacts, with the trajectory attached as LLM and TOOL spans or imported in ATIF; a tool-call evaluation guide exists. No user simulator.\n- **LangSmith API + MCP (LangChain, Inc.)**: Multi-turn online evaluators measure the trajectory of tool calls across a thread, and datasets can hold expected trajectories. No user simulator and no named divergence metric in the pages read.\n- **MLflow Tracing (MLflow Project, a Series of LF Projects, LLC)**: ToolCallCorrectness and ToolCallEfficiency judge TOOL spans in a trace, scorers can parse the whole trace for trajectory and routing, and ConversationSimulator (experimental, 3.10) plays a user. No divergence-between-runs metric.\n- **Overmind (Overmind Technology Ltd (England and Wales, company 16585996))**: Stock checks cover tool selection, tool arguments and goal state, and live trajectory and plan-drift scoring is mentioned with bounded attempts. No user simulator.\n\n## Couldn't check\n\nThe docs were closed to us, or silent. These carry no mark either way.\n\n- **Pydantic Logfire (Pydantic Services Inc.)**: pydantic-evals may score trajectories, but its docs were not opened and the Logfire pages read show no trajectory metric or user simulator.\n\n- not this job: Agent Tree (No legal entity found. The LICENSE, plugin.json and marketplace.json name \"Mike Pollow, Digitalroots\"): No scoring.\n- not this job: agents-tree (No legal entity found. The LICENSE and pyproject.toml name Karel Blavka): No scoring.\n- not this job: BotGauge (BotGauge AI Inc. (Delaware, United States)): Built-in judges score spans, traces or sessions against criteria; no trajectory metric or simulator in the pages read.\n- not this job: Helicone AI Gateway + MCP (Helicone, Inc.): No trajectory metric, tool-call judge or simulator in the docs index.\n- not this job: Langfuse API + MCP (Langfuse GmbH (a ClickHouse, Inc. company)): Evaluators see one observation and do not load sibling or child observations; multi-span evaluation is promised for the new data model. No trajectory metric or user simulator in the docs.\n- not this job: Navige (Navige Ltd): No trajectory scoring.\n- not this job: Prefactor (Prefactor Pty Ltd): No trajectory scoring or simulator.\n- not this job: Respan API + MCP (Keywords AI Inc. (d/b/a Respan)): No trajectory metric, tool-choice judge or simulator in the docs index.\n- not this job: W\u0026B Weave (Weights and Biases, LLC): Scorers are custom functions; no trajectory metric, tool-choice judge or user simulator appears in the pages read.\n\n## What stops agents finishing\n\n- metrics score one run against a static expectation only\n- no simulation of a user\n- tool calls not in the trace so nothing to judge\n\n## Around it\n\nBefore: agent-observability.debug-failed-run. After: agent-observability.offline-eval-regression.\n\n## Where this comes from\n\nWeight 5 of 100 in Agent observability \u0026 evals: 6 of 86 accounts: the DeepEval dev.to post (ToolCorrectness at 20 per cent), the Opik trajectory metric request, the DeepEval divergence proposal, the Cekura thread's wrong-tool support agent, the Lucidic thread on session-level evals, Phoenix multi-hop evals; LangWatch Scenario appears only in its staff's posts\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/agent-observability.agent-trajectory-eval.json, or the MCP tool get_use_case with id agent-observability.agent-trajectory-eval. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Judge whether the agent took the right steps, not only gave the right answer",
        "url": ""
      }
    ],
    "description": "Judge whether the agent took the right steps, not only gave the right answer: 2 companies fully do it, 10 with a limit, 20 carry the tag. Every listing in Agent observability \u0026 evals with the reason and a source.",
    "facts": null,
    "h1": "Judge whether the agent took the right steps, not only gave the right answer",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/agent-observability.agent-trajectory-eval/",
    "published": "",
    "section": "agents",
    "title": "Judge whether the agent took the right steps, not only gave the right…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/agent-observability.agent-trajectory-eval/"
  },
  "tokens": {
    "markdown": 1800,
    "slim": 380
  },
  "version": 1
}
