Niches · Agent observability & evals

Know what a run, step or feature cost

No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.

0companies, full fit
18companies, fit with a limit
25companies carry the tag
9share of the category's work
10first-hand accounts

What counts as done

A cost per trace and per span, including cached tokens and models released in the last 30 days, and a per-step ranking for one run without reading the tree by hand.

Measured, for our own tests: the cost matches the provider's bill within 5 per cent. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: how much did this run cost, which step burned the tokens, why is the cost double what the provider billed, cost per conversation turn

Does the job, with a limit 18

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Agent Tree C
No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots"
The pane shows the main session's model, effort, context and cost and each sub-agent's context and usage live, but nothing is ranked by step or kept afterwards. source
agents-tree B
No legal entity found. The LICENSE and pyproject.toml name Karel Blavka
The detail view and --json give request and token counts per agent and context tokens, with no cost figure and no persistence. source
Arize Phoenix BB
Arize AI, Inc.
Cost is derived from OpenInference token counts, including cache_read and cache_write, and rolled up to trace and project. Provider or gateway cost attributes are dropped (issue 15240) and per-step cost is per span, not ranked (HN 49093357). source
BotGauge E
BotGauge AI Inc. (Delaware, United States)
Logs show cost per session and trace and troubleshooting covers zero-cost traces, but cached tokens and gateway cost are not mentioned. source
Braintrust API + MCP B
Braintrust Data, Inc.
Cost comes from token counts and model pricing, cache_read and cache_creation tokens are mapped, and a span's own metrics.estimated_cost overrides it. Preset charts add up duplicated token metrics, so a copied call shows twice; per-step ranking needs sql_query. source
Helicone AI Gateway + MCP E
Helicone, Inc.
Gateway requests are priced from the Model Registry, direct integrations from a best-effort table; sessions roll cost up per interaction and HQL exposes provider_total_cost. Context-tier pricing was not applied for OpenAI, Azure or OpenRouter (issue 5690). source
HoneyHive D
HoneyHive Inc.
Sessions sum metadata.cost and tokens from child events, but the aggregation page says cost is a value you set on child events; cached tokens are not mentioned. source
Laminar API + MCP C
LMNR AI, Inc.
Every LLM span carries input, output, cache read, cache write and reasoning costs with long-context and service-tier rates, rolled up to trace columns you can rank in SQL; custom model costs exist. A user had to ask for OpenRouter usage to be read (issue 706). source
Langfuse API + MCP BB
Langfuse GmbH (a ClickHouse, Inc. company)
usage_details buckets cache reads and writes separately and OpenRouter or LiteLLM cost is captured, but a gateway that appends a cost field breaks the mapping, cache tokens were double counted (issue 12306) and root rows showed $0.00 (issue 16077). source
LangSmith API + MCP BB
LangChain, Inc.
Automatic cost from tokens for OpenAI, Anthropic and LangChain calls with cache-read subtypes, custom costs for other runs, totals per trace and per parent run in the tree; the gateway records its own cost. Thread totals need metadata on every child run. source
LangWatch B
Reasoning Engine B.V. (doing business as LangWatch)
Cost tracking per model call down to the token is advertised and OTel ingestion applies cost enrichment, but the cost tutorial itself was not read, so cache tokens and per-step ranking are unverified. source
MLflow Tracing C
MLflow Project, a Series of LF Projects, LLC
Token usage and cost are logged per span and aggregated in the experiment dashboard, as the Claude Code page shows. The cost tracking page itself answered 403, so cache-token handling and gateway-reported cost could not be read. source
Navige E
Navige Ltd
Each call is priced from the provider's usage with cached input counted separately, OpenRouter's reported cost is used as is, unknown prices show unknown not zero, and transactions give cost per task with a percentile estimate. No per-step ranking inside a run and no prompt content. source
Opik API + MCP BB
Comet ML, Inc.
Span and trace cost in USD, aggregated from spans and readable as total_estimated_cost, with provider and model settable by hand. Unsupported models return None, and cache tokens are not mentioned on the page. source
Overmind C
Overmind Technology Ltd (England and Wales, company 16585996)
Usage is normalised from GenAI attributes and cost budgets are stock evaluation checks, but no trace-level cost field, cache-token handling or price table appears in the pages read. source
Pydantic Logfire B
Pydantic Services Inc.
Tokens and cost by model, provider and agent in the LLMs view and per-trace in the panel; SQL can rank spans by cost. Cache-token handling and gateway-reported cost were not in the pages read. source
Respan API + MCP BB
Keywords AI Inc. (d/b/a Respan)
Every gateway call writes a log with tokens and cost retrievable by X-Respan-Log-Id, the Claude Code plugin records cache tokens, and SQL ranks cost per model. Cache pricing for SDK traces and per-step ranking within one trace were not shown. source
W&B Weave B
Weights and Biases, LLC
Automatic cost for supported integrations from token usage and built-in pricing, custom costs for others, available in the trace tree and via include_costs. TypeScript has no cost tracking and cached tokens are not mentioned. source

Couldn't check 1

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
Galileo API + MCP D
Galileo Technologies, Inc.
Model pricing settings and integration costs pages are listed in the docs index but were not opened.

Doesn't do this job 3

Listings in the category that do something else, with the reason.

ListingWhy
DeepEval B
Confident AI, Inc.
No cost or token accounting for traced runs. source
Gauge CLI B
Gauge
Runs are billed in Gauge credits; no token or model cost per session is reported. source
Prefactor B
Prefactor Pty Ltd
No token or model cost fields in the documentation; cost does not appear outside billing. source

What stops agents finishing

  • new model missing from the price table
  • cost only derived from tokens so gateway prices are ignored
  • cached tokens double counted
  • per-step cost only as a dashboard aggregate

Around it

Before: agent-observability.instrument-app, inference.cached-long-session. After: agent-observability.offline-eval-regression.

Where this comes from

Weight 9 of 100 in Agent observability & evals: 10 of 86 accounts: the sub-agent cost CLI thread, Langfuse issues on cache double counting and $0 root rows, Phoenix issues on gateway-reported cost, Helicone tier pricing, Laminar OpenRouter usage, Opik judge token usage

The same as data: /api/v1/use-cases/agent-observability.cost-per-run.json, or the MCP tool get_use_case with id agent-observability.cost-per-run. Other jobs in this category: /api/v1/niches?category=agent-observability.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.