Niches · Agent observability & evals

Score production traces automatically with an LLM judge

One company's docs show a full fit.

1companies, full fit
14companies, fit with a limit
24companies carry the tag
9share of the category's work
9first-hand accounts

What counts as done

Each new trace or chosen span gets a score, the judge runs on the configured model, the judge's own token cost is visible, and scores can be filtered and alerted on.

Measured, for our own tests: each score arrives within 10 minutes of the trace or span. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: flag bad answers in production, run a judge on every trace, score just the retrieval span, alert me when quality drops

Does the job 1

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
LangWatch B
Reasoning Engine B.V. (doing business as LangWatch)
Monitors score each trace or thread as it arrives with a chosen judge model, preconditions and sampling; the judge sees formatted_trace with tool calls and results, and Instant Evals publish their token price. source

Does the job, with a limit 14

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Arize Phoenix BB
Arize AI, Inc.
Evals are provider agnostic (OpenAI, Azure, Anthropic, Bedrock and Vertex via LiteLLM) and span annotations are logged by span_id, but server-side evaluators run only on dataset experiments; continuous scoring of incoming traces is your own loop. source
BotGauge E
BotGauge AI Inc. (Delaware, United States)
Judges run at span, trace or session scope with sampling and API filters; traces wait 2 to 30 minutes (default 10) after their last span, spans are judged at once, retries and quota pauses are shown. The judge model is BotGauge's, not the owner's choice. source
Braintrust API + MCP B
Braintrust Data, Inc.
Online scoring rules at trace or span scope with filters, sampling and a 30 second idle timeout; rule activity shows scorer LLM cost; judges run on any configured provider including Bedrock and custom endpoints. Duplicate AI SDK spans doubled scoring runs and cost (992). source
Galileo API + MCP D
Galileo Technologies, Inc.
Log stream metrics run automatically on incoming traces with sampling, and custom LLM-as-judge metrics target session, trace, LLM or tool spans on any configured integration, custom ones included. Judge cost visibility was not read. source
HoneyHive D
HoneyHive Inc.
Online evaluations run Python or LLM evaluators asynchronously on ingested traces matching event filters, with a sampling percentage, on workspace-configured providers. Judge cost is not shown and the provider list page was not read. source
Laminar API + MCP C
LMNR AI, Inc.
Signals run a trace-analysis agent on each finished trace under triggers, filters and a sample rate, write events to SQL and alert Slack or email. The judge model is not the owner's choice in the docs read, and Signals need an enterprise licence when self-hosted. source
Langfuse API + MCP BB
Langfuse GmbH (a ClickHouse, Inc. company)
Observation-level evaluators run on a worker with filters and the project's LLM connection (Azure, Bedrock, Vertex, OpenAI-compatible). Trace-level evaluators stop on 2026-11-16, a GLM judge failed on non-JSON output (13502) and Bedrock Claude on missing tool calls (9537). source
LangSmith API + MCP BB
LangChain, Inc.
Run-level and thread-level LLM-as-judge evaluators with filters, sampling, a weekly spend limit that pauses the evaluator, and a decision-model option. The provider list for the judge was not on the page read, and scored traces are auto-upgraded to paid extended retention. source
MLflow Tracing C
MLflow Project, a Series of LF Projects, LLC
Automatic evaluation runs registered judges on traces or sessions as they are logged, with a sample rate and filter string, but the judge must be an AI Gateway endpoint to run from the UI, only traces under an hour old are picked up, and judge cost is not shown. source
Opik API + MCP BB
Comet ML, Inc.
Online rules score traces, threads or spans with filters, sampling, a USD cost limit and an agent-as-a-judge that reads the trace; providers include Bedrock, Azure, Vertex, Ollama and any OpenAI-compatible endpoint. A user could not find judge token usage (issue 4017). source
Overmind C
Overmind Technology Ltd (England and Wales, company 16585996)
Live span scoring writes Span.eval_attributes with bounded attempts, judged by Jev, a decision model, with a generative judge_model only for explanations; the judge is not the owner's choice and no arrival-to-score time is given. source
Prefactor B
Prefactor Pty Ltd
Hosted judges score other agents' runs on models from LLM providers you connect, with a Preparing status and a pass or fail verdict on the Quality tab. Scope is the run, not a span, and no arrival-to-verdict time is given. source
Pydantic Logfire B
Pydantic Services Inc.
The FAQ describes live evaluations grading production traffic in the background from pydantic-evals, but that page was not read, so target level, judge model and cost visibility are unverified. source
Respan API + MCP BB
Keywords AI Inc. (d/b/a Respan)
Behaviours score every new span with span-02, Respan's decision model, firing above 65 per cent confidence, with custom plain-language definitions on the Team plan. The judge is not a model of the owner's choosing and past spans are not rescored. source

Couldn't check 2

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
Helicone AI Gateway + MCP E
Helicone, Inc.
The docs index lists an Eval Scores page and score submission endpoints but no judge runner; those pages were not opened.
W&B Weave B
Weights and Biases, LLC
The guardrails and monitors page describes inline guardrails that need code changes; a background monitor over incoming traces was not found in the pages read.

Doesn't do this job 5

Listings in the category that do something else, with the reason.

ListingWhy
Agent Tree C
No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots"
No judge. source
agents-tree B
No legal entity found. The LICENSE and pyproject.toml name Karel Blavka
No judge. source
DeepEval B
Confident AI, Inc.
Production scoring is Confident AI's job; the framework runs offline. source
Gauge CLI B
Gauge
No production trace ingestion. source
Navige E
Navige Ltd
Outcomes are reported by you or judged by hand; no LLM judge. source

What stops agents finishing

  • judge requires a tool-calling or strict-JSON model
  • span-level evaluation not offered or broken on self-host
  • judge cost invisible
  • evaluator silently never fires

Around it

Before: agent-observability.instrument-app. After: agent-observability.offline-eval-regression, notifications.alert.

Where this comes from

Weight 9 of 100 in Agent observability & evals: 9 of 86 accounts: Langfuse issues on judge model configuration (GLM, Bedrock), span-level evaluators failing or not triggering, Phoenix multi-hop span evals, Opik judge usage and nested online rules, Braintrust duplicate scoring

The same as data: /api/v1/use-cases/agent-observability.online-llm-judge.json, or the MCP tool get_use_case with id agent-observability.online-llm-judge. Other jobs in this category: /api/v1/niches?category=agent-observability.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.