Niches · Agent observability & evals
Rerun a dataset through a prompt or model change and get a score
Four or more companies' docs show a full fit.
What counts as done
A run over the whole dataset completes and reports per-row pass or fail and an aggregate score, reproducible against the same dataset and prompt version, with a non-zero exit code usable as a CI gate.
Measured, for our own tests: the run takes under 20 minutes for 300 rows. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: did my prompt change make things worse
, run the eval suite in CI
, compare this model against the old one on our questions
, turn these failing traces into test cases
Does the job 5
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| HoneyHive D HoneyHive Inc. | CI regression detection runs a baseline, calls compare_runs() on the PR and exits non-zero on degradation, with a GitHub Actions workflow; LLM evaluators use any provider configured in the workspace. source |
| Langfuse API + MCP BB Langfuse GmbH (a ClickHouse, Inc. company) | Experiments in CI: raise RegressionError on a threshold, the GitHub Action fails the job by default, and an approved baseline catches newly failing cases. Judges run on Azure, Bedrock, Vertex or any OpenAI-compatible connection. source |
| LangSmith API + MCP BB LangChain, Inc. | The pytest integration logs each test's pass feedback to a dataset and raises assertion errors locally, so the pytest exit code gates CI; datasets, experiments and LLM, code or decision-model evaluators are documented. source |
| LangWatch B Reasoning Engine B.V. (doing business as LangWatch) | Experiments run from CI with a timeout and exit_on_failure=True, and agent test suites run with --wait, which exits non-zero when a run fails and stops after 45 minutes by default. source |
| MLflow Tracing C MLflow Project, a Series of LF Projects, LLC | @mlflow.test turns evaluate() scorers into pytest assertions recorded under one run, so the pytest exit code gates CI; judges take openai, azure, anthropic, bedrock, gemini, ollama or gateway model URIs. source |
Does the job, with a limit 10
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Arize Phoenix BB Arize AI, Inc. | pytest plugin records pass annotations and the pytest exit code gates CI; async_evaluate_dataframe takes a concurrency setting. But AsyncExecutor requeued timed-out tasks forever (issue 14312). source |
| Braintrust API + MCP B Braintrust Data, Inc. | bt eval returns non-zero on exceptions, reporters define pass or fail, and eval-action comments on the PR. Judges can use Bedrock, Azure or a custom endpoint. A few hundred tasks went from 1.5 to 5 minutes (1394) and /logs3 returned 500s mid-run (1414). source |
| DeepEval B Confident AI, Inc. | deepeval test run plugs into pytest with assert_test, AsyncConfig sets max_concurrent and ignore_errors, results land in local JSON, and any judge works (OpenAI, Azure, Ollama, Anthropic, Gemini, LiteLLM). Claude with extended thinking crashed the judge (issue 2946). source |
| Galileo API + MCP D Galileo Technologies, Inc. | Experiments run prompts or code against datasets with metrics through configured LLM integrations, and a unit-test page is listed; it was not read, so exit codes and timeouts are unverified. source |
| Gauge CLI B Gauge | Eval cases with pass criteria rerun on each PR through the GitHub App or gauge run -o json, and failed runs refund credits. The dataset is tasks for coding agents using your product, not your own app's inputs; --verify once rejected the --files flag (issue 23). source |
| Laminar API + MCP C LMNR AI, Inc. | Evaluations run datapoints through an executor and evaluators in parallel, store scores and compare runs in groups; agent evaluations run in containers with lmnr-cli. No exit code or per-row timeout appears in the pages read. source |
| Opik API + MCP BB Comet ML, Inc. | evaluate() scores a dataset with any LiteLLM or OpenAI-compatible judge and test suites carry assertions with runs_per_item and pass_threshold, but no CI exit code or timeout handling appears in the pages read and the pytest page is empty. source |
| Overmind C Overmind Technology Ltd (England and Wales, company 16585996) | run_evaluation applies matrix rows (JSON validity, tool selection, exact match, cost and latency budgets, Jev judgement) to 1 to 6 targets and separates scores, abstentions and technical errors; runs spend credits. No CI exit code or duration figure is documented. source |
| Pydantic Logfire B Pydantic Services Inc. | pydantic-evals is code-first and feeds datasets and experiment comparisons in the Logfire UI, per the FAQ; the evals pages were not read, so exit codes, timeouts and judge choice are unverified. source |
| W&B Weave B Weights and Biases, LLC | Evaluation runs a dataset through the app with scorers and compares runs in the UI; LLM judges are your own client code. No CI exit code or timeout handling in the pages read, and TypeScript lacks class scorers. source |
Couldn't check 1
The docs were closed to us, or silent. These carry no mark either way.
| Listing | Why |
|---|---|
| Respan API + MCP BB Keywords AI Inc. (d/b/a Respan) | The docs index lists prompt management and span-02 decisions but no dataset runner or experiment pages, so the obs.evals and obs.datasets tags could not be checked. |
Doesn't do this job 6
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Agent Tree C No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots" | No evaluation. source |
| agents-tree B No legal entity found. The LICENSE and pyproject.toml name Karel Blavka | No evaluation. source |
| BotGauge E BotGauge AI Inc. (Delaware, United States) | Evals are judges over incoming traces; there is no dataset runner or CI gate. source |
| Helicone AI Gateway + MCP E Helicone, Inc. | Datasets can be curated and exported and scores submitted by API, but there is no eval runner, judge or CI gate. source |
| Navige E Navige Ltd | No datasets or evaluators. source |
| Prefactor B Prefactor Pty Ltd | Judges score recorded runs; there is no dataset runner or CI gate. source |
What stops agents finishing
- judge model limited to one provider
- eval run hangs on timeouts without failing
- eval endpoint returns 500s mid-run
- no exit code or machine-readable result for CI
Around it
Before: agent-observability.debug-failed-run, agent-harnesses.fix-failing-ci. After: agent-observability.online-llm-judge, finetune.sft-from-traces.
Where this comes from
Weight 13 of 100 in Agent observability & evals: 14 of 86 accounts: the Ask HN evals thread (Braintrust, promptfoo plus scripts, bootstrapped judges), the DeepEval dev.to post, Braintrust SDK issues on eval run speed and 500s, Phoenix and DeepEval issues on hanging or crashing evaluators
The same as data: /api/v1/use-cases/agent-observability.offline-eval-regression.json, or the MCP tool get_use_case with id agent-observability.offline-eval-regression. Other jobs in this category: /api/v1/niches?category=agent-observability.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.