{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/agent-observability.offline-eval-regression/",
    "json": "https://www.anchorterminal.com/niches/agent-observability.offline-eval-regression/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/agent-observability.offline-eval-regression/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/agent-observability.offline-eval-regression/index.min.md"
  },
  "markdown": "Four or more companies' docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 5 | 10 | 20 | 13 | 14 |\n\n## What counts as done\n\nA run over the whole dataset completes and reports per-row pass or fail and an aggregate score, reproducible against the same dataset and prompt version, with a non-zero exit code usable as a CI gate.\n\nMeasured, for our own tests: the run takes under 20 minutes for 300 rows.\n\nHow agents ask for it: did my prompt change make things worse; run the eval suite in CI; compare this model against the old one on our questions; turn these failing traces into test cases\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **HoneyHive (HoneyHive Inc.)**: CI regression detection runs a baseline, calls compare_runs() on the PR and exits non-zero on degradation, with a GitHub Actions workflow; LLM evaluators use any provider configured in the workspace.\n- **Langfuse API + MCP (Langfuse GmbH (a ClickHouse, Inc. company))**: Experiments in CI: raise RegressionError on a threshold, the GitHub Action fails the job by default, and an approved baseline catches newly failing cases. Judges run on Azure, Bedrock, Vertex or any OpenAI-compatible connection.\n- **LangSmith API + MCP (LangChain, Inc.)**: The pytest integration logs each test's pass feedback to a dataset and raises assertion errors locally, so the pytest exit code gates CI; datasets, experiments and LLM, code or decision-model evaluators are documented.\n- **LangWatch (Reasoning Engine B.V. (doing business as LangWatch))**: Experiments run from CI with a timeout and exit_on_failure=True, and agent test suites run with --wait, which exits non-zero when a run fails and stops after 45 minutes by default.\n- **MLflow Tracing (MLflow Project, a Series of LF Projects, LLC)**: @mlflow.test turns evaluate() scorers into pytest assertions recorded under one run, so the pytest exit code gates CI; judges take openai, azure, anthropic, bedrock, gemini, ollama or gateway model URIs.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Arize Phoenix (Arize AI, Inc.)**: pytest plugin records pass annotations and the pytest exit code gates CI; async_evaluate_dataframe takes a concurrency setting. But AsyncExecutor requeued timed-out tasks forever (issue 14312).\n- **Braintrust API + MCP (Braintrust Data, Inc.)**: bt eval returns non-zero on exceptions, reporters define pass or fail, and eval-action comments on the PR. Judges can use Bedrock, Azure or a custom endpoint. A few hundred tasks went from 1.5 to 5 minutes (1394) and /logs3 returned 500s mid-run (1414).\n- **DeepEval (Confident AI, Inc.)**: deepeval test run plugs into pytest with assert_test, AsyncConfig sets max_concurrent and ignore_errors, results land in local JSON, and any judge works (OpenAI, Azure, Ollama, Anthropic, Gemini, LiteLLM). Claude with extended thinking crashed the judge (issue 2946).\n- **Galileo API + MCP (Galileo Technologies, Inc.)**: Experiments run prompts or code against datasets with metrics through configured LLM integrations, and a unit-test page is listed; it was not read, so exit codes and timeouts are unverified.\n- **Gauge CLI (Gauge)**: Eval cases with pass criteria rerun on each PR through the GitHub App or gauge run -o json, and failed runs refund credits. The dataset is tasks for coding agents using your product, not your own app's inputs; --verify once rejected the --files flag (issue 23).\n- **Laminar API + MCP (LMNR AI, Inc.)**: Evaluations run datapoints through an executor and evaluators in parallel, store scores and compare runs in groups; agent evaluations run in containers with lmnr-cli. No exit code or per-row timeout appears in the pages read.\n- **Opik API + MCP (Comet ML, Inc.)**: evaluate() scores a dataset with any LiteLLM or OpenAI-compatible judge and test suites carry assertions with runs_per_item and pass_threshold, but no CI exit code or timeout handling appears in the pages read and the pytest page is empty.\n- **Overmind (Overmind Technology Ltd (England and Wales, company 16585996))**: run_evaluation applies matrix rows (JSON validity, tool selection, exact match, cost and latency budgets, Jev judgement) to 1 to 6 targets and separates scores, abstentions and technical errors; runs spend credits. No CI exit code or duration figure is documented.\n- **Pydantic Logfire (Pydantic Services Inc.)**: pydantic-evals is code-first and feeds datasets and experiment comparisons in the Logfire UI, per the FAQ; the evals pages were not read, so exit codes, timeouts and judge choice are unverified.\n- **W\u0026B Weave (Weights and Biases, LLC)**: Evaluation runs a dataset through the app with scorers and compares runs in the UI; LLM judges are your own client code. No CI exit code or timeout handling in the pages read, and TypeScript lacks class scorers.\n\n## Couldn't check\n\nThe docs were closed to us, or silent. These carry no mark either way.\n\n- **Respan API + MCP (Keywords AI Inc. (d/b/a Respan))**: The docs index lists prompt management and span-02 decisions but no dataset runner or experiment pages, so the obs.evals and obs.datasets tags could not be checked.\n\n- not this job: Agent Tree (No legal entity found. The LICENSE, plugin.json and marketplace.json name \"Mike Pollow, Digitalroots\"): No evaluation.\n- not this job: agents-tree (No legal entity found. The LICENSE and pyproject.toml name Karel Blavka): No evaluation.\n- not this job: BotGauge (BotGauge AI Inc. (Delaware, United States)): Evals are judges over incoming traces; there is no dataset runner or CI gate.\n- not this job: Helicone AI Gateway + MCP (Helicone, Inc.): Datasets can be curated and exported and scores submitted by API, but there is no eval runner, judge or CI gate.\n- not this job: Navige (Navige Ltd): No datasets or evaluators.\n- not this job: Prefactor (Prefactor Pty Ltd): Judges score recorded runs; there is no dataset runner or CI gate.\n\n## What stops agents finishing\n\n- judge model limited to one provider\n- eval run hangs on timeouts without failing\n- eval endpoint returns 500s mid-run\n- no exit code or machine-readable result for CI\n\n## Around it\n\nBefore: agent-observability.debug-failed-run, agent-harnesses.fix-failing-ci. After: agent-observability.online-llm-judge, finetune.sft-from-traces.\n\n## Where this comes from\n\nWeight 13 of 100 in Agent observability \u0026 evals: 14 of 86 accounts: the Ask HN evals thread (Braintrust, promptfoo plus scripts, bootstrapped judges), the DeepEval dev.to post, Braintrust SDK issues on eval run speed and 500s, Phoenix and DeepEval issues on hanging or crashing evaluators\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/agent-observability.offline-eval-regression.json, or the MCP tool get_use_case with id agent-observability.offline-eval-regression. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Rerun a dataset through a prompt or model change and get a score",
        "url": ""
      }
    ],
    "description": "Rerun a dataset through a prompt or model change and get a score: 5 companies fully do it, 10 with a limit, 20 carry the tag. Every listing in Agent observability \u0026 evals with the reason and a source.",
    "facts": null,
    "h1": "Rerun a dataset through a prompt or model change and get a score",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/agent-observability.offline-eval-regression/",
    "published": "",
    "section": "agents",
    "title": "Rerun a dataset through a prompt or model change and get a score |…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/agent-observability.offline-eval-regression/"
  },
  "tokens": {
    "markdown": 1950,
    "slim": 380
  },
  "version": 1
}
