Niches · Agent observability & evals
Run the trace store yourself and keep it healthy
No company's docs show a full fit yet. The listings below do the job with a limit, or not at all.
What counts as done
The stack runs from the documented compose or Helm files, with disk growth and retention controls documented.
Measured, for our own tests: it ingests at the team's volume, and after 30 days the UI still loads a trace list in under 5 seconds. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: self-host it with docker compose
, keep traces in our region
, disk is full after three months
, upgrade broke the UI
Does the job, with a limit 8
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Arize Phoenix BB Arize AI, Inc. | Docker, Compose and Helm with SQLite or Postgres; PHOENIX_DEFAULT_RETENTION_POLICY_DAYS and per-project retention policies since 9.0, infinite by default. Before that a user hit 44 GB and an unusable UI (issue 8420). source |
| Helicone AI Gateway + MCP E Helicone, Inc. | An all-in-one Docker image and Kubernetes guide exist, with S3, Better Auth and the ClickHouse and Postgres stack from the listing. No retention or disk-growth setting appears on the Docker page. source |
| Laminar API + MCP C LMNR AI, Inc. | Compose runs frontend, app server, Postgres, ClickHouse and Quickwit; Helm adds RabbitMQ, Redis and a consumer. Signals and alerts need a licence key, no retention setting is documented, and ClickHouse system logs grew without bound on a small instance (issue 2176). source |
| Langfuse API + MCP BB Langfuse GmbH (a ClickHouse, Inc. company) | Compose and Helm with Postgres, ClickHouse, Redis and S3; retention is per project with nightly deletes and data is kept indefinitely by default. One user grew a terabyte ClickHouse on a laptop and another lost the UI after an image upgrade (issue 12172). source |
| LangWatch B Reasoning Engine B.V. (doing business as LangWatch) | Apache 2.0 self-hosting with Docker or Helm is claimed and retention is configurable per organisation, team or project on paid and self-hosted plans, with 49 days on Free. The component list and disk guidance were not in the pages read. source |
| MLflow Tracing C MLflow Project, a Series of LF Projects, LLC | SQLite by default, a Compose bundle with Postgres and RustFS, and a Helm chart; trace archival moves old span payloads to artifact storage under a retention policy. Authentication is off unless basic-auth is enabled. An HN user calls it well established. source |
| Opik API + MCP BB Comet ML, Inc. | Local Compose or Kubernetes with ClickHouse, Zookeeper, MySQL, Redis and MinIO; a tiered-storage TTL moves trace data to cold S3 after a set number of weeks. The open-source build has no authentication. source |
| Overmind C Overmind Technology Ltd (England and Wales, company 16585996) | Compose with six Celery worker profiles under AGPL, but startup validation requires Modal, OpenRouter, AWS checkpoint and sandbox credentials even to run; no retention or disk guidance. source |
Doesn't do this job 14
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Agent Tree C No legal entity found. The LICENSE, plugin.json and marketplace.json name "Mike Pollow, Digitalroots" | Nothing to host. source |
| agents-tree B No legal entity found. The LICENSE and pyproject.toml name Karel Blavka | Nothing to host. source |
| BotGauge E BotGauge AI Inc. (Delaware, United States) | Hosted only; no self-hosting pages. source |
| Braintrust API + MCP B Braintrust Data, Inc. | Only SaaS or BYOC, and BYOC needs the Enterprise plan with Braintrust operating the data plane; there is no documented Compose or Helm stack to run yourself. source |
| DeepEval B Confident AI, Inc. | Nothing to host; runs are written to local JSON or SQLite stores. source |
| Galileo API + MCP D Galileo Technologies, Inc. | On-prem and VPC deployment are Enterprise only per the pricing notes; no Compose or Helm files are documented. source |
| Gauge CLI B Gauge | Local mode runs in Docker with results in .gauge/gauge.sqlite; there is no trace store to host. source |
| HoneyHive D HoneyHive Inc. | Self-hosting is BYOC or BYOVPC on AWS, deployed by HoneyHive with Terraform and Helm for Enterprise; there is no stack to run from documented files. source |
| LangSmith API + MCP BB LangChain, Inc. | Self-hosted LangSmith is an Enterprise add-on with a licence key; the ClickHouse or SmithDB, Postgres, Redis and blob stack is documented but not available to run otherwise. One builder made a local tool because LangSmith needs a cloud account. source |
| Navige E Navige Ltd | Hosted only. source |
| Prefactor B Prefactor Pty Ltd | Hosted only. source |
| Pydantic Logfire B Pydantic Services Inc. | Self-hosted Logfire is an Enterprise deployment; the Personal, Team and Growth plans are cloud only. source |
| Respan API + MCP BB Keywords AI Inc. (d/b/a Respan) | The trace platform is hosted only; the open-source gateway is a proxy with a request log, not a trace store. source |
| W&B Weave B Weights and Biases, LLC | Multi-tenant or Dedicated Cloud only on the plans read; the trace server left the public repository in October 2026. source |
What stops agents finishing
- self-hosting is Enterprise only
- ClickHouse or S3 growth with no retention setting
- upgrade path breaks the trace tables
- auth off by default on the self-hosted build
- cloud account required to see one's own traces
Around it
Before: agent-observability.instrument-app. After: agent-observability.debug-failed-run.
Where this comes from
Weight 10 of 100 in Agent observability & evals: 11 of 86 accounts: Langfuse self-hosters (ClickHouse stack, terabyte disk, upgrade breakage), Phoenix at 44 GB, Laminar ClickHouse logs, MLflow as the established self-hosted option, the Ask HN comparison thread, LangSmith lock-in and cloud-account complaints that led to a local-first alternative
The same as data: /api/v1/use-cases/agent-observability.self-host-retain.json, or the MCP tool get_use_case with id agent-observability.self-host-retain. Other jobs in this category: /api/v1/niches?category=agent-observability.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.