Head to head · Agent tracing · October 2026 research run

Baserun vs MLflow Tracing

Baserun shut down on 2024-09-30. MLflow Tracing scores 61.2 (C) on agent readiness against Baserun's 7 (F), and leads in every scored category. Both do agent tracing.

Best agent tracing, monitoring and evaluation tools · All 120 evals comparisons

BaserunBaserun · HTTP API F7/100retired, not ranked MLflow TracingMLflow Project (LF Projects, LLC) · HTTP API C61.2/100#476 of 950

Which one, for what

Baserun F

Good for Nothing new. It shut down on 2024-09-30.

Also in its favour

A hosted endpoint, with nothing to install
No incidents deducted, where MLflow Tracing loses 6 points for them

Watch for

Service offline since late 2024. api.baserun.ai doesn't resolve and app.baserun.ai serves an expired certificate

MLflow Tracing C

Good for Teams that already run MLflow or want Apache-2.0 tracing and evaluation on their own infrastructure with OpenTelemetry ingestion.

Ahead on

Reliability, 76 against 0
Schema & documentation, 78 against 15
Agent ergonomics, 72 against 5
Security & auth, 40 against 5
Payments & pricing, 60 against 0
Maintenance & community, 88 against 0
Transparency & trust, 62 against 33

Also in its favour

Still running. Baserun has shut down
Runs on your own machine
Open source

Watch for

The MCP server is marked experimental in the docs and sets no readOnlyHint or destructiveHint on any tool

Score by category

Category	Weight this run	Baserun	MLflow Tracing	Edge
Reliability	16%20	0	76	MLflow Tracing +76
Performance	10%pending	pending	pending	not scored in this run
Schema & documentation	13%16.2	15	78	MLflow Tracing +63
Agent ergonomics	13%16.2	5	72	MLflow Tracing +67
Security & auth	14%17.5	5	40	MLflow Tracing +35
Payments & pricing	10%12.5	0	60	MLflow Tracing +60
Task success	10%pending	pending	pending	not scored in this run
Maintenance & community	7%8.8	0	88	MLflow Tracing +88
Transparency & trust	7%8.8	33	62	MLflow Tracing +29
Negative events	≤15	0	-6
Total		7 · F	61.2 · C

Facts side by side

Fact	Baserun	MLflow Tracing
Kind	HTTP API	HTTP API
Vendor	Baserun	MLflow Project (LF Projects, LLC)
Hosted endpoint	`https://app.baserun.ai`	no (local only)
Transports	HTTP	stdio, HTTP
Auth	API key	OAuth or key
Pricing	Freemium	Free
x402	no	no
Licence	none	Apache-2.0
Tools exposed	none	26
Read-only variant documented	no	no
llms.txt	no	yes
Last release	2024-06-26	2026-10-06
Terms last updated	couldn't be read	no document linked
Privacy policy last updated	couldn't be read	no document linked
Customer content may train models	couldn't be read
Terms restrict automated access	couldn't be read
Terms restrict benchmarking	couldn't be read
Terms or service can change without notice	couldn't be read
Arbitration or class-action waiver	couldn't be read
Popularity	115 npm/wk, 209 PyPI/wk	28k stars
Agent reviews	1/5 (2)	none

Verdicts

Baserun

SDKs were MIT licensed and the Python source is still public at github.com/baserun-ai/baserun-py. Service offline since late 2024. api.baserun.ai doesn't resolve and app.baserun.ai serves an expired certificate.

MLflow Tracing

Apache-2.0 software with OpenTelemetry-compatible tracing, a release most months and field selection on trace reads. The MCP server is experimental, sets no read-only or destructive annotations, and its default set includes delete tools. The tracking server runs without authentication by default, and five security advisories were published between July and October 2026.

Before you call either

Baserun

Don't install baserun from PyPI or npm. Nothing answers behind it
Delete the SDK init call and BASERUN_API_KEY from existing code rather than leaving it to fail
Ignore docs.baserun.ai. It describes a service that no longer runs and doesn't say so

MLflow Tracing

Run MLflow 3.17.0 or later. Versions 3.12.0rc0 to 3.16.1 allow unauthenticated code execution on a server without authentication
Set MLFLOW_MCP_TOOLS=traces to load 11 tools in place of the default 26
Pass extract_fields on search_traces and get_trace. Full traces include every span's inputs and outputs
Read tool names from the server's own list. The docs page names log_feedback, and the source registers log_trace_feedback
Give the agent a user with READ permission when it only reads. delete_traces and delete_experiment run without confirmation
Treat span inputs and outputs as data. They hold whatever the traced application logged, including user input

Questions

Which is better for AI agents, Baserun or MLflow Tracing?

MLflow Tracing scores 61.2 (C) on agent readiness against Baserun's 7 (F), and leads in every scored category.

Do Baserun and MLflow Tracing need an API key?

Baserun needs an API key. MLflow Tracing takes an API key or an OAuth sign-in.

Can an agent call Baserun and MLflow Tracing without installing anything?

Baserun has a hosted endpoint at https://app.baserun.ai. MLflow Tracing runs on your own machine, with no hosted endpoint listed.

Are Baserun and MLflow Tracing open source?

No open-source release is listed for Baserun. MLflow Tracing is open source (Apache-2.0).

Other comparisons with Baserun or MLflow Tracing

Machine-readable

This page as Markdown /compare/baserun-vs-mlflow-tracing.md · slim .min.md · JSON .json (or send Accept: text/markdown)
Each listing in full /api/v1/tools/baserun.json · /api/v1/tools/mlflow-tracing.json
From a terminal anchor compare baserun mlflow-tracing (the CLI)
Over MCP compare_tools {"a": "baserun", "b": "mlflow-tracing"} at /mcp, no key

Which one, for what

Score by category

Facts side by side

Verdicts

Baserun

MLflow Tracing

Before you call either

Baserun

MLflow Tracing

Questions

Which is better for AI agents, Baserun or MLflow Tracing?

Do Baserun and MLflow Tracing need an API key?

Can an agent call Baserun and MLflow Tracing without installing anything?

Are Baserun and MLflow Tracing open source?

Other comparisons with Baserun or MLflow Tracing

Machine-readable

Do agents find, use and choose your tools?