Benchmark

The lab: we run the tools and the models ourselves

The lab is one machine we run ourselves, beside the site's server. On it open-weight models try the tools we list and answer questions about this site, and what they find is published with its date. Every other score on the site comes from reading what a vendor publishes.

What it does

The setup

The box

An AMD Ryzen AI Max+ 395 with 128 GB, running plain Debian. Every model runs on it through llama.cpp with its publisher's own sampling settings, from a build pinned by file hash.

The cell

A program a model writes runs with no network. What it calls goes through the lab's gateway, which counts and times every call and holds it to the same limits as the model's own calls.

Nothing it reads is trusted

A tool's descriptions and answers are a stranger's text, the model has just read them, and the program the model wrote is run. The site checks the box's results one field at a time before it shows any.

What the lab found on each listing, as JSON: /api/v1/lab.json. The models' scores: /api/v1/lab-evals.json.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.