Check an LLM's output before it is sent or stored
Four or more companies' docs show a full fit.
What counts as done
A supported or unsupported probability per response.
Measured, for our own tests: the probabilities match held-out human labels at least as well as an LLM judge on the caller's set, at under a tenth of the judge's cost. No docs we read state these figures, so they don't count against a listing here.
How agents ask for it: is every claim in this answer supported by the context
, does this draft follow policy
, is this retrieved passage relevant
, which of these two replies is better
Does the job 4
The docs show it meets the test above, and nothing we read says otherwise.
| Listing | Why |
|---|---|
| Celeris-1 Decision D Marqo Inc | predicate and score questions over a response plus context, 512 questions a request, $0.004 per million input tokens, far under a judge's cost. No independent accuracy measurement against human labels was found. source |
| Clef B Cloudflare, Inc. | Arize: 85.2 percent balanced accuracy on RAGTruth, within noise of an Opus 5 judge at 86.6 percent and $14.30 per 1,000, at $0.223 per 1,000; 91.5 percent on the Phoenix evaluator suites; wording-stable. source |
| OpenAI Decisions API A OpenAI OpCo, LLC | Arize: 81.5 percent on RAGTruth with intervals touching Jev's, 90.0 percent on the Phoenix suites, calibration error 0.042, wording-stable, $0.090 per 1,000 against $14.30 for an Opus 5 judge. source |
| Jev B TypeSafe AI, Inc. | Arize: 85.5 percent on RAGTruth with calibration error 0.023, within noise of an Opus 5 judge at 86.6 percent and $14.30, at $0.045 per 1,000; wording-stable (0.93 both ways); a verify_llm_outputs cookbook exists. source |
Does the job, with a limit 19
It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.
| Listing | Why |
|---|---|
| Decider BB Mark Marosi (Mapika) | noul and score questions over a context, with the README warning that calibration on hard items is the weak axis (ECE 0.15 to 0.31 on JevBench hard). No measurement against an LLM judge was found. source |
| Decision 3.0 B vLLM Semantic Router project | noul and score questions over context and answer in one call. No calibration figures on the cards and no measurement against an LLM judge; issue 6367 crashed the batched /v1/decisions route on a rejected state. source |
| Drex Max v1 C Nace AI, Inc. | noul and score over context and answer; vendor's Decision Index and Artificial Analysis scores only, no judge comparison. Licence cap. source |
| elephant-1 D Baseten Labs, Inc. | noul and score over context and answer, with 16 percent of training on judging and rubric grading. No comparison against an LLM judge on human labels was found. source |
| Granite Switch 4.1 B International Business Machines Corporation | Possible as a two-label or multi-label logprob read with the state as a chat turn; no typed answer, no calibration figure and no judge comparison. source |
| Kev BB Jared Palmer | Arize: 76.8 percent on RAGTruth against the 86.6 percent judge, 88.7 percent on the Phoenix suites, calibration error 0.055, and a coin flip on reversed wording. source |
| JEV-27B-VL C AutoTrust AI Lab | The card's agent-judge result (73.2 percent macro, AUROC 0.91) and multimodal judge tables are AutoTrust's own; one question per request and no human-label comparison. source |
| joey-1 C Baseten Labs, Inc. | noul and score over context and answer, with 16 percent of training on judging and rubric grading. No comparison against an LLM judge on human labels was found. source |
| Liquid d1 C Liquid AI, Inc. | Arize: 84.8 percent on RAGTruth, within noise of the 86.6 percent judge at $0.034 per 1,000, but calibration error 0.051 and the reversed-wording flip to 0.17 AUC. source |
| mammoth-1 D Baseten | noul and score over context and answer, with 16 percent of training on judging and rubric grading. No comparison against an LLM judge on human labels was found. source |
| Microsoft-Decision-1 E Microsoft | Microsoft says its Copilot team found it competitive with GPT-5.6 Luna for judging responses; no independent comparison on human labels was found. source |
| Drex 1.5 BB Nace AI, Inc. | noul and score over context and answer, 512 questions a request; no calibration figure and no comparison against an LLM judge on human labels. source |
| pplx-decider BB Perplexity AI, Inc. | Grouped questions change each other's answers, so independent candidates must go one per request, which multiplies calls; the docs say grouping cost up to 4.8 points on ranking tasks. No judge comparison. source |
| pplx-decider-flash B Perplexity AI, Inc. | One question per request, so no grouping effects, but 10.74 points behind the 27B on the index and the docs say to keep Flash only if it meets your accuracy bar on your own labelled sample. source |
| RightNow System Zero B RightNow AI, Inc. | boolean and score over context and answer, per-question refusals, uncalibrated probabilities and per-question fees; no human-label comparison. source |
| ScaleDown E Carbon ScaleDown, Inc. | noul and score per question, each its own call; 'constrained, calibrated decode' is claimed without a figure, and no judge comparison exists. source |
| SPEV D Sprag AI, Inc. | predicate and score over text, with many questions per request; Sprag's figures cover latency against OpenAI, not accuracy against human labels. source |
| kettei C Sqwish Ltd, trading as Sqwish Labs | A groundedness example and an evaluations endpoint that reports accuracy, log loss and ECE on your own dataset; no independent comparison against an LLM judge. source |
| Vela 2.0 BB vLLM Semantic Router project (with KR Labs) | A 'halu' span head marks unsupported claims with offsets and probabilities and RAGTruth F1 is published; no comparison against an LLM judge on the caller's labels and Choice confidence isn't calibrated by the usage guide's own note. source |
Doesn't do this job 3
Listings in the category that do something else, with the reason.
| Listing | Why |
|---|---|
| Laya BB Convai Innovations Pvt. Ltd. | Arize: 59.8 percent balanced accuracy on RAGTruth against an Opus 5 judge at 86.6 percent, and AUC 0.64 falling to 0.36 with the question reversed. source |
| GLiClass C Knowledgator Engineering Ltd. | No state-plus-question shape for a response against its context; GLiClass scores label strings against one text. Not aimed at this use case. source |
| Strands Decider 2B B Amazon Web Services, Inc. | Arize: 69.0 percent on RAGTruth against the 86.6 percent judge, and AUC 0.75 falling to 0.26 with the question reversed. source |
1 listings in the category were added after this reading and aren't marked yet.
What stops agents finishing
- reversed question wording inverts the answer on several open models
- grouped candidates change each other's answers, so one candidate per request multiplies calls
- context over 32k on long RAG bundles
Around it
Before: inference.llm.generate, vector-databases.retrieve. After: chat.send-message, human-review, evals.record-score.
Where this comes from
Weight 12 of 100 in Decision models: 5 of 41 accounts; the Arize RAGTruth and Phoenix evaluator runs, the OpenRouter verify step and the AWS judge and rerank desks
The same as data: /api/v1/use-cases/decision-models.verify-output.json, or the MCP tool get_use_case with id decision-models.verify-output. Other jobs in this category: /api/v1/niches?category=decision-models.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.