Niches · Decision models

Check an LLM's output before it is sent or stored

Four or more companies' docs show a full fit.

4companies, full fit
16companies, fit with a limit
30companies carry the tag
12share of the category's work
5first-hand accounts

What counts as done

A supported or unsupported probability per response.

Measured, for our own tests: the probabilities match held-out human labels at least as well as an LLM judge on the caller's set, at under a tenth of the judge's cost. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: is every claim in this answer supported by the context, does this draft follow policy, is this retrieved passage relevant, which of these two replies is better

Does the job 4

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
Celeris-1 Decision D
Marqo Inc
predicate and score questions over a response plus context, 512 questions a request, $0.004 per million input tokens, far under a judge's cost. No independent accuracy measurement against human labels was found. source
Clef B
Cloudflare, Inc.
Arize: 85.2 percent balanced accuracy on RAGTruth, within noise of an Opus 5 judge at 86.6 percent and $14.30 per 1,000, at $0.223 per 1,000; 91.5 percent on the Phoenix evaluator suites; wording-stable. source
OpenAI Decisions API A
OpenAI OpCo, LLC
Arize: 81.5 percent on RAGTruth with intervals touching Jev's, 90.0 percent on the Phoenix suites, calibration error 0.042, wording-stable, $0.090 per 1,000 against $14.30 for an Opus 5 judge. source
Jev B
TypeSafe AI, Inc.
Arize: 85.5 percent on RAGTruth with calibration error 0.023, within noise of an Opus 5 judge at 86.6 percent and $14.30, at $0.045 per 1,000; wording-stable (0.93 both ways); a verify_llm_outputs cookbook exists. source

Does the job, with a limit 19

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Decider BB
Mark Marosi (Mapika)
noul and score questions over a context, with the README warning that calibration on hard items is the weak axis (ECE 0.15 to 0.31 on JevBench hard). No measurement against an LLM judge was found. source
Decision 3.0 B
vLLM Semantic Router project
noul and score questions over context and answer in one call. No calibration figures on the cards and no measurement against an LLM judge; issue 6367 crashed the batched /v1/decisions route on a rejected state. source
Drex Max v1 C
Nace AI, Inc.
noul and score over context and answer; vendor's Decision Index and Artificial Analysis scores only, no judge comparison. Licence cap. source
elephant-1 D
Baseten Labs, Inc.
noul and score over context and answer, with 16 percent of training on judging and rubric grading. No comparison against an LLM judge on human labels was found. source
Granite Switch 4.1 B
International Business Machines Corporation
Possible as a two-label or multi-label logprob read with the state as a chat turn; no typed answer, no calibration figure and no judge comparison. source
Kev BB
Jared Palmer
Arize: 76.8 percent on RAGTruth against the 86.6 percent judge, 88.7 percent on the Phoenix suites, calibration error 0.055, and a coin flip on reversed wording. source
JEV-27B-VL C
AutoTrust AI Lab
The card's agent-judge result (73.2 percent macro, AUROC 0.91) and multimodal judge tables are AutoTrust's own; one question per request and no human-label comparison. source
joey-1 C
Baseten Labs, Inc.
noul and score over context and answer, with 16 percent of training on judging and rubric grading. No comparison against an LLM judge on human labels was found. source
Liquid d1 C
Liquid AI, Inc.
Arize: 84.8 percent on RAGTruth, within noise of the 86.6 percent judge at $0.034 per 1,000, but calibration error 0.051 and the reversed-wording flip to 0.17 AUC. source
mammoth-1 D
Baseten
noul and score over context and answer, with 16 percent of training on judging and rubric grading. No comparison against an LLM judge on human labels was found. source
Microsoft-Decision-1 E
Microsoft
Microsoft says its Copilot team found it competitive with GPT-5.6 Luna for judging responses; no independent comparison on human labels was found. source
Drex 1.5 BB
Nace AI, Inc.
noul and score over context and answer, 512 questions a request; no calibration figure and no comparison against an LLM judge on human labels. source
pplx-decider BB
Perplexity AI, Inc.
Grouped questions change each other's answers, so independent candidates must go one per request, which multiplies calls; the docs say grouping cost up to 4.8 points on ranking tasks. No judge comparison. source
pplx-decider-flash B
Perplexity AI, Inc.
One question per request, so no grouping effects, but 10.74 points behind the 27B on the index and the docs say to keep Flash only if it meets your accuracy bar on your own labelled sample. source
RightNow System Zero B
RightNow AI, Inc.
boolean and score over context and answer, per-question refusals, uncalibrated probabilities and per-question fees; no human-label comparison. source
ScaleDown E
Carbon ScaleDown, Inc.
noul and score per question, each its own call; 'constrained, calibrated decode' is claimed without a figure, and no judge comparison exists. source
SPEV D
Sprag AI, Inc.
predicate and score over text, with many questions per request; Sprag's figures cover latency against OpenAI, not accuracy against human labels. source
kettei C
Sqwish Ltd, trading as Sqwish Labs
A groundedness example and an evaluations endpoint that reports accuracy, log loss and ECE on your own dataset; no independent comparison against an LLM judge. source
Vela 2.0 BB
vLLM Semantic Router project (with KR Labs)
A 'halu' span head marks unsupported claims with offsets and probabilities and RAGTruth F1 is published; no comparison against an LLM judge on the caller's labels and Choice confidence isn't calibrated by the usage guide's own note. source

Doesn't do this job 3

Listings in the category that do something else, with the reason.

ListingWhy
Laya BB
Convai Innovations Pvt. Ltd.
Arize: 59.8 percent balanced accuracy on RAGTruth against an Opus 5 judge at 86.6 percent, and AUC 0.64 falling to 0.36 with the question reversed. source
GLiClass C
Knowledgator Engineering Ltd.
No state-plus-question shape for a response against its context; GLiClass scores label strings against one text. Not aimed at this use case. source
Strands Decider 2B B
Amazon Web Services, Inc.
Arize: 69.0 percent on RAGTruth against the 86.6 percent judge, and AUC 0.75 falling to 0.26 with the question reversed. source

1 listings in the category were added after this reading and aren't marked yet.

What stops agents finishing

  • reversed question wording inverts the answer on several open models
  • grouped candidates change each other's answers, so one candidate per request multiplies calls
  • context over 32k on long RAG bundles

Around it

Before: inference.llm.generate, vector-databases.retrieve. After: chat.send-message, human-review, evals.record-score.

Where this comes from

Weight 12 of 100 in Decision models: 5 of 41 accounts; the Arize RAGTruth and Phoenix evaluator runs, the OpenRouter verify step and the AWS judge and rerank desks

The same as data: /api/v1/use-cases/decision-models.verify-output.json, or the MCP tool get_use_case with id decision-models.verify-output. Other jobs in this category: /api/v1/niches?category=decision-models.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.