# Choose the next step for an agent acting in a UI, game or environment

> Choose the next step for an agent acting in a UI, game or environment: 5 companies fully do it, 13 with a limit, 30 carry the tag. Every listing in Decision models with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/decision-models.agent-step-choice/
- Markdown: https://www.anchorterminal.com/niches/decision-models.agent-step-choice/index.md (~1,900 tokens)
- Slim: https://www.anchorterminal.com/niches/decision-models.agent-step-choice/index.min.md (~380 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/decision-models.agent-step-choice/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

Four or more companies' docs show a full fit.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 5 | 13 | 30 | 7 | 6 |

## What counts as done

A move per call, chosen from the offered options.

Measured, for our own tests: each move in under 400 ms hosted or 200 ms local, the episode completed at least as often as a rule baseline, without always picking the first option.

How agents ask for it: which direction should the agent move; click, wait, retry or stop; which link on this page leads to the target; is the game state winning

## Does the job

The docs show it meets the test above, and nothing we read says otherwise.

- **Clef (Cloudflare, Inc.)**: Images on Workers AI (up to 4 per request, 16 megapixels each) and video through the weights or Clef-omni; Clef-flash median 38.8 ms, Clef 209 ms. No option-order control.
- **Decision 3.0 (vLLM Semantic Router project)**: Images (several, up to 1.6 megapixels) and video at 2 frames a second; d3-mini answers an image request in 112 ms median, d3 in 290 ms; permutation_average counters first-option bias.
- **Liquid d1 (Liquid AI, Inc.)**: Up to 8 base64 images a request on the hosted API at 0.20 s p50 (Arize); d1-2B adds video and runs on MLX and llama.cpp locally.
- **OpenAI Decisions API (OpenAI OpCo, LLC)**: Images as base64 or public URLs, conversation and tool outputs in input, 0.16 s p50 at Arize, and a voice guide for GPT-Live action selection. No option-order control.
- **pplx-decider (Perplexity AI, Inc.)**: Up to 8 base64 images a question at 2,048 tiles each, a browser-agent cookbook asking six questions a screenshot; images over the tile cap time out at 504 rather than failing fast.
- **pplx-decider-flash (Perplexity AI, Inc.)**: Images up to 8 a question on the hosted API and the Mac app; the 4B is built for latency, and the Mac app's examples label screenshots locally.

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Celeris-1 Decision (Marqo Inc)**: Up to 8 images a request at 25 megapixels, read in order as image_1, image_2. No published latency for the decision model and no control for option order.
- **Laya (Convai Innovations Pvt. Ltd.)**: Fast enough at 33 ms on a GPU, but the core checkpoints read text only (images route to the optional vega engine) and issue 131 shows a position bias on score options.
- **Decider (Mark Marosi (Mapika))**: Images in /v1/systemone since 1.10.0 on the CUDA Python path (GGUF is text only); the README's browser results are 22 click-only MiniWoB++ tasks and issue 9 shows a destructive click. 133 ms on MPS.
- **Drex Max v1 (Nace AI, Inc.)**: Images need the Python loader on an NVIDIA GPU (up to 4,194,304 pixels each); the GGUF, Q4, Kapitsa and MLX paths read text only. Public vision index 73.92 is the vendor's run. No latency figure for image requests.
- **elephant-1 (Baseten Labs, Inc.)**: Images are accepted, but the report says the vision encoder is frozen and training ran on text alone, so image decisions are untested by the authors. No per-decision latency on the card.
- **Kev (Jared Palmer)**: Text only, no image input; a chess demo and the permute endpoint cover order bias, and Kev-4B answers in 18 ms on an H100. Screenshot environments need a text description.
- **JEV-27B-VL (AutoTrust AI Lab)**: Reads screenshots and camera images: about 150 to 300 ms a decision in the Mario, Tetris and robot-arm demos and 239 ms a robot-arm step, on an 80 GB GPU. Over the 200 ms local target and no order control.
- **joey-1 (Baseten Labs, Inc.)**: Images are accepted, but the report says the vision encoder is frozen and training ran on text alone, so image decisions are untested by the authors. No per-decision latency on the card.
- **mammoth-1 (Baseten)**: Images are accepted, but the report says the vision encoder is frozen and training ran on text alone, so image decisions are untested by the authors. No per-decision latency on the card.
- **Drex 1.5 (Nace AI, Inc.)**: Images (up to 8 PNG or JPEG) are accepted on the Drex Max models only, as an images array or the TypeSafe multipart form. No latency figure for image requests.
- **RightNow System Zero (RightNow AI, Inc.)**: Images, 16 kHz WAV audio, PDFs and preview video as inline input, each with its own file fee (USD 0.011 an image); a measured latency table exists but no guarantee. Uncalibrated probabilities.
- **SPEV (Sprag AI, Inc.)**: One inline image per request plus audio and video; no latency figure, one image means no before-and-after comparison.
- **kettei (Sqwish Ltd, trading as Sqwish Labs)**: kettei-chub and kettei-fat read up to 8 pictures with the context; the default kettei-core is text only and no latency is published.
- **Strands Decider 2B (Amazon Web Services, Inc.)**: Text first; the Qwen3.5 v1 model answers over images zero-shot with the vision extra (Gemma 4 models are text only), 115 ms on a GPU, and the 55 percent first-answer preference measured by AWS Builder.
- **Jev (TypeSafe AI, Inc.)**: Images via the TypeSafe-Danger-Zone-Images header and replies under 400 ms through OpenRouter, but the jaggedness page records a lean to the first option and users saw Jev keep picking the first move in a six-room map and fail basic mazes.

- not this job: GLiClass (Knowledgator Engineering Ltd.): Text only; no image input in the pipeline or server.
- not this job: Granite Switch 4.1 (International Business Machines Corporation): Text and audio only; the model pages document no image input.
- not this job: Microsoft-Decision-1 (Microsoft): Text and JSON input only; OpenRouter lists text as the sole input modality.
- not this job: ScaleDown (Carbon ScaleDown, Inc.): Images and PDFs are OCR-extracted to text before classification, so the model never reads pixels; no screenshot decisions.
- not this job: Vela 2.0 (vLLM Semantic Router project (with KR Labs)): Text only; the card documents no image input.

1 listings in the category were added after this reading and aren't marked yet.

## What stops agents finishing

- first-option bias stalls exploration
- models fail multi-step reasoning such as mazes
- images need a CUDA loader on several open models

## Around it

Before: browser-automation.screenshot, computer-use.read-screen. After: browser-automation.click, computer-use.act.

## Where this comes from

Weight 7 of 100 in Decision models: 6 of 41 accounts, all small experiments (mazes, rooms, games, form filling); weighted below the business uses because none was in production

As data: https://www.anchorterminal.com/api/v1/use-cases/decision-models.agent-step-choice.json, or the MCP tool get_use_case with id decision-models.agent-step-choice. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
