# Pick a model, quantisation and context that fit the owner's memory and run fast enough

> Pick a model, quantisation and context that fit the owner's memory and run fast enough: 1 companies fully do it, 16 with a limit, 49 carry the tag. Every listing in Local AI with the reason and a source.

- Canonical: https://www.anchorterminal.com/niches/local-ai.fit-model-to-hardware/
- Markdown: https://www.anchorterminal.com/niches/local-ai.fit-model-to-hardware/index.md (~1,950 tokens)
- Slim: https://www.anchorterminal.com/niches/local-ai.fit-model-to-hardware/index.min.md (~430 tokens, same facts, less prose, for token-sensitive contexts)
- JSON: https://www.anchorterminal.com/niches/local-ai.fit-model-to-hardware/index.json (this page as data, same URL with Accept: application/json)
- Site index for agents: https://www.anchorterminal.com/llms.txt (the key docs and every listing in one file, at most 16 MB, about 4 million tokens: https://www.anchorterminal.com/llms-full.txt)
- API: https://www.anchorterminal.com/api/v1/index.json
- MCP: https://www.anchorterminal.com/mcp (the whole catalogue as tools, no key)
- DNS: dig +short TXT <slug>.grade.anchorterminal.com (the grade as a DNS TXT record, for a sandbox with no HTTP where names still resolve; also top., best.<category>. and <a>.vs.<b>. under the zone, and the same under grade.letme.dev)
- Licence: CC BY 4.0, credit Anchor Terminal and link this page (https://www.anchorterminal.com/terms/#data)
- Updated: 2026-11-15

One company's docs show a full fit.

| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |
| --- | --- | --- | --- | --- |
| 1 | 16 | 49 | 12 | 11 |

## What counts as done

The runner reports the loaded model's memory use and context size before the first request, and refuses or warns when weights plus KV cache exceed physical memory.

Measured, for our own tests: the model generates at 20 tokens per second or more at the chosen context.

How agents ask for it: what can I run on a 16 GB card; will a 27B model fit on my Strix Halo; why is my Mac thrashing when I load this model; how many tokens per second will I get

## Does the job

The docs show it meets the test above, and nothing we read says otherwise.

- **llama.cpp (ggml.ai, part of Hugging Face since 2026)**: --fit (on by default) adjusts unset context and offload to device memory with a margin; /props reports n_ctx. The maintainers publish Apple silicon tokens-per-second tables, and accounts report 50 tokens a second for 27B on 24 GB cards.

## Does the job, with a limit

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

- **Berth (Trevin Tindall (T-Crypt on GitHub))**: berth models indexes GGUFs across llama.cpp, LM Studio and Ollama stores with quant, size, parameters and context, and the daemon waits for the driver to show memory free before granting a card. Alpha; AMD and Windows experimental; no speed table.
- **BitRun (Abhishek Khanra (IshekKhal on GitHub))**: Publishes tokens per second, memory and context per model on named CPUs (M4 Pro, i5-12600K, WSL2), and the Cockpit shows context use and RAM. CPU ternary models only; no memory guard.
- **Docker Model Runner (Docker, Inc.)**: Variants for CPU/Vulkan, CUDA and ROCm are documented. An account on a 24 GB Mac saw the stored 16k context ignored, a 128k default allocate about 21 GB of KV cache and the host kernel-panic three times.
- **Foundry Local (Microsoft Corporation)**: Automatic selection of CPU, GPU or NPU variants and foundry model list --device. No memory use report, refusal or tokens-per-second table in the reference pages.
- **GPT4All (Nomic, Inc.)**: Settings expose device (Auto, Metal, CPU, GPU), GPU layers and context length. No memory guard or speed table; an account calls it slow but functional on 8 GB VRAM.
- **HyperQwen (syv.ai (Copenhagen, CVR DK44671247))**: Publishes tokens per second per profile on a named card at 250 W, memory needs (24 GB or two cards) and context per profile. One model and one card class only; figures are the project's own.
- **Jan (Menlo Research Pte Ltd)**: Fit to Hardware caps context size from model and memory, on by default only on Windows and Linux with a discrete GPU, off on macOS and integrated GPUs. GPU layer guidance is given; no speed table.
- **KoboldCpp (LostRuins (Concedo))**: The wiki gives RAM tables at 2048 context, --contextsize and --gpulayers guidance, and CUDA, Vulkan, Metal and ROCm-fork backends. No memory guard: crashes are answered with advice to reduce context. No speed table.
- **Lemonade (Advanced Micro Devices, Inc.)**: README publishes a backend matrix (Metal, CUDA, Vulkan, ROCm, XDNA2 NPU recipes) and /v1/models reports context_length; ctx_size -1 sizes automatically. No memory refusal or speed table in the pages read.
- **LM Studio (Element Labs, Inc.)**: System requirements give RAM and VRAM floors but no memory guard or speed table is documented. One account could not load LFM3 MLX models (safetensors unreadable), another restarted models when Copilot Chat failed.
- **LocalAI (Ettore Di Giacinto and the LocalAI team)**: The GPU acceleration page covers CUDA, ROCm, Intel, Vulkan and Metal images. No memory guard or tokens-per-second table.
- **MLX LM (Apple Inc.)**: --kv-bits quantises the cache and --max-kv-size rotates it to bound memory; a macOS 15 wired-memory note exists. No refusal when weights exceed memory and no speed table.
- **Ollama (Ollama Inc.)**: ollama ps reports size, CPU/GPU split and context after load. A model that does not fit spills to system memory (48%/52% CPU/GPU) rather than refusing. No tokens-per-second table is published.
- **TextGen (oobabooga)**: Models load with ctx_size and cache_type arguments over /v1/internal/model/load across llama.cpp, ExLlamaV3 and Transformers. No memory guard or speed table.
- **Underdog (Conway Research)**: The card publishes decode rates per task on a named machine (M5 Max, 48 GiB) and a 16 GB download of MLX 4-bit weights. Apple silicon only, no context length, no memory guidance beyond the measured machine.
- **vLLM (The Linux Foundation (vLLM is a PyTorch Foundation project))**: The memory page lists options when a model would run out of memory (gpu_memory_utilization, max-model-len, quantised KV) and the KV cache is preallocated at start. Aimed at GPU servers; no consumer hardware speed table read.

## Couldn't check

The docs were closed to us, or silent. These carry no mark either way.

- **Core (ZMJ, Inc., doing business as Ghost, 325 9th Street, San Francisco, CA 94103, United States)**: Fixed hardware (24 GB GDDR7, 64 GB DDR5) with a model picker in settings; nothing reports memory or speed.

- not this job: AnythingLLM (Mintplex Labs, Inc.): Not a runner; models load in Ollama, LM Studio, LocalAI or the built-in provider.
- not this job: Khoj (Khoj Inc.): Khoj points at Ollama, vLLM or LM Studio for chat; it does not load weights itself.
- not this job: LocalGhost (LocalGhost.ai Ltd (company number 17213100, registered in England and Wales, incorporated 12 May 2026)): Fixed stack: llama.cpp with Gemma 4 12B on a Debian box with an NVIDIA GPU; nothing to pick.
- not this job: Open WebUI (Open WebUI Inc.): Not a runner; models load in Ollama or another backend.
- not this job: PrivateGPT (PriBAI Technology Spain S.L.): Does not load weights; it proxies Ollama, LM Studio, llama-server or vLLM.
- not this job: screenpipe (Negentropy Labs, Inc. (dba Screenpipe)): Does not serve language models.

## What stops agents finishing

- stored context size not applied at load, so the packaged 128k default allocates a 21 GB KV cache and the host panics
- VRAM oversubscription spills to shared memory and the resident model runs five times slower
- a model format (MLX safetensors) the app lists but cannot load
- MoE model at 12 to 15 tokens per second on a 96 GB Mac, too slow for an agent

## Around it

Before: open-weights.big-model-on-big-box. After: local-ai.coding-agent-backend, local-ai.household-or-team-chat.

## Where this comes from

Weight 12 of 100 in Local AI: 11 of 50 accounts give hardware, model size and a speed or a memory failure; the buyer checks name memory and tokens per second

As data: https://www.anchorterminal.com/api/v1/use-cases/local-ai.fit-model-to-hardware.json, or the MCP tool get_use_case with id local-ai.fit-model-to-hardware. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.

---

Query this catalogue directly over MCP: https://www.anchorterminal.com/mcp (Streamable HTTP, no key). How: https://www.anchorterminal.com/agents/#mcp
