Niches · Local AI

Pick a model, quantisation and context that fit the owner's memory and run fast enough

One company's docs show a full fit.

1companies, full fit
16companies, fit with a limit
49companies carry the tag
12share of the category's work
11first-hand accounts

What counts as done

The runner reports the loaded model's memory use and context size before the first request, and refuses or warns when weights plus KV cache exceed physical memory.

Measured, for our own tests: the model generates at 20 tokens per second or more at the chosen context. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: what can I run on a 16 GB card, will a 27B model fit on my Strix Halo, why is my Mac thrashing when I load this model, how many tokens per second will I get

Does the job 1

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
llama.cpp D
ggml.ai, part of Hugging Face since 2026
--fit (on by default) adjusts unset context and offload to device memory with a margin; /props reports n_ctx. The maintainers publish Apple silicon tokens-per-second tables, and accounts report 50 tokens a second for 27B on 24 GB cards. source

Does the job, with a limit 16

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Berth B
Trevin Tindall (T-Crypt on GitHub)
berth models indexes GGUFs across llama.cpp, LM Studio and Ollama stores with quant, size, parameters and context, and the daemon waits for the driver to show memory free before granting a card. Alpha; AMD and Windows experimental; no speed table. source
BitRun E
Abhishek Khanra (IshekKhal on GitHub)
Publishes tokens per second, memory and context per model on named CPUs (M4 Pro, i5-12600K, WSL2), and the Cockpit shows context use and RAM. CPU ternary models only; no memory guard. source
Docker Model Runner C
Docker, Inc.
Variants for CPU/Vulkan, CUDA and ROCm are documented. An account on a 24 GB Mac saw the stored 16k context ignored, a 128k default allocate about 21 GB of KV cache and the host kernel-panic three times. source
Foundry Local B
Microsoft Corporation
Automatic selection of CPU, GPU or NPU variants and foundry model list --device. No memory use report, refusal or tokens-per-second table in the reference pages. source
GPT4All F
Nomic, Inc.
Settings expose device (Auto, Metal, CPU, GPU), GPU layers and context length. No memory guard or speed table; an account calls it slow but functional on 8 GB VRAM. source
HyperQwen C
syv.ai (Copenhagen, CVR DK44671247)
Publishes tokens per second per profile on a named card at 250 W, memory needs (24 GB or two cards) and context per profile. One model and one card class only; figures are the project's own. source
Jan C
Menlo Research Pte Ltd
Fit to Hardware caps context size from model and memory, on by default only on Windows and Linux with a discrete GPU, off on macOS and integrated GPUs. GPU layer guidance is given; no speed table. source
KoboldCpp B
LostRuins (Concedo)
The wiki gives RAM tables at 2048 context, --contextsize and --gpulayers guidance, and CUDA, Vulkan, Metal and ROCm-fork backends. No memory guard: crashes are answered with advice to reduce context. No speed table. source
Lemonade B
Advanced Micro Devices, Inc.
README publishes a backend matrix (Metal, CUDA, Vulkan, ROCm, XDNA2 NPU recipes) and /v1/models reports context_length; ctx_size -1 sizes automatically. No memory refusal or speed table in the pages read. source
LM Studio C
Element Labs, Inc.
System requirements give RAM and VRAM floors but no memory guard or speed table is documented. One account could not load LFM3 MLX models (safetensors unreadable), another restarted models when Copilot Chat failed. source
LocalAI B
Ettore Di Giacinto and the LocalAI team
The GPU acceleration page covers CUDA, ROCm, Intel, Vulkan and Metal images. No memory guard or tokens-per-second table. source
MLX LM C
Apple Inc.
--kv-bits quantises the cache and --max-kv-size rotates it to bound memory; a macOS 15 wired-memory note exists. No refusal when weights exceed memory and no speed table. source
Ollama D
Ollama Inc.
ollama ps reports size, CPU/GPU split and context after load. A model that does not fit spills to system memory (48%/52% CPU/GPU) rather than refusing. No tokens-per-second table is published. source
TextGen E
oobabooga
Models load with ctx_size and cache_type arguments over /v1/internal/model/load across llama.cpp, ExLlamaV3 and Transformers. No memory guard or speed table. source
Underdog F
Conway Research
The card publishes decode rates per task on a named machine (M5 Max, 48 GiB) and a 16 GB download of MLX 4-bit weights. Apple silicon only, no context length, no memory guidance beyond the measured machine. source
vLLM D
The Linux Foundation (vLLM is a PyTorch Foundation project)
The memory page lists options when a model would run out of memory (gpu_memory_utilization, max-model-len, quantised KV) and the KV cache is preallocated at start. Aimed at GPU servers; no consumer hardware speed table read. source

Couldn't check 1

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
Core F
ZMJ, Inc., doing business as Ghost, 325 9th Street, San Francisco, CA 94103, United States
Fixed hardware (24 GB GDDR7, 64 GB DDR5) with a model picker in settings; nothing reports memory or speed.

Doesn't do this job 6

Listings in the category that do something else, with the reason.

ListingWhy
AnythingLLM E
Mintplex Labs, Inc.
Not a runner; models load in Ollama, LM Studio, LocalAI or the built-in provider. source
Khoj E
Khoj Inc.
Khoj points at Ollama, vLLM or LM Studio for chat; it does not load weights itself. source
LocalGhost F
LocalGhost.ai Ltd (company number 17213100, registered in England and Wales, incorporated 12 May 2026)
Fixed stack: llama.cpp with Gemma 4 12B on a Debian box with an NVIDIA GPU; nothing to pick. source
Open WebUI D
Open WebUI Inc.
Not a runner; models load in Ollama or another backend. source
PrivateGPT C
PriBAI Technology Spain S.L.
Does not load weights; it proxies Ollama, LM Studio, llama-server or vLLM. source
screenpipe C
Negentropy Labs, Inc. (dba Screenpipe)
Does not serve language models. source

What stops agents finishing

  • stored context size not applied at load, so the packaged 128k default allocates a 21 GB KV cache and the host panics
  • VRAM oversubscription spills to shared memory and the resident model runs five times slower
  • a model format (MLX safetensors) the app lists but cannot load
  • MoE model at 12 to 15 tokens per second on a 96 GB Mac, too slow for an agent

Around it

Before: open-weights.big-model-on-big-box. After: local-ai.coding-agent-backend, local-ai.household-or-team-chat.

Where this comes from

Weight 12 of 100 in Local AI: 11 of 50 accounts give hardware, model size and a speed or a memory failure; the buyer checks name memory and tokens per second

The same as data: /api/v1/use-cases/local-ai.fit-model-to-hardware.json, or the MCP tool get_use_case with id local-ai.fit-model-to-hardware. Other jobs in this category: /api/v1/niches?category=local-ai.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.