Category · Models & inference
Local AI: models and assistants that run on your own hardware
Software that runs models on hardware the owner keeps, a laptop, a desktop or a home server. Model runners with a local API, chat apps, and personal assistants that work from the owner's own files, mail and records, with no cloud account needed. Compared on what models they run, what hardware they need, what leaves the machine, what an agent can call and the licence.
Best local AI models and assistants, the shortlist284 head-to-head comparisons
Capability keys inference.local · inference.open-weights · memory.user · memory.search · agent.mcp-client · All tools
letme.dev/inference.local picks the top-graded tool in this list and says how to call it direct (how letme works).
The same listing from the live API. Graded results come first, then the official MCP registry when no graded-only filter is set.
https://www.anchorterminal.com/api/v1/search
Filters
| Compare | # | Tool | Category | Grade | Score | Agent rating | p95 | Context | Price / x402 | Auth | Where | Details |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 386 | BerthTrevin Tindall (T-Crypt on GitHub) · HTTP API | Local AI | B | 68.4 | none | n/a | n/a | Free · OSS | API key | Local | ||
|
Berth is an open-source Go daemon and CLI that leases one GPU to local inference engines, image pipelines and games in turn, with a priority queue, measured VRAM handoffs and a preview OpenAI-compatible gateway on loopback. Top strength A lease is granted only after the driver shows the previous holder's memory free, and processes it did not start are reported, never killed Top weakness First released on 11 October 2026 and called alpha. Windows, AMD and multiple GPUs are experimental, and the gateway is a preview |
||||||||||||
| 429 | LemonadeAMD and the Lemonade community · HTTP API | Local AI | B | 67.6 | 3.0 (4) | n/a | n/a | Free · OSS | API key | Local | ||
|
Open-source local AI server from AMD and community contributors. It runs text, speech and image models on the owner's CPU, GPU or NPU behind OpenAI-, Anthropic- and Ollama-compatible APIs and an MCP endpoint on port 13305. Top strength Apache 2.0, with OpenAI, Anthropic Messages, Ollama and llama.cpp-compatible routes on one port (13305) Top weakness No authentication by default. With no key set, every route answers, including |
||||||||||||
| 495 | LocalAIEttore Di Giacinto and the LocalAI team · HTTP API | Local AI | B | 66.2 | 3.0 (2) | n/a | n/a | Free · OSS | OAuth or key | Local | ||
|
Open-source engine in Go, MIT licensed, that runs models on the owner's hardware behind OpenAI-, Anthropic-, Ollama- and ElevenLabs-compatible APIs on port 8080. Top strength MIT and Go, with Docker images for CUDA 12 and 13, ROCm, Intel oneAPI, Vulkan, Jetson and CPU, Linux binaries and a macOS app Top weakness No authentication by default. Loopback, LAN and VPN binds answer every caller, and keys set by environment variable grant full admin |
||||||||||||
| 511 | KoboldCppLostRuins (Concedo) · HTTP API | Local AI | B | 65.9 | 2.8 (4) | n/a | n/a | Free · OSS | None | Local | ||
|
Open-source program for running GGUF models on the owner's own computer, built on llama.cpp. One executable serves a web interface and KoboldAI, OpenAI, Ollama and Anthropic compatible APIs on port 5001. Top strength OpenAPI 3.0.3 document served at Top weakness With no |
||||||||||||
| 596 | Foundry LocalMicrosoft · SDK + MCP | Local AI | B | 64.3 | 3.0 (4) | n/a | n/a | Free · OSS | None | Local | ||
|
Microsoft's on-device model runtime, built on ONNX Runtime. Applications embed it through SDKs for C#, JavaScript, Python and Rust, and it can start an optional OpenAI-compatible server on localhost. A preview CLI is also available. Top strength SDK 2.1.0 published on npm, PyPI, NuGet and crates.io on 29 September 2026, with the SDK and the v2 native runtime under MIT in a public repository Top weakness The local server takes no credential, and its routes include model load and unload and |
||||||||||||
| 752 | JanMenlo Research · Platform | Local AI | C | 60.9 | 2.0 (2) | n/a | n/a | Free · OSS | API key | Local | ||
|
Open-source desktop app for running models locally or connecting to cloud models with the user's API keys. Top strength Apache-2.0, with installers for macOS, Windows and Linux plus Flathub and the Microsoft Store Top weakness 0.8.8's security fixes shipped on 28 October 2026 with advisories promised later, and none was published by 4 November |
||||||||||||
| 767 | screenpipeNegentropy Labs, Inc. (dba Screenpipe) · Platform | Local AI | C | 60.5 | 2.0 (2) | n/a | n/a | $21 / mo | API key | Local | ||
|
Desktop app and CLI from Negentropy Labs, Inc. (Screenpipe) that record the owner's screen and audio on macOS, Windows and Linux and serve the history to agents through a local REST API and an MCP server. Top strength Every one of the 34 stdio MCP tools has a typed input schema and annotations, 22 of them readOnlyHint. Top weakness The 34 tool definitions run to about 36,000 characters in the source, with no toolsets, and only the HTTP transport's three tools as a smaller set. |
||||||||||||
| 835 | LM StudioElement Labs, Inc. · HTTP API | Local AI | C | 58.8 | 2.5 (2) | n/a | n/a | Free | API key | Local | ||
|
Desktop app and headless daemon from Element Labs for running open-weight models on the owner's machine with llama.cpp and MLX, plus the Splash engine on Apple silicon M3 or newer since 0.4.25. Top strength OpenAI-compatible chat completions, responses, completions and embeddings, Anthropic-compatible /v1/messages and a native /api/v1, all on one port Top weakness Authentication is off by default, so any local process can call the server |
||||||||||||
| 882 | Docker Model RunnerDocker · HTTP API | Local AI | C | 57.6 | none | n/a | n/a | Free · OSS | None | Local | ||
|
Docker's open-source tool for pulling and running open models from Docker Hub, OCI registries or Hugging Face. It runs through Docker Desktop, Docker Engine or a standalone `dmr` binary, with local OpenAI-, Anthropic- and Ollama-compatible APIs. Top strength OpenAI-, Anthropic- and Ollama-compatible routes on one local port, so existing clients for those three APIs work with a changed base URL Top weakness No credential on the API. The docs say any client that can reach it, including other containers, can pull, load and run models |
||||||||||||
| 911 | PrivateGPTZylon (PriBAI Technology Spain S.L.) · HTTP API | Local AI | C | 56.9 | none | n/a | n/a | Free · OSS | None | Local | ||
|
PrivateGPT is Zylon's open-source server that puts an Anthropic-style messages API, document ingestion, retrieval with citations, built-in tools and MCP connectors in front of any OpenAI-compatible local model server. The owner runs it. Top strength Messages API modelled on Anthropic's, so Claude Code and Anthropic SDKs work by changing the base URL Top weakness Authentication is off by default, the server binds 0.0.0.0 and CORS allows every origin |
||||||||||||
| 1004 | HyperQwensyv.ai (syv-ai on GitHub) · HTTP API | Local AI | C | 54.6 | none | n/a | n/a | Free · OSS | None | Local | ||
|
HyperQwen is syv.ai's open-source serving setup for Qwen3.8-27B on one 24 GB NVIDIA GPU. A patch series against a pinned vLLM, a requantisation pipeline and Docker Compose profiles run vLLM's OpenAI-compatible API on port 18020. Top strength Prebuilt image on ghcr.io with an immutable tag per commit, built by CI that applies every patch and runs the build-time tests Top weakness Serves Qwen3.8-27B and checkpoints derived from it only. Other models are on the roadmap |
||||||||||||
| 1012 | MLX LMApple · HTTP API | Local AI | C | 54.4 | none | n/a | n/a | Free · OSS | None | Local | ||
|
Open-source Python package and command-line tools from Apple's MLX team for running, quantising and fine-tuning language models on Apple silicon. `mlx_lm.server` exposes a local HTTP API modelled on OpenAI's chat completions. Top strength MIT, with no telemetry, analytics or update check found in the source Top weakness
|
||||||||||||
| 1068 | Open WebUIOpen WebUI Inc. · Platform | Local AI | D | 52.8 | 2.5 (2) | n/a | n/a | Free | API key | Local | ||
|
Self-hosted web interface for chatting with models, from Open WebUI Inc., with a Python (FastAPI) back end and a Svelte front end. Top strength 14 releases in the 90 days to 11 November 2026, 0.11.1 to 0.13.3, each with a dated changelog section. Top weakness 129 reviewed advisories cover flaws fixed from 24 November 2025 to 25 August 2026, 1 Critical and 56 High among them, with 19 more fixed in 0.11.4. |
||||||||||||
| 1095 | vLLMvLLM project (PyTorch Foundation) · HTTP API | Local AI | D | 52.2 | none | n/a | n/a | Free · OSS | None | Local | ||
|
vLLM is an open-source inference and serving engine for open-weight language models. `vllm serve` runs an HTTP server with OpenAI-compatible, Anthropic Messages, embedding, reranking and transcription routes on the owner's own GPUs or CPUs. Top strength OpenAI chat, completions, responses and embeddings, Anthropic Top weakness
|
||||||||||||
| 1159 | OllamaOllama Inc. · HTTP API | Local AI | D | 50.1 | 2.5 (2) | n/a | n/a | $20 / mo | None | Local | ||
|
Open-source model runner for macOS, Windows and Linux, with a local API and a library of downloadable models. Top strength An OpenAPI 3.1 file for the 15 native operations and llms.txt with 68 links to Markdown pages Top weakness No credential on the local API, and any caller that reaches it can pull, push, create and delete models |
||||||||||||
| 1191 | llama.cppggml.ai (Hugging Face) · HTTP API | Local AI | D | 48.9 | 2.5 (2) | n/a | n/a | Free · OSS | None | Local | ||
|
Open-source C/C++ engine for running GGUF models locally, with a web interface and compatible model APIs. Top strength MIT, with no telemetry or update check in the source, and Top weakness API keys are off by default and CORS reflects any origin with credentials, so a web page can call a keyless server on localhost |
||||||||||||
| 1262 | AnythingLLMMintplex Labs · Platform | Local AI | E | 45.9 | 2.0 (2) | n/a | n/a | $50 / mo | API key | Local | ||
|
Open-source app for chatting with documents using local or hosted models. Available as a desktop app or a self-hosted server. Top strength MIT server with desktop builds for macOS, Windows and Linux and Docker images for amd64 and arm64 Top weakness One kind of API key, admin-equivalent across every endpoint, with no scopes or expiry, stored in plain text |
||||||||||||
| 1336 | TextGenoobabooga · Platform | Local AI | E | 42.4 | 2.5 (4) | n/a | n/a | Free · OSS | API key | Local | ||
|
Open-source desktop and browser app for running language models on the owner's hardware, formerly text-generation-webui. With `--api` it serves an OpenAI and Anthropic-compatible local API with tool calling, vision, embeddings and image generation. Top strength AGPL-3.0, with analytics switched off in the source and a README that states zero telemetry Top weakness No release since v4.9 on 20 May 2026 and no code commit on |
||||||||||||
| 1345 | BitRunAbhishek Khanra (IshekKhal on GitHub) · HTTP API | Local AI | E | 42 | 2.8 (4) | n/a | n/a | Free · OSS | None | Local | ||
|
BitRun is an open-source command-line runner for 1-bit and 1.58-bit ternary language models such as BitNet b1.58, Falcon3 1.58-bit and Bonsai on CPUs, with a model downloader and a local OpenAI-compatible server. Top strength One archive per platform (macOS arm64, Linux amd64, Windows amd64) with llama-server, so no compiler, Python or Docker is needed Top weakness First released on 12 October 2026. Three pre-release versions in four days, none since, one maintainer and no CI workflow |
||||||||||||
| 1392 | KhojKhoj Inc. · Platform | Local AI | E | 39.5 | 1.0 (2) | n/a | n/a | Free · OSS | OAuth or key | Local | ||
|
Open-source personal AI application with a Python server and a web interface, for search and chat over a person's own documents. Self-hosted only since Khoj Cloud closed on 15 April 2026. Top strength AGPL-3.0-or-later for the server and every client, self-hosted with nothing to buy and no account or card. Top weakness No stable release since 1.42.10 (15 July 2025) and no tag since 2.0.0-beta.28 (26 March 2026). Last commit on master 2 August 2026. |
||||||||||||
| 1411 | GPT4AllNomic, Inc. · Platform | Local AI | F | 37.6 | 1.0 (2) | n/a | n/a | Free · OSS | None | Local | ||
|
Desktop app from Nomic that runs GGUF models on Windows, macOS and Linux through Nomic's fork of llama.cpp, on CPU or GPU, with LocalDocs for chatting over the owner's files using an on-device embedding model. Top strength MIT, with installers for Windows x64 and ARM64, macOS 12.6 or later and Linux, and published minimum and recommended hardware Top weakness No release since 24 February 2025 and no commit to main since 27 May 2025 |
||||||||||||
| 1460 | LocalGhostLocalGhost · Platform | Local AI | F | 30.1 | 2.0 (3) | n/a | n/a | Free · OSS | Token | Local | ||
|
Open-source personal AI server that runs on a Debian machine its owner keeps, archiving photos, location, notes, voice and health data and reading them with local models, with its own Android app as the only client. Top strength MIT licence for the server and the Android app, with release notes and setup guides in the repository. Top weakness No API, MCP server or credential for any agent. The box answers every caller but its enrolled phone with a 503. |
||||||||||||
| 1507 | UnderdogConway Research · Platform | Local AI | F | 18.2 | 2.0 (2) | n/a | n/a | Free | None | Local | ||
|
Underdog is a personal AI app from Conway Research that runs Conway's own open-weight models on the owner's Mac with Apple silicon. Top strength Conway's cards say the app runs Woof, Saluki and Bark on the Mac, with Bark's audio kept on the device. Top weakness No API, MCP server, CLI or SDK from Conway, so an outside agent has nothing documented to call. |
||||||||||||
| 1548 | CoreGhost (ZMJ, Inc.) · Platform | Local AI | F | 6.6 | 2.0 (1) | n/a | n/a | Paid | OAuth or key | Local | ||
|
Core is a personal AI computer from Ghost that runs open-weight language models and a memory of the owner's apps, files and devices at home, used through Ghost's apps and an OpenAI Responses-compatible endpoint. Batch 1 is sold out. Top strength Model inference and memory run on Core, and the privacy policy says there is no remote model fallback. Top weakness The Responses-compatible endpoint has no published address, port, authentication method, API reference or error documentation. |
||||||||||||
Nothing matches these filters. .
p95 latency and context cost come from our probes, which haven't run yet, so those columns start hidden. Grades run from AA to F, and agent-ready means BB or better. Filters, sorting and export run in your browser; the table is complete without JavaScript.
How we test this category
In this run, public evidence against the published checklist, read as software the owner runs on their own hardware (the local-software lines for Reliability and the self-hosted rule for Payments). When the task suites run, the same small open model, prompts and documents on the same machine through each listing's local API or MCP server. We check the setup steps, tokens per second, memory use, whether answers cite the right file and what a network monitor sees leave the machine.
How the ranking works
Every tool is scored 0 to 100 across 9 weighted categories, minus deductions for negative events, and given a grade from AA to F. Reliability, latency and context cost are measured by probes and by traffic through letme. Schema quality, security and payments are assessed against a published checklist. Task success comes from a fixed task suite. Payments, including x402, are one category of the nine. Read the methodology.
For companies
Do agents find, use and choose your tools?
An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.