Niches · Local AI

Keep a 32k to 128k token agent session without silent truncation or a crash

One company's docs show a full fit.

1companies, full fit
10companies, fit with a limit
26companies carry the tag
8share of the category's work
8first-hand accounts

What counts as done

A 60k token prompt is accepted whole and the server reports the active context length through its API.

Measured, for our own tests: the response shows no sign of a truncated prefix, and the same tool works at 60k as at 5k. No docs we read state these figures, so they don't count against a listing here.

How agents ask for it: set the context to 64k for Claude Code, why did the model forget the start of the task, run a 100k token repository read locally, stop the model looping once the conversation is long

Does the job 1

The docs show it meets the test above, and nothing we read says otherwise.

ListingWhy
HyperQwen C
syv.ai (Copenhagen, CVR DK44671247)
CTX=long gives 150k and CTX=huge 240k with measured decode rates and cold time-to-first-token at 66k and 100k. vLLM sizes the KV pool at start and the README states the pool size per arm. source

Does the job, with a limit 10

It does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.

ListingWhy
Docker Model Runner C
Docker, Inc.
Context is configured per model, but the account shows the stored value not always applied at load, with the packaged 128k default used instead. source
Jan C
Menlo Research Pte Ltd
Context size is a per-model setting and Fit to Hardware caps it where enabled. Nothing documents the default, what happens past it, or an API field reporting the active value. source
KoboldCpp B
LostRuins (Concedo)
--contextsize sets the window and /api/extra/true_max_context_length reports it over the API. Smart Context and context shift drop the oldest half when full rather than rejecting, so long sessions lose their prefix silently. source
Lemonade B
Advanced Micro Devices, Inc.
ctx_size is set per load and /v1/models reports the active context_length. The Strix Halo issue shows a compute crash past 33k while 262k was advertised, with no maintainer reply when read. source
llama.cpp D
ggml.ai, part of Hugging Face since 2026
--ctx-size defaults to the model's trained length, /props and /slots expose n_ctx, responses carry a truncated flag. One account got 128k on 16 GB; another hit a tool loop past 30k until a parser fix. source
MLX LM C
Apple Inc.
A rotating fixed-size KV cache keeps generation going by discarding older tokens, which is silent truncation of the prefix; the quantised cache disables batching. No API field reports the active context. source
Ollama D
Ollama Inc.
OLLAMA_CONTEXT_LENGTH or num_ctx sets 64k and ollama ps shows the allocated context. The default is 4k below 24 GiB and prompts past it are truncated with no error, per two accounts. source
PrivateGPT C
PriBAI Technology Spain S.L.
context_window must be set explicitly in the model profile to avoid overflow; the backend decides the rest. source
TextGen E
oobabooga
ctx_size is set per load (32768 in the example) and --parallel N gives llama.cpp slots. No field reports the active context and nothing says whether over-length prompts are rejected. source
vLLM D
The Linux Foundation (vLLM is a PyTorch Foundation project)
--max-model-len fixes the window and the KV pool is sized at start, with a Context Extension feature page. Whether an over-length request is rejected or truncated was not read. source

Couldn't check 6

The docs were closed to us, or silent. These carry no mark either way.

ListingWhy
Foundry Local B
Microsoft Corporation
No context length setting or report appears in the REST, CLI or SDK reference pages read.
Core F
ZMJ, Inc., doing business as Ghost, 325 9th Street, San Francisco, CA 94103, United States
No context lengths are stated for any listed model.
GPT4All F
Nomic, Inc.
Context Length is a setting, but the docs state no default, no behaviour past it and no API field for the active value.
LM Studio C
Element Labs, Inc.
The context length page (/docs/app/advanced/context-length) returns 404 and the developer docs read do not state the default or whether the API reports the active value.
LocalAI B
Ettore Di Giacinto and the LocalAI team
Per-model context size configuration and its default were not found in the pages read.
Underdog F
Conway Research
No context length is stated on the card.

Doesn't do this job 7

Listings in the category that do something else, with the reason.

ListingWhy
AnythingLLM E
Mintplex Labs, Inc.
Context length belongs to the configured LLM provider. source
Berth B
Trevin Tindall (T-Crypt on GitHub)
Context belongs to the engine; Berth only saves and restores its KV slots across switches. source
BitRun E
Abhishek Khanra (IshekKhal on GitHub)
The largest catalogue context is 8,192 tokens. source
Khoj E
Khoj Inc.
Context handling belongs to the configured model server. source
LocalGhost F
LocalGhost.ai Ltd (company number 17213100, registered in England and Wales, incorporated 12 May 2026)
No agent sessions. source
Open WebUI D
Open WebUI Inc.
Context length belongs to the connected backend; the interface sets none of its own. source
screenpipe C
Negentropy Labs, Inc. (dba Screenpipe)
Not a model server. source

What stops agents finishing

  • default context of 2k or 4k with prompts silently truncated rather than rejected
  • hardware backend crashes with a compute error past about 33k tokens while advertising 262k
  • model repeats the same tool call once the context is long

Around it

Before: local-ai.fit-model-to-hardware. After: local-ai.coding-agent-backend.

Where this comes from

Weight 8 of 100 in Local AI: 8 of 50 accounts hit a context limit: the 2k or 4k default, a crash past 33k, or tool loops past 30k

The same as data: /api/v1/use-cases/local-ai.long-context-agent-session.json, or the MCP tool get_use_case with id local-ai.long-context-agent-session. Other jobs in this category: /api/v1/niches?category=local-ai.

Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person. A mark says what the docs show, not what a run measured.

For companies

Do agents find, use and choose your tools?

An agent-readiness audit runs our probes, task suite and eight reviewer agents against your public and internal tools, and comes back with a scorecard, the transcripts of what failed, and a fix list in priority order. From $2,500, re-run included. We never take payment to move a rank. We do help companies earn one.