{
  "data": null,
  "kind": "anchor.page",
  "links": {
    "api": "https://www.anchorterminal.com/api/v1/index.json",
    "html": "https://www.anchorterminal.com/niches/local-ai.long-context-agent-session/",
    "json": "https://www.anchorterminal.com/niches/local-ai.long-context-agent-session/index.json",
    "llms": "https://www.anchorterminal.com/llms.txt",
    "markdown": "https://www.anchorterminal.com/niches/local-ai.long-context-agent-session/index.md",
    "mcp": "https://www.anchorterminal.com/mcp",
    "slim": "https://www.anchorterminal.com/niches/local-ai.long-context-agent-session/index.min.md"
  },
  "markdown": "One company's docs show a full fit.\n\n| Companies, full fit | With a limit | Carry the tag | Share of the category's work | First-hand accounts |\n| --- | --- | --- | --- | --- |\n| 1 | 10 | 26 | 8 | 8 |\n\n## What counts as done\n\nA 60k token prompt is accepted whole and the server reports the active context length through its API.\n\nMeasured, for our own tests: the response shows no sign of a truncated prefix, and the same tool works at 60k as at 5k.\n\nHow agents ask for it: set the context to 64k for Claude Code; why did the model forget the start of the task; run a 100k token repository read locally; stop the model looping once the conversation is long\n\n## Does the job\n\nThe docs show it meets the test above, and nothing we read says otherwise.\n\n- **HyperQwen (syv.ai (Copenhagen, CVR DK44671247))**: CTX=long gives 150k and CTX=huge 240k with measured decode rates and cold time-to-first-token at 66k and 100k. vLLM sizes the KV pool at start and the README states the pool size per arm.\n\n## Does the job, with a limit\n\nIt does the job, but something in the docs or a user's account will get in an agent's way. The limit is named in each row.\n\n- **Docker Model Runner (Docker, Inc.)**: Context is configured per model, but the account shows the stored value not always applied at load, with the packaged 128k default used instead.\n- **Jan (Menlo Research Pte Ltd)**: Context size is a per-model setting and Fit to Hardware caps it where enabled. Nothing documents the default, what happens past it, or an API field reporting the active value.\n- **KoboldCpp (LostRuins (Concedo))**: --contextsize sets the window and /api/extra/true_max_context_length reports it over the API. Smart Context and context shift drop the oldest half when full rather than rejecting, so long sessions lose their prefix silently.\n- **Lemonade (Advanced Micro Devices, Inc.)**: ctx_size is set per load and /v1/models reports the active context_length. The Strix Halo issue shows a compute crash past 33k while 262k was advertised, with no maintainer reply when read.\n- **llama.cpp (ggml.ai, part of Hugging Face since 2026)**: --ctx-size defaults to the model's trained length, /props and /slots expose n_ctx, responses carry a truncated flag. One account got 128k on 16 GB; another hit a tool loop past 30k until a parser fix.\n- **MLX LM (Apple Inc.)**: A rotating fixed-size KV cache keeps generation going by discarding older tokens, which is silent truncation of the prefix; the quantised cache disables batching. No API field reports the active context.\n- **Ollama (Ollama Inc.)**: OLLAMA_CONTEXT_LENGTH or num_ctx sets 64k and ollama ps shows the allocated context. The default is 4k below 24 GiB and prompts past it are truncated with no error, per two accounts.\n- **PrivateGPT (PriBAI Technology Spain S.L.)**: context_window must be set explicitly in the model profile to avoid overflow; the backend decides the rest.\n- **TextGen (oobabooga)**: ctx_size is set per load (32768 in the example) and --parallel N gives llama.cpp slots. No field reports the active context and nothing says whether over-length prompts are rejected.\n- **vLLM (The Linux Foundation (vLLM is a PyTorch Foundation project))**: --max-model-len fixes the window and the KV pool is sized at start, with a Context Extension feature page. Whether an over-length request is rejected or truncated was not read.\n\n## Couldn't check\n\nThe docs were closed to us, or silent. These carry no mark either way.\n\n- **Foundry Local (Microsoft Corporation)**: No context length setting or report appears in the REST, CLI or SDK reference pages read.\n- **Core (ZMJ, Inc., doing business as Ghost, 325 9th Street, San Francisco, CA 94103, United States)**: No context lengths are stated for any listed model.\n- **GPT4All (Nomic, Inc.)**: Context Length is a setting, but the docs state no default, no behaviour past it and no API field for the active value.\n- **LM Studio (Element Labs, Inc.)**: The context length page (/docs/app/advanced/context-length) returns 404 and the developer docs read do not state the default or whether the API reports the active value.\n- **LocalAI (Ettore Di Giacinto and the LocalAI team)**: Per-model context size configuration and its default were not found in the pages read.\n- **Underdog (Conway Research)**: No context length is stated on the card.\n\n- not this job: AnythingLLM (Mintplex Labs, Inc.): Context length belongs to the configured LLM provider.\n- not this job: Berth (Trevin Tindall (T-Crypt on GitHub)): Context belongs to the engine; Berth only saves and restores its KV slots across switches.\n- not this job: BitRun (Abhishek Khanra (IshekKhal on GitHub)): The largest catalogue context is 8,192 tokens.\n- not this job: Khoj (Khoj Inc.): Context handling belongs to the configured model server.\n- not this job: LocalGhost (LocalGhost.ai Ltd (company number 17213100, registered in England and Wales, incorporated 12 May 2026)): No agent sessions.\n- not this job: Open WebUI (Open WebUI Inc.): Context length belongs to the connected backend; the interface sets none of its own.\n- not this job: screenpipe (Negentropy Labs, Inc. (dba Screenpipe)): Not a model server.\n\n## What stops agents finishing\n\n- default context of 2k or 4k with prompts silently truncated rather than rejected\n- hardware backend crashes with a compute error past about 33k tokens while advertising 262k\n- model repeats the same tool call once the context is long\n\n## Around it\n\nBefore: local-ai.fit-model-to-hardware. After: local-ai.coding-agent-backend.\n\n## Where this comes from\n\nWeight 8 of 100 in Local AI: 8 of 50 accounts hit a context limit: the 2k or 4k default, a crash past 33k, or tool loops past 30k\n\nAs data: https://www.anchorterminal.com/api/v1/use-cases/local-ai.long-context-agent-session.json, or the MCP tool get_use_case with id local-ai.long-context-agent-session. Draft, read 2026-11-12 from public docs and first-hand accounts, not yet reviewed by a person.\n",
  "meta": {
    "attribution": "Anchor Terminal (https://www.anchorterminal.com)",
    "attributionRequired": true,
    "credit": "CC BY 4.0, credit Anchor Terminal and link the page or API address you took it from (https://www.anchorterminal.com/terms/#data)",
    "docs": "https://www.anchorterminal.com/docs/",
    "generatedAt": "2026-11-16",
    "license": "CC-BY-4.0",
    "method": "https://www.anchorterminal.com/benchmark/",
    "methodology": "0.8",
    "openapi": "https://www.anchorterminal.com/openapi.json",
    "preview": false,
    "run": "2026-10-01",
    "runLabel": "October 2026 research run",
    "terms": "https://www.anchorterminal.com/terms/#data"
  },
  "page": {
    "breadcrumbs": [
      {
        "name": "Home",
        "url": "https://www.anchorterminal.com/"
      },
      {
        "name": "For agents",
        "url": "https://www.anchorterminal.com/agents/"
      },
      {
        "name": "Niches",
        "url": "https://www.anchorterminal.com/niches/"
      },
      {
        "name": "Keep a 32k to 128k token agent session without silent truncation or a crash",
        "url": ""
      }
    ],
    "description": "Keep a 32k to 128k token agent session without silent truncation or a crash: 1 companies fully do it, 10 with a limit, 26 carry the tag. Every listing in Local AI with the reason and a source.",
    "facts": null,
    "h1": "Keep a 32k to 128k token agent session without silent truncation or a crash",
    "image": "https://www.anchorterminal.com/assets/og/default.png",
    "path": "/niches/local-ai.long-context-agent-session/",
    "published": "",
    "section": "agents",
    "title": "Keep a 32k to 128k token agent session without silent truncation or a…",
    "toc": null,
    "updated": "2026-11-15",
    "url": "https://www.anchorterminal.com/niches/local-ai.long-context-agent-session/"
  },
  "tokens": {
    "markdown": 1650,
    "slim": 380
  },
  "version": 1
}
