AnythingLLM vs llama.cpp

AnythingLLM

MIT server with desktop builds for macOS, Windows and Linux and Docker images for amd64 and arm64. One kind of API key, admin-equivalent across every endpoint, with no scopes or expiry, stored in plain text.

llama.cpp

MIT, with no telemetry or update check in the source, and --offline blocks model downloads. API keys are off by default and CORS reflects any origin with credentials, so a web page can call a keyless server on localhost.

AnythingLLM

1. Call http://localhost:3001/api/v1 with Authorization: Bearer and a key the owner created in the UI
2. Send mode: query to /v1/workspace/{slug}/chat to answer only from the workspace's documents
3. Treat the key as admin. It can delete workspaces, users and documents
4. Read /api/docs on the instance for the endpoint list. Request bodies there are examples, not schemas
5. Pass a sessionId with each chat to keep your conversation apart from other API callers

llama.cpp

1. Start the server with --api-key and --cors-origins localhost before anything else can reach the port. Both are off by default
2. Pass n_predict or max_tokens. Generation is unbounded by default
3. Send response_fields to /completion to drop the fields you don't read
4. Wait and retry on a 503 unavailable_error. The model is still loading
5. Read the server README of the build you run. Behaviour changes between nightly builds without a changelog entry