Foundry Local vs llama.cpp

Foundry Local

The SDK and its native runtime are MIT, at 2.1.0 on four registries, and pick a CPU, GPU or NPU model variant automatically. The optional local server has no credential, its current routes aren't in the published REST reference, and telemetry is on by default with an opt-out.

llama.cpp

MIT, with no telemetry or update check in the source, and --offline blocks model downloads. API keys are off by default and CORS reflects any origin with credentials, so a web page can call a keyless server on localhost.

Foundry Local

1. Read the server URL from manager.urls[0] or foundry server status. The port is dynamic unless the owner sets web.urls or foundry server start --port
2. Send the model ID that GET /v1/models returns, not the alias. The alias resolves to a hardware-specific variant
3. Check supportsToolCalling before sending tools. Support differs by variant, and issue #1183 reports Qwen tool calling failing on QNN
4. Set your own request timeout. Inference has no built-in one, and cancellation takes effect only after the current generation step
5. Ask the owner to set ORT_TELEMETRY_DISABLED=1 or disableNonessentialTelemetry before the manager is created if telemetry must be off

llama.cpp

1. Start the server with --api-key and --cors-origins localhost before anything else can reach the port. Both are off by default
2. Pass n_predict or max_tokens. Generation is unbounded by default
3. Send response_fields to /completion to drop the fields you don't read
4. Wait and retry on a 503 unavailable_error. The model is still loading
5. Read the server README of the build you run. Behaviour changes between nightly builds without a changelog entry