llama.cpp vs MLX LM

llama.cpp

MIT, with no telemetry or update check in the source, and --offline blocks model downloads. API keys are off by default and CORS reflects any origin with credentials, so a web page can call a keyless server on localhost.

MLX LM

MIT, with no telemetry found in the source, and the tests passed on the last eight pushes to main. mlx_lm.server has no API key option, answers any origin by default and loads whichever model a request names, and its own docs say it is not recommended for production.

llama.cpp

1. Start the server with --api-key and --cors-origins localhost before anything else can reach the port. Both are off by default
2. Pass n_predict or max_tokens. Generation is unbounded by default
3. Send response_fields to /completion to drop the fields you don't read
4. Wait and retry on a 503 unavailable_error. The model is still loading
5. Read the server README of the build you run. Behaviour changes between nightly builds without a changelog entry

MLX LM

1. Keep mlx_lm.server on 127.0.0.1 and pass --allowed-origins with the origins you trust. There is no API key, and the default answers every origin
2. Treat any caller as able to load any model. The model and adapters request fields accept any Hugging Face repository or local path
3. Send max_tokens or max_completion_tokens when you need more than 512 tokens, the server default
4. Read errors as {"error": "<text>"} with 400 for a bad field and 404 for a model that failed to load. They are not OpenAI error objects
5. Poll GET /health before the first request. It answers 503 with unavailable when the generation thread has stopped