Ollama

Ollama packages llama.cpp into a service with a model library: ollama pull qwen2.5-coder:3b, and a minute later there is an HTTP server answering chat requests.^[1] It runs as a background service on Windows, macOS and Linux, keeps models loaded for a configurable time, and exposes both its own API and an OpenAI-compatible one.

What it is for

The helper tier on a desktop with a modest card, where the ease of install matters more than the last few percent of throughput. It is also what many desktop chat clients expect to find.

Things to know

• Two APIs, not one. The OpenAI-compatible shim covers the common fields and silently ignores others. chat_template_kwargs is one it drops, so a thinking model keeps thinking; the native /api/chat with "think": false is the switch.^[2]
• The model name matters. Unlike a bare llama-server, Ollama resolves the model field to a pulled tag and errors on an unknown one.
• One model at a time on a small card. A 6 GB card holds either the text helper or the vision model; a request for the other evicts the first and the next call waits for a reload. Keep tags in config, send no keep_alive or num_ctx unless you mean to change what stays loaded, and never cause a load from a background job.
• OLLAMA_KEEP_ALIVE=-1 on the server pins the loaded model for good; a per-request value overrides it for that model.

History

• 2026-10-09: article written and published.
• 2026-10-09: entry created.