AnythingLLM

AnythingLLM is a self-hosted chat application: workspaces, uploaded documents chunked and embedded for retrieval, and a chat that sends the retrieved passages plus the question to whatever model server is configured.^[1] It runs as a container and can embed documents with a small model inside that container, so nothing leaves the machine.

What it is for

Talking to a local model directly, and asking questions over a set of documents without writing any code.

Things to know

• It is a pass-through. With its provider pointed at a llama-server, every chat is a request to that server, with retrieval latency on top. It shares the server's single slot with every other caller, and listing it as a fallback for that server is listing the same process twice.^[2]
• It cannot set model flags. Its LocalAI-style provider sends model, messages and temperature and nothing else. A reasoning model behind it keeps thinking unless the server turns that off by default, which is exactly why a server-side default exists.
• Token cap. A trivial question once cost over a hundred tokens of hidden reasoning against a 4K cap; with thinking off server-side the same question is eight tokens.
• A manual docker stop on a container with restart: unless-stopped survives reboots. If it is missing, check that before anything else.

History

• 2026-10-09: article written and published.
• 2026-10-09: entry created.