llama-swap routes local AI models behind one name

llama-swap lets a client keep calling one model name while you switch the local model behind it with a single API call, but the server that ships with llama.cpp can already switch models by itself. llama.cpp is the open-source engine that runs language models on your own CPU or GPU, and its llama-server program serves them over an OpenAI-compatible API.

llama-swap is a local LLM (large language model) router that sits in front of model servers such as llama-server. You give it each server’s start command. It starts a server when a request needs one, stops it after an idle limit, and passes requests through. Named profiles pick which server answers a model name. The cost is one more always-on proxy.

Tested: commit 7761aa13360ea379cb89366d07c2d08aa9f1ed10 (release v255) on September 14, 2026, on a CPU-only Linux virtual machine with llama.cpp serving model A (Qwen2.5-Coder 0.5B) and model B (Qwen3 0.6B). Workflow: use two profiles to switch the model name coding from A to B and back, wait past a 30-second idle limit, then kill the running model server and send more requests. Result: every answered request came from the model the active profile named, and idle model servers stopped. llama.cpp’s own router also switched a name from A to B and freed idle memory. Requests sent right after a kill failed, so turn on your client’s retry; a request one second later got an answer. Setup: one Go binary and a YAML file holding the llama-server command for each model. Verdict: adopt with care, if you switch model setups often.

Set up llama-swap with your coding agent

Paste this prompt into Claude Code, or another coding agent that can run commands, to install the latest llama-swap with Homebrew and send one request through it to a local model. It asks before it installs llama.cpp or downloads a model.

Set up llama-swap (https://github.com/mostlygeek/llama-swap) on this computer so one local address can start and switch llama.cpp models. Go one step at a time.

1. Install the latest version with Homebrew on macOS or Linux:
   brew tap mostlygeek/llama-swap && brew install llama-swap
   Without Homebrew, stop and ask me. The other official source is the release binaries at https://github.com/mostlygeek/llama-swap/releases.
   Run `llama-swap --version`. It should print "version: 262" or newer. Stop if it prints anything else.
2. llama-swap starts a model server that you supply. Run `llama-server --version`. If it is missing, tell me you want to run `brew install llama.cpp` and wait for my yes.
3. Ask me for the path to a GGUF model file. If I have none, offer to download qwen2.5-coder-0.5b-instruct-q4_k_m.gguf (about 490 MB) from https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF and wait for my yes.
4. Write swap.yaml in a new folder, with the real model path:
   models:
     coding: {cmd: "llama-server --port ${PORT} --model /path/to/model.gguf", ttl: 300}
5. Start `llama-swap --config swap.yaml --listen 127.0.0.1:6300` in the background, send one chat request, show me the answer, then stop llama-swap:
   curl http://127.0.0.1:6300/v1/chat/completions -H 'content-type: application/json' -d '{"model":"coding","messages":[{"role":"user","content":"Write a Python function clamp(value, low, high). Return only code."}],"max_tokens":128}'
   If a request returns HTTP 502 with an empty body, the model server exited. Send the request once more; llama-swap starts a new model server.
6. Do not install a background service or edit my shell profile. Report what you installed or changed and what failed. Do not retry a failure more than once without asking.

llama-swap profiles switch the model behind one name

In llama-swap’s YAML config, each model ID (here a and b) holds the command that starts one llama-server. A profile maps the name the client calls, coding, to one of those IDs. ttl is the idle limit in seconds.

profiles:
  code:    {pins: {coding: a}}
  explain: {pins: {coding: b}}
models:
  a: {cmd: "llama-server --model model-a.gguf --alias a --port ${PORT}", ttl: 30}
  b: {cmd: "llama-server --model model-b.gguf --alias b --port ${PORT}", ttl: 30}

The commands are shortened. llama-swap runs exactly the commands you write, with the flags you choose.

Start llama-swap, then pick a profile and call the shared name with curl:

$ llama-swap --config swap.yaml --listen 127.0.0.1:6300

# in a second terminal
$ curl -X PUT http://127.0.0.1:6300/api/profiles/active \
    -H 'content-type: application/json' -d '{"name":"code"}'
$ curl http://127.0.0.1:6300/v1/chat/completions \
    -H 'content-type: application/json' \
    -d '{"model":"coding","messages":[{"role":"user","content":"Write a Python function clamp(value, low, high). Return only code."}],"temperature":0,"seed":4242,"max_tokens":128,"stream":false}'

With code active, model A answered (response trimmed):

{"model": "a", "choices": [{"finish_reason": "stop", "message": {"content": "```python\ndef clamp(value, low, high): … return max(low, min(value, high))\n```"}}]}

After we switched to explain, the same request went to model B:

{"model": "b", "choices": [{"finish_reason": "length", "message": {"content": "", "reasoning_content": "Okay, I need to write a Python function called clamp …"}}]}

Model B, a Qwen3 model that reasons before answering, spent its whole 128-token limit thinking about clamp and never got to writing it. It did the same when we called it directly, without a router. For short answers, add /no_think to the prompt, which Qwen3’s model card says turns thinking off, or raise max_tokens; the card suggests 32,768 output tokens. Switching back to code sent the next request to model A, with no restart of the client or llama-swap. The first request after each switch also started the new model server; it ran 2.2 to 3.7 seconds longer than the next request to the same model, which used a different prompt. The docs say one profile can switch several names at once; this test covers one name.

llama.cpp’s own router also switched models and freed idle memory

Started without a model, llama-server runs as a router: it reads a preset file in INI format (plain key = value lines) listing models and their settings, and starts a separate server process for whichever model a request names. We pointed one entry, stable-coding, at model A (settings shortened):

[*]
sleep-idle-seconds = 30
[stable-coding]
model = model-a.gguf

$ llama-server --models-preset native.ini --models-max 1 --port 6200

To move stable-coding to model B, we edited the model line and called GET /models?reload=1 so the router reread the file. The next request with the same name came from model B. This test covers llama.cpp’s router in one direction, A to B; the return to A is tested on llama-swap only.

Both routers used the same model files and settings. Memory figures are resident memory (RSS), the RAM the operating system counted for one process in one snapshot, in MiB (mebibytes, about a million bytes each).

Step	llama-swap	llama.cpp router
Switch one model name	coding: A → B → A, by profile API call	stable-coding: A → B, by file edit and reload
After 30 seconds idle	Model server process ended	Model server process kept, asleep, at 68.2 MiB
Router process memory (RSS) while idle	127.9 MiB	12.1 MiB
Next request	Started a new model server and answered	Reloaded the model in the sleeping server and answered

llama.cpp’s sleep mode unloads the model and its cache from RAM but keeps the server process running. Model A’s server fell from 558.7 MiB loaded to 68.2 MiB asleep, which is what “freed idle memory” means here. llama-swap ends the model server instead. Its own process was about ten times the size of llama.cpp’s router process, and the test doesn’t show what fills it. Both router figures are small next to the 558.7 MiB a loaded 0.5-billion-parameter model took, and single snapshots can’t show which setup uses less memory overall.

llama-swap and llama.cpp’s router both failed the request sent right after a crash

We killed llama-swap’s model server three times, and each time the request sent right after the kill got HTTP 502 with an empty body. When we waited one second and sent another request, llama-swap started a new model server and answered 9 to 11 seconds after the kill.

We killed the model server behind llama.cpp’s router once. The next request got HTTP 500 with the message proxy error: Failed to read connection, and a request one second later got an answer from a new server. Neither router resent the failed request on its own.

llama-swap v255 has no setting that retries that request or explains the 502. Its code returns an empty 502 when the connection to the model server fails and writes the cause to its log (upstream process exited unexpectedly). There is no workaround inside llama-swap. If your client stops on a 502, turn on the client’s retry option or wrap the call in a retry; the next request starts a new model server.

llama-swap ships often, mostly from one maintainer, and users report mixed-server setups

llama-swap is MIT-licensed and started in October 2024. Releases are numbered builds; GitHub’s release list shows 3 to 12 releases a month in the year to October 2026. When we tested, about 83% of the project’s GitHub contributions came from its lead maintainer, so most maintenance depends on one person.

The README says llama-swap works with any OpenAI- or Anthropic-compatible server, including vLLM and llama.cpp forks. In a Reddit thread about local setups, the author runs llama.cpp beside the fork ik_llama.cpp, and another user wrote on X that their models on SGLang, another model server, still go through llama-swap. We tested only llama.cpp, the OpenAI chat route and small CPU models; mixed servers, GPUs, Anthropic routes and heavy concurrent load are untested here.

Use llama-swap when you switch model setups often

Adopt llama-swap with care if you point one OpenAI-compatible client at several local models and switch between them often enough to justify one more always-on proxy. It moved one model name between models with a single API call and stopped idle model servers by itself. But llama.cpp’s own router also switches models and frees idle memory, and llama-swap answers a request that hits a crashed model server with an empty 502 that no setting changes, so retry from the client.

Use it when you change setups during the day, or when you want idle model servers to exit completely. Skip it when one model covers your work, when an occasional llama.cpp preset edit is enough, or when your client can’t retry. If a llama-swap release note says the proxy now retries a request that hits a crashed model server, the main caveat is gone.