Self-hosted inference orchestrators compared: LocalAI, exo, GPUStack, vLLM
A September 2026 survey of self-hosted inference orchestrators for OpenAI-compatible endpoints. It compares Ollama, llama.cpp, vLLM, LiteLLM, LocalAI, exo, Xinference, GPUStack, NVIDIA Dynamo, llm-d, SkyPilot, dstack, and CoderAI across modalities, multi-machine support, auto-discovery, cache-aware routing, ops consoles, cloud burst, non-LLM fan-out, training, Kubernetes, platforms, and signed images. The author, who built CoderAI, gives honest recommendations for each situation.
The parts nobody else has are the three-tier escalation — a model runs on your card, then on a machine you own, then on a RunPod GPU rented by the second with a price cap and a budget, chosen per model and switchable to "only when busy" — the fan-out of image, video, embedding, speech, transcription and OCR requests over every machine that has the model, a video pipeline relayed part by part, and LoRA/QLoRA training data-parallel across nodes from the same server.
- nextime
for whoever is reading and rightly say "it seems llm generated", you are right, sorry, wasn't supposed to really post it and it did by error, my bad for not supervision my test while i was doing it.
- SahAssar
Seems generated. Also why no llamafile?
- hypfer
This feels agentically generated.
The blog, the post here, the (auto?)killed LLM comment.