AISBFWhy more than one engine
A 7B safetensors model and a 2.78-trillion-parameter mixture-of-experts checkpoint are not the same problem. The first wants PyTorch. The second wants an engine that keeps the dense trunk resident and streams routed experts off NVMe, because it will never fit anywhere else.
So CoderAI treats the runtime as a per-model decision. Every model entry in models.json can carry a backend pin; the front proxy routes the request to an engine node that declares the matching capability, and the engine dispatches to the right backend class. CoderAI owns the whole lifecycle of each external engine: cloning and building it, downloading its weights, supervising the process, sharing VRAM with the other tenants, and tearing it down.
The engine matrix
backend | Engine | Shape | What it is for |
|---|---|---|---|
transformers | PyTorch + HF Transformers | in-process | safetensors models on CUDA, with 4/8-bit quantization and offload |
gguf | llama.cpp | in-process | GGUF models on CUDA or Vulkan — the AMD/Intel path |
ds4 | ds4 / DwarfStar | HTTP subprocess | DeepSeek-V4, from-scratch native C/CUDA engine with its own OpenAI server |
colibri | colibri | native wire protocol | GLM-5.2, DeepSeek-V4 and Kimi-K3 — a pure-C MoE engine streaming experts from disk |
k3 | kimi-k3-in-c | native wire protocol | Kimi-K3 (2.78T params) on CPU, in as little as ~8 GB RAM |
kt | ktransformers via SGLang | HTTP subprocess | CPU+GPU heterogeneous MoE: DeepSeek, Kimi, Qwen, GLM, MiniMax |
vllm | vLLM | HTTP subprocess, isolated venv | continuous batching + paged KV for high aggregate throughput |
runpod | RunPod | remote HTTP | a GPU rented by the second — see the RunPod guide |
How an engine is chosen
Selection is deliberately conservative, because guessing wrong means loading tens of gigabytes onto the wrong device. In order:
- An explicit
backendpin on the model entry wins outright. - An enabled engine's
model_idalias — if you configuredcolibri.model_idand the request names it, colibri serves it. - An unambiguous name claim — e.g. a GGUF whose architecture only ds4 can read. Engines that could collide with everything else (
kt,vllm,runpod) never claim by name; they must be pinned.
{
"text_models": [
{ "id": "glm-5.2", "backend": "colibri", "enabled": true },
{ "id": "kimi-k3", "backend": "k3", "enabled": true },
{ "id": "Qwen/Qwen3.5-9B","backend": "vllm", "enabled": true },
{ "id": "cloud-70b", "backend": "runpod", "enabled": true, "runpod": { "mode": "pods" } }
]
}Isolated environments
Its own venv, its own torch
vLLM pins its own torch/CUDA build, which cannot coexist with the main CoderAI environment. It therefore runs in an isolated virtualenv built from requirements-vllm.txt. The same pattern is used for pyannote diarization, NVIDIA NeMo and CrisperWhisper.
Built on first use
ds4, colibri and k3 are C engines. CoderAI clones and builds them on first use, applies the patches it needs (a resident serve loop for k3), and manages the resulting binary as a supervised subprocess.
Sharing one GPU
External engines participate in the same VRAM accounting as local models: they are eviction-tracked, they take part in thermal throttling, and a cross-engine swap gate batches same-model requests before handing the GPU over, so two engines on one card do not thrash.
Engines as nodes
The front proxy is torch-free: it supervises engine subprocesses and forwards /v1/* to them over localhost HTTP. Each engine reports its state — resident models, VRAM, in-flight tasks, thermal state — and the front aggregates that into the engines and Tasks pages.
vLLM appears there as a first-class node next to nvidia and radeon, with the same health, task and VRAM reporting. When RunPod is enabled it gets its own box too, so remote work is visible in the same place as local work.
Further reading
The repository carries a per-engine guide under docs/: deepseek-ds4.md, glm-colibri.md, kimi-k3.md, ktransformers.md, vllm.md, runpod.md and frontend-engine-split.md.