AISBFThree words, three mechanisms
Orchestration is one front routing every request to the right runtime, card, machine or cloud. Distribution is one model — or one request's work — spread over several machines' cards. Escalation is the same model running on your card, then on a machine you own, then on a GPU rented by the second (remote execution, RunPod). This page is the first two.
| Need | Mechanism | Where it is set |
|---|---|---|
| More cards, one admin: a model runs somewhere on the network, requests routed by capability and load | Cluster nodes — other CoderAI installs used as engines of this one | Settings → Cluster; per model: Engine / card |
| One GGUF too big for any single machine | llama.cpp RPC — cards on other machines join the layer split, or the row split (tensor parallel) on a fast fabric | Settings → Cluster → RPC servers; per model: Cards on other machines |
| One HF model too big for any single machine, with continuous batching | vLLM on Ray (pipeline parallel across nodes; tensor parallel may span them on 10 GbE / RDMA) / SGLang multi-node | Settings → vLLM / ktransformers; per model: the vLLM / kt block |
| Images, video, embeddings, speech, transcription, OCR: many units of work in one request | Fan-out — the parts over every engine and node that has the model | per model: Distribute |
| A video model too big for one card | Pipeline parts — text encoder, low-noise expert, VAE on different nodes | per model: Pipeline parts on other machines |
| A LoRA / QLoRA trained faster than one card allows | Data-parallel training over torch.distributed | per model: LoRA training nodes; per job: nodes |
| Boxes on one LAN that should just find each other | mDNS discovery + one shared cluster token; members auto-join as nodes | Settings → Cluster: token, Find nodes on the LAN, Auto-join |
| A conversation whose model is loaded in several places | Prefix-cache routing — its next turn goes back where its KV cache is warm | Settings → Cluster: Prefix-cache routing |
| Several machines for one capability or one host model, with failover | Pools of remotes and hosts | Settings → Remote capabilities; per model: More machines |
No impossible pairs
The model form narrows the Engine / card and Backend selects to what fits: pick a backend and the engines that cannot run it grey out, pin an engine and the backends it cannot run grey out; a wrong pair already selected shows a red note and Save refuses. The same table is applied again on the server.
Cluster nodes: another CoderAI as an engine of this one
The front already runs one engine process per GPU and knows nothing of their insides: it polls a state endpoint, hands each engine the models it owns, proxies by capability and load. A node is that, one network hop away — a whole CoderAI with its own front, cards, thermal protection and admin. The node needs only an API token from its Tokens page; the head needs the node's URL and that token (Settings → Cluster; verify its certificate with the system store, a pasted PEM, or not at all on a LAN you trust).
One engine, many cards
The node's front answers /cluster/state from its own registry: loaded models, VRAM summed over its engines, the union of their capabilities. A node with an NVIDIA and a Radeon card offers both transformers and gguf and picks the card itself.
Models it owns
Pinned by name from the model page, or assigned by the head like to any engine; the entries are pushed to the node and registered there under a path that exists on it, the HuggingFace id CoderAI recovers, or the path on the node you set.
Load, unload, route
Requests go to the node with its token, never the caller's credentials; Load and Unload from the head's model page reach it; files it generates are addressed through the head. Node changes apply on the next poll, no restart.
One GGUF over several machines
llama.cpp's RPC backend makes a card on another machine one more ggml device: a machine runs rpc-server (Settings → Cluster → RPC servers: bind address, port, device, threads — started, watched and advertised by its front), and a model names the servers under Cards on other machines. The layer split then spreads the model over local and remote cards alike; the ratio reads local cards first, then the servers in the listed order, and the automatic split counts remote memory too.
Per token only the activations between the layers on each side cross the wire, so decode over a LAN is workable; loading and prompt processing are wire-bound — 1 GbE hurts, 10 GbE is fine. The protocol is unauthenticated: LAN or WireGuard only. The bundled llama-cpp-python is built with the RPC backend and rpc-server comes from the same llama.cpp tree (the protocol is versioned); the coderai-llama pod image lends its card with one environment variable.
vLLM on Ray, SGLang on nnodes
For weights that are not GGUF the servers span machines themselves; CoderAI does the choreography. Per model (or as defaults): tensor parallel inside a machine, pipeline parallel across them, the nodes as name | start command | stop command | gpus. CoderAI starts the Ray head, runs each node's command with {ray_address} filled in — the coderai-vllm image joins as a Ray worker when started with CODERAI_RAY_ADDRESS — waits until Ray sees tensor × pipeline GPUs, launches, and tears it all down with the service. SGLang ranks 1… are started the same way before rank 0.
Every generation capability: distributing the work
Tick Distribute on a model and the front cuts a request into parts and sends them concurrently to every engine that has the model, here and on the nodes, then merges: n images or videos (each part with its own seed offset), embedding and rerank lists (re-indexed), a text to speak (by sentences, joined with ffmpeg), a recording to transcribe (cut at silences, timestamps shifted back), a batch of documents to OCR. A part that fails is retried once elsewhere. One generation is never split.
Pipeline parts on other machines is the heavy one for video: text encoder, the Wan 2.2 low-noise expert and the VAE each on a named node. The head encodes the prompt where the text encoder is, runs the high-noise steps, hands the latents over at the expert boundary; the node finishes and, if it holds the VAE, returns the finished video. The far side runs the ordinary generation endpoint with a hand-off block, so loading, acceleration and LoRAs happen as usual there.
LoRA / QLoRA training over several machines
Every LoRA trained on a model can run on this machine and on cluster nodes at once: each on its own card with its share of the samples, the adapter's gradients — a few MB — averaged after every step, so all machines hold the same adapter and the result is the whole cluster's work. Rank 0 is the machine that took the job; it inlines the images, sends the same job to each node, broadcasts its initial adapter, trains, saves. Peers keep nothing and are cancelled with it. SD1.5, SDXL, Z-Image, the flow-DiT family and Wan, 4-bit included; gloo by default, NCCL on request.
Zero-config on a LAN: a shared token and mDNS
The manual way — a token per node, its URL on the head — still works and is what you use across subnets or VPNs. On one LAN there is a shorter way, the same on every box: type one cluster token, tick Find nodes on the LAN. Each install announces itself as a _coderai._tcp service (name, version, port, capabilities) carrying an HMAC fingerprint of the token, never the token; peers with the same fingerprint are members and, with auto-join (on by default), become cluster nodes of this head with no row to fill in. A node accepts the shared token on its cluster endpoints like one of its own API tokens; a listed node with a blank token field gets it too. A member that stops announcing leaves; one listed by hand under the same name is never duplicated. Every box can have all three on and be a head of the others and a node of the others at once.
mDNS is link-local multicast: it does not cross routers, and a container only sees it on the host's network — coderai-docker --host-network. A capability image joins with -e CODERAI_CLUSTER_TOKEN=… -e CODERAI_DISCOVERY=1. The Cluster page lists everything seen on the link, member or not, joined or not.
Prefix-cache-aware routing
A model resident in more than one place — two local engines, a local engine and a node, replicas after a fan-out — is served by the least busy holder, local before remote on a tie. For a conversation, though, the better place is where its earlier turns went: that engine's KV / prefix cache already holds the shared opening, and the prompt is processed from the first new token rather than from scratch. So the front keeps a small map conversation → engine and prefers that engine while it is alive, capable and still holds the model. The key is an X-Session-Id header or the OpenAI user field when the client sends one, else a hash of the conversation's stable opening. A pin still wins; a preference never routes to an engine that cannot serve. On by default, TTL 30 minutes.
Operating it: metrics, usage, logs, recovery
GET /metrics
Per completed request: coderai_requests_total and coderai_request_seconds labelled by API key name, model, kind, status and the engine or node that served it. Gauges: engines and nodes up, in-flight, VRAM free/total, loaded models, RPC servers, discovery peers/members, RunPod spend per model and period, coderai_info{version}. Guarded like the other telemetry unless Serve /metrics without a credential is on.
By key, by model
The same counters as tables on the Cluster page: requests, errors, wall time, kinds, where they ran — since the front started; Prometheus is the durable store.
Every node's tail from the head
The Cluster page shows each node's recent output (its front and every engine line it re-emits) with the log button on the node row, and this front's own next to the engines.
A node that comes back
keeps its assignment; when it answers again the head pushes its assigned models again and they load there without anyone clicking. Local engines have had this since the supervisor: a crashed engine is respawned, a crash-looping one quarantined.
Signed images
Every published image is signed with cosign against the key pair in packaging/cosign.pub; the private half never leaves the release machine. cosign verify --key https://raw.githubusercontent.com/nextime/coderai/master/packaging/cosign.pub ghcr.io/nextime/coderai:latest prints the signed digest and the version and git annotations of the build.
Everything on one page
Admin → Cluster lists the engines here, every node with its engines, VRAM, last error and log, everything seen over mDNS, the RPC servers (this machine's and the nodes'), the hosts behind backend: host models, the capability remotes, the pods, and usage by API key and by model — each with whether it answers right now.