AISBFYour GPUs first. Your other machines second. A rented card third.
CoderAI is one OpenAI-compatible endpoint for text, images, video, speech, embeddings and OCR. Every model runs where you decide — on your card, on another box you own, or on a GPU rented by the second when nothing else fits — with a price cap and a budget, per model. Nothing else self-hosted does that.
GPL-3 · ghcr.io/nextime/coderai · cosign-signed · no account, no telemetry, no cloud unless you ask for one
Your cards
One engine per GPU, NVIDIA and AMD side by side, models pinned or balanced by VRAM and load. Thermal protection, RAM caps, smart KV caching.
Your other machines
Type one cluster token on every box: they find each other over mDNS and become engines of one front. One model can span their cards.
A GPU by the second
Only when the model does not fit or your card is busy: a RunPod pod for exactly as long as it is used, capped in $/hour and by budget, with a spend ledger.
Ten minutes to a first answer.
One NVIDIA card (or AMD/Intel with --vulkan), Docker, and the image. The first run creates the config, models and cache folders and prints the admin login.
Pull and run
docker pull ghcr.io/nextime/coderai:latest
curl -fsSLO https://raw.githubusercontent.com/nextime/coderai/master/packaging/linux/run_oci.sh
chmod +x run_oci.sh
./run_oci.sh --nvidia -dAdd a model
Open http://localhost:8776/admin, paste a HuggingFace id or a path to a GGUF, pick where it runs. The form refuses combinations that cannot work.
Call it like OpenAI
curl http://localhost:8776/v1/chat/completions \
-H "Authorization: Bearer $TOKEN" \
-d '{"model":"qwen3-8b","messages":[{"role":"user","content":"hi"}]}'Any OpenAI client works. Windows: the installer (Docker Desktop + WSL2).
What only CoderAI does.
We surveyed the field — Ollama, LocalAI, exo, GPUStack, Xinference, vLLM, Dynamo, SkyPilot — and closed most of what they had that we lacked. These are the ones that go the other way.
Rent a GPU from inside the API
Per model: runs locally, bursts when busy, or rents always. Pods start on demand, are killed when idle, reaped when forgotten; $/hour and VRAM caps, monthly budgets, a ledger you can read on the /metrics page. SkyPilot and dstack burst jobs; no inference server does this.
Every modality fans out
Tick Distribute and n images, an embedding list, a long text to speak, a recording to transcribe or a batch of documents to OCR is split over every machine that has the model and merged back. Everyone else distributes language models only.
A video pipeline in parts
Text encoder, denoising expert and VAE on different machines, latents handed over at the boundary. And the classics: a GGUF over several boxes through llama.cpp RPC, an HF model over vLLM on Ray.
LoRA on every card you own
LoRA and QLoRA training data-parallel across your machines from the same server that serves the result — adapter gradients averaged every step, fine on 1 GbE.
And the rest, in one box.
The breadth LocalAI has, with per-model engine choice and the operational hardening the enterprise tools keep for LLMs only.
Every runtime
llama.cpp GGUF (CUDA and Vulkan), transformers, vLLM, ktransformers, ds4, colibri, kimi-k3-in-c — chosen per model, with prefix-cache-aware routing across engines and nodes.
Images, video, audio
SDXL, Flux, Z-Image, Wan, LTX-2; TTS, voice cloning, music, stems; five STT stacks, diarization, speaker recognition, face swap — all on the same API and the same placement rules.
Embeddings, rerank, OCR
Text, image and geolocation embeddings, rerankers, document OCR with structured output — the parts a RAG or a search pipeline actually needs, local.
Made to be run
Prometheus /metrics with usage per API key, node logs from the head, automatic recovery of nodes and engines, thermal and RAM protection, GPU swap gating, signed images.
Where it stands.
Stars as of September 2026. Read the columns, not the row lengths: the ones on the right are the reasons this page exists.
| Stars | All modalities | Multi-machine | Auto-discovery | Non-LLM fan-out | Rent GPU on demand | Distributed LoRA | |
|---|---|---|---|---|---|---|---|
| CoderAI | new | yes | yes | yes | yes | yes | yes |
| Ollama | 181k | text | no | no | no | no | no |
| LocalAI | 49k | yes | yes | yes | no | no | no |
| exo | 47k | text | Macs | yes | no | no | no |
| GPUStack | 5.7k | partly | yes | no | no | no | no |
| Xinference | 9.6k | yes | yes | no | no | no | no |
| vLLM | 92k | text | yes | no | no | no | no |
What they have that we do not, still: Kubernetes-native deployment, Apple Silicon/MLX, a one-click model catalogue, and years of community. The full comparison · how the gaps were closed.
What is true today.
We would rather you read this here than find it out.
CoderAI is one person's project, in production for that person's own work every day: an RTX 3090 and a Radeon in one box, a RunPod budget, a broker in front. The RunPod escalation path has been exercised live across every model type. The multi-machine paths — nodes, RPC, fan-out, the video relay, distributed training — are tested against fakes, two processes and a localhost rpc-server; as of this week nothing has crossed a real cable between two GPU machines, and the write-ups say so. The Windows installer is unsigned and has not yet been run on a Windows machine. If you try any of that first, you will be the first, and an issue with what you saw is worth more to this project than a star.
One endpoint. Every model. Wherever it fits.
Complete orchestration, distribution and escalation of remotizable advanced inference.