AISBF

Renting a GPU is not only for language models

A language model on a rented GPU is a solved shape: vLLM and llama.cpp both ship a server image that already speaks the OpenAI API, so CoderAI only has to rent the machine and point at it. Every other capability — image generation, video, speech, transcription, OCR, embeddings, face swap — has no such image, because those endpoints are CoderAI's own.

So the far side is CoderAI itself, trimmed down. Ten published pod images carry the same codebase with only the Python dependencies one capability needs, sharing a common core layer so ten images are not ten downloads. They are public on GHCR: a pod pulls one without credentials.

Getting the weights onto a pod

A pod is disposable storage, so how it obtains a model matters as much as which GPU it runs on.

LoRA and QLoRA adapters travel the same way. A local adapter is content-addressed and pushed to the pod, because "trained on my machine" and "served on a rented one" should not be mutually exclusive.

Test runs

Placement has many moving parts — image, weights, adapters, auth, budgets — and most of them fail minutes into a pod boot, in a log nobody is watching. Every model page has a Test run button:

POST /v1/models/test  {"model": "…", "where": "auto" | "local" | "runpod"}

It sends the smallest real request for the model's kind through the front, so it exercises the same routing, auth and provisioning a real request would. where: "runpod" refuses to fall back to local, so a pod that is not actually reachable fails the test instead of quietly passing somewhere else.

Inputs that can be synthesised honestly are: an embedding, a small image, a short TTS line, a spoken sentence for transcription, a rendered page for OCR — and the result is checked against what went in, so a model returning fluent nonsense fails as surely as one returning nothing. Kinds that cannot be synthesised honestly (a video generation is minutes of GPU; a face swap needs faces) report a reachability check and say plainly that no generation was run.

To test one model without moving its capability, tools/runpod_model_test.py pins a single model entry and restores the catalogue afterwards. This matters more than it sounds: a capability remote is a production routing switch, and one left enabled after a test will send real traffic to a rented pod.

What a booting pod tells you

RunPod publishes no pod-log API. Their REST OpenAPI spec lists twenty-three routes — pods, billing, endpoints, volumes — and none of them serve logs; GraphQL refuses introspection. The web console is a human surface. Code that "tries several log routes" is guessing at URLs that do not exist.

So a CoderAI pod reports on itself. Its entrypoint prints timestamped phases from the first instant the container exists, and serves the same record at GET /boot, readable the moment the port opens:

[boot +0s] container started
[boot +0s] profile=embeddings image=ghcr.io/nextime/coderai-embeddings:0.2.12
[boot +0s] seed models: [{"path":"BAAI/bge-m3", …}]
[boot +1s] gpu: NVIDIA RTX 2000 Ada, 16376 MiB
[boot +1s] starting uvicorn on 0.0.0.0:8000
[boot +0.4s] registering 1 seeded model(s)

That seed line is the one that matters: a pod serving nothing is the most common failure, and it is visible here in one look instead of after a boot timeout.

Nineteen images, three kinds

Every image is the same codebase with only the dependencies its capability needs, all public on GHCR under ghcr.io/nextime/coderai-*. RunPod refuses an image above a size it does not document — 12.4 GB has booted, 14.5 GB was refused — so what goes into each one is measured, not assumed.

Sixteen of the nineteen have completed a real inference on a rented GPU on the current build. faceswap needs real faces to test; the two engine images need a hundred gigabytes of weights on a network volume.

The LLM servers as CoderAI pods

engine: vllm and engine: llamacpp run the upstream vllm/vllm-openai and ggml-org/llama.cpp images: well-tested, but not CoderAI — no TLS on the direct path, no boot record, no seeding, no local LoRA adapters, nothing on the Tasks page. Two more images run the same servers inside a CoderAI pod:

• coderai-llama — llama-cpp-python compiled with CUDA for every GPU generation RunPod rents (Ampere through Blackwell), on the torch-free core: the CUDA GGUF backend asks NVML about the card instead of torch, which took the image from 11.6 GB to 5.3 GB. Local LoRAs work.
• coderai-vllm — vLLM in its own venv, driven by CoderAI's vLLM backend; the local vLLM settings (dtype, quantisation, memory utilisation, extra args) travel to the pod and tensor parallelism follows the pod's gpu_count. It installs vLLM's +cu129 build, because PyPI's vLLM wheel is a CUDA 13 build and RunPod's hosts run CUDA 12.8 drivers — the pod dies at engine start with "The NVIDIA driver on your system is too old (found version 12080)" otherwise.
• coderai-engines — ds4, colibri and kimi-k3-in-c, compiled in a CUDA 12.8 toolchain stage for sm_80 through sm_120a rather than copied from a machine whose builds target one card and one CUDA. coderai-engines-kt carries ktransformers (SGLang + kt-kernel, which dispatches AMX / AVX-512 / AVX2 at run time) with the C compiler Triton needs on the pod. Both come with a warning the UI shows the moment you pick them: their models are 100 GB to 1.5 TB, and a pod without a network volume downloads all of it on every cold start. volume_path points a pod at weights already on the volume.

Direct to the pod, with the pod's own TLS

RunPod's HTTP proxy (<pod>-8000.proxy.runpod.net) sits behind Cloudflare: TLS for free, but a 100-second idle limit per request and a body-size cap. A voice clone through it died with a 524 at 516 seconds while the pod was still working. RunPod also maps a raw TCP port straight to the machine — no proxy, no cutoff, and no TLS.

So a CoderAI pod brings its own. Each install has a CA, made once with openssl and kept beside the pod registry; each pod gets a short-lived certificate signed by it at provision time, sent as environment exactly like the bearer token; uvicorn on the pod starts on it, and this side verifies pod connections against that CA and nothing else. There is deliberately no hostname check — the pod's public IP is not known when the certificate is issued, and a certificate the CA did not sign fails before any name is compared.

What the last live round found

Every image was rebuilt with the TLS-aware boot script and re-run on rented cards before :latest moved. What failed, and what changed because of it:

• The root cause was above the tail. A pod's 503 carried the last six lines of vLLM's output — the wrapper's own traceback, "see root cause above". The worker now surfaces the lines that name an error first.
• A pod terminated behind the pool's back stayed "healthy" until the next timed probe, and the next request died on connection refused. The backend now drops such a pod immediately and retries once on a fresh one.
• Two 404s at second zero. Text-to-speech on one card and OCR on another failed with a 404 issued the same second the pod was declared ready, and passed unchanged a minute later: RunPod's proxy answers 404 until its route settles. A fresh pod's not-found and unavailable answers are now retried for its first minute.
• pip could not resolve sglang-kt's forty exact pins in an hour; uv did in seconds. A per-venv marker switches the resolver, and wheels are cached across builds because PyPI's CDN throttles a machine that has pulled 100 GB in a day to ~400 KB/s.
• Docker lost 500 GB twice — overlay2 layers of old flattened images pinned by mount records of containers the daemon no longer knew about, invisible to every docker prune. tools/docker_leak_audit.py walks the on-disk metadata the way a fresh daemon would and, with the daemon stopped, removes what nothing references.