AISBF

Two modes, chosen per model

The RunPod backend uses no local VRAM. It runs as a thin HTTP proxy inside the primary engine, exactly like the ds4 and ktransformers HTTP backends — which means a CoderAI instance with no GPU and no local models at all is a valid deployment.

Account setup

Settings → RunPod. The Settings page is split into category tabs; RunPod is one of them.

Test key & list GPUs verifies the key and pulls the live GPU catalogue with on-demand and spot prices. It rents nothing.

Putting a model on RunPod

Set the model's backend to runpod and fill in the sub-form. In models.json:

{
  "id": "cloud-qwen-72b",
  "backend": "runpod",
  "enabled": true,
  "runpod": {
    "mode": "pods",
    "served_model": "Qwen/Qwen3.5-72B-Instruct",
    "cloud_types": ["SECURE", "COMMUNITY"],
    "selection_criteria": "cheaper",
    "min_vram_gb": 48,
    "max_hourly_usd": 1.20,
    "allow_spot": false,
    "min_pods": 0,
    "max_pods": 2,
    "idle_timeout_s": 300,
    "boot_timeout_s": 300,
    "load_timeout_s": 600,
    "ctx": 32768,
    "cost_limit_usd": 50,
    "cost_period": "week"
  }
}

What happens on a request

client ─▶ front proxy ─▶ primary engine ─▶ RunpodBackend ─┬─▶ https://<pod>-8000.proxy.runpod.net/v1
                                                         └─▶ https://api.runpod.ai/v2/<eid>/openai/v1

Never pay for a pod you forgot

A rented pod nobody is watching bills forever. CoderAI treats this as a standing invariant rather than a cleanup step.

Every pod it creates is named coderai-<deployment_id>-<model>-<random>. A maintenance loop on the primary engine runs every ~15 seconds and, roughly every 30 seconds plus once at startup, lists the account's pods and terminates any pod carrying this deployment's tag that is not in a live pool and not currently being provisioned. That covers pods orphaned by a crash, a kill -9, or a container restart mid-provision. A clean shutdown tears down every pool as well.

Budgets

Two independent guards, because they fail differently:

• Rate caps (max_hourly_usd per model, global_max_hourly_usd account-wide) refuse to start a pod whose price, added to the running total, would breach the ceiling. This bounds your burn rate.
• Spend budgets (cost_limit_usd + cost_period, and the global pair) refuse to start once cumulative spend in the trailing window is exhausted. This bounds your bill.

Spend is persisted to a ledger next to the config. A pod's final cost is booked when it is torn down, and the live cost of still-running pods is added on top for enforcement and display, so nothing is counted twice. When a budget blocks provisioning the request fails rather than quietly overspending.

Spillover: local first, cloud on overflow

{
  "id": "my-local-model",
  "backend": "gguf",
  "runpod_spillover": {
    "enabled": true,
    "on_concurrency_full": true,
    "on_no_gpu": true,
    "on_local_error": false,
    "target": { "mode": "serverless", "endpoint_id": "abc123",
                "served_model": "Qwen/Qwen3.5-72B-Instruct" }
  }
}

The spill is a guarded reverse-proxy in the front: it rewrites the request's model to the target's served name, adds the bearer token, and forwards. If the spill itself fails, the request falls back to the local path rather than erroring.

Watching it

Reaching the pod: direct or proxy

By default a CoderAI pod is reached at its public ip:port over TLS the pod brings itself (a per-install CA signs a certificate per pod); the upstream vLLM / llama.cpp images go through RunPod's Cloudflare-fronted proxy, which is the only TLS they have. The proxy has a 100-second idle limit per request — enough to kill a voice clone or a cold engine's first token — and a body-size cap; the direct path has neither. direct_tcp on the model's runpod block forces either way; the model page offers it as Auto / Direct TCP / RunPod proxy.

Multi-GPU pods

gpu_count (1–8) rents one machine with N cards. min_vram_gb is checked against the total, max_hourly_usd against the whole pod, and the engine is told to shard across every card — vLLM --tensor-parallel-size N, llama.cpp --split-mode layer. For a model too big for any single card.

Engine pods

A RunPod-only model has backend: runpod, so the engine goes on the runpod block: engine: ds4, colibri, k3 or kt (and coderai-vllm / coderai-llama for the LLM servers as CoderAI pods). The pod arrives with that engine switched on and this install's settings for it forwarded — context, expert cache, extra args, the ds4 download variant — and volume_path points it at weights already on the volume. ds4 downloads its variant when nothing is there; colibri, k3 and kt never download.

Field notes

These come from live testing against a real account, not from documentation:

• Community cloud is the flaky tier. Expect "no longer any instances available", and pods that are created but never boot. Secure booted reliably. Selecting both pools with selection_criteria: "faster" prefers Secure and falls back only when needed.
• Cold start is expensive in wall-clock, not money. A successful cold pods request took about nine and a half minutes end to end — through two unavailable GPUs and one failed boot — and cost roughly half a cent.
• Choose idle_timeout_s deliberately. Too short and you re-pay the cold start; too long and you rent an idle GPU. Minutes for interactive use; for batch work, pinning min_pods: 1 for the duration is cheaper than repeated cold starts.
• Gated models need HF_TOKEN in the pod's env, or the pod boots, fails the download, and dies at the load timeout.