AISBFThe backend is a per-model choice
POST /v1/audio/transcriptions keeps the OpenAI shape, but the engine behind it is whatever that model's entry declares in its backend key. Different jobs want genuinely different engines: verbatim transcription with disfluencies is not the same task as a fast CPU-only draft, and neither is speech translation.
backend | Engine | Notes |
|---|---|---|
| unset | whisper.cpp server → faster-whisper → whispercpp | the default fallback chain |
whisper-server | whisper.cpp GGUF | a per-model runner; starting it counts as a model load, so it evicts for VRAM and is itself evictable |
whisper-hf / crisperwhisper | CrisperWhisper | verbatim transcription with word timestamps; long-form overlap windowing; isolated venv |
wav2vec2 | Wav2Vec2 (HF) | FP16 on GPU |
vosk | Vosk | CPU, per-language model directory |
nemo | NVIDIA NeMo — Canary / Parakeet | isolated venv; the translation-capable family |
Each model entry can also declare a languages allow-list — a request for a language the model was not trained on is rejected with the list of what it does support, instead of silently returning nonsense — and supports_translation, which gates the target_language parameter. /v1/models surfaces both.
Beyond the transcript
Word timestamps
A top-level words[] with per-word timings, for subtitle alignment, dubbing and editing.
Who spoke when
Runs pyannote and tags every segment with a speaker. num_speakers is a hint when you know the answer. Also available on its own at /v1/audio/diarization, with srt and vtt output.
Translation
Speech in, translated text out, in one pass — on a model whose family supports it.
Speaker recognition
Diarization tells you there were three speakers. It does not tell you which one is the judge. Enrolled voiceprints do.
| Endpoint | Purpose |
|---|---|
POST /v1/audio/speakers | Enrol a named speaker from a clip. Re-enrolling averages the voiceprint. |
GET /v1/audio/speakers · DELETE /v1/audio/speakers/{name} | List and remove enrolments. |
POST /v1/audio/speaker-identify | Best match among the enrolled, or unknown below the threshold. |
POST /v1/audio/speaker-verify | Is this the same person as name? A yes/no against one enrolment. |
POST /v1/audio/speaker-embeddings | Raw voiceprint vectors. A window/step pair switches to sliding-window mode, one embedding per window. |
Backends: ecapa (SpeechBrain ECAPA-TDNN, ungated, the default), pyannote, and wespeaker. Enrolment and identification must use the same backend — the vectors are not interchangeable.
Putting the two together
Call /v1/audio/diarization with identify=true and the turns come back labelled with enrolled names instead of SPEAKER_00. That is the difference between a diarized transcript and a usable one.
The rest of the audio surface
Separation and repair
/v1/audio/stems splits vocals from instrumental, or into four stems, with Demucs. /v1/audio/cleanup denoises with DeepFilterNet (VoiceFixer as fallback) and then applies hum removal, click repair and loudness normalisation as DSP. Without those packages both return 501 naming what to install, or accept fallback_mode for a best-effort ffmpeg pass.
Speech synthesis
Kokoro TTS, F5-TTS zero-shot voice cloning from a reference clip, and Seed-VC voice conversion with a singing mode. Named voice profiles can be extracted straight from a video file and reused by name.
Pipelines
/v1/pipelines/audio-dub transcribes, translates, clones the voice and replaces the track. /v1/pipelines/audio-understand transcribes and then answers a question about the recording with a text model.
Dubbing a song
/v1/pipelines/audio-music-dub sings a track in another language over its own backing. Demucs isolates the vocal; the lyrics are transcribed from that stem rather than the full mix; a text model adapts them to stay singable — same syllable count per line, rhyme kept where it can be; F5-TTS re-sings them using the isolated original vocal as the cloning reference, so the dub keeps the original singer’s timbre with no voice profile to set up; an optional pitch-conditioned Seed-VC pass pulls the delivery from spoken toward sung; and ffmpeg remixes the result over the instrumental.
Stages that need an optional dependency report a skipped status with a reason rather than failing or faking it, and the response says whether the run was complete or degraded.
Running several at once
STT models are VRAM-eviction-tracked like everything else, and keep_resident keeps a small one co-resident so short clips do not pay a load each time. Because one weights file can be registered several times under different aliases pinned to different engine nodes, a single whisper-large-v3 GGUF can be served as whisper0, whisper1 and whisper2 across two GPUs, transcribing three files in parallel.