Self-Host Whisper for Speech-to-Text on a GPU Server

Speech recognition is one of the few AI workloads where self-hosting is plainly cheaper than an API at modest volume, because the model is small and the work batches well. It is also a workload where the obvious implementation runs several times slower than the good one, and where two features people assume are built in, accurate word timings and speaker labels, are separate models entirely.

Start with the economics, because they are unusually favourable here. Transcription APIs bill per minute of audio, so cost scales linearly with volume and never stops climbing. A self-hosted deployment has a fixed monthly cost and a throughput ceiling instead, which means there is a crossover point, and for anyone transcribing meetings or calls routinely it arrives early.

MassiveGRID GPU instances with PyTorch, CUDA, cuDNN and the Hugging Face stack pre-installed · RTX 4000 Ada 20 GB from $449.99/mo · A100 40 GB from $1,649/mo or $2.26/hr · placement in any of 85+ metros across 30+ countries

GPU cloud · Dedicated GPU servers · Dedicated VPS for CPU transcription

Pick the Implementation Before the Model

Whisper is a model, and several projects run it. Which one you choose changes speed and memory more than which model size you pick, so decide this first.

Start with faster-whisper on a GPU or whisper.cpp on a CPU. Reaching for the reference implementation and then concluding that transcription needs an expensive card is a common and avoidable detour.

Model Size and What It Costs

Two things worth knowing before defaulting to the largest model. Quality gains are strongly non-linear: the step from small to medium is usually larger than medium to large for clear English. And quantisation to int8 roughly halves the memory with a small accuracy cost, which often makes the large model fit where you expected to need medium.

So test on your own audio. Accuracy on clean studio recordings and accuracy on a noisy conference call are different problems, and the model that wins one may not win the other.

Setting It Up

python3 -m venv /opt/whisper
/opt/whisper/bin/pip install faster-whisper

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")

segments, info = model.transcribe(
    "meeting.mp3",
    beam_size=5,
    vad_filter=True,
    language="en",
)

print(f"detected {info.language} with probability {info.language_probability:.2f}")
for s in segments:
    print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")

Three arguments there matter more than they look.

vad_filter=True runs voice activity detection first and skips silence. On recordings with long gaps, which describes most meetings and calls, this alone can halve the work. It also suppresses a well-known failure where the model invents text during silence.

language="en" skips language detection. Setting it when you know the language is faster and avoids a misdetection that garbles the whole transcript.

compute_type is where quantisation happens. Use float16 on a modern GPU, int8_float16 if memory is tight, and int8 on a CPU.

Throughput and Batching

Transcription speed is usually quoted as a real-time factor: how many minutes of audio you process per minute of wall clock. On a modern GPU with faster-whisper, the large model runs many times faster than real time, and batching pushes that considerably further.

The important architectural point is that this is a queue workload, not an interactive one. Nobody waits synchronously for an hour-long recording. So the right shape is a job queue, a worker holding the model in memory, and a callback when the transcript is ready.

Loading the model takes seconds and transcribing a short file takes less. A service that loads the model per request spends most of its time loading, which is the most common performance mistake in self-hosted transcription. Load once at start-up and keep it resident.

Word Timestamps and Speaker Labels

Two features are frequently assumed to be part of Whisper and are not.

Whisper produces segment-level timestamps that are approximate. If you need accurate word-level timing, for subtitles or for search within audio, that requires a forced alignment pass, which is what WhisperX adds.

Speaker diarisation, labelling who spoke when, is a separate model entirely. The commonly used one requires accepting terms and authenticating to download it, which is a practical hurdle worth discovering during planning rather than during deployment. Diarisation is also the least reliable part of the pipeline: overlapping speech and similar voices produce errors that no configuration fixes.

If your requirement is "a transcript with speaker names", budget for that being the hard part rather than the transcription.

Does This Need a GPU at All?

Often not, and this is worth checking before spending. whisper.cpp with a quantised model transcribes usefully on ordinary server CPUs, at a real-time factor that is fine for a queue.

The honest test is whether anybody is waiting. Overnight transcription of the day's recordings is a CPU job. Live captioning is not.

The Privacy Argument

This is usually the actual reason for self-hosting, and it is a strong one. Recordings of meetings, medical consultations, legal calls and support conversations are among the most sensitive data an organisation holds, and sending them to a transcription API means sending exactly that.

For anything under GDPR, professional privilege or sector rules on health data, keeping audio on infrastructure you control removes a transfer and a processor from the analysis rather than papering over them. Our guide to GPU hosting in Europe covers the jurisdictional side, and the same reasoning applies to the storage: recordings and transcripts both need a retention policy, because a transcript is the recording in a more searchable form.

Infrastructure for a Queue Workload

Transcription suits hourly GPU billing well, because the load is intermittent by nature. The A100 40 GB at $2.26 an hour reaches its $1,649 monthly price at around 730 hours, so a workload running a few hours a day is substantially cheaper hourly.

For a permanent service, the RTX 4000 Ada at $449.99 a month with 20 GB comfortably holds the large model with room for batching, and is the cheapest sensible dedicated option. Every instance is a whole card rather than a time slice, which matters for predictable queue drain rates.

PyTorch, CUDA, cuDNN and the Hugging Face stack are pre-installed, so setup is the code above rather than a driver project. Storage sits on Ceph with three-way replication, which is worth having under an audio archive nobody can recreate. For CPU-only transcription, a Dedicated VPS with guaranteed cores keeps queue times predictable, and our CPU inference guide covers what else runs well without a GPU.