Self-Host an LLM API with vLLM on a GPU Server

The difference between a hobby inference setup and a production one is what happens on the second concurrent request. vLLM batches requests continuously into the same forward pass and manages the KV cache in pages, which is why one GPU can serve many sessions rather than queueing them. This guide covers installing it, the flags that decide whether it fits in memory, running it as a service, and measuring what the card actually delivers.

Ollama processes requests largely one at a time. With a single user that is invisible; with eight, seven of them wait. vLLM was built for the second case. It admits and evicts sequences at every decoding step so they share the same forward pass, and it stores the KV cache in fixed-size blocks rather than one contiguous reservation per sequence. Those two mechanisms are the whole reason a single card can serve a real user base.

vLLM comes pre-installed on MassiveGRID GPU instances, alongside PyTorch, CUDA, cuDNN, the NVIDIA container runtime, Jupyter and the Hugging Face stack.

A100 40 GB — $1,649/mo, 16 vCPU, 120 GB RAM
A100 80 GB — $2,499/mo, 24 vCPU, 240 GB RAM
H100 80 GB — $3,999/mo, 32 vCPU, 480 GB DDR5, 100 Gbps

What Continuous Batching Actually Does

Static batching waits for a batch to fill, runs it, and waits for the slowest sequence to finish before starting the next. Every finished sequence leaves its slot idle until the whole batch completes.

Continuous batching evicts finished sequences and admits waiting ones at every decoding step, so the GPU stays saturated. Paired with PagedAttention, which stores the KV cache in fixed-size blocks rather than one contiguous reservation per sequence, memory waste from over-allocated context drops from a large fraction to a few percent.

The practical effect is that throughput scales with concurrency until the cache pool is full, rather than being flat.

Prerequisites

An NVIDIA GPU with compute capability 7.0 or newer, which covers everything from V100 onward. The A100 is 8.0, the RTX 6000 Ada is 8.9, and the H100 is 9.0. FP8 quantization needs 8.9 or newer, so it is available on Ada and Hopper cards but not on A100.

Confirm the driver first, because every later error is confusing if this is wrong:

nvidia-smi
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

Installation

Install into a virtual environment rather than system Python. vLLM pins specific PyTorch and CUDA builds, and letting it manage them in isolation avoids a class of dependency conflicts that are tedious to unpick:

python3 -m venv /opt/vllm
/opt/vllm/bin/pip install --upgrade pip
/opt/vllm/bin/pip install vllm

Serve a model. The first run downloads weights from Hugging Face, which for an 8B model is a few minutes on a fast link:

/opt/vllm/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

Gated models need a token in the environment as HF_TOKEN. Set HF_HOME to a path with room, because the default cache under the home directory fills quietly and then fails mid-download.

The Flags That Matter

The mistake to avoid is leaving --max-model-len at the model default. A model advertising 128k context will try to size its cache pool for 128k, and on an 80 GB card that either fails at startup or leaves almost no room for concurrency. Set it deliberately.

If --enable-prefix-caching is available in your release, turn it on for any workload with a long shared system prompt. Retrieval-augmented setups that prepend the same instructions to every request see the largest benefit.

Running It as a Service

[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=vllm
Environment="HF_HOME=/var/lib/vllm/hf"
Environment="HF_TOKEN=your-token-here"
ExecStart=/opt/vllm/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 16384 --max-num-seqs 64 \
  --gpu-memory-utilization 0.90 --enable-prefix-caching
Restart=on-failure
RestartSec=15
TimeoutStartSec=600

[Install]
WantedBy=multi-user.target

Write that to /etc/systemd/system/vllm.service, then enable it:

useradd --system --home /var/lib/vllm --create-home vllm
systemctl daemon-reload
systemctl enable --now vllm
journalctl -u vllm -f

The long TimeoutStartSec is deliberate. Loading a large model and building the cache pool can take several minutes, and the default timeout will kill a healthy startup and leave you debugging the wrong problem.

Using the API

vLLM speaks the OpenAI API, so existing clients need only a base URL:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}],
    "max_tokens": 200
  }'

From Python, point the official client at it:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-used")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarise this alert: disk 94% full"}],
)
print(resp.choices[0].message.content)

Start the server with --api-key to require a bearer token. That is worth doing even behind a proxy, so an internal misconfiguration does not expose an open endpoint.

Reverse Proxy and Access Control

Bind vLLM to localhost and let nginx terminate TLS:

server {
    listen 443 ssl;
    http2 on;
    server_name inference.example.com;

    ssl_certificate     /etc/letsencrypt/live/inference.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/inference.example.com/privkey.pem;

    location /v1/ {
        proxy_pass         http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_buffering    off;
        proxy_read_timeout 600s;
        limit_req          zone=inference burst=20 nodelay;
    }
}

Define the rate limit zone in the http block, for example limit_req_zone $binary_remote_addr zone=inference:10m rate=10r/s. Inference requests are expensive enough that a single misbehaving client can consume a GPU on its own, and a rate limit is cheaper than an incident.

Measuring Throughput

Do not guess at capacity. vLLM ships a serving benchmark that reports request throughput, time to first token and inter-token latency under a chosen concurrency:

/opt/vllm/bin/vllm bench serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --base-url http://127.0.0.1:8000 \
  --dataset-name random \
  --num-prompts 200 \
  --max-concurrency 32

Three numbers are worth recording. Time to first token is what a user perceives as responsiveness. Inter-token latency determines whether streamed output reads smoothly. Total output tokens per second is the number that decides how many users one card supports.

Run the benchmark at the concurrency you expect, then again at double it. The point where time to first token climbs sharply is where the queue is forming, and that is your real capacity ceiling regardless of what average throughput says.

Common Failures

Dedicated GPUs and Real Failover

An OpenAI-compatible endpoint is easy to adopt and therefore easy to depend on. Once an application defaults to it, an outage is not a degraded feature, it is a failed request path.

MassiveGRID GPU instances are dedicated rather than time-sliced, so the VRAM you sized for stays yours and benchmark numbers hold in production. They run on the same Proxmox high-availability cluster as the rest of the platform, with automatic failover between nodes, Ceph triple-replicated NVMe storage for your model cache and adapters, and 12 Tbps DDoS protection in front of the network. The H100 tier includes 100 Gbps networking for pulling large weights and moving datasets.

Deploy in any of the 85+ metros on our datacenter map to keep inference close to the application calling it, and to keep data in a jurisdiction you can defend. Compare GPU configurations, or size the card first with our VRAM planning guide.