Run Stable Diffusion on a GPU Server: Setup and Sizing

Image generation sizes differently from text generation. A language model holds its weights and streams tokens; a diffusion model holds smaller weights and then allocates memory in proportion to resolution and batch size. That means the same card is comfortable at one setting and out of memory at the next, so resolution and batch size have to be decided before the hardware is.

There is a second difference that matters for choosing hardware. Diffusion is comparatively compute-heavy and light on memory bandwidth, where token generation is the reverse. That inverts the usual card ranking: the workstation card whose GDDR6 memory penalises language models is well matched here, and it costs considerably less per gigabyte of VRAM.

MassiveGRID GPU options for image generation: RTX 4000 Ada 20 GB from $449.99/mo · RTX 6000 Ada 48 GB from $1,420.85/mo · A100 40 GB from $1,649/mo or $2.26/hr · H100 80 GB from $3,999/mo. PyTorch, CUDA and the Hugging Face stack pre-installed.

Dedicated GPU servers · GPU cloud with hourly billing

What the Model Generation Changes

The three families in common use have very different footprints, and picking hardware for the wrong one is the usual mistake.

Treat the workable column as the figure that matters. Every family runs in less memory than shown, using offloading and attention slicing, at a speed penalty that turns interactive work into batch work.

What Actually Consumes VRAM

Four things, and only the first is fixed:

Weights. The UNet or transformer, the text encoders and the VAE. Loading base and refiner together doubles part of this, which is why SDXL workflows are heavier than the headline number.

Activations, scaling with resolution. Attention memory grows faster than linearly with pixel count, so doubling each dimension is considerably more than four times the cost. This is why 1024×1024 is a different proposition from 512×512 on the same card.

Batch size, multiplying everything. Four images at once is roughly four times the activation memory. Batching is the main throughput lever and the main cause of out-of-memory failures.

Extras. ControlNet adds a second network. Upscalers and img2img at high resolution allocate large intermediate tensors. LoRAs are small individually and add up.

The practical consequence: decide resolution and batch size first, then choose the card. Choosing the card first produces a configuration that works for single 512-pixel images and fails on the first real request.

Setting It Up

ComfyUI is the better choice on a server, because a workflow is a JSON graph you can version, run headless and call over an API. Web interfaces built for interactive use are harder to automate.

apt install -y python3-venv git
git clone https://github.com/comfyanonymous/ComfyUI /opt/comfyui
cd /opt/comfyui
python3 -m venv venv
./venv/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
./venv/bin/pip install -r requirements.txt

Confirm the GPU is visible before troubleshooting anything else, because a silent fall back to CPU presents as extreme slowness rather than as an error:

./venv/bin/python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
nvidia-smi

Run it as a service bound to localhost, with a reverse proxy in front. An exposed image generation endpoint is compute someone else will happily spend:

[Unit]
Description=ComfyUI
After=network-online.target

[Service]
User=comfy
WorkingDirectory=/opt/comfyui
ExecStart=/opt/comfyui/venv/bin/python main.py --listen 127.0.0.1 --port 8188
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Model files are large and numerous. Keep them on a separate volume and symlink, rather than filling the root filesystem and discovering it during a generation run.

The Flags That Buy You Memory

When a model does not fit, these trade speed for VRAM. Reach for them in this order, because the early ones cost little:

Sequential offload deserves a warning. It moves parts of the model between system RAM and VRAM as needed, which makes almost anything run on almost any card and can slow generation by an order of magnitude. It is a way to prove a workflow, not a way to serve one.

Matching Card to Workload

Image generation is a case where the RTX 6000 Ada is genuinely well matched, which is not true for text. Diffusion is more compute-bound and less bandwidth-bound than token generation, so the GDDR6 penalty that hurts language models matters less, and 48 GB of memory for less than an A100 40 GB is good value.

Throughput and What to Measure

Benchmark your own workflow rather than trusting published figures, because step count, sampler, resolution and extras change the answer by multiples.

Three numbers matter. Time per image at batch one, which is what a waiting user experiences. Images per minute at your largest working batch, which is your capacity. And the batch size at which you run out of memory, which is your actual ceiling and worth knowing before a queue finds it for you.

Watch memory during a run rather than inferring it afterwards:

nvidia-smi dmon -s um
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 2

If utilisation sits well below full while generation is slow, the bottleneck is elsewhere: model loading from disk, image encoding on the CPU, or a workflow serialising steps that could batch.

The Part That Is Not Technical

Two things belong in the plan before this goes anywhere near users, and they are the reasons self-hosted image generation projects get stopped rather than the reasons they fail technically.

Model licences differ and some restrict commercial use. Check the licence for every checkpoint and LoRA you load, not just the base model. This is a genuine commercial risk that is easy to check and easy to overlook.

An open generation endpoint will be used to generate things you do not want associated with your infrastructure. Authentication, rate limiting and a record of who generated what are not optional for anything reachable beyond your own network.

Dedicated Cards and Hourly Billing

Image generation is bursty in a way that suits hourly billing. An experimentation phase, a batch of production renders, then idle. The A100 40 GB at $2.26 an hour reaches its $1,649 monthly price at around 730 hours, so anything running less than roughly three weeks a month is cheaper hourly.

For a steady endpoint, monthly is both cheaper and reserves the capacity. Dedicated GPU servers start at $449.99 a month for the RTX 4000 Ada with 20 GB, which is the sensible entry point for SDXL, and $1,420.85 for the RTX 6000 Ada with 48 GB, which is where batching and ControlNet stop being constrained.

Every instance is a dedicated card rather than a time slice, which matters here because a shared GPU changes both throughput and available memory, and a workflow tuned to the edge of VRAM will fail intermittently. PyTorch, CUDA, cuDNN, Docker with the NVIDIA container runtime and the Hugging Face stack come pre-installed, on Proxmox high-availability clusters with Ceph triple-replicated NVMe storage for the model library. Compare the cards for other workloads in our GPU selection guide.