Two deployments of the same model on the same card can differ twentyfold in what they cost to serve, and the gap is not hardware. It comes from how many requests share each pass over the weights, how much memory is left for attention state, and whether long prompts are allowed to stall everyone else. This works through each, ending in the one figure worth comparing.
A GPU serving one request at a time is being wasted, and the waste is large. Generating tokens for a single user leaves most of the card idle between memory accesses, which is why the difference between a naive deployment and a tuned one is not a few percent but often an order of magnitude in requests served per hour.
Why Batching Is Nearly Free
Token generation is memory-bandwidth-bound, not compute-bound. Each step reads the whole model's weights to produce one token per sequence. Reading those weights for one sequence costs the same as reading them for thirty-two, because it is the same read.
That is the entire economics of inference serving. Throughput scales almost linearly with batch size until memory runs out, while per-request latency barely moves. A card serving one user and a card serving twenty-four cost the same per hour.
Static batching, where a batch is assembled and run to completion, wastes most of this because sequences finish at different lengths and the whole batch waits for the longest. Continuous batching evicts finished sequences and admits new ones every step, which is what modern inference servers do and what makes the difference. Our guide to serving an LLM API with vLLM covers the server side.
The KV Cache Is Your Real Capacity Limit
Weights are a fixed cost, loaded once. The variable cost is the KV cache, which holds attention state for every token of every active sequence, and it is what decides how many concurrent requests fit.
Two consequences follow, and both are counterintuitive.
First, your concurrency limit is set by context length, not by request count. Sixty-four requests at 1,000 tokens and eight at 8,000 tokens consume similar cache. A capacity plan expressed in requests per second without a context length attached is meaningless.
Second, quantizing the model buys concurrency as well as fit. Halving the weight footprint hands the freed memory to the cache, so a 4-bit model on a 40 GB card serves substantially more concurrent sequences than the same model at 16-bit, quite apart from whether it fits at all. Our guide to GPU memory requirements for local LLMs covers the arithmetic.
The Knobs That Matter
| Setting | Effect | Cost |
|---|---|---|
| Raise max batch size | Large throughput gain | More cache memory; OOM if too high |
| Cap max context length | More concurrent sequences | Long prompts rejected |
| Raise GPU memory fraction | More cache for the same card | Less headroom; risk of OOM under spikes |
| Quantize weights | More cache, faster reads | Small quality loss, needs evaluation |
| Enable prefix caching | Large gain on shared prompts | Cache memory; useless if prompts differ |
| Speculative decoding | Lower latency per request | A second model in memory |
| Tensor parallelism | Fits models beyond one card | Interconnect overhead; needs NVLink |
Prefix caching deserves emphasis because it is the largest win available to most deployments and the most often left off. If every request shares a long system prompt, or a RAG pipeline prepends the same retrieved context to many queries, the prefill for that shared portion is computed once instead of per request. On a chat product with a 2,000-token system prompt, this is not a tuning detail.
Prefill and Decode Compete
Two phases with opposite characteristics share one card. Prefill processes the whole prompt at once and is compute-bound. Decode generates one token per sequence per step and is memory-bound.
They interfere. A large prompt arriving mid-stream stalls token generation for everyone already connected, which users experience as the response freezing. This is the mechanism behind most complaints about inconsistent latency in a self-hosted deployment.
Two mitigations. Chunked prefill splits a long prompt across several steps so decode continues in between, trading a little prefill latency for much steadier output. And separating traffic classes helps: long-prompt batch work and interactive chat on the same endpoint will always degrade each other, so route them to different instances if both matter.
Sharing One Card Between Tenants
Three approaches, and the right one is usually not the one people reach for first.
One server, many adapters. Load the base model once and attach several LoRA adapters, so a dozen fine-tunes share one copy of the weights. This is by far the most efficient way to serve multiple tenants a customised model, and it is an application-level feature rather than an infrastructure one. Our guide to fine-tuning on a single GPU covers producing those adapters.
MIG, on cards that support it. Partitions an A100 or H100 into hardware-isolated instances with dedicated memory and compute slices. Real isolation, and each partition is small, so it suits several modest models rather than one large one. Partitioning is a deliberate reconfiguration, not something to change per workload.
Time-slicing. Multiple processes share a card by interleaving. Simple, no isolation, and unpredictable latency because one tenant's spike is everyone's spike. Acceptable for development, not for anything with a response-time commitment.
Running two full copies of the same model on one card to serve two tenants is the option to avoid: you have paid for the weights twice in the memory that should have been KV cache.
Measure Cost Per Million Tokens
Utilisation and requests per second are not comparable across models or hardware. Cost per million output tokens is, and it is the number that makes a build-versus-buy conversation tractable.
The arithmetic is simple: hourly instance cost divided by tokens generated per hour, scaled to a million. An A100 40GB at $2.26 per hour sustaining 1,200 output tokens per second across a batch produces 4.32 million tokens an hour, which is about $0.52 per million. The same card serving one request at a time might manage 60 tokens per second, or 0.216 million an hour, which is $10.46 per million. Same hardware, same model, twenty times the cost, entirely down to batching.
Measure it under your own traffic shape rather than with a synthetic benchmark, because the mix of prompt lengths and output lengths moves the result more than any single setting. Then re-measure after every change; several of the knobs above interact, and two individually good changes can combine badly.
Pick the Card by Throughput, Not Price
Because throughput scales with memory bandwidth and with how much cache fits, the more expensive card is frequently cheaper per token.
MassiveGRID's GPU cloud instances make that comparable directly. An A100 40GB with 16 vCPUs, 120 GB of RAM and 512 GB of NVMe delivers 312 TFLOPS of FP16 at $2.26 per hour or $1,649 a month. An A100 80GB with 24 vCPUs and 240 GB of RAM is $3.42 or $2,499, and the extra 40 GB goes almost entirely to KV cache, so concurrency rises more than the price does. An H100 80GB with 32 vCPUs, 480 GB of DDR5 and 989 TFLOPS is $5.48 or $3,999, and on a saturated endpoint it is typically the lowest cost per token of the three. Multi-GPU configurations with NVLink and InfiniBand are available for models that need tensor parallelism.
Run the cost-per-million calculation on each before committing to a monthly term. Hourly on-demand pricing exists precisely so that measurement is cheap. Our analysis of renting against buying covers the longer horizon, and the GPU infrastructure page covers the custom configurations.
Instances can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.