CPU-Only LLM Inference: What Actually Works Without a GPU

Language models run without a GPU. The useful question is which workloads tolerate the memory bandwidth a CPU has, and that line falls in a more generous place than most people assume. Embeddings, reranking, classification and batch summarisation all sit comfortably on the cheap side of it, which means a lot of AI infrastructure spending is aimed at the wrong tier.

The reason that line is generous comes down to what actually limits inference, and it is not the thing most people assume. Once you know why a CPU sits roughly an order of magnitude behind a data centre GPU rather than a hundred times behind, the sensible workload split becomes obvious and the hardware budget drops sharply.

MassiveGRID Cloud VPS and Dedicated VPS: per-resource pricing at $2.87 per core, $0.80 per GB RAM and $0.01 per GB SSD · Proxmox HA clustering with automatic failover · Ceph 3x replicated NVMe · placement in any of 85+ metros across 30+ countries

Cloud VPS from $1.99/mo · Dedicated VPS for guaranteed cores · GPU instances when CPU is not enough

Why CPUs Are Slow at This

Not for the reason people assume. Token generation is not compute-bound, it is memory-bound: producing each token requires reading every active parameter out of memory, so the ceiling is bandwidth divided by model size.

That reframes the comparison usefully. A modern multi-channel server platform delivers somewhere in the region of 200 to 400 GB/s of memory bandwidth. An A100 delivers around 1.9 TB/s and an H100 around 3.4 TB/s. So a CPU is not hundreds of times slower for generation, it is roughly five to fifteen times slower, and it costs a small fraction as much.

Two practical consequences follow. Memory channels matter more than core count, so a many-core CPU on a narrow memory bus underperforms a modest one on a wide bus. And past a certain point adding cores stops helping at all, because they are all waiting on the same memory.

What Runs Well on CPU

This is the part worth internalising, because several of these are quietly the majority of real AI work.

Embeddings deserve emphasis. In a retrieval system the embedding model is small, the work batches perfectly, and it runs once per document rather than once per user request. Buying GPU capacity for an embedding tier is a common and expensive misallocation, and our RAG stack guide keeps that tier on CPU deliberately.

What to Actually Run

llama.cpp is the mature path, and the GGUF format it uses is built for quantised CPU inference:

apt install -y build-essential cmake libcurl4-openssl-dev
git clone https://github.com/ggml-org/llama.cpp /opt/llama.cpp
cd /opt/llama.cpp
cmake -B build -DLLAMA_CURL=ON
cmake --build build --config Release -j$(nproc)

It ships an OpenAI-compatible server, so application code does not need to know it is talking to a CPU:

./build/bin/llama-server \
  --model /models/llama-3.1-8b-instruct-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  --threads $(nproc) \
  --ctx-size 8192 \
  --batch-size 512

Set --threads to physical cores rather than logical ones. Hyper-threaded siblings contend for the same memory path, and on a memory-bound workload they frequently make things slower rather than faster. Test both on your own host, because the answer varies by platform.

Quantisation Matters More Here

On a GPU, quantisation is mostly about fitting in VRAM. On a CPU it directly buys speed, because a smaller model means fewer bytes read per token.

Q4_K_M is the practical default. Below 4-bit, quality falls off faster than size does, so when a model does not fit the better move is almost always a smaller model at 4-bit rather than the same model at 3-bit.

Sizing a CPU Inference Host

Memory is the constraint, and the arithmetic is simple: enough RAM to hold the model plus the KV cache plus the operating system, with headroom so the page cache can keep the weights resident.

Use a Dedicated VPS rather than a shared-core plan for anything with a latency expectation. Inference saturates the cores it is given, so on shared vCPU it is both a noisy neighbour and a victim of one, and generation speed becomes unpredictable in a way that is hard to explain to users.

Measuring Rather Than Guessing

Benchmark on the actual host, because platform differences here are large:

./build/bin/llama-bench \
  -m /models/llama-3.1-8b-instruct-Q4_K_M.gguf \
  -p 512 -n 128 -t 8,16,24

That reports prompt processing and generation separately, and the distinction matters. Prompt processing is compute-bound and parallelises across cores. Generation is memory-bound and does not. A host that ingests a long document quickly may still produce tokens slowly, which is exactly the profile that suits summarisation and does not suit chat.

Sweeping the thread count is worth the minutes. The best value is frequently below the total core count, and finding it is free performance.

The Honest Comparison With a GPU

Take an 8B model at 4-bit. On a well-provisioned CPU host expect roughly 10 to 20 tokens a second. On a mid-range GPU expect 50 to 150.

Reading speed is around 5 to 8 tokens a second, so both are faster than a person reads. The difference is felt in time to first token and in what happens under concurrency: a CPU serving a second simultaneous request roughly halves throughput per user, where a GPU with continuous batching absorbs it.

So the decision rule is about shape rather than speed. One user at a time, or a queue, and CPU is fine at a twentieth of the cost. Several concurrent users expecting immediacy, and it is not.

A Sensible Architecture

Split the workload by what it actually needs, rather than putting everything on the most expensive tier.

Embeddings, reranking, classification and batch summarisation go on a Dedicated VPS at tens of dollars a month. Interactive generation, if a human genuinely waits for it, goes on a GPU instance, hourly at $2.26 for an A100 40 GB if demand is intermittent or monthly at $1,649 if it is steady.

That split routinely cuts the bill by most of its value, because the GPU ends up sized for the one stage that needs it rather than for the whole pipeline. Both tiers sit on the same Proxmox high-availability clusters with Ceph triple-replicated NVMe storage, and CPU, RAM and storage scale independently, so adding the memory a larger quantised model needs does not mean buying cores you will not use.

If you conclude you do need a GPU, size it with our VRAM requirements guide and serve it with vLLM. If you are still exploring, Ollama on a VPS runs on either.