Proxmox Cluster Sizing: Nodes, Capacity and Failure Domains

Meeting the voting requirement and being able to survive a failure are two different tests, and only the first is documented anywhere prominent. The second is arithmetic about memory and free storage that has to be done before hardware is ordered, because the answer changes how many machines you buy. This is that calculation, plus the dependencies that make nodes fail together.

Three nodes gives you quorum. It does not tell you whether the cluster can absorb losing one, which is a capacity question with an answer most people have not calculated. A cluster that stays quorate while being unable to run its workloads has satisfied the documentation and failed the purpose.

Quorum Is the Floor, Not the Design

The voting arithmetic is settled and covered elsewhere: an odd number of nodes, a majority required, three as the practical minimum. Our walkthrough for building a Proxmox HA cluster covers that, along with fencing and the shared storage it depends on.

What that arithmetic does not address is the question this post is about. When a node fails, its guests restart on the survivors. Do the survivors have the memory? If not, HA will try, some guests will fail to start, and you will discover which ones by finding out what stopped working.

The N+1 Arithmetic

Reserve one node's worth of capacity across the cluster, and express it as a per-node utilisation ceiling.

The pattern is the argument for more, smaller nodes. On three nodes you pay a third of your hardware for redundancy; on seven you pay a seventh. The same total memory arranged as seven machines rather than three gives you roughly 28% more usable capacity at the same redundancy, plus a smaller blast radius per failure.

Memory is the binding constraint, not CPU. Proxmox will happily oversubscribe cores, and a busy period on a degraded cluster is slow. Memory is not compressible: a guest with 32 GB assigned either has somewhere to go or does not start.

Do the sum with real numbers rather than nominal ones. Subtract what the hypervisor itself needs, and on a hyper-converged cluster subtract what Ceph's OSDs consume, which is not small.

Ceph Changes the Sum

A hyper-converged cluster has a second capacity question that is easy to get wrong in the expensive direction.

With size 3 replication, usable storage is a third of raw. That much is well known. What catches people is that recovery needs somewhere to put re-replicated data: when a node fails, Ceph rebuilds its copies onto the survivors, so a cluster already near capacity cannot restore redundancy and sits degraded until hardware is replaced.

So the storage ceiling is lower than the arithmetic suggests. Plan to stay under roughly 70% of usable capacity in steady state on a small cluster, and note that Ceph itself starts warning near 85% and stops writes at the full ratio, which takes the cluster down rather than degrading it.

Node count matters here too. Three nodes with size 3 means one copy per node and no spare node to recover onto, so a node failure leaves you degraded by definition until it returns. Five nodes gives recovery somewhere to go. That is the strongest single argument for five over three, and it is a storage argument rather than a quorum one. Our walkthroughs for setting up Ceph on Proxmox and choosing a storage backend cover the pool settings.

Failure Domains Above the Node

Node-level redundancy assumes nodes fail independently. Frequently they do not, and the shared dependency decides what a single failure actually costs.

Four to check, in ascending order of how often they are overlooked. A power feed, where three nodes on one circuit are one breaker away from a total outage. A network switch, where a single top-of-rack switch means a switch reboot is a cluster partition. A rack, whose loss takes everything in it. And the storage network, which on a hyper-converged cluster is as critical as power.

Ceph can be told about this. Setting the CRUSH failure domain to rack rather than host means replicas are placed in different racks, so a rack loss leaves a complete copy. It requires enough racks to satisfy the rule, which is the constraint that makes people leave it at host.

The honest position for a small cluster in one rack: you are protected against node failure and not against rack failure, and the answer to the second is a second site rather than a cleverer CRUSH map. Say so in the design document rather than implying more.

Hardware Shape Per Node

Given a capacity target, the per-node specification follows from a few rules that are cheap to follow and expensive to retrofit.

Memory: the largest single number, and the one to over-provide. Adding memory later means downtime per node; buying it now costs money once.

Network: separate interfaces for cluster, storage and guest traffic, with 10 GbE or better on storage. Corosync sharing a link with a backup job is the most common cause of nodes being fenced spuriously. Our guide to Proxmox networking covers the separation.

Disks: NVMe for Ceph, several per node rather than one large one, since recovery parallelism follows OSD count. Enterprise drives with power-loss protection, because Ceph's write pattern punishes consumer SSDs severely.

Boot device: mirrored, and separate from the OSDs. A node whose boot drive failed is a node down for a rebuild.

Uniformity: identical nodes where you can. Mixed capacities mean the smallest node bounds what can migrate to it, and unequal Ceph node weights complicate recovery.

The Growth Path

Clusters grow, and the sequence matters because some steps are disruptive and others are not.

Three to five is the most valuable expansion available: it takes you from no recovery headroom to real headroom, and it lowers the redundancy tax from 33% to 20%. Five to seven is incremental by comparison. Adding nodes is online, so growth is not the hard part.

What is hard is going from one large node to a cluster, because guests must move onto shared storage. Building on shared storage from the start, even for a single node you intend to grow, avoids a migration later. That single decision is worth more than any amount of capacity planning.

Adding an even-numbered node is safe but adds no fault tolerance, so treat four as a step towards five rather than a destination. And where two nodes is the budget, a QDevice on a third machine supplies the tie-breaking vote without a third full node, which is a legitimate arrangement rather than a hack.

What to Verify Before You Trust the Plan

Capacity planning on paper is a hypothesis. Two tests confirm it, and both belong in the commissioning window rather than in an incident.

Fail a node deliberately, with the cluster loaded to its planned ceiling, and confirm every guest restarts. Not a graceful shutdown: pull the power, so fencing is exercised too. Count the guests afterwards against what was running before.

Then pull a disk and watch Ceph recover, checking that the cluster reaches HEALTH_OK rather than settling at degraded for want of somewhere to put the data. Time both. Those two numbers are your real recovery characteristics, and they are usually worse than the plan assumed. Our runbook for maintenance without downtime covers the same discipline applied to planned work.

Or Buy the Capacity Already Sized

Every number above is a hardware commitment made in advance, which is the awkward part: you buy for the peak and the failure case, and run below both.

MassiveGRID's platform runs Proxmox high-availability clustering with automatic failover over Ceph storage that replicates every block three times across independent NVMe drives, sized and maintained with the headroom above already accounted for, behind a 100% uptime SLA. A private cloud gives you dedicated capacity without commissioning it, and per-resource pricing at $2.87 per CPU core, $0.80 per GB of RAM and $0.01 per GB of SSD per month means the redundancy tax is not yours to carry. Proxmox support from $99 per node per month up to $449 and custom tiers covers the design review for a cluster you build yourself, which is the cheapest point at which to catch a sizing error.

Infrastructure can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore, which is also how you get the second failure domain that a single rack cannot provide.