Build a Proxmox HA Cluster: Quorum, Fencing and Failover

Proxmox high availability is a chain of four mechanisms rather than a setting: corosync for membership, quorum for deciding who may act, a watchdog for removing a node that may not, and the HA manager for restarting the workload elsewhere. Miss a link and the cluster either fails to recover or recovers on both sides at once and corrupts shared storage. This guide covers all four, and the failover testing that proves they work.

Take them in order, because each depends on the one before. Corosync establishes which nodes can see each other. Quorum turns that membership into permission to run workloads. The watchdog enforces the absence of permission, by resetting a node that no longer holds it. Only then can the HA manager start a virtual machine elsewhere, knowing the original cannot still be writing to the same disk.

This is the architecture MassiveGRID runs. Every server on the platform sits on a Proxmox HA cluster with Ceph triple-replicated NVMe storage and automatic failover between nodes.

How our high availability works · Proxmox support from $99/node/month
HA Private Cloud · Colocation

Three Nodes Is the Real Minimum

Proxmox will form a cluster with two nodes and it will not survive a failure the way people expect. Quorum requires more than half of the votes, so a two-node cluster needs two votes to be quorate. Lose one node and the survivor holds one vote out of two, which is not a majority, so it stops rather than risk both halves writing to the same storage believing the other is dead.

That behaviour is correct. Split brain on shared storage destroys data, and refusing to act is the safe failure. But it means a two-node cluster gives you management convenience, not availability.

Three nodes tolerate one failure with two votes out of three. Four nodes still tolerate only one, because losing two leaves two out of four, which is a tie. Five tolerate two. The pattern is that odd numbers are efficient and even numbers waste a node:

If three nodes are genuinely not available, a QDevice gives a two-node cluster a tie-breaking third vote from a small external machine that runs nothing else. It needs no Proxmox installation, just network reachability from both nodes and a location that will not fail with either of them.

# on the external arbiter
apt install corosync-qnetd

# on one cluster node
apt install corosync-qdevice
pvecm qdevice setup 10.10.0.9
pvecm status

Give Corosync Its Own Network

This is the decision that separates clusters that behave from clusters that fence themselves at random, and it is the one most often skipped.

Corosync exchanges membership messages on a timer. It is indifferent to bandwidth and extremely sensitive to latency and jitter. Put it on the same interface as VM traffic, backups or Ceph replication, and a large backup job can delay those messages past the token timeout. Corosync then concludes the node is gone, quorum is recalculated, and a perfectly healthy node self-fences in the middle of the working day. The logs will show a token loss and no cause, because the cause was a backup.

Build the cluster with a dedicated link, and a second one for redundancy. Corosync supports up to eight rings and will fail over between them:

# first node
pvecm create production \
  --link0 address=10.10.0.1 \
  --link1 address=10.20.0.1

# each additional node
pvecm add 10.10.0.1 \
  --link0 address=10.10.0.2 \
  --link1 address=10.20.0.2

pvecm status
corosync-cfgtool -s

Keep corosync latency under a few milliseconds and steady. A separate physical link at 1 GbE is better than a shared 25 GbE link, because the metric that matters is jitter rather than throughput.

How Fencing Actually Works Here

Coming from VMware or from Linux HA stacks, people look for STONITH agents and IPMI credentials. Proxmox does not work that way. It uses watchdog-based self-fencing.

Every node with HA active arms a watchdog timer and refreshes it while it holds quorum. Lose quorum and the node stops refreshing. The watchdog expires and resets the machine. Meanwhile the remaining quorate nodes wait out a fixed interval, long enough that the lost node must already have reset itself, then restart its workloads.

The elegance is that it needs no out-of-band access and no credentials for a BMC that may itself be unreachable. The consequence is that recovery is not instant: the sequence has to wait for the watchdog to fire before anything can safely start elsewhere. Expect roughly a minute or two from failure to the workload running again, not seconds.

By default Proxmox uses the softdog kernel module, which is adequate. A hardware watchdog, IPMI or a TCO device on server-grade boards, is more trustworthy because it survives a kernel that has stopped scheduling. Configure it in /etc/default/pve-ha-manager and confirm the module is loaded rather than assuming it.

Shared Storage Is Not Optional

HA restarts a VM on another node. That node must be able to read the VM's disk, which means the disk cannot live on local storage.

ZFS replication deserves a clear warning. It is asynchronous, so a failover loses everything written since the last replication run. At a one-minute schedule that is up to a minute of writes gone. For a build agent that is fine. For a database it is a data-loss event, and it will be discovered during the incident rather than during planning.

Configuring HA

Define groups first, so recovery targets are deliberate rather than arbitrary. A group with a priority list keeps a workload near the nodes that suit it:

ha-manager groupadd db-nodes --nodes "node1:2,node2:2,node3:1"
ha-manager add vm:120 --group db-nodes --max_restart 2 --max_relocate 2
ha-manager status

Higher numbers win. Setting nofailback=1 on a group stops a VM migrating back the moment its preferred node returns, which matters because a node that has just rebooted is the node most likely to reboot again.

The two restart counters are worth setting explicitly. max_restart is how many times HA tries locally before relocating; max_relocate is how many other nodes it tries. Left unbounded, a VM that crashes because of its own configuration will migrate around the cluster indefinitely, which turns one broken VM into cluster-wide noise.

Test Failover Before You Need It

An untested HA configuration is a belief, not a capability. Test all three failure modes, because they exercise different code paths.

Graceful maintenance, which should move workloads with no downtime at all:

ha-manager crm-command node-maintenance enable node2

Abrupt loss of a node, which is the one people skip and the one that matters. Pull the power, or from a console on the node itself:

echo b > /proc/sysrq-trigger

Then watch recovery from another node and time it:

watch -n 2 'ha-manager status; pvecm status | head -20'

Third, and least intuitive, test network partition rather than node death. Block the corosync link on one node and confirm it fences itself and that the majority side recovers its workloads. This is where a cluster with corosync sharing a busy interface reveals itself.

Record the observed recovery time. That number, not the marketing figure, is what you can promise the business.

Mistakes That Cost Clusters

What HA Does Not Give You

Proxmox HA restarts a virtual machine. It does not preserve its memory state, its open connections or its in-flight transactions. Everything running at the moment of failure is lost, and the guest comes up as though it had been powered off abruptly.

So the guest needs to survive an unclean shutdown, which means journalled filesystems, a database configured to recover, and applications that reconnect rather than hang. And HA is not backup: it protects against a node failing, not against a deletion, a bad deployment or ransomware. Proxmox Backup Server covers that, and it is a separate project from this one.

Or Run on a Cluster That Is Already Proven

Everything above is achievable, and the cost is measured in the hardware for a third node, a dedicated corosync network, and the operational discipline to test failover on a schedule rather than after an incident.

MassiveGRID has run Proxmox HA with Ceph in production for years, across a partner footprint spanning 85+ metros in 30+ countries. Workloads sit on clusters with automatic failover and three-way replicated NVMe storage, and the corosync design, watchdog configuration and failover testing are already done. Because CPU, RAM and storage scale independently, growth does not mean rebuilding the cluster.

For clusters you own, Proxmox support plans start at $99 per node per month, with HA configuration and failover management included at $249 alongside a four-hour critical response SLA. Next, size the storage layer with our Proxmox Ceph setup guide.