Proxmox Maintenance Without Downtime: A Node-by-Node Runbook

The commands for patching a cluster are all short and all documented. What turns a routine window into an incident is sequencing: starting the second node before the first has genuinely rejoined, rebalancing storage for a ten-minute reboot, or leaving a flag set afterwards. This is the order, with the check that gates each step.

Patching a cluster is not a technical problem so much as a sequencing one. The tools all work; what goes wrong is doing things in the wrong order, on too many nodes at once, or without checking that the previous step finished. This is the sequence, and the checks between each step.

The Order That Does Not Bite

One node at a time, always, and never begin the next until the cluster is fully healthy again. On a three-node cluster that means quorum can tolerate exactly one node down, so two nodes in maintenance simultaneously is a self-inflicted outage.

Before touching anything, confirm the cluster is healthy rather than assuming it. A cluster that was already degraded before maintenance is how a routine patch becomes an incident.

pvecm status                     # quorum, and every node present
ceph -s                          # HEALTH_OK, not HEALTH_WARN
pvesh get /cluster/resources --type vm --output-format yaml | grep -c 'status: running'

Take a note of that running count. It is what you compare against afterwards, and it catches the guest that quietly failed to come back.

Drain the Node Before You Touch It

Proxmox has a maintenance mode for HA-managed guests that migrates them off and, importantly, remembers to bring them back.

ha-manager crm-command node-maintenance enable pve-02
watch -n5 'ha-manager status'    # wait until nothing is left on pve-02

Two things this does not cover. Guests not under HA management are not moved, so migrate them by hand. And containers cannot live-migrate at all, which is the constraint that shapes the whole window.

qm migrate 101 pve-01 --online              # VM, no interruption
pct migrate 201 pve-01 --restart             # container, brief restart

Plan the container restarts explicitly. A cluster where every service runs in a container has no zero-downtime maintenance path, whatever the cluster documentation implies, and knowing that in advance is better than discovering it at 02:00. Our comparison of LXC containers and VMs on Proxmox covers choosing the guest type with this in mind.

Tell Ceph You Are Coming Back

This is the step most often skipped and the one that causes the most damage. When a node's OSDs go down, Ceph starts rebalancing data to restore replica counts. For a ten-minute reboot that rebalance is pure waste: it saturates the storage network, degrades guest I/O, and then has to be undone when the node returns.

ceph osd set noout
ceph osd set norebalance
# ... reboot the node, wait for OSDs to come back up ...
ceph osd unset norebalance
ceph osd unset noout

Set the flags, do the work, unset them. The unset is not optional: a cluster left with noout set will not recover from a genuine disk failure, and the flag is easy to forget because nothing appears to be wrong. Make checking for stray flags part of the post-maintenance verification.

Patch, Reboot, and the Kernel Question

With the node drained and Ceph told to sit still, the update itself is unremarkable.

apt update && apt full-upgrade
pveversion -v                     # what you now have
reboot

Use full-upgrade, not upgrade. Proxmox package transitions frequently need to remove or replace packages, and plain upgrade holds those back, leaving a half-updated node that behaves oddly.

A kernel update needs a reboot to take effect, and there is no live patching here. Since you have drained the node anyway, reboot rather than deferring: a cluster whose nodes are running three different kernels for months is harder to reason about than one that reboots on a schedule. If a new kernel misbehaves, the previous one is still installed and selectable at boot, so pin the working version and investigate on a node you have drained again rather than in a hurry.

Bring It Back and Verify

Do not disable maintenance mode until the node has genuinely rejoined everything, which takes longer than the boot.

pvecm status                      # node present, quorum intact
ceph -s                           # all OSDs up and in, HEALTH_OK
ceph osd stat
ha-manager crm-command node-maintenance disable pve-02

Then compare the running guest count against the number you noted, check that no Ceph flags are still set, and confirm the node is actually accepting workloads rather than sitting quorate but empty. Only after that does the next node's turn begin.

The Mistakes That Turn Maintenance Into an Incident

Major version upgrades deserve their own treatment. Read the upgrade notes, upgrade one node, and run mixed for long enough to be confident before continuing. Proxmox supports mixed versions during an upgrade window precisely so you do not have to commit the whole cluster in one move.

What This Does Not Protect You From

A careful maintenance sequence protects against the failures maintenance causes. It does nothing about the ones it reveals.

Take a backup before starting, and know that it restores. A node that does not come back from a reboot is an uncommon but real outcome, particularly after a firmware or kernel change, and at that point the question is whether the guests can be recovered elsewhere rather than whether the node can be fixed. Our guide to Proxmox Backup Server covers the verification and restore-testing side.

Also budget more time than the steps suggest. The commands take minutes; waiting for Ceph to report clean, and for a drained node to genuinely rejoin, takes as long as it takes. Maintenance windows that assume the happy path are how people end up unsetting flags in a hurry.

Or Let Somebody Else Hold the Window

None of this is difficult. It is exacting, it happens on evenings and weekends, and the cost of one skipped verification is disproportionate to the effort of doing it.

MassiveGRID runs this sequence on its own platform, where Proxmox high-availability clustering with automatic failover sits over Ceph storage replicating every block three times across independent NVMe drives, backed by a 100% uptime SLA. Proxmox support covers clusters you own, from $99 per node per month up to $449 and custom tiers, including patch windows and the upgrade planning. NOC services provide the monitoring that tells you a node did not come back before a customer does.

Infrastructure can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.