Kubernetes on Proxmox: Nodes, Networking and Storage

Proxmox restarts a failed virtual machine and Kubernetes reschedules a failed pod, and both are right. Configure them without considering the interaction and one hardware fault triggers two recoveries that take longer than either layer alone. This guide covers the VM template settings that matter, where the control planes belong, which layer should own which failure, and why a Proxmox snapshot is not a backup of a cluster.

Worth stating plainly first: this is not the same as Kubernetes managing virtual machines, and it is not a nested-virtualisation trick. Proxmox provides the compute the cluster runs on, the same way a cloud provider would, and the cluster neither knows nor cares. What you gain is everything a hypervisor gives you underneath a distributed system, and what you take on is one more layer with opinions about failure.

MassiveGRID runs Proxmox with Ceph in production. Placement in any of 85+ metros across 30+ countries, with auto-provisioning in New York, London, Frankfurt and Singapore.

Proxmox support — from $99/node/month · Managed Kubernetes from about $25.37/mo
HA Private Cloud · Colocation

Why Run Kubernetes on Proxmox at All

Because you get to keep the layer underneath. Bare-metal Kubernetes is more efficient and gives you nothing to snapshot, no live migration, and no way to run anything that is not a container on the same hardware.

Proxmox underneath means node rebuilds are template clones, an upgrade can be snapshotted before it starts, and the database that refuses to be containerised runs as a VM on the same cluster. The cost is a virtualisation layer's overhead, which for most workloads is a few percent and worth paying.

The Node Template

Build one template and clone it. Doing this by hand per node guarantees drift, and drift in a Kubernetes cluster surfaces as one node behaving differently under load.

Settings that matter on the VM itself:

Two of these are worth dwelling on. Disable ballooning: Kubernetes calculates allocatable memory once and does not expect it to move, so a balloon reclaiming memory produces evictions that make no sense against the numbers you can see. And leave disk cache on none for control plane nodes, because etcd fsyncs every write and a writeback cache can acknowledge a write the hardware has not committed.

Inside the guest, prepare it once: swap off, overlay and br_netfilter loaded, the bridge and forwarding sysctls set, containerd installed with the systemd cgroup driver, and qemu-guest-agent running. Then convert to a template.

qm clone 9000 111 --name k8s-cp1 --full
qm set 111 --ipconfig0 ip=10.0.0.11/24,gw=10.0.0.1
qm start 111

Regenerate the machine ID after cloning, or the nodes will share one and the kubelet will report duplicate identities:

truncate -s 0 /etc/machine-id
systemd-machine-id-setup

Placing the Control Plane

Three control plane VMs on one Proxmox host is a cluster that survives a VM failure and not a host failure, which is the failure that actually happens. Spread them across three Proxmox nodes.

Proxmox has no true anti-affinity rule, so this is a placement decision you make and then protect. HA groups with a priority list keep each control plane VM preferring its own host:

ha-manager groupadd k8s-cp1-pref --nodes "pve1:3,pve2:2,pve3:1"
ha-manager add vm:111 --group k8s-cp1-pref --max_relocate 2
ha-manager status

Do the same for the other two with their priorities inverted. Without this, a maintenance window that migrates VMs can quietly land two control planes on one host, and nothing will tell you until that host fails.

The Two Recovery Layers

Here is the interaction to get right. When a Proxmox node fails, Proxmox HA waits out the watchdog interval and restarts its VMs elsewhere, typically a minute or two. Meanwhile Kubernetes marks the missing node NotReady after its own timeout and starts rescheduling pods.

Both happen, and that is fine. What matters is not fighting them:

Let Kubernetes own the pods. Do not shorten Proxmox HA timers hoping for faster pod recovery. Kubernetes reschedules faster than a VM can boot, so pod recovery is already handled before the VM returns.

Let Proxmox own the nodes. Proxmox HA restores cluster capacity, which is what you actually want from it. Set nofailback=1 on the HA groups so a rebooted host does not immediately pull its VM back, because a host that just rebooted is the host most likely to reboot again.

Do not put workers under Proxmox HA at all, if the cluster has spare capacity. A failed worker's pods move in seconds; restarting the VM adds nothing except a machine rejoining later. Reserve Proxmox HA for the control planes, where losing quorum is the real risk.

Storage

Three layers can hold persistent volumes and they behave differently.

If Proxmox already runs Ceph, expose it to Kubernetes with a CSI driver rather than building Rook on top. You would be running distributed storage over distributed storage, tripling replication for no gain, and the write amplification is real.

Point the CSI driver at the Ceph monitors with a dedicated pool and a restricted key rather than the admin key:

pveceph pool create k8s-rbd --size 3 --min_size 2 --pg_autoscale_mode on
ceph auth get-or-create client.k8s \
  mon 'profile rbd' \
  osd 'profile rbd pool=k8s-rbd' \
  mgr 'profile rbd pool=k8s-rbd'

Keep min_size at 2. Setting it to 1 permits writes with a single surviving copy, which is how a disk failure becomes data loss rather than a non-event. Our Proxmox Ceph guide covers the rest of that layer.

Networking and LoadBalancer Services

A Linux bridge on the Proxmox hosts is all the cluster needs, VLAN-aware if you are separating traffic. Keep Kubernetes node traffic off the same interface as Ceph replication and Proxmox corosync, because Ceph recovery saturating a shared link will delay corosync and fence a healthy host.

The part with no default answer is LoadBalancer services. There is no cloud provider to allocate an address, so those services stay Pending forever unless you provide an implementation. MetalLB in layer 2 mode is the usual choice: give it a pool of unused addresses on the node network and it answers ARP for them.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lan-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.0.0.200-10.0.0.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: lan
  namespace: metallb-system
spec:
  ipAddressPools:
    - lan-pool

Reserve that range outside your DHCP scope. An address MetalLB hands out that DHCP also assigns produces an intermittent fault that is genuinely unpleasant to trace.

Snapshots Are Not Backups of a Cluster

Proxmox snapshots are the best thing about this architecture and the easiest to misuse. Snapshotting a control plane VM before an upgrade is excellent. Rolling one back on its own is not, because etcd members will have moved on and the restored member returns with stale state that it then tries to reconcile.

Two rules. Snapshot all three control planes together, at the same point, if you intend to roll back. And keep etcd snapshots separately, because that is what actually restores cluster state:

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key

Persistent volume data needs its own treatment again. A Proxmox backup of the VMs does not capture Ceph RBD volumes that Kubernetes provisioned, because those are not VM disks. Use a Kubernetes-aware tool for those, or accept that your cluster backup covers configuration and not data.

Upgrades Become Boring

This is where the virtualisation layer pays for itself. Snapshot the node, drain it, upgrade it, uncheck it, and if anything is wrong roll the snapshot back rather than debugging a half-upgraded kubelet:

kubectl drain k8s-w1 --ignore-daemonsets --delete-emptydir-data
qm snapshot 121 pre-upgrade
# upgrade kubeadm, kubelet, kubectl on the node
kubectl uncordon k8s-w1

On bare metal the equivalent recovery is a reinstall. That difference is the practical case for running Kubernetes on Proxmox rather than under it.

A Cluster Underneath That Is Already Built

Everything above assumes a healthy Proxmox cluster: corosync on its own low-jitter network, Ceph on enterprise NVMe with power-loss protection, watchdog fencing tested, and three nodes rather than two. Those are the areas where a mistake costs data rather than time, and they are unrelated to Kubernetes.

MassiveGRID has run Proxmox with Ceph in production for years, so that layer arrives built, tuned and monitored, with automatic failover and three-way replicated NVMe underneath. For clusters you own, Proxmox support starts at $99 per node per month, with Ceph and HA management at $249 and a four-hour critical response SLA.

If Kubernetes itself is the part you would rather not operate, managed Kubernetes gives you an HA control plane, ingress with TLS termination, autoscaling and dynamic persistent volumes already configured, billed on the RAM and CPU your pods use rather than on server size, from about $25.37 a month. The Proxmox design is covered in our HA cluster guide, and the Kubernetes side in our three-node control plane guide.