Proxmox HA: Automatic VM Failover Across Nodes

What you'll build and why

On top of the cluster from the prerequisite guide, this adds automatic recovery: if the node running a VM disappears without warning (power loss, hardware failure, a kernel panic), Proxmox detects it, fences the dead node, and restarts the VM on a surviving node -- typically within about a minute, with nobody touching anything.

Don't build this if: you haven't verified your cluster is actually quorate yet (do that first -- HA on a shaky cluster just gives you a shaky cluster that also occasionally thrashes VMs around). You also don't need this for VMs where a few minutes of downtime genuinely doesn't matter -- HA adds real operational complexity (a second failure mode to understand and troubleshoot) for a benefit that's only worth it when some of what you're running actually needs to survive an unplanned outage unattended. If nothing you run meets that bar yet, plain backups (see Back it up and maintain it, or the CLI reference guide's vzdump section) are the better first investment.

This guide covers HA for VMs whose disks live on ZFS storage replication -- the no-extra-hardware path -- with a note on the NFS alternative. It does not cover backups (a different problem: HA protects against a node dying, not against you or a guest OS destroying data) or scheduled snapshots beyond what replication itself uses internally.

How it works

Two things have to both be true for HA to actually recover a VM: the VM's disk has to be reachable from more than one node, and Proxmox has to be able to fence (definitively stop) the dead node before starting the VM anywhere else -- without fencing, a node that's merely unreachable (not actually dead) could still be running the VM when a second copy starts elsewhere, corrupting shared state.

Storage: this guide uses ZFS storage replication -- each node keeps its own local ZFS copy of the VM's disk, and Proxmox replicates changes to the other nodes on a schedule (as often as once a minute). It's asynchronous, so a failure can lose whatever changed since the last replication run -- acceptable for most homelab workloads, not for something that can't tolerate any write loss. The alternative is a shared NFS target (a 4th machine or NAS everyone mounts the same disk from) -- synchronous by nature, but a single point of failure unless that NFS box is itself redundant.

Fencing: every HA-managed node runs a watchdog timer. Under normal operation, Proxmox's ha-manager resets it continuously; if a node stops responding, the watchdog isn't reset and the node reboots itself after a timeout, guaranteeing it's genuinely not running anything before another node starts the VM elsewhere. A hardware watchdog (a real timer chip, independent of a hung OS) is more reliable than the software fallback (softdog), which depends on the same kernel that might be the thing that's hung.

Node Affinity rules (the current mechanism -- see the note on old terminology below) decide which nodes a resource is allowed to run on, and how strictly. A rule can be:

• positive (default) -- the resource prefers these nodes
• negative -- the resource avoids these nodes
• strict or non-strict (default) -- strict means the resource only ever runs on the listed nodes, stopping if none are available; non-strict means it prefers them but will run anywhere in the cluster if all preferred nodes are down

If you've read an older guide (including an earlier version of this one) that talks about "HA Groups": that's the same idea under Proxmox VE 9's previous name. Groups are deprecated and replaced by Node Affinity rules as of Proxmox VE 9.0 -- the old "restricted" group maps to a strict rule, "unrestricted" maps to non-strict.

   Before a failure:                  After pve3 dies unexpectedly:
┌────────┐ ┌────────┐ ┌────────┐   ┌────────┐ ┌────────┐   pve3
│  pve1  │ │  pve2  │ │  pve3  │   │  pve1  │ │  pve2  │   (fenced,
│        │ │        │ │  VM101 │   │  VM101 │ │        │    self-
│ ZFS    │ │ ZFS    │ │ ZFS    │   │ ZFS    │ │ ZFS    │    rebooted
│ replica│◄┤ replica│◄┤ (live) │   │ (now   │ │ replica│    via
│        │ │        │ │        │   │ live)  │ │        │    watchdog)
└────────┘ └────────┘ └────────┘   └────────┘ └────────┘
Replication: pve3 -> pve1, pve2    ha-manager restarts VM101 on
on a schedule (e.g. every          pve1 using pve1's replicated
15 minutes)                        copy -- current up to the last
successful replication run.

Before you start

Prerequisites: the cluster from the prerequisite guide, quorate and confirmed healthy.

Decision: ZFS replication or a shared NFS target?

Steps

Step 1: Create a ZFS pool on each node (skip if you already have one)

Datacenter → node → Disks → ZFS → Create: ZFS, select a free (non-boot) disk, use the exact same pool name on every node -- replication matches storage by name across nodes, and a mismatch fails silently later.

Step 2: Register the ZFS pool as cluster-wide storage

Datacenter → Storage → Add → ZFS:

• ID: a name for this storage entry, e.g. local-zfs-data
• ZFS Pool: the pool name from Step 1
• Nodes: leave as "All (no restrictions)"

Step 3: Put the VM's disk on that storage

Select the VM → Hardware → its disk → Move Storage → the local-zfs-data storage, format raw. (For a new VM, just pick this storage when creating its disk.)

Step 4: Configure replication

VM → Replication → Add:

• Target: another cluster node (add a second entry for a third node, for resilience against two failures)
• Schedule: */15 (every 15 minutes) is a reasonable default -- the format is a subset of systemd calendar events, minimum interval one minute, maximum once a week

Expected output: the first run does a full copy (can take a while depending on disk size); after that, only changed blocks transfer. Replication traffic uses the cluster's configured migration network by default, or the management network if none is set -- if you want replication on its own dedicated link, configure that in Datacenter → Options, not per-job.

Step 5: Create a Node Affinity rule

From a shell on any node:

$ ha-manager rules add node-affinity my-vms --resources vm:101 --nodes pve1,pve2,pve3

This creates a non-strict (default), positive (default) rule named my-vms that prefers all three nodes for VM 101 -- effectively "run anywhere in the cluster, no restriction," the common case for a homelab. For a rule that keeps a resource off one specific node (say, one you want reserved for other work):

$ ha-manager rules add node-affinity keep-off-pve3 --resources vm:101 --affinity negative --nodes pve3

The web UI equivalent lives under Datacenter → HA → Rules; this guide uses the CLI because the exact rule syntax is easier to state precisely and copy-paste correctly than to describe as a sequence of clicks.

Step 6: Add the VM as an HA-managed resource

$ ha-manager add vm:101
$ ha-manager set vm:101 --state started

Expected output: ha-manager status (or Datacenter → HA → Resources in the web UI) lists vm:101 with state started, associated with the rule from Step 5.

Verify it works

• Datacenter → HA → Resources shows vm:101 with state started
• The VM's Replication tab shows a recent successful sync, not an error
• Failure test: power off the node the VM is currently running on (pull the power -- you're testing an unplanned loss, not a graceful shutdown, which HA handles differently). Watch Datacenter → HA → Status on a surviving node. Expected: within roughly a minute, the downed node is fenced and the VM starts on a surviving node, using that node's most recent replicated copy -- confirm it boots and reflects data current as of the last successful replication run, not necessarily the very latest write
• When the downed node reboots and rejoins, it does not also try to start its own stale local copy of the VM (fencing prevents this by design)

Secure it

• What's exposed: nothing new beyond the base cluster -- HA doesn't open any additional network-facing service. The watchdog and fencing mechanisms are local to each node.
• Default credentials: none specific to HA; see the cluster guide's Secure it section for the shared root-password/cluster-join concerns that still apply.
• Least privilege: anyone with permission to change HA resource state (ha-manager set, or the equivalent UI permission) can stop a production VM cluster-wide -- scope that permission the same way you'd scope any other destructive admin action, if more than one person has access to this cluster.
• Update strategy: the same Proxmox VE security-advisory channel as the base cluster (proxmox.com/en/security); update one node at a time, and be aware that HA can migrate a VM mid-upgrade if you take a node offline for patching without first setting its resources to ignored or migrating them off deliberately.
• The failure that hurts most: believing HA is a backup strategy. It isn't -- it protects against a node dying, not against a bad rm -rf, a botched update inside the guest, or ransomware. Losing sight of that distinction is the single most common way people discover, too late, that they had no real data protection.

Back it up and maintain it

What matters: the HA rule and resource configuration lives in the same replicated cluster filesystem (/etc/pve) as the rest of cluster config -- no separate backup needed for that. The VM data itself still needs a real backup path (Proxmox Backup Server or scheduled vzdump, covered in the CLI reference guide) to storage outside this cluster -- replication protects against losing a node, not against losing the whole cluster, deleting a VM by mistake, or guest-level corruption.

Update cadence: monthly at minimum, same as the base cluster, one node at a time.

What to monitor: each VM's Replication status for failed jobs (a silently-failing replication job means your "protected" VM is actually running on stale or no redundancy), and ha-manager status after any maintenance to confirm every resource is back to its expected state and node.

Troubleshooting

Logs: journalctl -u pve-ha-crm -n 100 (on whichever node is the current HA manager) and journalctl -u pve-ha-lrm -n 100 (on the node running the resource) are the first places to look.

Undo

To remove HA management from a VM without deleting the VM itself: ha-manager remove vm:101 (or Datacenter → HA → Resources, select it, Remove). To delete a Node Affinity rule: ha-manager rules remove <rule-name>. To stop replicating a VM's disk: remove the job from its Replication tab -- this does not delete the already-replicated copies on target nodes, which you can separately delete from each node's ZFS storage if you want the space back.

Go further

• Proxmox VE Command-Line Reference -- vzdump/qmrestore for the real backup path this guide deliberately doesn't cover
• Set HA restart priorities (a numeric priority on each resource) so critical services come back before less important ones when several VMs fail over at once
• If you outgrew the ZFS-replication trade-off, revisit the NFS-shared-storage alternative from Before you start
• Configure a dedicated replication network (Datacenter → Options) once replication traffic is heavy enough to want its own link separate from migration/management traffic

Resources

Official documentation:

• High Availability -- Node Affinity rules, fencing, watchdog configuration
• Storage Replication -- ZFS replication scheduling, requirements, limitations
• Proxmox VE Administration Guide

Source and releases:

• Proxmox VE source repositories
• Proxmox VE Security Advisories

Community:

• Proxmox VE Community Forum
• r/Proxmox

Go deeper:

• High Availability wiki page again, specifically its section on migrating from the old Groups syntax, if you have existing HA config from before Proxmox VE 9.0

Related DaemonPress projects:

• Build a 3-Node Proxmox VE Cluster -- the prerequisite this guide builds on
• Proxmox VE Command-Line Reference

---

Last verified: 2026-09-21, checked against official Proxmox VE documentation (see Resources above).

Changelog:

• 2026-09-21 -- Full rewrite: the original's core walkthrough used HA Groups, deprecated and replaced by Node Affinity rules as of Proxmox VE 9.0 -- this was a real factual error in the previous version, not a style update. Replaced Steps 5-6 with the current CLI-based rule/resource workflow, added the old-terminology troubleshooting entry, an ASCII before/after failover diagram, Secure it, Back it up and maintain it, Undo, and a full Resources section. Expanded troubleshooting from 3 to 6 entries. Moved from homelab category to virtualization-containers. Dropped the backup tag (this guide doesn't teach backup, only mentions it as a separate next step); added networking (replication-network configuration is now substantively covered).
• 2026-09-21 (earlier) -- Original version published, using the now-deprecated HA Groups workflow.