Choosing storage on Proxmox VE:
ZFS, LVM, NFS and Ceph

Two classes: file vs block

File-level (POSIX) storage holds any content: images, ISOs, templates, backups, snippets. ZFS is the most advanced here — snapshots and clones. Block storage holds large raw images; ISOs and backups usually do not live there. Ceph RADOS is distributed block, presented as RBD.

Configuration sits in /etc/pve/storage.cfg and is shared across cluster nodes. Shared storage is physically reachable from every node; local is configured everywhere, but the content differs per host.

Type table (pvesm)

Footnotes from the guide: qcow2 snapshots on NFS can stall a running VM and on large disks take minutes or hours. Types with snapshots also support thin provisioning. If storage fills up, guests get IO errors — watch over-provisioning. Shared in the table means nodes physically see the same image; local is in storage.cfg everywhere, but the content is per host. That distinction decides whether live migration copies memory or the whole volume.

Local ZFS and LVM-thin

ZFS (zfspool): VM images as zvols (block). Native snapshots and clones. Not shared — live-migrating the disk needs --with-local-disks (it copies the volume). Excellent on an NVMe node, on PBS and wherever you want checksums and scrub.

LVM-thin: a local thin pool, snapshots yes, shared no. The PVE installer often creates local-lvm. Thick LVM on a shared LUN (iSCSI/FC) slices the LUN into virtual disks; from PVE 9, snapshots as volume chains.

NFS, iSCSI and Ceph

NFS: shared, flexible content. Snapshots via qcow2 or a chain — with the performance caveat. A good shelf for ISOs and templates; production VM disks only if the array and network can carry them.

iSCSI / FC: LUN + LVM, shared, multipath usually required. Snapshots from PVE 9 as a chain (technology preview in the 9 docs).

Ceph RBD: shared, snapshots, HCI or dedicated OSDs. Comparison with vSAN: vSAN vs Ceph. We leave CephFS for ISOs, backups and CTs, not as the only VM-disk datastore.

Snapshots as volume chains

From Proxmox VE 9, snapshots as volume chains work on block storage that could not snapshot on its own (iSCSI, FC). Layers are separate volumes. The documentation marks this as a technology preview. Internal qcow2 snapshots still stall the VM on create and delete.

How we choose in a design

• 3+ nodes, HCI, live migrate without copying the disk: Ceph RBD.
• The array already exists: NFS or iSCSI/FC + LVM.
• One or two nodes, NVMe, local snapshots: ZFS; HA still needs quorum (QDevice) and a plan for the disk on failover.
• Backups: the pbs plugin, not a directory on the same ZFS as the VMs — see ransomware backup.
• Btrfs: technology preview (best-effort) in the pvesm table. We do not put production on it.

Multipath, overprovision, PBS

The pvesm guide on iSCSI/FC: LVM slices the LUN into virtual disks, and multipath is usually required in a cluster. Without it, path failover looks like “the disk vanished”. Thin provisioning comes with snapshot-capable types — when storage fills, guests get IO errors. We watch overprovision; we do not “leave it for later”.

The pbs plugin is shared and does not hold VM snapshots in the qcow2 sense — it is the backup target. A directory on the same ZFS as VM disks is not 3-2-1. Btrfs in the pvesm table is a technology preview (best-effort) — we do not put production on it. ZFS over iSCSI is in the table as shared with snapshots; that is a separate backend, not ordinary zfspool.

Live migration vs storage type

Shared (NFS, RBD, iSCSI/FC) — live migration copies memory, not the disk. Local ZFS/LVM-thin — --with-local-disks copies the volume over the cluster network; on a large zvol that is minutes. HA after a node dies does not magically move local ZFS to a neighbour. Flag details: live migration. HCI vs vSAN: vSAN vs Ceph.

Content types in pvesm

The pvesm guide assigns each storage a content list: images (VM disks), rootdir (containers), iso, vztmpl, backup, snippets, import. Not every plugin takes every content type: LVM/RBD are mainly block images; ISOs and vzdump go on a directory, NFS or CephFS. PBS has its own plugin and is not an ISO shelf.

In a design we split: VM disks on RBD/ZFS/LVM, ISOs and templates on NFS or CephFS, backups on PBS. Mixing backups with VM disks on the same zfspool breaks 3-2-1, even when ZFS snapshots “look like a copy”.