Skip to main content

2 posts tagged with "Migration"

View All Tags

Stable Release Version v3.1.0

· 11 min read

Version v3.1.0 is the Proxmox release. The platform now speaks Proxmox VE natively: point it at any existing PVE 8 or 9 node or cluster with a single API token, and every node is discovered, linked, and managed from the same panel, API, and billing pipeline as your KVM fleet. There is nothing to install inside Proxmox and no agent to maintain - the driver works entirely over the Proxmox REST API, with one optional one-command bootstrap on a node for the features that need host access (VPC WebSSH and exact per-NIC metering). Nothing changes for existing KVM hypervisors, instances, plans, or balances.

  • [Feature] Native Proxmox VE driver - full instance lifecycle on PVE 8.0+ and 9.x: deploy, power, suspend and resume, reinstall, resize, destroy, ISO mounting, cloud-init, GPU passthrough, VM tags, and per-tenant resource pools. Agentless by design; see the section below.
  • [Feature] One-token cluster onboarding - give the panel one reachable member IP and one API token, and it discovers every node in the cluster, links them all, verifies SDN and FRR prerequisites, provisions console credentials, and self-checks the token's privileges. Failures are listed per node instead of aborting the lot.
  • [Feature] VPC networking on Proxmox SDN - VPCs are realized as PVE SDN EVPN zones, vnets, and subnets, with NAT gateways, hot attach and detach of VPC NICs, and automatic reconciliation. Security groups translate to the PVE firewall with accept and drop rules, IP sets, and per-VM rulesets.
  • [Feature] Backups and snapshots - vzdump backup, restore, and delete per instance, optional live-restore, native PVE scheduled backup jobs with full retention control, file-level restore (browse a backup and download individual files), and VM snapshots with optional RAM state, rollback, and delete.
  • [Feature] Live migration - move a running VM between cluster nodes from the panel, with an eligibility precheck, local-disk handling, target storage selection, bandwidth limits, and a dedicated migration network option.
  • [Feature] High availability via the PVE CRM - instances enroll into Proxmox's own HA stack on deploy, with per-group tuning and placement rules; the panel mirrors CRM state instead of running a second watchdog.
  • [Feature] Full guest-service parity - Docker deployments, managed databases (including S3 backups, point-in-time recovery, and replication), load balancers with Let's Encrypt, Kubernetes clusters, WebSSH, SSH key injection, and password resets all work on Proxmox instances through the QEMU guest agent.
  • [Feature] Exact bandwidth metering - per-NIC counters are read from the host, so VPC (east-west) traffic and public traffic are billed separately and never double-counted, with reset-safe accumulation across reboots.
  • [Feature] Node Issues dashboard - operational failures on a Proxmox node (metering, host access, proxy installs, firewall gaps) surface as first-class admin issues that reopen if they recur, instead of hiding in a log file.
  • [Improvement] Fleet-scale collection - metrics, statistics, security-group reconciliation, and HA reconciliation all fan out per node onto a worker pool, so a large cluster is collected in parallel rather than serially.

Native Proxmox VE support

v3.1.0 introduces a second hypervisor backend alongside KVM. A hypervisor group is now either a KVM group or a Proxmox cluster, and both kinds run side by side on one panel with the same instances, plans, billing, and user experience.

The design principle is API-only: the driver talks to the Proxmox REST API and nothing else. Your cluster keeps looking like a normal Proxmox cluster - VMs created by the panel are plain QEMU VMs you can see in the PVE web UI, tasks the panel starts are ordinary PVE tasks (UPIDs the panel tracks and can cancel), and nothing on the node is patched or replaced.

Onboarding an existing cluster

Create a hypervisor group of type Proxmox, paste one member's address and one API token, and the panel calls the cluster status endpoint to discover every node. Each node is linked individually, storage is discovered per node with its real PVE plugin type, an @pve console credential is provisioned for VNC, and SDN plus FRR prerequisites are verified so VPC problems are caught at link time rather than at first deploy. The token itself is self-checked: the panel reports the PVE version and the privileges the token actually holds, and flags the guest-agent privilege gap that PVE 9 introduced, right on the node's detail page.

Supported: Proxmox VE 8.0 and later, including 9.x, standalone nodes and clusters. Group names must match the PVE cluster name so the group is an unambiguous mirror of the datacenter, and groups are kept homogeneous - one group is either KVM or Proxmox, never a mix.

Images and deploys

Operating system images are built once per cluster into a vzdump archive, then every deploy restores that archive into the customer's VM ID. There are no template VMs parked on your nodes and no full-clone storms: what lands is a plain VM, on the storage you chose, with cloud-init applied. Archives are labeled with the upstream image name, can be purged with one command, and image builds are excluded from orphan detection.

Cloud-init works two ways: the default NoCloud seed ISO (full feature set, including multi-user and multi-IP layouts), or PVE's native cloud-init as a per-group opt-in for simple single-user, single-IP instances where a config drive is unnecessary.

Existing VMs on the cluster are not stranded either: the orphan importer can adopt any VM the panel did not create - including raw, ZFS, LVM, and Ceph-backed disks - and attach it to a user as a managed instance.

Storage

All common PVE storage backends are supported and detected with their real capabilities: directory, NFS, LVM, LVM-thin, ZFS, and Ceph RBD, including onboarding an external Ceph cluster directly from the panel. Extra volumes allocate through the PVE storage API with the correct format per backend, hot-plug with IO limits, and resize live (grow-only, as QEMU requires). Capability gates are honest: operations a given backend cannot do, such as snapshots on plain LVM, are refused with a clear message instead of failing halfway.

Networking, VPCs, and security groups

VPCs on a Proxmox group are built on PVE SDN: an EVPN controller and zone per VPC with a dedicated VRF VNI, a vnet and subnet per VPC subnet, and a NAT gateway with SNAT semantics that hold cluster-wide. VPC NICs hot attach and detach on running VMs. A reconciler keeps the SDN objects healthy and converges NAT gateway activation, and VPC create and destroy push eagerly so the fabric follows the panel immediately.

Security groups compile to the PVE firewall: cluster-level groups and IP sets, per-VM rulesets with both accept and drop actions, anti-spoofing IP filters, and firewall flags only on the NICs they belong to. Reconciliation is fingerprint-based, so unchanged nodes are skipped, and a cron backstop repairs drift. The datacenter firewall is enabled safely, preserving management access rather than default-dropping the hosts.

The instance network tab also gains guest IP discovery on Proxmox: addresses are read live from the QEMU guest agent, filtered of loopback and link-local noise, for both admins and users.

Consoles and WebSSH

Three console paths ship for Proxmox instances:

  • VNC (noVNC) through a PVE login ticket with single-use, server-side session tokens. PVE credentials and tickets never reach the browser.
  • Serial terminal (xterm.js over termproxy), tenant-scoped.
  • WebSSH - a real SSH shell in the browser. Public-IP guests are dialed directly. VPC-only guests are reached through proxmox-ssh-proxy, a small node-side service that enters the VPC's VRF on the host, because the guest's route only exists there.

The proxy is the one place host access is needed, and it bootstraps with one command on one node: the panel mints a short-lived, single-use install command, the script fetches the binary from your panel (never a third-party artifact), installs the service, opens the master's port in the PVE firewall using the egress address the node itself observed, and the panel keeps every node's proxy current automatically from then on. Install state is verified against what is actually on the node, so a half-finished install shows up as fixable instead of hiding.

Backups, scheduled jobs, and file restore

Per-instance backups run through vzdump with completion tracked to the panel's backup queue, restore (optionally live-restore, so the VM boots while data streams back), redirect-to-storage on restore, and delete. Cluster-level scheduled backup jobs are managed from the panel with the full keep-last, keep-hourly, keep-daily, keep-weekly, keep-monthly, and keep-yearly retention set, targeting specific VMs, a resource pool, or the whole cluster. With Proxmox Backup Server storage, file-level restore lets admins and users browse a backup's filesystem and download individual files without restoring the VM.

Snapshots and Forge

VM snapshots are first-class on capable storage: create with or without RAM state, list, rollback, and delete, from both admin and user instance pages. Forge (the checkpoint-try-rollback workflow) rides the same mechanism with a reserved snapshot slot, protected from name collisions with user snapshots.

Live migration

Admins can live-migrate a Proxmox instance between cluster nodes with a precheck that refuses ineligible targets and local hardware blockers, handles local disks, and passes through target storage, bandwidth limit, migration network, and online mode. On PVE 9, conntrack-state migration is available as an opt-in. Migration success updates the instance's node assignment and flushes routing caches; a failed migration leaves the instance exactly where it was.

High availability

Proxmox groups delegate HA to the PVE cluster resource manager - the layer that actually owns fencing and recovery - instead of duplicating it. Instances enroll on deploy and unenroll on destroy, groups expose HA tuning (and node-affinity placement, using HA groups on PVE 8 and node-affinity rules on PVE 9), and a reconciler mirrors CRM state back into the panel so the instance page shows the real HA state. The KVM HA watchdog is untouched, and a Proxmox-side failure can never stall KVM monitoring.

Guest services: Docker, databases, load balancers, Kubernetes

Everything that previously required the slave agent's SSH path now runs over the QEMU guest agent on Proxmox instances:

  • Docker: deploy, control, status, and logs.
  • Managed databases: the full suite - configure, restart, password reset, backup tooling, full and incremental backups to S3, point-in-time recovery, restore, upgrade, replication, and batched health checks.
  • Load balancers: configuration, Let's Encrypt issue and renew, and telemetry setup.
  • Kubernetes: kubectl execution, control-plane certificate upload and renewal, and admin kubeconfig reissue, routed through the Proxmox adapter for Proxmox-hosted clusters.
  • Instance basics: SSH key inject and remove with port detection, password resets, and WireGuard configuration sync.

Large payloads are delivered safely: small files write through the agent directly, larger ones are pulled by the guest from a single-use, checksum-verified URL.

Metering, billing, and scale

Metrics and statistics feed the same billing pipeline as KVM, with the units and counters aligned so plan enforcement and charts behave identically. Two things are worth calling out:

  • Exact per-NIC bandwidth. The PVE API only exposes aggregate VM counters, which cannot split public from VPC traffic. v3.1.0 reads per-NIC tap counters from the host instead, so VPC traffic and public traffic are metered separately and never double-billed, with reset-safe accumulation that keeps mid-day reboots billable.
  • Parallel collection. Metrics, instance statistics, HA reconciliation, and security-group reconciliation fan out per node and per group onto a dedicated worker pool. Collection time scales with your largest node, not your node count.

Operational visibility

A new Node Issues dashboard (admin, under Proxmox) surfaces operational failures on a node as structured issues: what failed, on which node, how many times, first and last seen. Issues reopen automatically if the condition recurs after being marked solved, and each issue carries a retry action. Long-running PVE tasks are tracked by UPID and can be cancelled from the panel, and a stuck panel task cancels its PVE counterpart when marked failed.

Security hardening in this release

The Proxmox driver went through dedicated security review during the cycle. Notable fixes shipping in v3.1.0: a WireGuard configuration path that could execute as root in the guest was closed, WebSSH sessions no longer expose PVE authentication cookies, console ACLs are revoked rather than accruing, ISO URL fetching is SSRF-hardened, console routes are tenant-scoped with negative-authorization tests, and backup restore and file-restore endpoints verify the backup belongs to the instance before acting.

Beta Release Version v2.2.6

· 6 min read

Version v2.2.6 makes local storage a first-class, pluggable layer. Until now a local instance disk meant a qcow2 file. This release adds four block-device backends - LVM thin, LVM thick, ZFS thin, and ZFS thick - that slot in alongside qcow2, Ceph, and NFS with the full lifecycle: deploy, resize, snapshots, incremental backups, restores, image creation, and live migration. Around them sit three operator-facing safety features: a guided storage wizard that probes the hypervisor before anything is saved, a collapse watchdog that suspends only the at-risk instances before a thin pool can corrupt, and a live Storage Health dashboard. The Virtualizor importer now lands migrated VMs straight onto LVM/ZFS with no qcow2 conversion step, and the admin panel picked up a round of polish along the way. This is a beta build - feature-complete for the storage work and safe to stage, but not yet promoted to stable.

  • [Feature] LVM and ZFS Storage Backends - Hypervisor storage now supports four new local pool types alongside qcow2 files, Ceph, and NFS: LVM thin, LVM thick, ZFS thin, and ZFS thick. Instances on these pools run on raw block devices for near-native disk performance, with the thin variants providing over-provisioning and the ZFS variants bringing checksumming and inline compression. Plans can target a specific backend, so an operator can sell "NVMe block storage" and "standard storage" tiers from the same hypervisor.
  • [Feature] Guided Storage Wizard - Adding a storage pool is now a five-step wizard that probes the hypervisor live before anything is saved: it verifies the tooling is installed, the volume group or pool exists, and reports real capacity, so a mistyped pool name can no longer create a broken storage row. LVM and ZFS appear in the picker with a Beta tag for this release.
  • [Feature] Storage Collapse Watchdog - Every hypervisor now watches its pools every 30 seconds. When a thin pool or ZFS pool approaches exhaustion, the platform suspends only the affected instances before their disks corrupt, alerts every administrator by mail, and surfaces the event live in the Storage Health dashboard. Recovery is operator-controlled: fix the pool, then resume the instances from where they stopped.
  • [Feature] Storage Health Dashboard - A per-pool health view in the admin panel shows live data and metadata usage, ZFS pool state and fragmentation, and warn/critical thresholds, fed by the same probes the watchdog uses, surfaced inline on the storage list.
  • [Feature] Native LVM/ZFS Virtualizor Import - The Virtualizor importer no longer forces every disk through a qcow2 conversion. When the source VM lives on LVM or ZFS, the slave probes the pool and the importer lands the instance natively on the matching backend, preserving the storage tuple and skipping the conversion pass entirely.
  • [Improvement] Incremental Backups Everywhere - The QEMU dirty-bitmap incremental backup pipeline now covers all seven backends, including raw block devices, writing the same qcow2 backup chains to your existing backup storage. Existing qcow2 and Ceph backup behavior is unchanged.
  • [Improvement] Live Migration for Block Pools - Live migration works between LVM/ZFS pools across hypervisors, pre-creating the destination volume and block-copying storage as part of the move.
  • [Improvement] Admin Panel Polish - The VPN Gateway, Managed Database, and Load Balancer plan create/edit pages were rebuilt on the shared sectioned-card layout used by Instance Plans (and a latent dropdown bug on those pages was fixed), more list pages gained the standard collapsible filter panel, the in-browser SSH terminal now stays connected for as long as the tab is open, and the frontend now builds through Vite.

LVM and ZFS Storage Backends

Until now local instance disks meant qcow2 files. v2.2.6 adds four block-device backends - LVM thin, LVM thick, ZFS thin, and ZFS thick - that slot in alongside the existing qcow2, Ceph, and NFS options with the full feature set: deploy, resize, snapshots (thin pools and ZFS), incremental backups, restores, image creation, and live migration. Because plans can target a specific backend, an operator can offer differentiated storage tiers from a single hypervisor.

Pools are added through the new guided wizard, which talks to the hypervisor before saving: it checks that lvm2 or zfsutils are installed (both are installed when you update the hypervisor from the admin panel), that the volume group, thin pool, or zpool actually exists, and shows live capacity. Existing storage rows, plans, and instances are untouched - qcow2, Ceph, and NFS behave exactly as on 2.2.5, and nothing is migrated or rewritten on upgrade.

The Collapse Watchdog

Thin provisioning's dark side is pool exhaustion: when an over-provisioned pool fills up, every VM on it can corrupt at once. v2.2.6 ships a two-layer defense. First, LVM thin pools are configured to auto-extend at 80% via dmeventd. If that cannot help - the volume group is out of space, or a ZFS pool faults - a watchdog on each hypervisor catches the threshold crossing within 30 seconds, pauses only the instances on the affected pool, mails every administrator, and raises a live alert in the Storage Health dashboard. Paused instances keep their RAM and disk state; once the operator extends the pool or replaces the failed device, the instances resume from where they stopped. Degraded ZFS mirrors raise an early warning while the instances keep running.

Storage Health Dashboard

The Storage Health view in the admin panel shows per-pool data and metadata usage, ZFS pool state and fragmentation, and warn/critical thresholds - fed by the same probes the watchdog uses and broadcast live over WebSocket. In this release the health values are surfaced inline on the storage list, so pool state lives in one place.

Native LVM/ZFS Virtualizor Import

Migrating off Virtualizor no longer means rewriting every disk to qcow2. When a source VM lives on LVM or ZFS, the slave probes the pool and the importer lands the instance natively on the matching backend, preserving the storage tuple and skipping the conversion pass entirely. qcow2 sources continue to import as qcow2.

Known Beta Notes

  • LVM and ZFS appear with a Beta tag in the storage wizard; exercise them on a test hypervisor before offering them to customers.
  • Pool auto-extend depends on free space in the underlying volume group or zpool - the watchdog is the safety net, not a substitute for capacity planning.
  • Found a problem? Report it referencing v2.2.6-beta along with the hypervisor's storage type and the relevant Storage Health alert.