One post tagged with "HA"

Version v3.1.0 is the Proxmox release. The platform now speaks Proxmox VE natively: point it at any existing PVE 8 or 9 node or cluster with a single API token, and every node is discovered, linked, and managed from the same panel, API, and billing pipeline as your KVM fleet. There is nothing to install inside Proxmox and no agent to maintain - the driver works entirely over the Proxmox REST API, with one optional one-command bootstrap on a node for the features that need host access (VPC WebSSH and exact per-NIC metering). Nothing changes for existing KVM hypervisors, instances, plans, or balances.

• [Feature] Native Proxmox VE driver - full instance lifecycle on PVE 8.0+ and 9.x: deploy, power, suspend and resume, reinstall, resize, destroy, ISO mounting, cloud-init, GPU passthrough, VM tags, and per-tenant resource pools. Agentless by design; see the section below.
• [Feature] One-token cluster onboarding - give the panel one reachable member IP and one API token, and it discovers every node in the cluster, links them all, verifies SDN and FRR prerequisites, provisions console credentials, and self-checks the token's privileges. Failures are listed per node instead of aborting the lot.
• [Feature] VPC networking on Proxmox SDN - VPCs are realized as PVE SDN EVPN zones, vnets, and subnets, with NAT gateways, hot attach and detach of VPC NICs, and automatic reconciliation. Security groups translate to the PVE firewall with accept and drop rules, IP sets, and per-VM rulesets.
• [Feature] Backups and snapshots - vzdump backup, restore, and delete per instance, optional live-restore, native PVE scheduled backup jobs with full retention control, file-level restore (browse a backup and download individual files), and VM snapshots with optional RAM state, rollback, and delete.
• [Feature] Live migration - move a running VM between cluster nodes from the panel, with an eligibility precheck, local-disk handling, target storage selection, bandwidth limits, and a dedicated migration network option.
• [Feature] High availability via the PVE CRM - instances enroll into Proxmox's own HA stack on deploy, with per-group tuning and placement rules; the panel mirrors CRM state instead of running a second watchdog.
• [Feature] Full guest-service parity - Docker deployments, managed databases (including S3 backups, point-in-time recovery, and replication), load balancers with Let's Encrypt, Kubernetes clusters, WebSSH, SSH key injection, and password resets all work on Proxmox instances through the QEMU guest agent.
• [Feature] Exact bandwidth metering - per-NIC counters are read from the host, so VPC (east-west) traffic and public traffic are billed separately and never double-counted, with reset-safe accumulation across reboots.
• [Feature] Node Issues dashboard - operational failures on a Proxmox node (metering, host access, proxy installs, firewall gaps) surface as first-class admin issues that reopen if they recur, instead of hiding in a log file.
• [Improvement] Fleet-scale collection - metrics, statistics, security-group reconciliation, and HA reconciliation all fan out per node onto a worker pool, so a large cluster is collected in parallel rather than serially.

Native Proxmox VE support​

v3.1.0 introduces a second hypervisor backend alongside KVM. A hypervisor group is now either a KVM group or a Proxmox cluster, and both kinds run side by side on one panel with the same instances, plans, billing, and user experience.

The design principle is API-only: the driver talks to the Proxmox REST API and nothing else. Your cluster keeps looking like a normal Proxmox cluster - VMs created by the panel are plain QEMU VMs you can see in the PVE web UI, tasks the panel starts are ordinary PVE tasks (UPIDs the panel tracks and can cancel), and nothing on the node is patched or replaced.

Onboarding an existing cluster​

Create a hypervisor group of type Proxmox, paste one member's address and one API token, and the panel calls the cluster status endpoint to discover every node. Each node is linked individually, storage is discovered per node with its real PVE plugin type, an @pve console credential is provisioned for VNC, and SDN plus FRR prerequisites are verified so VPC problems are caught at link time rather than at first deploy. The token itself is self-checked: the panel reports the PVE version and the privileges the token actually holds, and flags the guest-agent privilege gap that PVE 9 introduced, right on the node's detail page.

Supported: Proxmox VE 8.0 and later, including 9.x, standalone nodes and clusters. Group names must match the PVE cluster name so the group is an unambiguous mirror of the datacenter, and groups are kept homogeneous - one group is either KVM or Proxmox, never a mix.

Images and deploys​

Operating system images are built once per cluster into a vzdump archive, then every deploy restores that archive into the customer's VM ID. There are no template VMs parked on your nodes and no full-clone storms: what lands is a plain VM, on the storage you chose, with cloud-init applied. Archives are labeled with the upstream image name, can be purged with one command, and image builds are excluded from orphan detection.

Cloud-init works two ways: the default NoCloud seed ISO (full feature set, including multi-user and multi-IP layouts), or PVE's native cloud-init as a per-group opt-in for simple single-user, single-IP instances where a config drive is unnecessary.

Existing VMs on the cluster are not stranded either: the orphan importer can adopt any VM the panel did not create - including raw, ZFS, LVM, and Ceph-backed disks - and attach it to a user as a managed instance.

Storage​

All common PVE storage backends are supported and detected with their real capabilities: directory, NFS, LVM, LVM-thin, ZFS, and Ceph RBD, including onboarding an external Ceph cluster directly from the panel. Extra volumes allocate through the PVE storage API with the correct format per backend, hot-plug with IO limits, and resize live (grow-only, as QEMU requires). Capability gates are honest: operations a given backend cannot do, such as snapshots on plain LVM, are refused with a clear message instead of failing halfway.

Networking, VPCs, and security groups​

VPCs on a Proxmox group are built on PVE SDN: an EVPN controller and zone per VPC with a dedicated VRF VNI, a vnet and subnet per VPC subnet, and a NAT gateway with SNAT semantics that hold cluster-wide. VPC NICs hot attach and detach on running VMs. A reconciler keeps the SDN objects healthy and converges NAT gateway activation, and VPC create and destroy push eagerly so the fabric follows the panel immediately.

Security groups compile to the PVE firewall: cluster-level groups and IP sets, per-VM rulesets with both accept and drop actions, anti-spoofing IP filters, and firewall flags only on the NICs they belong to. Reconciliation is fingerprint-based, so unchanged nodes are skipped, and a cron backstop repairs drift. The datacenter firewall is enabled safely, preserving management access rather than default-dropping the hosts.

The instance network tab also gains guest IP discovery on Proxmox: addresses are read live from the QEMU guest agent, filtered of loopback and link-local noise, for both admins and users.

Consoles and WebSSH​

Three console paths ship for Proxmox instances:

• VNC (noVNC) through a PVE login ticket with single-use, server-side session tokens. PVE credentials and tickets never reach the browser.
• Serial terminal (xterm.js over termproxy), tenant-scoped.
• WebSSH - a real SSH shell in the browser. Public-IP guests are dialed directly. VPC-only guests are reached through proxmox-ssh-proxy, a small node-side service that enters the VPC's VRF on the host, because the guest's route only exists there.

The proxy is the one place host access is needed, and it bootstraps with one command on one node: the panel mints a short-lived, single-use install command, the script fetches the binary from your panel (never a third-party artifact), installs the service, opens the master's port in the PVE firewall using the egress address the node itself observed, and the panel keeps every node's proxy current automatically from then on. Install state is verified against what is actually on the node, so a half-finished install shows up as fixable instead of hiding.

Backups, scheduled jobs, and file restore​

Per-instance backups run through vzdump with completion tracked to the panel's backup queue, restore (optionally live-restore, so the VM boots while data streams back), redirect-to-storage on restore, and delete. Cluster-level scheduled backup jobs are managed from the panel with the full keep-last, keep-hourly, keep-daily, keep-weekly, keep-monthly, and keep-yearly retention set, targeting specific VMs, a resource pool, or the whole cluster. With Proxmox Backup Server storage, file-level restore lets admins and users browse a backup's filesystem and download individual files without restoring the VM.

Snapshots and Forge​

VM snapshots are first-class on capable storage: create with or without RAM state, list, rollback, and delete, from both admin and user instance pages. Forge (the checkpoint-try-rollback workflow) rides the same mechanism with a reserved snapshot slot, protected from name collisions with user snapshots.

Live migration​

Admins can live-migrate a Proxmox instance between cluster nodes with a precheck that refuses ineligible targets and local hardware blockers, handles local disks, and passes through target storage, bandwidth limit, migration network, and online mode. On PVE 9, conntrack-state migration is available as an opt-in. Migration success updates the instance's node assignment and flushes routing caches; a failed migration leaves the instance exactly where it was.

High availability​

Proxmox groups delegate HA to the PVE cluster resource manager - the layer that actually owns fencing and recovery - instead of duplicating it. Instances enroll on deploy and unenroll on destroy, groups expose HA tuning (and node-affinity placement, using HA groups on PVE 8 and node-affinity rules on PVE 9), and a reconciler mirrors CRM state back into the panel so the instance page shows the real HA state. The KVM HA watchdog is untouched, and a Proxmox-side failure can never stall KVM monitoring.

Guest services: Docker, databases, load balancers, Kubernetes​

Everything that previously required the slave agent's SSH path now runs over the QEMU guest agent on Proxmox instances:

• Docker: deploy, control, status, and logs.
• Managed databases: the full suite - configure, restart, password reset, backup tooling, full and incremental backups to S3, point-in-time recovery, restore, upgrade, replication, and batched health checks.
• Load balancers: configuration, Let's Encrypt issue and renew, and telemetry setup.
• Kubernetes: kubectl execution, control-plane certificate upload and renewal, and admin kubeconfig reissue, routed through the Proxmox adapter for Proxmox-hosted clusters.
• Instance basics: SSH key inject and remove with port detection, password resets, and WireGuard configuration sync.

Large payloads are delivered safely: small files write through the agent directly, larger ones are pulled by the guest from a single-use, checksum-verified URL.

Metering, billing, and scale​

Metrics and statistics feed the same billing pipeline as KVM, with the units and counters aligned so plan enforcement and charts behave identically. Two things are worth calling out:

• Exact per-NIC bandwidth. The PVE API only exposes aggregate VM counters, which cannot split public from VPC traffic. v3.1.0 reads per-NIC tap counters from the host instead, so VPC traffic and public traffic are metered separately and never double-billed, with reset-safe accumulation that keeps mid-day reboots billable.
• Parallel collection. Metrics, instance statistics, HA reconciliation, and security-group reconciliation fan out per node and per group onto a dedicated worker pool. Collection time scales with your largest node, not your node count.

Operational visibility​

A new Node Issues dashboard (admin, under Proxmox) surfaces operational failures on a node as structured issues: what failed, on which node, how many times, first and last seen. Issues reopen automatically if the condition recurs after being marked solved, and each issue carries a retry action. Long-running PVE tasks are tracked by UPID and can be cancelled from the panel, and a stuck panel task cancels its PVE counterpart when marked failed.

Security hardening in this release​

The Proxmox driver went through dedicated security review during the cycle. Notable fixes shipping in v3.1.0: a WireGuard configuration path that could execute as root in the guest was closed, WebSSH sessions no longer expose PVE authentication cookies, console ACLs are revoked rather than accruing, ISO URL fetching is SSRF-hardened, console routes are tenant-scoped with negative-authorization tests, and backup restore and file-restore endpoints verify the backup belongs to the instance before acting.

Upgrade notes​

• Standard update path: back up, update the panel, run migrations. New tables and columns for Proxmox (instances' internal IDs and tags, SDN bookkeeping, backup jobs, tap-counter snapshots, node issues) are created by the migrations and do not touch existing rows.
• Queue workers: Proxmox background work runs on a dedicated proxmox queue. Ensure your supervisor configuration includes a worker for it (the shipped supervisor template does).
• KVM-only installations are unaffected. No behavior changes unless a Proxmox group is created.
• To try it: create a hypervisor group of type Proxmox, paste a member IP and an API token with the documented privileges, and deploy. The optional node bootstrap command (for VPC WebSSH and exact VPC metering) is shown on the node's page after linking.