8 posts tagged with "Backups"

Version v3.1.6 is a large release across Kubernetes, networking and databases. The headline is node-pool plan rotation: changing a pool's instance plan used to be rejected outright while the pool had live workers, so the only way through was to scale to zero and back, a full capacity outage for that pool. Now the plan change is accepted and the platform rotates the workers for you, provisioning replacements before draining anything. Alongside it, System DNS turns provisioning into something that produces real hostnames rather than bare IP addresses, and managed databases on those names present publicly trusted TLS certificates that renew themselves. Load balancers gain host-based routing and can serve internal VIPs for Kubernetes Services. The release also carries a broad set of reliability fixes across Kubernetes operation locking, reverse DNS, backups and the admin panel.

Kubernetes​

• [Feature] Node-Pool Plan Rotation - A pool's instance plan can now be changed while it has live workers. Saving the change starts a surge rotation: new workers on the new plan are provisioned and confirmed ready first, then old ones are cordoned, drained through an escalating ladder, and destroyed, one wave at a time, so the pool never runs below capacity. Scale-down ordering is preserved across the rotation, so the pool does not scramble which node leaves next. Downsizing to a smaller plan is allowed with an explicit warning rather than blocked - it is your cluster - but it takes a second, separate confirmation naming the consequence, so a single click can never start one.
• [Feature] Rotation Visibility - The pool list shows a drift badge counting how many workers are still on the previous plan, with a Rotate action to start or resume a rotation. The decision data arrives with the page rather than from a probe request, so opening the page cannot consume the rate limit that governs the rotation endpoint itself.
• [Feature] Worker Pool Identity - Worker nodes now carry a hypervisor.io/node-pool label, following the same convention as the major managed Kubernetes providers, so kubectl get nodes -L hypervisor.io/node-pool shows which pool each node belongs to. Panel hostnames include the pool name too, so the instance lists in both panels no longer show several pools' worth of identically-shaped names. Renaming a pool heals the label on the next reconciliation. None of this costs anything in the database.
• [Feature] Internal Load Balancers for Services - A Kubernetes Service of type LoadBalancer annotated <prefix>internal: "true" now gets a private VIP inside the VPC instead of a public IP, and the cloud controller reports that private address back to the cluster. Internal services no longer have to be exposed publicly to be reachable, and no change to the cloud controller was required.
• [Improvement] One Transport for Long Orchestrations - Rolling upgrades, control-plane upgrades and plan rotations no longer run as queue jobs with a fixed ceiling. They run as detached console processes with a task row, an operation lock and a stale-task reconciler behind them, so a long rotation cannot be killed part-way by a queue timeout.
• [Fix] Kubernetes Operation Locking - Operation locks now correctly detect an already-held lock across the Redis client the platform ships, so concurrent scale, upgrade and rotation operations on one cluster are properly serialised. Lock acquisition also moved outside the surrounding database transactions, so a rolled-back operation releases its lock immediately instead of waiting for the timeout.

DNS​

• [Feature] System DNS - Every public instance, load balancer, managed database and Kubernetes control plane now receives a hostname automatically. Instances are named from their IP in the familiar cloud style (vm-203-0-113-7.cloud1.example.com); named resources use their own name (db-prod-mysql, lb-frontend, k8s-myapp-cp). Records are created at provision time, follow the resource if its IP changes, and are removed when it is destroyed. Where a reverse zone allows it, a matching PTR record is set so forward and reverse agree, and a value the customer set themselves is never overwritten. DNS problems never block or fail provisioning.
• [Feature] Delegation Wizard - Base domains are onboarded through a four-step wizard: enter the domain and nameservers, delegate at your registrar, then verify. Verification is real rather than advisory - the panel writes a random record into the zone and confirms both that your registrar delegates the domain and that public resolvers can see that record through the delegation, before the domain publishes anything. A daily re-check moves a domain that loses its delegation to a degraded state, where existing records keep resolving but no new resources are assigned to it, and restores it automatically when the delegation returns. A problem local to the panel, such as its own DNS tooling being unavailable, will never degrade your domains.
• [Feature] Trusted TLS for Managed Databases - A base domain can hold a Let's Encrypt wildcard certificate covering every name under it, issued over the DNS-01 challenge using the zone the panel already controls. Public managed databases receive it automatically and load it without a restart, so customers can connect with --ssl-mode=VERIFY_IDENTITY or sslmode=verify-full against the public trust store instead of trusting a self-signed certificate. One wildcard per domain keeps issuance well inside Let's Encrypt's rate limits, and renewal runs daily from 30 days before expiry, so a failed attempt has a month of retries behind it rather than being a countdown.
• [Feature] PowerDNS Deployment Kit - A self-contained kit ships with the release for operators who need authoritative DNS to point System DNS and reverse DNS at. It stands up a primary and any number of secondaries replicated by signed zone transfers, where a new zone propagates to every secondary without touching them. One script drives the whole fleet over SSH from the primary: it prints a readiness table per node and changes nothing until every node passes, then converges each node and verifies replication by querying each one directly, failing loudly if any node is not actually serving the zone.
• [Improvement] Reverse DNS Zone Forms - The reverse zone forms now show the hostname each automatic PTR format actually produces, built live from the prefix and domain you are typing, instead of four opaque option labels. Zone type is chosen from cards and the zone suffix follows the choice.
• [Fix] PowerDNS Reverse DNS - Corrected the call into the PowerDNS client when setting or rebuilding a PTR record, and added test coverage over that path. ClouDNS providers were not affected.
• [Fix] Automatic PTR Format Labels - The second and third automatic PTR format options in the reverse zone form now describe the output they actually produce, and the form previews the resulting hostname for each option. If you use format 2 or 3 on an existing zone, check the preview against what you expect before your next change.

Load Balancers​

• [Feature] Host-Based Routing - Load balancer rules can now match on the request host as well as the path, so one load balancer can serve several hostnames to different backends. The match-type options were consolidated at the same time.
• [Improvement] Self-Healing Configuration - A load balancer left stranded in the configuring state now recovers on its own instead of needing a manual sync, with the grace period derived from how long the agent-side configuration can legitimately take rather than an arbitrary number.
• [Fix] HTTPS Redirect and Form Wipes - The HTTPS-redirect option on port 80 now applies correctly, and a real-time update arriving while a load balancer configuration form is open no longer discards what you were typing.

Databases and Backups​

• [Feature] Detached Backup Execution - Managed database backups no longer run inside a single blocking connection to the guest. A backup taking more than about 58 minutes used to be killed mid-upload by a transport timeout that scaled with nothing, and the task showed no movement at all between "running" and completion, so a healthy 13 GB backup was indistinguishable from a hang. Backups now run detached with the panel polling progress, reporting transferred bytes live, and the ceiling is a policy you set rather than an artefact of how the command was run. A genuinely dead guest is now detected in about two minutes.
• [Feature] Backup Run-History Retention - Backup run records are now pruned on a schedule with a configurable retention period, so the history table stays a useful size on long-lived installs.
• [Improvement] Honest Backup Schedules - The backup settings page now shows the schedule actually in effect rather than a placeholder that could differ from it, and validates a custom schedule when you save it. New installs default to daily.
• [Improvement] Faster Failures on Broken Egress - The database agent scripts now fail immediately with a named reason when the guest has no route to the internet, instead of hanging until a timeout. The message points at the VPC NAT gateway, which is the usual cause.
• [Fix] PostgreSQL Incremental Backups - PostgreSQL incrementals are markers over continuous WAL archiving and upload no object of their own. The completion check now recognises that shape and records them correctly, and a marker whose WAL archiving is not actually running is reported as a failure rather than a success.
• [Fix] Phantom Restore Keys - A storage key recorded against those marker rows could make restore and retention treat an object that does not exist as downloadable. Marker rows no longer carry one.

Platform and Admin​

• [Improvement] Smaller Update Rollback Snapshots - The snapshot taken before an application update now skips logs, caches, images and existing backups, and skips walking those directories at all rather than listing and discarding them. Snapshots on a busy install were hundreds of megabytes of log files.
• [Improvement] Quieter Admin Lists - The VPC and load balancer list pages no longer reload on every NAT gateway heartbeat. Updates are batched, so a page that was reloading dozens of times a minute now settles.
• [Fix] Hypervisor Health Flag - The automatic health flag raised when a node stops reporting now clears again on the node's next successful metrics poll, so a node that recovers from a transient blip returns to the deployment pool by itself. Allow Deployments remains the operator-controlled switch, and the admin page now labels which is which.
• [Fix] Blank Task Status - Task status could arrive in a form the column could not store, leaving the dashboard blank for tasks that were running. The column has been widened and the accepted values are validated where they arrive.
• [Fix] Missing Hypervisor Uptime - KVM nodes showed "-" for uptime on the hypervisors list and the dashboard.
• [Fix] VPC Real-Time Updates - Aligned three broadcast channel names with what the panel subscribes to, so VPC views update live again.
• [Fix] Missing User Timezone Crashed Instance Metrics - Accounts created without a timezone caused the instance metrics endpoints to fail. Timezone resolution now falls back through the account, the system default and UTC in one place, existing accounts are backfilled, and a setting that nothing had ever written - so subuser invitations always fell back to UTC regardless of the configured default - now reads the correct one.
• [Fix] Supervisor Workers Pointed at a Missing PHP - The queue worker configuration hardcoded PHP 8.3 while the installers have defaulted to 8.4 for some time, so on a fresh install no queue worker would start, and every application update reapplied the mismatch. The PHP version is now substituted where the configuration is deployed, including the unversioned path that EL hosts use.
• [Fix] Build Pipeline Hardening - The release build now enforces the compatibility flag when targeting multiple PHP versions, records the flags used with each artifact, and adds a verification step plus a smoke test on the target host.
• [Fix] Billing and Provisioning Debt - Volume backup charges now apply the account balance policy consistently, user records created through every path carry the timestamps the schema requires, and VNC port allocation can no longer hand the same port to two instances.

Infrastructure Agent​

• [Improvement] PHP Entrypoint Pinning - The agent now selects its PHP interpreter explicitly, preferring 8.4 and falling back to 8.3, verifying the candidate actually carries the extensions it needs before committing to it. The update process repairs already-deployed nodes.
• [Fix] Proxy Survives an Unlinked Node - The SSH proxy no longer fails to start on a node that has not yet been linked to a master, and linking, relinking or unlinking now takes effect without a restart. Unlinking correctly revokes trust.
• [Fix] Stale Binaries After an Update - An incremental update could leave the previous version of a running binary in place, because a running executable cannot be overwritten in place. The update now replaces them correctly.
• [Fix] Missing VNC Port Tolerated - A null or zero VNC port arriving from the master no longer produces invalid guest XML or a firewall error.

Integrations​

• [Improvement] OpenTofu Provider and MCP Server - Both gained the node-pool rotation endpoint, and the MCP tool for updating a node pool gained the instance plan field it was missing, so a plan change and its rotation can be driven from infrastructure-as-code or from an AI agent as well as from the panel.

Upgrade notes​

• Deploy the infrastructure agent before the master. The certificate installation command the master sends reaches a route that only exists in the new agent.
• Three migrations run on upgrade: the task status column widening, the System DNS schema, and the user timezone backfill.
• Reverse DNS on PowerDNS providers is worth a quick check after upgrading, and if you use automatic PTR format 2 or 3, confirm the form's preview matches the naming you expect.

Version v3.1.2 is the trust and hardening release that follows the Proxmox debut in v3.1.0. It modernizes how people get into the panel - social sign-in for users and enforceable OIDC single sign-on for admins - and how you sell capacity, with private locations and a built-in request-access workflow. Operators get a per-node deployment readiness engine and a substantially tougher load balancer. Underneath, this cycle ran two full platform security sweeps plus dedicated audits of the backup system and managed databases, on both the master and the hypervisor agent.

• [Feature] Sign in with Google, Microsoft, or GitHub - users can register and log in through OAuth, link and unlink providers from their profile, and auto-link to an existing account only when the provider asserts a verified email.
• [Feature] Admin single sign-on (OIDC) - bind admin logins to your identity provider with strict subject binding and no just-in-time provisioning, optionally enforce SSO for all admin password logins, and keep a time-limited break-glass path for IdP outages.
• [Feature] Private locations with request access - lock any location to selected accounts. Locked regions stay visible in the catalog with a lock treatment, users request access in one click, and admins approve or deny from a dedicated queue with email notifications both ways.
• [Feature] Node deployment readiness - every hypervisor now carries a live readiness checklist (agent, storage, network, capacity, deploy gates) surfaced as a dashboard card, a fleet list badge, and a per-node checklist with failure-specific fix hints.
• [Feature] Load balancers on allocated static IPs, captured error state (full detail for admins, a subtle banner for users), and per-frontend idle timeouts - TCP frontends now default to one-hour timeouts with kernel keepalives, so SSH and database sessions through an LB no longer drop at 50 seconds.
• [Feature] Proxmox surface expansion - VM snapshots, instance tags, guest-agent IP discovery, and backup file-restore in the user API; node issues, scheduled backup jobs, and live migration in the admin API; and admin edits to resources, topology, boot order, and NICs now push live to running VMs.
• [Feature] Security group drop rules on KVM - rule actions are honored end to end, so explicit drop rules override broader accepts, matching the Proxmox behavior.
• [Improvement] Teams - instance password mails go to the account owner with every instance-manage member in CC. Admin task queue gains one-click pruning and clean deletion.
• [Security] Two platform-wide security sweeps, defense-in-depth guardrails for the AI assistant, a backup-system audit in three phases, and a managed-database hardening batch. Details below.

Sign in with Google, Microsoft, and GitHub​

The login and registration pages now offer OAuth sign-in for Google, Microsoft, and GitHub. Each provider is enabled individually in the admin settings with its own client credentials; nothing shows on the login page until a provider is configured and switched on.

The linking rules are deliberately conservative, because OAuth auto-linking is a classic account-takeover vector:

• An OAuth identity auto-links to an existing account only when the provider asserts the email as verified. Google must present a true email_verified claim, GitHub only ever returns primary-and-verified addresses, and Microsoft sign-ins are validated against the tenant-verified UPN with the known cross-tenant takeover patterns (nOAuth) explicitly rejected.
• Sign-ups that arrive without a usable verified email go through a complete-profile step instead of silently creating a half-formed account, and the account write is transactional so a double submit cannot orphan a user.
• Logged-in users manage linked providers from their profile: connect, view, and unlink, with relinking handled safely.

Admin single sign-on (OIDC)​

Admin access can now be delegated to your identity provider - Okta, Entra ID, Keycloak, or any OIDC-compliant IdP:

• Strict binding. An admin's IdP identity binds on sub (subject), never on mutable claims, and there is no just-in-time provisioning - only pre-existing admin accounts can bind. The first bind is forensically logged, and stale identities are deleted rather than left dangling.
• Enforcement. Once your IdP is verified, you can require SSO for all admin password logins. The enforcement policy carries a lockout interlock so you cannot switch it on in a state that would lock every admin out.
• Break-glass. For IdP outages, php artisan admin:sso-break-glass opens a time-limited bypass that expires on its own. It is a deliberate, logged, console-only action.
• Setup UI. A new Authentication settings tab covers both features, including an OIDC discovery test that validates your issuer before anything is enforced. HTTPS is required and JWT verification is always on.

Private locations and request access​

Locations (hypervisor groups) can now be restricted per account. The catalog stays honest about what exists:

• Locked regions render on every create surface - the deploy modal, Cloud Service, self-provisioning, VPC and Kubernetes pickers - with a frosted lock treatment and the region name still visible, instead of vanishing from the catalog.
• Users hit Request access on a locked location, confirm, and the request lands in a new admin queue with a navigation badge. Admins approve or deny inline; both outcomes notify the user by mail. Access states are tracked per account as available, requested, or locked.
• Admin user pages gain a Cloud Service tab consolidating the account's location grants, inline approve and deny, and the account's provisioning limits.
• A per-account cloud provisioning switch cleanly disables self-service provisioning for an account without touching its running services, and the billing-exemption logic was made consistent across every surface that renders a deploy button.

Access enforcement is server-side on every create path, across web, API, queue, and AI-assistant surfaces. The lock UI is presentation; the gate is in the services.

Node deployment readiness​

Answering "why is nothing deploying to this node" used to mean reading logs. Now every hypervisor - KVM and Proxmox - carries a readiness engine that evaluates the conditions a deploy actually requires: agent reachability, storage presence and free capacity, subnet availability, deploy flags, maintenance and lock state.

• The admin dashboard shows a fleet readiness card.
• The hypervisor list badges each node ready, pending, or blocked.
• The node detail page renders the full checklist, and every failed check carries a specific fix hint tied to the actual failure, not a generic message.
• Adding a Proxmox node now runs its first cluster reconcile synchronously, so a freshly linked node reports honest readiness immediately instead of waiting for the next cron pass.

The checks mirror the real deploy gates - a node the checklist calls ready is a node the scheduler will actually use.

Load balancer improvements​

• Static IP deploys. User load balancers can deploy onto allocated static IPs, so an LB's address can be planned, firewalled, and DNS'd before it exists.
• Error surfacing. LB provisioning and sync failures are captured as a last-error state: admins see the full detail on the LB page, users see a subtle banner that something is being worked on - operational detail stays internal.
• Long-lived TCP sessions. TCP-mode frontends previously inherited HTTP-tuned 50-second idle timeouts, which silently killed idle SSH, database, and message-queue connections through the LB. TCP frontends and backends now default to one-hour timeouts with kernel TCP keepalives on both sides, websocket tunnels get a matching post-upgrade timeout, and every frontend gains an optional idle timeout field (30 to 86400 seconds) in both the user and admin panels for workloads that need more or less.
• Kubernetes LB fixes. Service LBs honor the managed-loadbalancer-public-ip annotation, weighted routing-rule backends materialize correctly with collision-free ACL names, port 80 stays plaintext under global SSL mode, and NodePort backends are health-checked over TCP.
• Plan enforcement. Standalone LB deploys enforce the location's plan-group offering, closing a path where an LB could deploy from a plan the region does not sell.

Proxmox, continued​

v3.1.0 shipped the driver; v3.1.2 finishes the surfaces around it:

• User API: VM snapshots (list, create with optional RAM state, rollback, delete), instance tags, guest-agent IP discovery, and backup file-restore browse and download.
• Admin API: node issues (list, retry, resolve), PVE scheduled backup jobs, and live migration with precheck. Route binder failures return real 404s instead of leaking existence.
• Live VM edits. Admin changes to resources, CPU topology, boot order, and NICs push to the running VM where PVE allows it, with CPU flags, secure boot, and TPM handling brought to parity with KVM.

All new endpoints are covered by the API manifest and mirrored in the OpenTofu provider and MCP server coverage gates.

Reliability: Kubernetes, VPN gateways, VPC​

• Kubernetes: worker-pool scale-up crash fixed, long jobs no longer double-execute after 90-second queue redelivery, control-plane and worker plan pickers are scoped to the region's plan groups, node selection prefers the NAT-active hypervisor, and deploys survive recycled-IP ARP staleness and transient agent transport blips. Control-plane LB deploys from the queue were failing on an authentication-context assumption; provisioning gates now evaluate the acting user everywhere.
• VPN gateways: peer key pairs auto-generate as the UI always promised, and road-warrior clients receive the VPC's private DNS resolver.
• VPC on KVM: cross-node NAT egress now installs the correct default route on non-active nodes and repairs it in the periodic sync, the VPC bridge joins a firewalld zone so nftables cannot silently reject its traffic, and ICMP redirects are suppressed on VPC veths - closing a class of "works from one node, dead from another" reports.
• Node provisioning: fresh hypervisors install required CLIs rather than only upgrading existing ones, Debian contrib is enabled across both source layouts for ZFS, and half-merged /usr systems are repaired so kernel modules and ufw work on broken base images.

Managed database hardening​

The managed database service went through a dedicated audit. Highlights: six critical backup, restore, and HA defects fixed; incremental backup chain source pinning so a restore can never mix chains; encryption keys moved off process argv; a watchdog that rescues clusters stuck in configuring with init-phase visibility; callback token lifecycle hardening with a localhost guard; PostgreSQL cluster self-heal; replica resync credentials forwarded correctly; and the admin password revealed on the detail pages where operators actually need it.

Backup system audit​

A three-phase audit of the backup pipeline shipped on both sides:

• Master: failure alerting is throttled and queue-routed so it always sends, repeated failures auto-pause a plan instead of burning nightly cycles, prune notifications report what was actually pruned, backup sizes are captured from the agent callback, and remote restores gained a direct download path while a dead restore path was removed.
• Agent: a credential leak into backup artifacts was stopped, silently truncated backups are now detected and failed, backup and restore state files are no longer world-readable, and qcow restores verify the staged artifact and use tmp-then-rename so a partial download can never replace a disk.

Security sweeps​

Two platform-wide sweeps (2026-07-29 and 2026-07-30) ran during this cycle, with every finding remediated before release. The notable classes:

• Billing integrity: top-up capture is now bound to its originating transaction, closing a credit-fraud path; credit adds are validated; backup debits are atomic.
• Tenant scoping: SSH sessions, S3 access keys, Kubernetes certificate renewal, and VPC selection are all bound to the owning tenant; state-changing restore moved off GET.
• Auth: password-reset throttling, no exception reflection to clients, OAuth and email uniqueness guarantees, and the Microsoft cross-tenant (nOAuth) rejections described above.
• Secrets at rest and in transit: queue payloads carrying secrets are encrypted, failed-job rows are pruned, Kubernetes join credentials no longer travel through cloud-init user data, notification channel secrets are no longer serialized into events, and WireGuard AllowedIPs are validated before any privileged guest execution.
• AI assistant guardrails (three phases of defense in depth): streaming egress redaction of configured secrets, knowledge-base audience scoping that fails closed, untrusted-data framing around tool output with prompt-injection guards, and redaction of persisted tool calls and audit logs so the assistant's own storage cannot become the leak.
• Dependencies: dompdf bumped for CVE-2026-56722.

Version v3.1.0 is the Proxmox release. The platform now speaks Proxmox VE natively: point it at any existing PVE 8 or 9 node or cluster with a single API token, and every node is discovered, linked, and managed from the same panel, API, and billing pipeline as your KVM fleet. There is nothing to install inside Proxmox and no agent to maintain - the driver works entirely over the Proxmox REST API, with one optional one-command bootstrap on a node for the features that need host access (VPC WebSSH and exact per-NIC metering). Nothing changes for existing KVM hypervisors, instances, plans, or balances.

• [Feature] Native Proxmox VE driver - full instance lifecycle on PVE 8.0+ and 9.x: deploy, power, suspend and resume, reinstall, resize, destroy, ISO mounting, cloud-init, GPU passthrough, VM tags, and per-tenant resource pools. Agentless by design; see the section below.
• [Feature] One-token cluster onboarding - give the panel one reachable member IP and one API token, and it discovers every node in the cluster, links them all, verifies SDN and FRR prerequisites, provisions console credentials, and self-checks the token's privileges. Failures are listed per node instead of aborting the lot.
• [Feature] VPC networking on Proxmox SDN - VPCs are realized as PVE SDN EVPN zones, vnets, and subnets, with NAT gateways, hot attach and detach of VPC NICs, and automatic reconciliation. Security groups translate to the PVE firewall with accept and drop rules, IP sets, and per-VM rulesets.
• [Feature] Backups and snapshots - vzdump backup, restore, and delete per instance, optional live-restore, native PVE scheduled backup jobs with full retention control, file-level restore (browse a backup and download individual files), and VM snapshots with optional RAM state, rollback, and delete.
• [Feature] Live migration - move a running VM between cluster nodes from the panel, with an eligibility precheck, local-disk handling, target storage selection, bandwidth limits, and a dedicated migration network option.
• [Feature] High availability via the PVE CRM - instances enroll into Proxmox's own HA stack on deploy, with per-group tuning and placement rules; the panel mirrors CRM state instead of running a second watchdog.
• [Feature] Full guest-service parity - Docker deployments, managed databases (including S3 backups, point-in-time recovery, and replication), load balancers with Let's Encrypt, Kubernetes clusters, WebSSH, SSH key injection, and password resets all work on Proxmox instances through the QEMU guest agent.
• [Feature] Exact bandwidth metering - per-NIC counters are read from the host, so VPC (east-west) traffic and public traffic are billed separately and never double-counted, with reset-safe accumulation across reboots.
• [Feature] Node Issues dashboard - operational failures on a Proxmox node (metering, host access, proxy installs, firewall gaps) surface as first-class admin issues that reopen if they recur, instead of hiding in a log file.
• [Improvement] Fleet-scale collection - metrics, statistics, security-group reconciliation, and HA reconciliation all fan out per node onto a worker pool, so a large cluster is collected in parallel rather than serially.

Native Proxmox VE support​

v3.1.0 introduces a second hypervisor backend alongside KVM. A hypervisor group is now either a KVM group or a Proxmox cluster, and both kinds run side by side on one panel with the same instances, plans, billing, and user experience.

The design principle is API-only: the driver talks to the Proxmox REST API and nothing else. Your cluster keeps looking like a normal Proxmox cluster - VMs created by the panel are plain QEMU VMs you can see in the PVE web UI, tasks the panel starts are ordinary PVE tasks (UPIDs the panel tracks and can cancel), and nothing on the node is patched or replaced.

Onboarding an existing cluster​

Create a hypervisor group of type Proxmox, paste one member's address and one API token, and the panel calls the cluster status endpoint to discover every node. Each node is linked individually, storage is discovered per node with its real PVE plugin type, an @pve console credential is provisioned for VNC, and SDN plus FRR prerequisites are verified so VPC problems are caught at link time rather than at first deploy. The token itself is self-checked: the panel reports the PVE version and the privileges the token actually holds, and flags the guest-agent privilege gap that PVE 9 introduced, right on the node's detail page.

Supported: Proxmox VE 8.0 and later, including 9.x, standalone nodes and clusters. Group names must match the PVE cluster name so the group is an unambiguous mirror of the datacenter, and groups are kept homogeneous - one group is either KVM or Proxmox, never a mix.

Images and deploys​

Operating system images are built once per cluster into a vzdump archive, then every deploy restores that archive into the customer's VM ID. There are no template VMs parked on your nodes and no full-clone storms: what lands is a plain VM, on the storage you chose, with cloud-init applied. Archives are labeled with the upstream image name, can be purged with one command, and image builds are excluded from orphan detection.

Cloud-init works two ways: the default NoCloud seed ISO (full feature set, including multi-user and multi-IP layouts), or PVE's native cloud-init as a per-group opt-in for simple single-user, single-IP instances where a config drive is unnecessary.

Existing VMs on the cluster are not stranded either: the orphan importer can adopt any VM the panel did not create - including raw, ZFS, LVM, and Ceph-backed disks - and attach it to a user as a managed instance.

Storage​

All common PVE storage backends are supported and detected with their real capabilities: directory, NFS, LVM, LVM-thin, ZFS, and Ceph RBD, including onboarding an external Ceph cluster directly from the panel. Extra volumes allocate through the PVE storage API with the correct format per backend, hot-plug with IO limits, and resize live (grow-only, as QEMU requires). Capability gates are honest: operations a given backend cannot do, such as snapshots on plain LVM, are refused with a clear message instead of failing halfway.

Networking, VPCs, and security groups​

VPCs on a Proxmox group are built on PVE SDN: an EVPN controller and zone per VPC with a dedicated VRF VNI, a vnet and subnet per VPC subnet, and a NAT gateway with SNAT semantics that hold cluster-wide. VPC NICs hot attach and detach on running VMs. A reconciler keeps the SDN objects healthy and converges NAT gateway activation, and VPC create and destroy push eagerly so the fabric follows the panel immediately.

Security groups compile to the PVE firewall: cluster-level groups and IP sets, per-VM rulesets with both accept and drop actions, anti-spoofing IP filters, and firewall flags only on the NICs they belong to. Reconciliation is fingerprint-based, so unchanged nodes are skipped, and a cron backstop repairs drift. The datacenter firewall is enabled safely, preserving management access rather than default-dropping the hosts.

The instance network tab also gains guest IP discovery on Proxmox: addresses are read live from the QEMU guest agent, filtered of loopback and link-local noise, for both admins and users.

Consoles and WebSSH​

Three console paths ship for Proxmox instances:

• VNC (noVNC) through a PVE login ticket with single-use, server-side session tokens. PVE credentials and tickets never reach the browser.
• Serial terminal (xterm.js over termproxy), tenant-scoped.
• WebSSH - a real SSH shell in the browser. Public-IP guests are dialed directly. VPC-only guests are reached through proxmox-ssh-proxy, a small node-side service that enters the VPC's VRF on the host, because the guest's route only exists there.

The proxy is the one place host access is needed, and it bootstraps with one command on one node: the panel mints a short-lived, single-use install command, the script fetches the binary from your panel (never a third-party artifact), installs the service, opens the master's port in the PVE firewall using the egress address the node itself observed, and the panel keeps every node's proxy current automatically from then on. Install state is verified against what is actually on the node, so a half-finished install shows up as fixable instead of hiding.

Backups, scheduled jobs, and file restore​

Per-instance backups run through vzdump with completion tracked to the panel's backup queue, restore (optionally live-restore, so the VM boots while data streams back), redirect-to-storage on restore, and delete. Cluster-level scheduled backup jobs are managed from the panel with the full keep-last, keep-hourly, keep-daily, keep-weekly, keep-monthly, and keep-yearly retention set, targeting specific VMs, a resource pool, or the whole cluster. With Proxmox Backup Server storage, file-level restore lets admins and users browse a backup's filesystem and download individual files without restoring the VM.

Snapshots and Forge​

VM snapshots are first-class on capable storage: create with or without RAM state, list, rollback, and delete, from both admin and user instance pages. Forge (the checkpoint-try-rollback workflow) rides the same mechanism with a reserved snapshot slot, protected from name collisions with user snapshots.

Live migration​

Admins can live-migrate a Proxmox instance between cluster nodes with a precheck that refuses ineligible targets and local hardware blockers, handles local disks, and passes through target storage, bandwidth limit, migration network, and online mode. On PVE 9, conntrack-state migration is available as an opt-in. Migration success updates the instance's node assignment and flushes routing caches; a failed migration leaves the instance exactly where it was.

High availability​

Proxmox groups delegate HA to the PVE cluster resource manager - the layer that actually owns fencing and recovery - instead of duplicating it. Instances enroll on deploy and unenroll on destroy, groups expose HA tuning (and node-affinity placement, using HA groups on PVE 8 and node-affinity rules on PVE 9), and a reconciler mirrors CRM state back into the panel so the instance page shows the real HA state. The KVM HA watchdog is untouched, and a Proxmox-side failure can never stall KVM monitoring.

Guest services: Docker, databases, load balancers, Kubernetes​

Everything that previously required the slave agent's SSH path now runs over the QEMU guest agent on Proxmox instances:

• Docker: deploy, control, status, and logs.
• Managed databases: the full suite - configure, restart, password reset, backup tooling, full and incremental backups to S3, point-in-time recovery, restore, upgrade, replication, and batched health checks.
• Load balancers: configuration, Let's Encrypt issue and renew, and telemetry setup.
• Kubernetes: kubectl execution, control-plane certificate upload and renewal, and admin kubeconfig reissue, routed through the Proxmox adapter for Proxmox-hosted clusters.
• Instance basics: SSH key inject and remove with port detection, password resets, and WireGuard configuration sync.

Large payloads are delivered safely: small files write through the agent directly, larger ones are pulled by the guest from a single-use, checksum-verified URL.

Metering, billing, and scale​

Metrics and statistics feed the same billing pipeline as KVM, with the units and counters aligned so plan enforcement and charts behave identically. Two things are worth calling out:

• Exact per-NIC bandwidth. The PVE API only exposes aggregate VM counters, which cannot split public from VPC traffic. v3.1.0 reads per-NIC tap counters from the host instead, so VPC traffic and public traffic are metered separately and never double-billed, with reset-safe accumulation that keeps mid-day reboots billable.
• Parallel collection. Metrics, instance statistics, HA reconciliation, and security-group reconciliation fan out per node and per group onto a dedicated worker pool. Collection time scales with your largest node, not your node count.

Operational visibility​

A new Node Issues dashboard (admin, under Proxmox) surfaces operational failures on a node as structured issues: what failed, on which node, how many times, first and last seen. Issues reopen automatically if the condition recurs after being marked solved, and each issue carries a retry action. Long-running PVE tasks are tracked by UPID and can be cancelled from the panel, and a stuck panel task cancels its PVE counterpart when marked failed.

Security hardening in this release​

The Proxmox driver went through dedicated security review during the cycle. Notable fixes shipping in v3.1.0: a WireGuard configuration path that could execute as root in the guest was closed, WebSSH sessions no longer expose PVE authentication cookies, console ACLs are revoked rather than accruing, ISO URL fetching is SSRF-hardened, console routes are tenant-scoped with negative-authorization tests, and backup restore and file-restore endpoints verify the backup belongs to the instance before acting.

Version v3.0.0 is a foundation release. The platform - both the master and the hypervisor agent - now runs on Laravel 13 on PHP 8.3, and this build adds a broad round of precision and edge-case hardening across the systems that run quietly in the background. It also ships two ways to drive the platform programmatically: the Hypervisor.io OpenTofu / Terraform provider, so your instances, networks, and clusters can be managed as code, and a remote Model Context Protocol (MCP) server, so AI agents can operate the same account conversationally. Nothing changes for your existing instances, plans, balances, or configuration.

• [Feature] OpenTofu / Terraform provider - manage your account as Infrastructure as Code. The iaas provider (now released at v0.2.1, source hypervisor-io/iaas) exposes 56 resources and data sources - instances, VPCs and subnets, Kubernetes clusters, managed databases, load balancers, storage, DNS, VPN, S3, and more - over the user REST API, with full CRUD, import, and reference-driven dependencies. Authentication is your existing IP-locked API token, and both users and admins can use it. See the section below.
• [Feature] MCP server - hand your account to AI agents. A remote, stateless Streamable HTTP Model Context Protocol server exposes the platform as 364 tools (302 user + 62 curated admin) over the same REST API, with Bearer API-token pass-through auth, confirm-gated destructive operations, idempotency keys, and async convergence. Any MCP client can drive it. See the section below.
• [Platform] Laravel 13 across the stack - master and the hypervisor agent both move to Laravel 13 on PHP 8.3, on stable, security-audited dependencies. A pure runtime modernization; your data and settings are untouched.
• [Improvement] Billing precision - hourly billing is refined for long-running resources, unusual calendar edge cases, storage metering cadence, and exact fractional-credit accounting.
• [Improvement] Backup & restore robustness - incremental chains, in-place volume restore, and retention pruning were hardened for edge cases across Ceph and block backends.
• [Improvement] Migration & networking polish - cold and live migration edge cases, reverse DNS for uncommon IPv6 forms, upload/download rate-limit symmetry, and quota handling under concurrency were all tightened.
• [Improvement] Metrics accuracy - the usage pipeline that feeds billing now handles counter-reset edge cases more precisely.

Infrastructure as Code (OpenTofu / Terraform)​

v3.0.0 introduces the Hypervisor.io OpenTofu / Terraform provider - the whole platform, declared in HCL and converged with tofu apply.

You write the resources you want; the iaas provider translates them into calls against the user REST API using your existing IP-locked API token, and OpenTofu tracks the state. It covers 56 resources and data sources - instances, VPCs and subnets, security groups, Kubernetes clusters and node pools, managed databases, load balancers, storage volumes, DNS, VPN gateways, S3 buckets, projects, and the catalog data sources you reference for plans, images, and regions. Every resource supports full create / read / update / delete, tofu import <addr> <uuid> to adopt existing infrastructure, and normal Terraform references so one resource can depend on another.

Both users and admins can use it: a user manages their own resources with a user token; an operator uses an admin-scoped token for the broader surface. Because the token is validated against the IP it was registered with, run tofu from a stable egress IP (a CI runner, bastion, or workstation).

The provider is released at v0.2.1. Declare it with source hypervisor-io/iaas and let OpenTofu fetch it:

terraform {
  required_providers {
    iaas = {
      source = "hypervisor-io/iaas"
    }
  }
}

• Provider and full documentation: the repository is github.com/hypervisor-io/terraform-provider-iaas and the reference docs walk through getting started, the resource catalog, and common patterns.
• Publishing to the public OpenTofu / Terraform registry is being finalized; that hypervisor-io/iaas source is the install path once it lands, and you can build the provider from the repository in the meantime.

AI agents (Model Context Protocol server)​

v3.0.0 also ships the Hypervisor.io MCP server - the same platform, exposed to AI agents. Where the OpenTofu provider is the declarative path, the MCP server is the conversational one: point any MCP client at it and an agent can create instances, assign VPCs, and manage the rest of your infrastructure in natural language.

It is a remote, stateless Streamable HTTP server, so there is nothing to install locally - connect over HTTP and authenticate with a Bearer API token that is passed straight through to the platform, IP-locked exactly like the provider's. It exposes 364 tools - 302 user tools plus 62 curated admin tools - covering instances, VPCs and subnets, Kubernetes, managed databases, load balancers, storage, DNS, VPN, S3, and the catalog lookups agents need to reference plans, images, and regions.

Because agents act on their own, the server is built to be safe by construction:

• Confirm-gated destructive operations - deletes and other irreversible actions require an explicit confirmation step before they run.
• Idempotency keys - a retried call does not double-create, so a flaky connection cannot spawn duplicate instances.
• Async convergence - long-running operations return a task the agent can poll to completion instead of blocking.
• A curated admin allowlist - the 62 admin tools are a deliberately safe subset; billing, user deletion, and hypervisor destruction are not exposed.

The server is backed by the same tested Go client as the OpenTofu provider, so both consumers reach the API through one audited code path.

• Server and full documentation: the repository is github.com/hypervisor-io/iaas-mcp-server and the reference docs cover connecting a client, authentication, and the tool catalog.

The user / admin REST API is the single source of truth for all three surfaces - the API itself, the OpenTofu provider, and the MCP server - and a manifest-driven CI gate keeps them in lockstep, so no endpoint can ship without matching provider and MCP coverage.

Version v2.2.6 makes local storage a first-class, pluggable layer. Until now a local instance disk meant a qcow2 file. This release adds four block-device backends - LVM thin, LVM thick, ZFS thin, and ZFS thick - that slot in alongside qcow2, Ceph, and NFS with the full lifecycle: deploy, resize, snapshots, incremental backups, restores, image creation, and live migration. Around them sit three operator-facing safety features: a guided storage wizard that probes the hypervisor before anything is saved, a collapse watchdog that suspends only the at-risk instances before a thin pool can corrupt, and a live Storage Health dashboard. The Virtualizor importer now lands migrated VMs straight onto LVM/ZFS with no qcow2 conversion step, and the admin panel picked up a round of polish along the way. This is a beta build - feature-complete for the storage work and safe to stage, but not yet promoted to stable.

• [Feature] LVM and ZFS Storage Backends - Hypervisor storage now supports four new local pool types alongside qcow2 files, Ceph, and NFS: LVM thin, LVM thick, ZFS thin, and ZFS thick. Instances on these pools run on raw block devices for near-native disk performance, with the thin variants providing over-provisioning and the ZFS variants bringing checksumming and inline compression. Plans can target a specific backend, so an operator can sell "NVMe block storage" and "standard storage" tiers from the same hypervisor.
• [Feature] Guided Storage Wizard - Adding a storage pool is now a five-step wizard that probes the hypervisor live before anything is saved: it verifies the tooling is installed, the volume group or pool exists, and reports real capacity, so a mistyped pool name can no longer create a broken storage row. LVM and ZFS appear in the picker with a Beta tag for this release.
• [Feature] Storage Collapse Watchdog - Every hypervisor now watches its pools every 30 seconds. When a thin pool or ZFS pool approaches exhaustion, the platform suspends only the affected instances before their disks corrupt, alerts every administrator by mail, and surfaces the event live in the Storage Health dashboard. Recovery is operator-controlled: fix the pool, then resume the instances from where they stopped.
• [Feature] Storage Health Dashboard - A per-pool health view in the admin panel shows live data and metadata usage, ZFS pool state and fragmentation, and warn/critical thresholds, fed by the same probes the watchdog uses, surfaced inline on the storage list.
• [Feature] Native LVM/ZFS Virtualizor Import - The Virtualizor importer no longer forces every disk through a qcow2 conversion. When the source VM lives on LVM or ZFS, the slave probes the pool and the importer lands the instance natively on the matching backend, preserving the storage tuple and skipping the conversion pass entirely.
• [Improvement] Incremental Backups Everywhere - The QEMU dirty-bitmap incremental backup pipeline now covers all seven backends, including raw block devices, writing the same qcow2 backup chains to your existing backup storage. Existing qcow2 and Ceph backup behavior is unchanged.
• [Improvement] Live Migration for Block Pools - Live migration works between LVM/ZFS pools across hypervisors, pre-creating the destination volume and block-copying storage as part of the move.
• [Improvement] Admin Panel Polish - The VPN Gateway, Managed Database, and Load Balancer plan create/edit pages were rebuilt on the shared sectioned-card layout used by Instance Plans (and a latent dropdown bug on those pages was fixed), more list pages gained the standard collapsible filter panel, the in-browser SSH terminal now stays connected for as long as the tab is open, and the frontend now builds through Vite.

LVM and ZFS Storage Backends​

Until now local instance disks meant qcow2 files. v2.2.6 adds four block-device backends - LVM thin, LVM thick, ZFS thin, and ZFS thick - that slot in alongside the existing qcow2, Ceph, and NFS options with the full feature set: deploy, resize, snapshots (thin pools and ZFS), incremental backups, restores, image creation, and live migration. Because plans can target a specific backend, an operator can offer differentiated storage tiers from a single hypervisor.

Pools are added through the new guided wizard, which talks to the hypervisor before saving: it checks that lvm2 or zfsutils are installed (both are installed when you update the hypervisor from the admin panel), that the volume group, thin pool, or zpool actually exists, and shows live capacity. Existing storage rows, plans, and instances are untouched - qcow2, Ceph, and NFS behave exactly as on 2.2.5, and nothing is migrated or rewritten on upgrade.

The Collapse Watchdog​

Thin provisioning's dark side is pool exhaustion: when an over-provisioned pool fills up, every VM on it can corrupt at once. v2.2.6 ships a two-layer defense. First, LVM thin pools are configured to auto-extend at 80% via dmeventd. If that cannot help - the volume group is out of space, or a ZFS pool faults - a watchdog on each hypervisor catches the threshold crossing within 30 seconds, pauses only the instances on the affected pool, mails every administrator, and raises a live alert in the Storage Health dashboard. Paused instances keep their RAM and disk state; once the operator extends the pool or replaces the failed device, the instances resume from where they stopped. Degraded ZFS mirrors raise an early warning while the instances keep running.

Storage Health Dashboard​

The Storage Health view in the admin panel shows per-pool data and metadata usage, ZFS pool state and fragmentation, and warn/critical thresholds - fed by the same probes the watchdog uses and broadcast live over WebSocket. In this release the health values are surfaced inline on the storage list, so pool state lives in one place.

Native LVM/ZFS Virtualizor Import​

Migrating off Virtualizor no longer means rewriting every disk to qcow2. When a source VM lives on LVM or ZFS, the slave probes the pool and the importer lands the instance natively on the matching backend, preserving the storage tuple and skipping the conversion pass entirely. qcow2 sources continue to import as qcow2.

Known Beta Notes​

• LVM and ZFS appear with a Beta tag in the storage wizard; exercise them on a test hypervisor before offering them to customers.
• Pool auto-extend depends on free space in the underlying volume group or zpool - the watchdog is the safety net, not a substitute for capacity planning.
• Found a problem? Report it referencing v2.2.6-beta along with the hypervisor's storage type and the relevant Storage Health alert.

Version v2.2.3 is a major feature release headlined by Managed Kubernetes, a fully integrated Kubernetes-as-a-Service offering that runs alongside Instances, Volumes, Load Balancers, and Databases. Customers can spin up a control plane (single-node or 3-node HA), attach workers in per-purpose pools, expose Kubernetes Services through the bundled in-cluster cloud controller manager, autoscale workloads end to end with the cluster autoscaler, and roll the cluster forward to a newer Kubernetes version, all without touching the slave host. The release also ships a redesigned master backup pipeline with pluggable storage drivers and Grandfather-Father-Son retention, a new System Health dashboard widget, scheduled task health tracking, an app-wide timezone setting, team-member permissions for Kubernetes resources, retry for failed cluster creates, and a long list of reliability and performance improvements including a 70% reduction in peak load for the hot-path cron loop that runs every 30 seconds against the entire fleet.

• [Feature] Managed Kubernetes - Create production-grade Kubernetes clusters directly from the control panel. Choose single-node or 3-node HA control plane, pick instance plans and subnets for control plane and workers separately, and bring up the cluster with a bundled HAProxy load balancer for the Kubernetes API. Real-time progress streams to the cluster show page via WebSocket; downloaded kubeconfig points at the right private or public endpoint automatically.
• [Feature] Worker Node Pools - Each cluster has a default worker pool and supports unlimited additional pools, each with their own instance plan, labels, taints, autoscaling bounds, and drain settings. Useful for GPU nodes, memory-optimized workloads, or isolating tenants in a single cluster.
• [Feature] Cluster Autoscaler - Bundled cluster autoscaler binary speaks the Hypervisor API directly. Policy-driven scaling on CPU + memory pressure of pending pods, per-pool aware, and respecting each pool's min/max bounds. Manifest generated on demand from the cluster show page, customers grab the YAML and apply with kubectl apply -f -. Controller token refreshes on a rolling schedule so long-lived clusters never need a manual re-issue.
• [Feature] In-Cluster Cloud Controller Manager - Services of type LoadBalancer provision and tear down a real Hypervisor load balancer per service. Service annotations control listener port, backend mode (TCP / HTTP / per-port hybrid), session stickiness, multi-cert SNI, routing rules, and traffic split between subset endpoints.
• [Feature] Worker and Control Plane Rolling Upgrades - Upgrade Workers card on the Workers tab provisions new workers at the target version, drains old ones, repeats. Upgrade Control Plane card does the same for CPs via surge-replace strategy, etcd-quorum-safe at every step. Cluster card shows CP version and worker baseline as two distinct lines with a "mid-upgrade" badge when they diverge.
• [Feature] Retry Failed Cluster Create - A new "Retry create" button on the cluster page tears down partially provisioned artifacts and re-runs the bootstrap on the same row. Cluster name, slug, and identity certificates are preserved so any kubeconfig the user already downloaded stays valid. No more delete-and-recreate after a transient quota or capacity precondition fails.
• [Feature] Cluster Security Groups - Three auto-managed security groups per cluster (LB-only, CP-only, worker-only). Default rules expose the Kubernetes API on :443 via the LB and lock down direct access to CP nodes' :6443 from outside the cluster. Admins and users layer additional rules through a familiar Inbound / Outbound sub-tabbed interface.
• [Feature] Restricted Kubeconfig - Downloaded kubeconfig issued at cluster create exposes only worker nodes to kubectl get nodes. Control-plane VMs are hidden from end users in the Compute list, billing reports, monitoring tiles, and the cluster Nodes tab.
• [Feature] Master Backup Pipeline Redesign - Service-oriented orchestrator with pluggable storage drivers (Local, S3-compatible, Rsync over SSH, NFS), a singleton lock that survives long uploads, Grandfather-Father-Son retention, email + webhook notifications, and a configurable cron expression. Multiple destinations supported per install. Admin pages cover Destinations, Runs, Settings, and Scheduler Health.
• [Feature] Scheduled Task Health Tracking - Every scheduled task is observed via a unified health surface. Per-task tracking of last run, duration, exit code, and consecutive failures. Compact admin Scheduler page with a slide-in drawer per task, friendly task names, and a daily prune to keep the audit table compact. Dashboard tile shows healthy / degraded / failed scheduled-task counts at the top of every page.
• [Feature] System Health Dashboard Widget - Single compact strip on the admin dashboard showing four critical metrics at a glance: most recent successful master backup, scheduled-task health rollup, in-flight long-running tasks, and queue worker failed-job count. Replaces two separate tiles from earlier releases.
• [Feature] Application Timezone Setting - Pick any IANA timezone from a new dropdown under Admin > System > Settings > General. Applied app-wide on boot (Carbon, model date casts, scheduler firing times, direct PHP date functions). Default for customers signing up via self-registration and billing-API user creations, unless explicitly overridden. Existing users keep their own timezone selection.
• [Feature] Kubernetes Team Permissions - New kubernetes.* permission family with three tiers (view, manage, delete) granted through the existing team-member invitation flow. Predefined roles get sensible defaults from the migration. Custom roles need to be granted the new permissions explicitly.
• [Feature] Admin Destructive Controls for Clusters - Dedicated section for safe escape hatches when a cluster has gone wrong. Suspend locks out the customer while preserving forensics. Reset State clears stuck-operation flags. Force Cleanup bypasses normal teardown for clusters with zombie resources. Separate rate limits keep destructive (5/hour) and recovery (20/hour) actions distinct.
• [Feature] AWS-Style Node Drill-Down - Clicking a node on the cluster Nodes tab opens a side drawer with capacity gauges (CPU / RAM / disk), pod listing with pagination and search, taints section, and modern dark/light surface styling.
• [Feature] Cluster-Managed Resource Lockdown - Worker instances and the CP load balancer carry a clear "Cluster-managed" badge and a read-only banner in the user's Compute and Load Balancers lists. Direct power cycle, plan change, or LB rule edit is blocked at the controller. Manage them through the cluster page instead.
• [Feature] Live Load Balancer Filtering - User-side Load Balancers index now supports AJAX live filtering by name, status, and VPC. Useful for customers running dozens of LBs across multiple VPCs.
• [Feature] Cluster Activity Feed - User dashboard activity feed now translates Kubernetes audit-log actions into friendly sentences ("Created cluster prod-01", "Upgraded workers to 1.35.0") alongside the other resource types.
• [Feature] Pre-Flight Quota and Capacity Guards - Cluster create form rejects at submit time when load balancer quota is exhausted, when the chosen VPC has no NAT Gateway (needed for control-plane image pulls), or when the CP subnet is not private. Clear messages name the limit and point at the affected field instead of failing deep in the bootstrap chain.

Version v2.2.2 is a major feature release. Headline additions: Volume Snapshots and Backups (on-storage rollback marks and disaster-recovery exports for block storage volumes), Orphan VM Import (adopt KVM domains that already exist on a hypervisor as Hypervisor-managed instances without downtime or disk migration), and User Self-Registration with built-in CAPTCHA support for Cloudflare Turnstile and Google reCAPTCHA v2 / v3, signed email verification, and a post-verification onboarding top-up flow. Alongside the headliners: a comprehensive overhaul of the instance backup experience, a redesigned Forge tab, a polished tasks progress view, and the carry-over polish from earlier 2.2.x - RustFS as the new self-hosted object storage default, a fully reworked hypervisor self-update flow, and richer AI Assistant diagnostics for admins.

• [Feature] Volume Snapshots and Backups - Block storage volumes now have a dedicated Snapshots & Backups tab on every volume detail page. Take instant on-storage snapshots for rollback, or export full backups to your existing remote backup storage (S3-compatible, NFS, or local) for disaster recovery. Both support in-place rollback and restore-to-new-volume modes.
• [Feature] Per-Volume Safety Controls - Volume operations enforce per-volume serialization (one operation at a time), a 5-minute cooldown between operations, per-plan caps on retained snapshots and backups, and a per-user inflight limit so a single user cannot saturate the queue.
• [Feature] Volume Backup Billing - Volume plans now expose per-GB / per-month credit pricing for retained snapshots and retained backups, plus configurable caps on how many of each a customer can keep on a single volume. Defaults bill nothing for retention but allow up to 5 snapshots and 10 backups per volume.
• [Feature] Orphan VM Import - A new admin workflow on every hypervisor manage page surfaces KVM domains that exist on libvirt but are not tracked by Hypervisor - for example, VMs from before the slave was attached, or VMs created out-of-band. Eligible (qcow2 file-backed) domains can be adopted as Hypervisor-managed instances in place. MAC, IPs, and disk paths are preserved; per-disk storage assignment in the import dialog means no mv of qcow2 files.
• [Feature] User Self-Registration with CAPTCHA - A complete public signup pipeline with three CAPTCHA providers (Cloudflare Turnstile, Google reCAPTCHA v2, Google reCAPTCHA v3), per-form toggles for Login and Register, signed email-verification links, a Verify-Your-Email page that auto-redirects via WebSocket the moment the user clicks the link in another tab, and an optional post-verification top-up onboarding modal.

We're excited to announce Version v2.2.1 of the Hypervisor Control Panel! This release completely rebuilds the instance backup experience with support for remote backup destinations and adds GPU passthrough for compute-heavy workloads.

You can now back up your instances directly to Amazon S3, RustFS, Wasabi, Backblaze B2, DigitalOcean Spaces, SFTP servers, FTP servers, or any rclone-supported backend -- no need to mount backup storage locally on each hypervisor.

• [Feature] Remote Backup Destinations - Configure S3, SFTP, FTP, or rclone-based backup targets from the admin panel. Credentials are stored encrypted and pushed to hypervisors only when needed.
• [Feature] Streaming Backups - Full and incremental backups stream directly to the remote destination. Restores stream back the same way. No temporary disk space required.
• [Feature] Running-VM Backups Without Downtime - Backups of running VMs use external snapshots so the VM continues running normally while the backup completes. Changes made during the backup window are merged back seamlessly.
• [Feature] Efficient Incremental Backups - Incremental backups capture only the blocks that changed since the last backup, dramatically reducing backup size and duration.
• [Feature] Parallel Backups - Multiple backups can now run concurrently on the same hypervisor without conflict.
• [Feature] GPU Passthrough - PCIe GPU passthrough for compute-heavy workloads like ML training, rendering, and video encoding. Hypervisors automatically discover installed GPUs and show availability in the admin panel.
• [Feature] GPU Instance Plans - Instance plans can now specify a GPU count, enabling GPU-backed instance deployment through the standard creation flow.