5 posts tagged with "Backups"

Version v3.0.0 is a foundation release. The platform - both the master and the hypervisor agent - now runs on Laravel 13 on PHP 8.3, and this build adds a broad round of precision and edge-case hardening across the systems that run quietly in the background. It also ships two ways to drive the platform programmatically: the Hypervisor.io OpenTofu / Terraform provider, so your instances, networks, and clusters can be managed as code, and a remote Model Context Protocol (MCP) server, so AI agents can operate the same account conversationally. Nothing changes for your existing instances, plans, balances, or configuration.

• [Feature] OpenTofu / Terraform provider - manage your account as Infrastructure as Code. The iaas provider (now released at v0.2.1, source hypervisor-io/iaas) exposes 56 resources and data sources - instances, VPCs and subnets, Kubernetes clusters, managed databases, load balancers, storage, DNS, VPN, S3, and more - over the user REST API, with full CRUD, import, and reference-driven dependencies. Authentication is your existing IP-locked API token, and both users and admins can use it. See the section below.
• [Feature] MCP server - hand your account to AI agents. A remote, stateless Streamable HTTP Model Context Protocol server exposes the platform as 364 tools (302 user + 62 curated admin) over the same REST API, with Bearer API-token pass-through auth, confirm-gated destructive operations, idempotency keys, and async convergence. Any MCP client can drive it. See the section below.
• [Platform] Laravel 13 across the stack - master and the hypervisor agent both move to Laravel 13 on PHP 8.3, on stable, security-audited dependencies. A pure runtime modernization; your data and settings are untouched.
• [Improvement] Billing precision - hourly billing is refined for long-running resources, unusual calendar edge cases, storage metering cadence, and exact fractional-credit accounting.
• [Improvement] Backup & restore robustness - incremental chains, in-place volume restore, and retention pruning were hardened for edge cases across Ceph and block backends.
• [Improvement] Migration & networking polish - cold and live migration edge cases, reverse DNS for uncommon IPv6 forms, upload/download rate-limit symmetry, and quota handling under concurrency were all tightened.
• [Improvement] Metrics accuracy - the usage pipeline that feeds billing now handles counter-reset edge cases more precisely.

Infrastructure as Code (OpenTofu / Terraform)​

v3.0.0 introduces the Hypervisor.io OpenTofu / Terraform provider - the whole platform, declared in HCL and converged with tofu apply.

You write the resources you want; the iaas provider translates them into calls against the user REST API using your existing IP-locked API token, and OpenTofu tracks the state. It covers 56 resources and data sources - instances, VPCs and subnets, security groups, Kubernetes clusters and node pools, managed databases, load balancers, storage volumes, DNS, VPN gateways, S3 buckets, projects, and the catalog data sources you reference for plans, images, and regions. Every resource supports full create / read / update / delete, tofu import <addr> <uuid> to adopt existing infrastructure, and normal Terraform references so one resource can depend on another.

Both users and admins can use it: a user manages their own resources with a user token; an operator uses an admin-scoped token for the broader surface. Because the token is validated against the IP it was registered with, run tofu from a stable egress IP (a CI runner, bastion, or workstation).

The provider is released at v0.2.1. Declare it with source hypervisor-io/iaas and let OpenTofu fetch it:

terraform {
  required_providers {
    iaas = {
      source = "hypervisor-io/iaas"
    }
  }
}

• Provider and full documentation: the repository is github.com/hypervisor-io/terraform-provider-iaas and the reference docs walk through getting started, the resource catalog, and common patterns.
• Publishing to the public OpenTofu / Terraform registry is being finalized; that hypervisor-io/iaas source is the install path once it lands, and you can build the provider from the repository in the meantime.

AI agents (Model Context Protocol server)​

v3.0.0 also ships the Hypervisor.io MCP server - the same platform, exposed to AI agents. Where the OpenTofu provider is the declarative path, the MCP server is the conversational one: point any MCP client at it and an agent can create instances, assign VPCs, and manage the rest of your infrastructure in natural language.

It is a remote, stateless Streamable HTTP server, so there is nothing to install locally - connect over HTTP and authenticate with a Bearer API token that is passed straight through to the platform, IP-locked exactly like the provider's. It exposes 364 tools - 302 user tools plus 62 curated admin tools - covering instances, VPCs and subnets, Kubernetes, managed databases, load balancers, storage, DNS, VPN, S3, and the catalog lookups agents need to reference plans, images, and regions.

Because agents act on their own, the server is built to be safe by construction:

• Confirm-gated destructive operations - deletes and other irreversible actions require an explicit confirmation step before they run.
• Idempotency keys - a retried call does not double-create, so a flaky connection cannot spawn duplicate instances.
• Async convergence - long-running operations return a task the agent can poll to completion instead of blocking.
• A curated admin allowlist - the 62 admin tools are a deliberately safe subset; billing, user deletion, and hypervisor destruction are not exposed.

The server is backed by the same tested Go client as the OpenTofu provider, so both consumers reach the API through one audited code path.

• Server and full documentation: the repository is github.com/hypervisor-io/iaas-mcp-server and the reference docs cover connecting a client, authentication, and the tool catalog.

The user / admin REST API is the single source of truth for all three surfaces - the API itself, the OpenTofu provider, and the MCP server - and a manifest-driven CI gate keeps them in lockstep, so no endpoint can ship without matching provider and MCP coverage.

Upgrade Notes​

Stable release - stage first and take a fresh master backup. This crosses a framework generation, so clear compiled views after deploying and pin your cache and session key prefixes; database changes run automatically and nothing existing is rewritten.

Version v2.2.6 makes local storage a first-class, pluggable layer. Until now a local instance disk meant a qcow2 file. This release adds four block-device backends - LVM thin, LVM thick, ZFS thin, and ZFS thick - that slot in alongside qcow2, Ceph, and NFS with the full lifecycle: deploy, resize, snapshots, incremental backups, restores, image creation, and live migration. Around them sit three operator-facing safety features: a guided storage wizard that probes the hypervisor before anything is saved, a collapse watchdog that suspends only the at-risk instances before a thin pool can corrupt, and a live Storage Health dashboard. The Virtualizor importer now lands migrated VMs straight onto LVM/ZFS with no qcow2 conversion step, and the admin panel picked up a round of polish along the way. This is a beta build - feature-complete for the storage work and safe to stage, but not yet promoted to stable.

• [Feature] LVM and ZFS Storage Backends - Hypervisor storage now supports four new local pool types alongside qcow2 files, Ceph, and NFS: LVM thin, LVM thick, ZFS thin, and ZFS thick. Instances on these pools run on raw block devices for near-native disk performance, with the thin variants providing over-provisioning and the ZFS variants bringing checksumming and inline compression. Plans can target a specific backend, so an operator can sell "NVMe block storage" and "standard storage" tiers from the same hypervisor.
• [Feature] Guided Storage Wizard - Adding a storage pool is now a five-step wizard that probes the hypervisor live before anything is saved: it verifies the tooling is installed, the volume group or pool exists, and reports real capacity, so a mistyped pool name can no longer create a broken storage row. LVM and ZFS appear in the picker with a Beta tag for this release.
• [Feature] Storage Collapse Watchdog - Every hypervisor now watches its pools every 30 seconds. When a thin pool or ZFS pool approaches exhaustion, the platform suspends only the affected instances before their disks corrupt, alerts every administrator by mail, and surfaces the event live in the Storage Health dashboard. Recovery is operator-controlled: fix the pool, then resume the instances from where they stopped.
• [Feature] Storage Health Dashboard - A per-pool health view in the admin panel shows live data and metadata usage, ZFS pool state and fragmentation, and warn/critical thresholds, fed by the same probes the watchdog uses, surfaced inline on the storage list.
• [Feature] Native LVM/ZFS Virtualizor Import - The Virtualizor importer no longer forces every disk through a qcow2 conversion. When the source VM lives on LVM or ZFS, the slave probes the pool and the importer lands the instance natively on the matching backend, preserving the storage tuple and skipping the conversion pass entirely.
• [Improvement] Incremental Backups Everywhere - The QEMU dirty-bitmap incremental backup pipeline now covers all seven backends, including raw block devices, writing the same qcow2 backup chains to your existing backup storage. Existing qcow2 and Ceph backup behavior is unchanged.
• [Improvement] Live Migration for Block Pools - Live migration works between LVM/ZFS pools across hypervisors, pre-creating the destination volume and block-copying storage as part of the move.
• [Improvement] Admin Panel Polish - The VPN Gateway, Managed Database, and Load Balancer plan create/edit pages were rebuilt on the shared sectioned-card layout used by Instance Plans (and a latent dropdown bug on those pages was fixed), more list pages gained the standard collapsible filter panel, the in-browser SSH terminal now stays connected for as long as the tab is open, and the frontend now builds through Vite.

LVM and ZFS Storage Backends​

Until now local instance disks meant qcow2 files. v2.2.6 adds four block-device backends - LVM thin, LVM thick, ZFS thin, and ZFS thick - that slot in alongside the existing qcow2, Ceph, and NFS options with the full feature set: deploy, resize, snapshots (thin pools and ZFS), incremental backups, restores, image creation, and live migration. Because plans can target a specific backend, an operator can offer differentiated storage tiers from a single hypervisor.

Pools are added through the new guided wizard, which talks to the hypervisor before saving: it checks that lvm2 or zfsutils are installed (both are installed when you update the hypervisor from the admin panel), that the volume group, thin pool, or zpool actually exists, and shows live capacity. Existing storage rows, plans, and instances are untouched - qcow2, Ceph, and NFS behave exactly as on 2.2.5, and nothing is migrated or rewritten on upgrade.

The Collapse Watchdog​

Thin provisioning's dark side is pool exhaustion: when an over-provisioned pool fills up, every VM on it can corrupt at once. v2.2.6 ships a two-layer defense. First, LVM thin pools are configured to auto-extend at 80% via dmeventd. If that cannot help - the volume group is out of space, or a ZFS pool faults - a watchdog on each hypervisor catches the threshold crossing within 30 seconds, pauses only the instances on the affected pool, mails every administrator, and raises a live alert in the Storage Health dashboard. Paused instances keep their RAM and disk state; once the operator extends the pool or replaces the failed device, the instances resume from where they stopped. Degraded ZFS mirrors raise an early warning while the instances keep running.

Storage Health Dashboard​

The Storage Health view in the admin panel shows per-pool data and metadata usage, ZFS pool state and fragmentation, and warn/critical thresholds - fed by the same probes the watchdog uses and broadcast live over WebSocket. In this release the health values are surfaced inline on the storage list, so pool state lives in one place.

Native LVM/ZFS Virtualizor Import​

Migrating off Virtualizor no longer means rewriting every disk to qcow2. When a source VM lives on LVM or ZFS, the slave probes the pool and the importer lands the instance natively on the matching backend, preserving the storage tuple and skipping the conversion pass entirely. qcow2 sources continue to import as qcow2.

Upgrade Notes​

This is a beta build. Stage it first - take a fresh master backup and snapshot a test hypervisor before upgrading.

1. Run database migrations. They add the storage-backend columns, plan subtype, and health-tracking fields; NFS pools are marked as shared storage automatically.
2. Update each hypervisor from the admin panel. The hypervisor update installs lvm2, thin-provisioning-tools, and zfsutils-linux, enables LVM thin auto-extension, and starts the storage watchdog.
3. Rebuild the admin frontend bundle for the storage wizard and Storage Health dashboard, and restart the WebSocket broadcaster so storage-health events are registered.

Nothing in this release rewrites existing qcow2, Ceph, or NFS storage - those pools, their plans, and their instances are untouched.

Known Beta Notes​

• LVM and ZFS appear with a Beta tag in the storage wizard; exercise them on a test hypervisor before offering them to customers.
• Pool auto-extend depends on free space in the underlying volume group or zpool - the watchdog is the safety net, not a substitute for capacity planning.
• Found a problem? Report it referencing v2.2.6-beta along with the hypervisor's storage type and the relevant Storage Health alert.

Version v2.2.3 is a major feature release headlined by Managed Kubernetes, a fully integrated Kubernetes-as-a-Service offering that runs alongside Instances, Volumes, Load Balancers, and Databases. Customers can spin up a control plane (single-node or 3-node HA), attach workers in per-purpose pools, expose Kubernetes Services through the bundled in-cluster cloud controller manager, autoscale workloads end to end with the cluster autoscaler, and roll the cluster forward to a newer Kubernetes version, all without touching the slave host. The release also ships a redesigned master backup pipeline with pluggable storage drivers and Grandfather-Father-Son retention, a new System Health dashboard widget, scheduled task health tracking, an app-wide timezone setting, team-member permissions for Kubernetes resources, retry for failed cluster creates, and a long list of reliability and performance improvements including a 70% reduction in peak load for the hot-path cron loop that runs every 30 seconds against the entire fleet.

• [Feature] Managed Kubernetes - Create production-grade Kubernetes clusters directly from the control panel. Choose single-node or 3-node HA control plane, pick instance plans and subnets for control plane and workers separately, and bring up the cluster with a bundled HAProxy load balancer for the Kubernetes API. Real-time progress streams to the cluster show page via WebSocket; downloaded kubeconfig points at the right private or public endpoint automatically.
• [Feature] Worker Node Pools - Each cluster has a default worker pool and supports unlimited additional pools, each with their own instance plan, labels, taints, autoscaling bounds, and drain settings. Useful for GPU nodes, memory-optimized workloads, or isolating tenants in a single cluster.
• [Feature] Cluster Autoscaler - Bundled cluster autoscaler binary speaks the Hypervisor API directly. Policy-driven scaling on CPU + memory pressure of pending pods, per-pool aware, and respecting each pool's min/max bounds. Manifest generated on demand from the cluster show page, customers grab the YAML and apply with kubectl apply -f -. Controller token refreshes on a rolling schedule so long-lived clusters never need a manual re-issue.
• [Feature] In-Cluster Cloud Controller Manager - Services of type LoadBalancer provision and tear down a real Hypervisor load balancer per service. Service annotations control listener port, backend mode (TCP / HTTP / per-port hybrid), session stickiness, multi-cert SNI, routing rules, and traffic split between subset endpoints.
• [Feature] Worker and Control Plane Rolling Upgrades - Upgrade Workers card on the Workers tab provisions new workers at the target version, drains old ones, repeats. Upgrade Control Plane card does the same for CPs via surge-replace strategy, etcd-quorum-safe at every step. Cluster card shows CP version and worker baseline as two distinct lines with a "mid-upgrade" badge when they diverge.
• [Feature] Retry Failed Cluster Create - A new "Retry create" button on the cluster page tears down partially provisioned artifacts and re-runs the bootstrap on the same row. Cluster name, slug, and identity certificates are preserved so any kubeconfig the user already downloaded stays valid. No more delete-and-recreate after a transient quota or capacity precondition fails.
• [Feature] Cluster Security Groups - Three auto-managed security groups per cluster (LB-only, CP-only, worker-only). Default rules expose the Kubernetes API on :443 via the LB and lock down direct access to CP nodes' :6443 from outside the cluster. Admins and users layer additional rules through a familiar Inbound / Outbound sub-tabbed interface.
• [Feature] Restricted Kubeconfig - Downloaded kubeconfig issued at cluster create exposes only worker nodes to kubectl get nodes. Control-plane VMs are hidden from end users in the Compute list, billing reports, monitoring tiles, and the cluster Nodes tab.
• [Feature] Master Backup Pipeline Redesign - Service-oriented orchestrator with pluggable storage drivers (Local, S3-compatible, Rsync over SSH, NFS), a singleton lock that survives long uploads, Grandfather-Father-Son retention, email + webhook notifications, and a configurable cron expression. Multiple destinations supported per install. Admin pages cover Destinations, Runs, Settings, and Scheduler Health.
• [Feature] Scheduled Task Health Tracking - Every scheduled task is observed via a unified health surface. Per-task tracking of last run, duration, exit code, and consecutive failures. Compact admin Scheduler page with a slide-in drawer per task, friendly task names, and a daily prune to keep the audit table compact. Dashboard tile shows healthy / degraded / failed scheduled-task counts at the top of every page.
• [Feature] System Health Dashboard Widget - Single compact strip on the admin dashboard showing four critical metrics at a glance: most recent successful master backup, scheduled-task health rollup, in-flight long-running tasks, and queue worker failed-job count. Replaces two separate tiles from earlier releases.
• [Feature] Application Timezone Setting - Pick any IANA timezone from a new dropdown under Admin > System > Settings > General. Applied app-wide on boot (Carbon, model date casts, scheduler firing times, direct PHP date functions). Default for customers signing up via self-registration and billing-API user creations, unless explicitly overridden. Existing users keep their own timezone selection.
• [Feature] Kubernetes Team Permissions - New kubernetes.* permission family with three tiers (view, manage, delete) granted through the existing team-member invitation flow. Predefined roles get sensible defaults from the migration. Custom roles need to be granted the new permissions explicitly.
• [Feature] Admin Destructive Controls for Clusters - Dedicated section for safe escape hatches when a cluster has gone wrong. Suspend locks out the customer while preserving forensics. Reset State clears stuck-operation flags. Force Cleanup bypasses normal teardown for clusters with zombie resources. Separate rate limits keep destructive (5/hour) and recovery (20/hour) actions distinct.
• [Feature] AWS-Style Node Drill-Down - Clicking a node on the cluster Nodes tab opens a side drawer with capacity gauges (CPU / RAM / disk), pod listing with pagination and search, taints section, and modern dark/light surface styling.
• [Feature] Cluster-Managed Resource Lockdown - Worker instances and the CP load balancer carry a clear "Cluster-managed" badge and a read-only banner in the user's Compute and Load Balancers lists. Direct power cycle, plan change, or LB rule edit is blocked at the controller. Manage them through the cluster page instead.
• [Feature] Live Load Balancer Filtering - User-side Load Balancers index now supports AJAX live filtering by name, status, and VPC. Useful for customers running dozens of LBs across multiple VPCs.
• [Feature] Cluster Activity Feed - User dashboard activity feed now translates Kubernetes audit-log actions into friendly sentences ("Created cluster prod-01", "Upgraded workers to 1.35.0") alongside the other resource types.
• [Feature] Pre-Flight Quota and Capacity Guards - Cluster create form rejects at submit time when load balancer quota is exhausted, when the chosen VPC has no NAT Gateway (needed for control-plane image pulls), or when the CP subnet is not private. Clear messages name the limit and point at the affected field instead of failing deep in the bootstrap chain.

Version v2.2.2 is a major feature release. Headline additions: Volume Snapshots and Backups (on-storage rollback marks and disaster-recovery exports for block storage volumes), Orphan VM Import (adopt KVM domains that already exist on a hypervisor as Hypervisor-managed instances without downtime or disk migration), and User Self-Registration with built-in CAPTCHA support for Cloudflare Turnstile and Google reCAPTCHA v2 / v3, signed email verification, and a post-verification onboarding top-up flow. Alongside the headliners: a comprehensive overhaul of the instance backup experience, a redesigned Forge tab, a polished tasks progress view, and the carry-over polish from earlier 2.2.x - RustFS as the new self-hosted object storage default, a fully reworked hypervisor self-update flow, and richer AI Assistant diagnostics for admins.

• [Feature] Volume Snapshots and Backups - Block storage volumes now have a dedicated Snapshots & Backups tab on every volume detail page. Take instant on-storage snapshots for rollback, or export full backups to your existing remote backup storage (S3-compatible, NFS, or local) for disaster recovery. Both support in-place rollback and restore-to-new-volume modes.
• [Feature] Per-Volume Safety Controls - Volume operations enforce per-volume serialization (one operation at a time), a 5-minute cooldown between operations, per-plan caps on retained snapshots and backups, and a per-user inflight limit so a single user cannot saturate the queue.
• [Feature] Volume Backup Billing - Volume plans now expose per-GB / per-month credit pricing for retained snapshots and retained backups, plus configurable caps on how many of each a customer can keep on a single volume. Defaults bill nothing for retention but allow up to 5 snapshots and 10 backups per volume.
• [Feature] Orphan VM Import - A new admin workflow on every hypervisor manage page surfaces KVM domains that exist on libvirt but are not tracked by Hypervisor - for example, VMs from before the slave was attached, or VMs created out-of-band. Eligible (qcow2 file-backed) domains can be adopted as Hypervisor-managed instances in place. MAC, IPs, and disk paths are preserved; per-disk storage assignment in the import dialog means no mv of qcow2 files.
• [Feature] User Self-Registration with CAPTCHA - A complete public signup pipeline with three CAPTCHA providers (Cloudflare Turnstile, Google reCAPTCHA v2, Google reCAPTCHA v3), per-form toggles for Login and Register, signed email-verification links, a Verify-Your-Email page that auto-redirects via WebSocket the moment the user clicks the link in another tab, and an optional post-verification top-up onboarding modal.

We're excited to announce Version v2.2.1 of the Hypervisor Control Panel! This release completely rebuilds the instance backup experience with support for remote backup destinations and adds GPU passthrough for compute-heavy workloads.

You can now back up your instances directly to Amazon S3, RustFS, Wasabi, Backblaze B2, DigitalOcean Spaces, SFTP servers, FTP servers, or any rclone-supported backend -- no need to mount backup storage locally on each hypervisor.

• [Feature] Remote Backup Destinations - Configure S3, SFTP, FTP, or rclone-based backup targets from the admin panel. Credentials are stored encrypted and pushed to hypervisors only when needed.
• [Feature] Streaming Backups - Full and incremental backups stream directly to the remote destination. Restores stream back the same way. No temporary disk space required.
• [Feature] Running-VM Backups Without Downtime - Backups of running VMs use external snapshots so the VM continues running normally while the backup completes. Changes made during the backup window are merged back seamlessly.
• [Feature] Efficient Incremental Backups - Incremental backups capture only the blocks that changed since the last backup, dramatically reducing backup size and duration.
• [Feature] Parallel Backups - Multiple backups can now run concurrently on the same hypervisor without conflict.
• [Feature] GPU Passthrough - PCIe GPU passthrough for compute-heavy workloads like ML training, rendering, and video encoding. Hypervisors automatically discover installed GPUs and show availability in the admin panel.
• [Feature] GPU Instance Plans - Instance plans can now specify a GPU count, enabling GPU-backed instance deployment through the standard creation flow.