5 posts tagged with "Kubernetes"

Version v3.1.7 opens the platform's telemetry to your customers: Metrics Export adds Prometheus-format scrape endpoints on the user API, so a customer can point their own Prometheus or Grafana at the panel and chart their instances, managed databases and load balancers with the API token they already have. The second headline is the Import Doctor: virtual machines imported from Hyper-V, VMware or other platforms - which typically blue-screen on first boot because their disk drivers were never armed for our hardware - are now detected automatically and repaired with one click. Around them, a Fix Monitoring action repairs broken telemetry on managed services in one step, Kubernetes plan rotation picked up the fixes from its first weeks in the field, and the user panel's list pages were made consistent end to end.

Metrics Export​

• [Feature] Prometheus Scrape Endpoints - GET /api/metrics returns every instance, managed database and load balancer the account owns as one Prometheus text exposition, with per-resource endpoints alongside it for narrowly scoped scrape jobs. Authentication is the same bearer token used everywhere else on the user API - there is no separate credential to mint. Metric families are stable and customer-facing (hv_instance_*, hv_db_*, hv_lb_*), with resource id and name labels and no internal topology in the output. Instances include Kubernetes worker nodes and VPN gateway backing VMs.
• [Feature] Built for Scrapers, Not Browsers - The endpoints answer HTTP 200 always: a backing store being unreachable surfaces as hv_resource_up 0 on the affected resources, never a 5xx that turns into an error storm in the customer's Prometheus. Scrapes are rate limited per token at 30 per minute, the account-wide endpoint caches its rendered body for 10 seconds to absorb multi-target fan-out, and the whole call runs under a hard time budget so one slow backend cannot stall the response. Subuser tokens see exactly the resources their team permissions grant, nothing more.
• [Feature] Documentation and Examples - The user API documentation gained a Metrics Export section with the full endpoint table and a ready-to-paste scrape_config snippet, and a feature guide ships with this release. The pipeline was validated end to end against a real Prometheus and Grafana stack before release.

Imported Virtual Machines​

• [Feature] Import Doctor - Customers who convert a VHDX or VMDK, write it over their instance disk and boot typically hit INACCESSIBLE_BOOT_DEVICE: installing the virtio drivers inside Hyper-V or VMware stages them but never arms them for boot, because that hardware never presents a virtio disk. The platform now fingerprints every instance disk after each cold start, and when a foreign operating system appears, the instance page shows a banner on both the user and admin panels. One click repairs it offline: the staged virtio storage drivers are armed directly in the guest's registry, stale UEFI boot entries are reset, and the firmware type is matched to what the disk actually uses. Guests where the full repair is not possible fall back to compatible SATA and e1000 emulation so they boot regardless. Detection is automatic; repair only ever runs when someone asks for it, with the guest shut off.
• [Fix] UEFI Firmware on Debian Hypervisors - UEFI guest definitions probed only the Red Hat and legacy Ubuntu OVMF firmware layouts, so on Debian hypervisors (and newer Ubuntu, which drops the legacy names too) every UEFI define failed with a missing-file error until symlinks were made by hand. Firmware is now resolved from ordered CODE/VARS pairs covering all three layouts, requiring both halves of a pair so a partial install can never mix incompatible images.

Monitoring and Managed Databases​

• [Feature] Fix Monitoring - Managed databases and load balancers gained a Fix Monitoring action that repairs the telemetry pipeline in one step: the metrics agent configuration is re-rendered and re-installed, the metrics exporter login and its grants are re-created, and the agent is restarted. It is available on the metrics tab in both panels - once per resource per 24 hours on the user side, unthrottled for admins - and a failed attempt never consumes the user's daily budget. A new monitoring:fix console command batches the same repair across every active managed service, for fleet-wide rollout after an endpoint or credential change.
• [Fix] Customer-Created Read-Only Roles - On PostgreSQL, the customer admin account could create a role but not grant it anything useful: granting membership in the monitoring and read-all roles failed with a permission error, so a read-only or Grafana login ended up empty. On MySQL and MariaDB the grant capability had been removed outright. The admin account now carries the delegation rights it needs on both engines - scoped so the hardening that removed superuser-level capabilities stays in force - and the Fix Monitoring action applies the same correction to existing databases.
• [Fix] Honest Instance Memory Graphs - Instance memory usage was derived from the guest's free-memory figure, which page cache drains toward zero on any warm Linux guest, so memory graphs crept toward 100% regardless of real pressure. The calculation now uses the guest's available-memory figure, which counts reclaimable cache as free - the same number free -m shows in its available column. Guests with older drivers keep the previous behaviour.

Kubernetes​

• [Feature] Cluster API Endpoint by Name - New public clusters serve their API endpoint by their System DNS name (k8s-<name>-cp.<your-domain>) instead of a bare IP: the name is baked into the API server certificate at bootstrap, the cluster page shows it, and generated kubeconfigs use it. The name is pinned at creation and never rewritten, since it lives inside a signed certificate distributed to customers. Existing clusters are untouched and keep working by IP.
• [Feature] Webhook Subscriptions API - The user API gained full management of webhook subscriptions (/api/webhook-subscriptions): create, list, update, delete, and per-subscription delivery history. Deliveries are HMAC-signed with a per-subscription secret, endpoints must be HTTPS, and subscriptions can filter to a single cluster. Kubernetes lifecycle events are the first event source.
• [Improvement] Plan Changes Lead Into Rotation - After changing a pool's instance plan, nothing pointed at the rotation that actually resizes the workers, so pools sat reporting drift until someone found the unlabeled icon. The edit form now says what saving will and will not do, a plan-changing save flows directly into the rotation dialog (including the separate downsize confirmation where it applies), and the drift badge itself became a clickable "Rotate now" action. Declining the dialog simply leaves the badge as the reminder.
• [Fix] Rotation Waves Apply Labels and Taints - Workers added by a plan rotation joined without their pool's labels and taints and showed no role in kubectl get nodes. Each rotation wave now applies the pool's labels and taints after the new workers are ready and before the old ones drain - exactly the moment draining reschedules pods onto them - and the role label appears promptly instead of up to a day later.
• [Fix] Rotation Dispatch Self-Heals - A rotation is launched as a detached background process, and that launch can die silently. A rotation still pending after three minutes is now redispatched once by the periodic janitor - rotations are resume-safe by design, and a healthy run is never touched - and every detached launch now leaves a forensic log so a failed spawn is diagnosable rather than a mystery.
• [Fix] Pool Deletion Cannot Strand Workers - Deleting a node pool could remove the pool record while its workers were still protected by scale-down guards, leaving live worker VMs attached to a deleted pool where no cleanup process could reach them. Pool deletion now marks every member for collection with those guards bypassed - they exist to protect a pool that is staying - and refuses to remove the pool record if any live member could not be marked. The cleanup and drain paths also learned to resolve a deleted pool's configuration, so members marked before the deletion still drain correctly.
• [Fix] Placement on Heavily Committed Nodes - A node whose allocated memory exceeded its physical total crashed the placement query for its whole group with a database range error, aborting scale-ups that had healthy capacity elsewhere. Placement and the rotation capacity gate now both measure headroom against the node's effective memory ceiling - the operator-set overcommit limit where one is configured, physical memory otherwise - so a sanctioned overcommit node counts its real headroom and an over-allocated one simply sorts last instead of taking the group down. The placement query also gained the standard maintenance, lock and deployment gates it was missing.
• [Fix] Certificate Auto-Renewal Now Scheduled - The cluster certificate renewal service shipped fully built but nothing ever ran it. It now runs daily, renewing certificates inside a 30-day window ahead of expiry.

Networking​

• [Fix] Private DNS on Dual-Homed Instances - On instances with both a public IP and a VPC interface, the first lookup of a private DNS name could stall for seconds: the VPC link shipped without its search domains, so the resolver had no reason to route private-zone queries to the VPC resolver and raced it against the public nameserver. The deploy payload now carries the VPC's DNS zones as search domains. Already-deployed guests pick this up on their next network configuration rebuild.
• [Fix] Private DNS on VPC-Only Instances - VPC-only instances could lose private-zone resolution entirely: a fallback public nameserver on the same link could permanently win the resolver's affinity, and the VPC's own resolver was not even running until the VPC had at least one zone. The gateway resolver now always runs - from VPC creation, zones or not - and VPC guests use it exclusively on that link, with public resolution forwarded upstream through it.

User Panel​

• [Improvement] Consistent Filtering Everywhere - The service list pages were brought up to one standard: URL-shareable filter state, status and facet filters, and debounced server-side search. Managed database filters that the server always supported are now in the UI; VPN gateway and scaling group search actually filters instead of doing nothing; volume search no longer breaks the table; pagination keeps your filters instead of dropping them. Columns that were already on the wire but never rendered are now shown - private IPs for databases and VPN gateways, location and plan for instances, distribution and region for images.
• [Improvement] Capacity Errors Reworded for Customers - When a deploy or scale-up fails for a capacity reason - no memory, storage or IP headroom where the resource was requested - the customer now sees a clear "contact support" message instead of raw infrastructure wording that named things they cannot see or fix. Admins are emailed the precise original error (throttled per distinct error), see it unchanged in the admin panel, and it is always logged.
• [Improvement] AI Assistant Out of Beta - The AI assistant has run long enough in production to drop the beta label. The badge and the settings-page warning are gone; nothing about its configuration changes.

Billing and WHMCS​

• [Feature] Create-Time Add-Ons - External billing provisioning now accepts additional IPv4 addresses (up to 20) and additional disk (up to 5000 GB) at instance creation. The extra disk is placed on the same storage pool as the primary and the capacity check covers the combined footprint. The WHMCS module exposes both as configurable options; they apply at creation only, by design.
• [Improvement] Hardened WHMCS Module - The module now keeps its own service-to-instance link table (auto-migrated, self-healing from the custom fields it also auto-creates), so suspend, terminate and upgrade no longer depend on an admin having manually created a custom field. Re-running CreateAccount on a linked service refuses to mint a duplicate instance, and terminating a service whose instance is already gone converges cleanly instead of failing forever. The client-area overview page was redesigned to match the panel, with live data, copyable IPs, and one-click SSO into the panel.
• [Fix] Server Host Resolution - A WHMCS server saved with only the IP Address field filled (Hostname blank) could never load plans or hypervisor groups into product configuration. All module variants now resolve hostname-or-IP consistently, honour the configured port, tolerate a pasted URL, and fail with a named message when both fields are empty.
• [Improvement] TLS Posture Made Explicit - The billing API client does not verify the master's TLS certificate, because masters are routinely addressed by IP or carry self-signed certificates and verification would break provisioning on those installs. This trade-off is now documented rather than implicit: where possible, point WHMCS at a hostname with a publicly trusted certificate.

Platform and Admin​

• [Fix] Route Debt Sweep - An inventory pass over every registered route closed a set of long-standing gaps: the kubeconfig acknowledgement button now works, users can delete their object storage access keys (the endpoint existed but was never routed), several links that led to pages that do not exist now redirect to the real pages, and two admin route groups gated on permission slugs that could never be granted are now grantable. Dead controllers, models and page stubs were removed.
• [Fix] Volume Attach Validation - Attaching a volume now verifies the target instance belongs to the same account and derives the instance's hypervisor group correctly (the previous check read a field instances do not have, and failed for every attach). Cluster-managed workers are rejected as attach targets, and subusers now see their account's volumes and instances on the volume pages.
• [Fix] Web SSH Stability - Web SSH sessions on remote or CDN-fronted masters appeared to disconnect frequently: overlapping readiness polls could each open a connection against a one-time token, and a rejected duplicate would paint a disconnect overlay over the live terminal. Polling is now single-flight and a stale socket can never steal the display from a live one.
• [Fix] Encrypted Secret Storage - Columns storing encrypted provider secrets were widened; a secret over 23 characters could previously be truncated at rest.
• [Improvement] PowerDNS Onboarding - Adding a System DNS domain whose zone already exists in PowerDNS now adopts the zone instead of failing, the delegation checker installs its DNS tooling where missing, and a zone error clears automatically once resolved. The wizard's guidance on public-suffix domains was corrected.

Upgrade notes​

• Two migrations run on upgrade: the imported-OS state column and the encrypted secret column widening.
• Deploy the master before the hypervisor agents for the VPC DNS fixes; each side tolerates the other being old. The agent release carries the memory metric fix, the Debian UEFI firmware resolution and the Import Doctor tooling - the agent installs its guest-inspection packages during provisioning or update.
• The read-only role grant fix applies to newly provisioned databases automatically. For existing databases, run php artisan monitoring:fix --type=db once (or use the Fix Monitoring button per database) to roll it out.
• Metrics Export needs no setup: it uses the metrics backends you already configured per hypervisor group. See the new Metrics Export guide for the customer-facing details and a scrape configuration example.

Version v3.1.6 is a large release across Kubernetes, networking and databases. The headline is node-pool plan rotation: changing a pool's instance plan used to be rejected outright while the pool had live workers, so the only way through was to scale to zero and back, a full capacity outage for that pool. Now the plan change is accepted and the platform rotates the workers for you, provisioning replacements before draining anything. Alongside it, System DNS turns provisioning into something that produces real hostnames rather than bare IP addresses, and managed databases on those names present publicly trusted TLS certificates that renew themselves. Load balancers gain host-based routing and can serve internal VIPs for Kubernetes Services. The release also carries a broad set of reliability fixes across Kubernetes operation locking, reverse DNS, backups and the admin panel.

Kubernetes​

• [Feature] Node-Pool Plan Rotation - A pool's instance plan can now be changed while it has live workers. Saving the change starts a surge rotation: new workers on the new plan are provisioned and confirmed ready first, then old ones are cordoned, drained through an escalating ladder, and destroyed, one wave at a time, so the pool never runs below capacity. Scale-down ordering is preserved across the rotation, so the pool does not scramble which node leaves next. Downsizing to a smaller plan is allowed with an explicit warning rather than blocked - it is your cluster - but it takes a second, separate confirmation naming the consequence, so a single click can never start one.
• [Feature] Rotation Visibility - The pool list shows a drift badge counting how many workers are still on the previous plan, with a Rotate action to start or resume a rotation. The decision data arrives with the page rather than from a probe request, so opening the page cannot consume the rate limit that governs the rotation endpoint itself.
• [Feature] Worker Pool Identity - Worker nodes now carry a hypervisor.io/node-pool label, following the same convention as the major managed Kubernetes providers, so kubectl get nodes -L hypervisor.io/node-pool shows which pool each node belongs to. Panel hostnames include the pool name too, so the instance lists in both panels no longer show several pools' worth of identically-shaped names. Renaming a pool heals the label on the next reconciliation. None of this costs anything in the database.
• [Feature] Internal Load Balancers for Services - A Kubernetes Service of type LoadBalancer annotated <prefix>internal: "true" now gets a private VIP inside the VPC instead of a public IP, and the cloud controller reports that private address back to the cluster. Internal services no longer have to be exposed publicly to be reachable, and no change to the cloud controller was required.
• [Improvement] One Transport for Long Orchestrations - Rolling upgrades, control-plane upgrades and plan rotations no longer run as queue jobs with a fixed ceiling. They run as detached console processes with a task row, an operation lock and a stale-task reconciler behind them, so a long rotation cannot be killed part-way by a queue timeout.
• [Fix] Kubernetes Operation Locking - Operation locks now correctly detect an already-held lock across the Redis client the platform ships, so concurrent scale, upgrade and rotation operations on one cluster are properly serialised. Lock acquisition also moved outside the surrounding database transactions, so a rolled-back operation releases its lock immediately instead of waiting for the timeout.

DNS​

• [Feature] System DNS - Every public instance, load balancer, managed database and Kubernetes control plane now receives a hostname automatically. Instances are named from their IP in the familiar cloud style (vm-203-0-113-7.cloud1.example.com); named resources use their own name (db-prod-mysql, lb-frontend, k8s-myapp-cp). Records are created at provision time, follow the resource if its IP changes, and are removed when it is destroyed. Where a reverse zone allows it, a matching PTR record is set so forward and reverse agree, and a value the customer set themselves is never overwritten. DNS problems never block or fail provisioning.
• [Feature] Delegation Wizard - Base domains are onboarded through a four-step wizard: enter the domain and nameservers, delegate at your registrar, then verify. Verification is real rather than advisory - the panel writes a random record into the zone and confirms both that your registrar delegates the domain and that public resolvers can see that record through the delegation, before the domain publishes anything. A daily re-check moves a domain that loses its delegation to a degraded state, where existing records keep resolving but no new resources are assigned to it, and restores it automatically when the delegation returns. A problem local to the panel, such as its own DNS tooling being unavailable, will never degrade your domains.
• [Feature] Trusted TLS for Managed Databases - A base domain can hold a Let's Encrypt wildcard certificate covering every name under it, issued over the DNS-01 challenge using the zone the panel already controls. Public managed databases receive it automatically and load it without a restart, so customers can connect with --ssl-mode=VERIFY_IDENTITY or sslmode=verify-full against the public trust store instead of trusting a self-signed certificate. One wildcard per domain keeps issuance well inside Let's Encrypt's rate limits, and renewal runs daily from 30 days before expiry, so a failed attempt has a month of retries behind it rather than being a countdown.
• [Feature] PowerDNS Deployment Kit - A self-contained kit ships with the release for operators who need authoritative DNS to point System DNS and reverse DNS at. It stands up a primary and any number of secondaries replicated by signed zone transfers, where a new zone propagates to every secondary without touching them. One script drives the whole fleet over SSH from the primary: it prints a readiness table per node and changes nothing until every node passes, then converges each node and verifies replication by querying each one directly, failing loudly if any node is not actually serving the zone.
• [Improvement] Reverse DNS Zone Forms - The reverse zone forms now show the hostname each automatic PTR format actually produces, built live from the prefix and domain you are typing, instead of four opaque option labels. Zone type is chosen from cards and the zone suffix follows the choice.
• [Fix] PowerDNS Reverse DNS - Corrected the call into the PowerDNS client when setting or rebuilding a PTR record, and added test coverage over that path. ClouDNS providers were not affected.
• [Fix] Automatic PTR Format Labels - The second and third automatic PTR format options in the reverse zone form now describe the output they actually produce, and the form previews the resulting hostname for each option. If you use format 2 or 3 on an existing zone, check the preview against what you expect before your next change.

Load Balancers​

• [Feature] Host-Based Routing - Load balancer rules can now match on the request host as well as the path, so one load balancer can serve several hostnames to different backends. The match-type options were consolidated at the same time.
• [Improvement] Self-Healing Configuration - A load balancer left stranded in the configuring state now recovers on its own instead of needing a manual sync, with the grace period derived from how long the agent-side configuration can legitimately take rather than an arbitrary number.
• [Fix] HTTPS Redirect and Form Wipes - The HTTPS-redirect option on port 80 now applies correctly, and a real-time update arriving while a load balancer configuration form is open no longer discards what you were typing.

Databases and Backups​

• [Feature] Detached Backup Execution - Managed database backups no longer run inside a single blocking connection to the guest. A backup taking more than about 58 minutes used to be killed mid-upload by a transport timeout that scaled with nothing, and the task showed no movement at all between "running" and completion, so a healthy 13 GB backup was indistinguishable from a hang. Backups now run detached with the panel polling progress, reporting transferred bytes live, and the ceiling is a policy you set rather than an artefact of how the command was run. A genuinely dead guest is now detected in about two minutes.
• [Feature] Backup Run-History Retention - Backup run records are now pruned on a schedule with a configurable retention period, so the history table stays a useful size on long-lived installs.
• [Improvement] Honest Backup Schedules - The backup settings page now shows the schedule actually in effect rather than a placeholder that could differ from it, and validates a custom schedule when you save it. New installs default to daily.
• [Improvement] Faster Failures on Broken Egress - The database agent scripts now fail immediately with a named reason when the guest has no route to the internet, instead of hanging until a timeout. The message points at the VPC NAT gateway, which is the usual cause.
• [Fix] PostgreSQL Incremental Backups - PostgreSQL incrementals are markers over continuous WAL archiving and upload no object of their own. The completion check now recognises that shape and records them correctly, and a marker whose WAL archiving is not actually running is reported as a failure rather than a success.
• [Fix] Phantom Restore Keys - A storage key recorded against those marker rows could make restore and retention treat an object that does not exist as downloadable. Marker rows no longer carry one.

Platform and Admin​

• [Improvement] Smaller Update Rollback Snapshots - The snapshot taken before an application update now skips logs, caches, images and existing backups, and skips walking those directories at all rather than listing and discarding them. Snapshots on a busy install were hundreds of megabytes of log files.
• [Improvement] Quieter Admin Lists - The VPC and load balancer list pages no longer reload on every NAT gateway heartbeat. Updates are batched, so a page that was reloading dozens of times a minute now settles.
• [Fix] Hypervisor Health Flag - The automatic health flag raised when a node stops reporting now clears again on the node's next successful metrics poll, so a node that recovers from a transient blip returns to the deployment pool by itself. Allow Deployments remains the operator-controlled switch, and the admin page now labels which is which.
• [Fix] Blank Task Status - Task status could arrive in a form the column could not store, leaving the dashboard blank for tasks that were running. The column has been widened and the accepted values are validated where they arrive.
• [Fix] Missing Hypervisor Uptime - KVM nodes showed "-" for uptime on the hypervisors list and the dashboard.
• [Fix] VPC Real-Time Updates - Aligned three broadcast channel names with what the panel subscribes to, so VPC views update live again.
• [Fix] Missing User Timezone Crashed Instance Metrics - Accounts created without a timezone caused the instance metrics endpoints to fail. Timezone resolution now falls back through the account, the system default and UTC in one place, existing accounts are backfilled, and a setting that nothing had ever written - so subuser invitations always fell back to UTC regardless of the configured default - now reads the correct one.
• [Fix] Supervisor Workers Pointed at a Missing PHP - The queue worker configuration hardcoded PHP 8.3 while the installers have defaulted to 8.4 for some time, so on a fresh install no queue worker would start, and every application update reapplied the mismatch. The PHP version is now substituted where the configuration is deployed, including the unversioned path that EL hosts use.
• [Fix] Build Pipeline Hardening - The release build now enforces the compatibility flag when targeting multiple PHP versions, records the flags used with each artifact, and adds a verification step plus a smoke test on the target host.
• [Fix] Billing and Provisioning Debt - Volume backup charges now apply the account balance policy consistently, user records created through every path carry the timestamps the schema requires, and VNC port allocation can no longer hand the same port to two instances.

Infrastructure Agent​

• [Improvement] PHP Entrypoint Pinning - The agent now selects its PHP interpreter explicitly, preferring 8.4 and falling back to 8.3, verifying the candidate actually carries the extensions it needs before committing to it. The update process repairs already-deployed nodes.
• [Fix] Proxy Survives an Unlinked Node - The SSH proxy no longer fails to start on a node that has not yet been linked to a master, and linking, relinking or unlinking now takes effect without a restart. Unlinking correctly revokes trust.
• [Fix] Stale Binaries After an Update - An incremental update could leave the previous version of a running binary in place, because a running executable cannot be overwritten in place. The update now replaces them correctly.
• [Fix] Missing VNC Port Tolerated - A null or zero VNC port arriving from the master no longer produces invalid guest XML or a firewall error.

Integrations​

• [Improvement] OpenTofu Provider and MCP Server - Both gained the node-pool rotation endpoint, and the MCP tool for updating a node pool gained the instance plan field it was missing, so a plan change and its rotation can be driven from infrastructure-as-code or from an AI agent as well as from the panel.

Upgrade notes​

• Deploy the infrastructure agent before the master. The certificate installation command the master sends reaches a route that only exists in the new agent.
• Three migrations run on upgrade: the task status column widening, the System DNS schema, and the user timezone backfill.
• Reverse DNS on PowerDNS providers is worth a quick check after upgrading, and if you use automatic PTR format 2 or 3, confirm the form's preview matches the naming you expect.

Version v3.1.5 closes out the 3.1 line. It adds no new product surface - it makes the surface added in v3.1.0 and v3.1.2 behave consistently. The admin REST API now matches the panel and its own documentation, load balancer routing and Proxmox firewall synchronization are more dependable, alert mail is delivered on its own queue, and every instance operation is fully localized. A platform review ran alongside this work; its outcomes are folded into the items below.

Upgrading is routine. There is nothing to reconfigure and no behavior to relearn.

• [Improvement] Admin REST API consistency - updating a user no longer requires resending a password, the acting administrator resolves the same way on every endpoint, and account credentials are excluded from responses. Load balancer security group rules are readable from the administrative API.
• [Improvement] Load balancer routing rules honor catch-all matches, applied after the specific rules and before the default backend, so precedence follows the order the rule list reads.
• [Improvement] Proxmox security group synchronization converges reliably, so firewall rule changes reach running VMs on every cycle, and an explicit deny takes precedence over a broader allow.
• [Improvement] Notification and alert mail - certificate lifecycle notices, autoscaler token renewals, the admin income digest, and invoice issuance now use the dedicated notifications queue rather than sharing one with live dashboard traffic.
• [Improvement] Full localization of instance operations, with a build-time guard so a new action cannot ship without its wording.
• [Improvement] Quota enforcement is identical from the panel and the API, and quota messages distinguish being at your limit from a brief collision with your own concurrent request.
• [Security] Tighter tenant boundaries on object storage keys and volume plans, stricter identity binding for cluster nodes, and payment capture bound to its originating transaction.
• [Security] Cluster join credentials are cleared on every terminal outcome, with kubernetes:scrub-cloudcfg provided for existing installs.
• [Operations] Managed MariaDB monitoring, scheduler health naming, and a single pinned PHP runtime for workers and cron.

Since v3.1.0​

If you are upgrading from the 3.0 line, the two releases between it and this one carry the feature work:

• v3.1.0 adds native Proxmox VE support. Point the platform at an existing PVE 8 or 9 node or cluster with a single API token and manage it beside your KVM fleet - deploys, VPC networking, security groups, backups, snapshots, live migration, consoles, HA, Docker, managed databases, load balancers, Kubernetes, and exact per-NIC bandwidth metering. Agentless, over the Proxmox API.
• v3.1.2 adds OAuth sign-in with Google, Microsoft and GitHub, enforceable admin OIDC single sign-on, private locations with a built-in request-access workflow, a per-node deployment readiness engine surfaced on the dashboard, the hypervisor list and each node's detail page, and a substantially tougher load balancer with static-IP deploys, captured error state, and long-lived TCP session support.

v3.1.5 builds directly on both: the Proxmox firewall and load balancer improvements below apply to the surfaces those releases introduced.

Kubernetes cluster credential hygiene​

Cluster join credentials are meant to be short-lived - they exist while a node is joining and are cleared once it settles. That clearing now runs on every terminal outcome, so a node that fails to join is cleaned up exactly like one that succeeds, and it runs from the periodic reconcilers as well as the join callback, covering a node that never reports back.

Because the behavior changed, v3.1.5 ships php artisan kubernetes:scrub-cloudcfg for existing installs. It is a dry run by default and requires an explicit --force to write, only touches clusters that are fully destroyed, holds a settle window on top of that, includes soft-deleted records, and logs everything it does. Operators upgrading from an earlier 3.1.x can run it once; new installs never accumulate this data.

Admin REST API​

Several /api/v1 endpoints behaved differently from the panel and from their own documentation. They now agree:

• Updating a user no longer requires a password. You can change a name, a role, or a quota without sending a credential, and an unchanged email address is accepted.
• The acting administrator resolves consistently across instance image listing, user update and delete, VPC enable and disable, and backup creation.
• Account secrets never appear in responses - API credentials and multi-factor state are excluded from every serialized user object, and the published examples match.

The API manifest, the OpenTofu provider, and the MCP server coverage gates remain in lockstep at 889 endpoints, verified in CI.

Tenant boundaries​

Object storage access keys resolve strictly within the owning account on every user-facing route, while the administrative surface keeps its intentionally cross-account view. Volume plans are enforced against the region that offers them, so a plan wired to one location cannot be provisioned into another and pricing stays aligned with your catalog. Cluster node identity is derived solely from the signed token a node presents when it joins.

None of this requires a schema change or any action on your part.

Localization​

Every user-facing string in the instance operation pipeline is now translatable, including task names and result messages assembled at runtime. A build-time guard reads the accepted actions directly from the service and fails the build if any of them lacks its wording, so the coverage cannot silently regress.

Version v3.1.2 is the trust and hardening release that follows the Proxmox debut in v3.1.0. It modernizes how people get into the panel - social sign-in for users and enforceable OIDC single sign-on for admins - and how you sell capacity, with private locations and a built-in request-access workflow. Operators get a per-node deployment readiness engine and a substantially tougher load balancer. Underneath, this cycle ran two full platform security sweeps plus dedicated audits of the backup system and managed databases, on both the master and the hypervisor agent.

• [Feature] Sign in with Google, Microsoft, or GitHub - users can register and log in through OAuth, link and unlink providers from their profile, and auto-link to an existing account only when the provider asserts a verified email.
• [Feature] Admin single sign-on (OIDC) - bind admin logins to your identity provider with strict subject binding and no just-in-time provisioning, optionally enforce SSO for all admin password logins, and keep a time-limited break-glass path for IdP outages.
• [Feature] Private locations with request access - lock any location to selected accounts. Locked regions stay visible in the catalog with a lock treatment, users request access in one click, and admins approve or deny from a dedicated queue with email notifications both ways.
• [Feature] Node deployment readiness - every hypervisor now carries a live readiness checklist (agent, storage, network, capacity, deploy gates) surfaced as a dashboard card, a fleet list badge, and a per-node checklist with failure-specific fix hints.
• [Feature] Load balancers on allocated static IPs, captured error state (full detail for admins, a subtle banner for users), and per-frontend idle timeouts - TCP frontends now default to one-hour timeouts with kernel keepalives, so SSH and database sessions through an LB no longer drop at 50 seconds.
• [Feature] Proxmox surface expansion - VM snapshots, instance tags, guest-agent IP discovery, and backup file-restore in the user API; node issues, scheduled backup jobs, and live migration in the admin API; and admin edits to resources, topology, boot order, and NICs now push live to running VMs.
• [Feature] Security group drop rules on KVM - rule actions are honored end to end, so explicit drop rules override broader accepts, matching the Proxmox behavior.
• [Improvement] Teams - instance password mails go to the account owner with every instance-manage member in CC. Admin task queue gains one-click pruning and clean deletion.
• [Security] Two platform-wide security sweeps, defense-in-depth guardrails for the AI assistant, a backup-system audit in three phases, and a managed-database hardening batch. Details below.

Sign in with Google, Microsoft, and GitHub​

The login and registration pages now offer OAuth sign-in for Google, Microsoft, and GitHub. Each provider is enabled individually in the admin settings with its own client credentials; nothing shows on the login page until a provider is configured and switched on.

The linking rules are deliberately conservative, because OAuth auto-linking is a classic account-takeover vector:

• An OAuth identity auto-links to an existing account only when the provider asserts the email as verified. Google must present a true email_verified claim, GitHub only ever returns primary-and-verified addresses, and Microsoft sign-ins are validated against the tenant-verified UPN with the known cross-tenant takeover patterns (nOAuth) explicitly rejected.
• Sign-ups that arrive without a usable verified email go through a complete-profile step instead of silently creating a half-formed account, and the account write is transactional so a double submit cannot orphan a user.
• Logged-in users manage linked providers from their profile: connect, view, and unlink, with relinking handled safely.

Admin single sign-on (OIDC)​

Admin access can now be delegated to your identity provider - Okta, Entra ID, Keycloak, or any OIDC-compliant IdP:

• Strict binding. An admin's IdP identity binds on sub (subject), never on mutable claims, and there is no just-in-time provisioning - only pre-existing admin accounts can bind. The first bind is forensically logged, and stale identities are deleted rather than left dangling.
• Enforcement. Once your IdP is verified, you can require SSO for all admin password logins. The enforcement policy carries a lockout interlock so you cannot switch it on in a state that would lock every admin out.
• Break-glass. For IdP outages, php artisan admin:sso-break-glass opens a time-limited bypass that expires on its own. It is a deliberate, logged, console-only action.
• Setup UI. A new Authentication settings tab covers both features, including an OIDC discovery test that validates your issuer before anything is enforced. HTTPS is required and JWT verification is always on.

Private locations and request access​

Locations (hypervisor groups) can now be restricted per account. The catalog stays honest about what exists:

• Locked regions render on every create surface - the deploy modal, Cloud Service, self-provisioning, VPC and Kubernetes pickers - with a frosted lock treatment and the region name still visible, instead of vanishing from the catalog.
• Users hit Request access on a locked location, confirm, and the request lands in a new admin queue with a navigation badge. Admins approve or deny inline; both outcomes notify the user by mail. Access states are tracked per account as available, requested, or locked.
• Admin user pages gain a Cloud Service tab consolidating the account's location grants, inline approve and deny, and the account's provisioning limits.
• A per-account cloud provisioning switch cleanly disables self-service provisioning for an account without touching its running services, and the billing-exemption logic was made consistent across every surface that renders a deploy button.

Access enforcement is server-side on every create path, across web, API, queue, and AI-assistant surfaces. The lock UI is presentation; the gate is in the services.

Node deployment readiness​

Answering "why is nothing deploying to this node" used to mean reading logs. Now every hypervisor - KVM and Proxmox - carries a readiness engine that evaluates the conditions a deploy actually requires: agent reachability, storage presence and free capacity, subnet availability, deploy flags, maintenance and lock state.

• The admin dashboard shows a fleet readiness card.
• The hypervisor list badges each node ready, pending, or blocked.
• The node detail page renders the full checklist, and every failed check carries a specific fix hint tied to the actual failure, not a generic message.
• Adding a Proxmox node now runs its first cluster reconcile synchronously, so a freshly linked node reports honest readiness immediately instead of waiting for the next cron pass.

The checks mirror the real deploy gates - a node the checklist calls ready is a node the scheduler will actually use.

Load balancer improvements​

• Static IP deploys. User load balancers can deploy onto allocated static IPs, so an LB's address can be planned, firewalled, and DNS'd before it exists.
• Error surfacing. LB provisioning and sync failures are captured as a last-error state: admins see the full detail on the LB page, users see a subtle banner that something is being worked on - operational detail stays internal.
• Long-lived TCP sessions. TCP-mode frontends previously inherited HTTP-tuned 50-second idle timeouts, which silently killed idle SSH, database, and message-queue connections through the LB. TCP frontends and backends now default to one-hour timeouts with kernel TCP keepalives on both sides, websocket tunnels get a matching post-upgrade timeout, and every frontend gains an optional idle timeout field (30 to 86400 seconds) in both the user and admin panels for workloads that need more or less.
• Kubernetes LB fixes. Service LBs honor the managed-loadbalancer-public-ip annotation, weighted routing-rule backends materialize correctly with collision-free ACL names, port 80 stays plaintext under global SSL mode, and NodePort backends are health-checked over TCP.
• Plan enforcement. Standalone LB deploys enforce the location's plan-group offering, closing a path where an LB could deploy from a plan the region does not sell.

Proxmox, continued​

v3.1.0 shipped the driver; v3.1.2 finishes the surfaces around it:

• User API: VM snapshots (list, create with optional RAM state, rollback, delete), instance tags, guest-agent IP discovery, and backup file-restore browse and download.
• Admin API: node issues (list, retry, resolve), PVE scheduled backup jobs, and live migration with precheck. Route binder failures return real 404s instead of leaking existence.
• Live VM edits. Admin changes to resources, CPU topology, boot order, and NICs push to the running VM where PVE allows it, with CPU flags, secure boot, and TPM handling brought to parity with KVM.

All new endpoints are covered by the API manifest and mirrored in the OpenTofu provider and MCP server coverage gates.

Reliability: Kubernetes, VPN gateways, VPC​

• Kubernetes: worker-pool scale-up crash fixed, long jobs no longer double-execute after 90-second queue redelivery, control-plane and worker plan pickers are scoped to the region's plan groups, node selection prefers the NAT-active hypervisor, and deploys survive recycled-IP ARP staleness and transient agent transport blips. Control-plane LB deploys from the queue were failing on an authentication-context assumption; provisioning gates now evaluate the acting user everywhere.
• VPN gateways: peer key pairs auto-generate as the UI always promised, and road-warrior clients receive the VPC's private DNS resolver.
• VPC on KVM: cross-node NAT egress now installs the correct default route on non-active nodes and repairs it in the periodic sync, the VPC bridge joins a firewalld zone so nftables cannot silently reject its traffic, and ICMP redirects are suppressed on VPC veths - closing a class of "works from one node, dead from another" reports.
• Node provisioning: fresh hypervisors install required CLIs rather than only upgrading existing ones, Debian contrib is enabled across both source layouts for ZFS, and half-merged /usr systems are repaired so kernel modules and ufw work on broken base images.

Managed database hardening​

The managed database service went through a dedicated audit. Highlights: six critical backup, restore, and HA defects fixed; incremental backup chain source pinning so a restore can never mix chains; encryption keys moved off process argv; a watchdog that rescues clusters stuck in configuring with init-phase visibility; callback token lifecycle hardening with a localhost guard; PostgreSQL cluster self-heal; replica resync credentials forwarded correctly; and the admin password revealed on the detail pages where operators actually need it.

Backup system audit​

A three-phase audit of the backup pipeline shipped on both sides:

• Master: failure alerting is throttled and queue-routed so it always sends, repeated failures auto-pause a plan instead of burning nightly cycles, prune notifications report what was actually pruned, backup sizes are captured from the agent callback, and remote restores gained a direct download path while a dead restore path was removed.
• Agent: a credential leak into backup artifacts was stopped, silently truncated backups are now detected and failed, backup and restore state files are no longer world-readable, and qcow restores verify the staged artifact and use tmp-then-rename so a partial download can never replace a disk.

Security sweeps​

Two platform-wide sweeps (2026-07-29 and 2026-07-30) ran during this cycle, with every finding remediated before release. The notable classes:

• Billing integrity: top-up capture is now bound to its originating transaction, closing a credit-fraud path; credit adds are validated; backup debits are atomic.
• Tenant scoping: SSH sessions, S3 access keys, Kubernetes certificate renewal, and VPC selection are all bound to the owning tenant; state-changing restore moved off GET.
• Auth: password-reset throttling, no exception reflection to clients, OAuth and email uniqueness guarantees, and the Microsoft cross-tenant (nOAuth) rejections described above.
• Secrets at rest and in transit: queue payloads carrying secrets are encrypted, failed-job rows are pruned, Kubernetes join credentials no longer travel through cloud-init user data, notification channel secrets are no longer serialized into events, and WireGuard AllowedIPs are validated before any privileged guest execution.
• AI assistant guardrails (three phases of defense in depth): streaming egress redaction of configured secrets, knowledge-base audience scoping that fails closed, untrusted-data framing around tool output with prompt-injection guards, and redaction of persisted tool calls and audit logs so the assistant's own storage cannot become the leak.
• Dependencies: dompdf bumped for CVE-2026-56722.

Version v2.2.3 is a major feature release headlined by Managed Kubernetes, a fully integrated Kubernetes-as-a-Service offering that runs alongside Instances, Volumes, Load Balancers, and Databases. Customers can spin up a control plane (single-node or 3-node HA), attach workers in per-purpose pools, expose Kubernetes Services through the bundled in-cluster cloud controller manager, autoscale workloads end to end with the cluster autoscaler, and roll the cluster forward to a newer Kubernetes version, all without touching the slave host. The release also ships a redesigned master backup pipeline with pluggable storage drivers and Grandfather-Father-Son retention, a new System Health dashboard widget, scheduled task health tracking, an app-wide timezone setting, team-member permissions for Kubernetes resources, retry for failed cluster creates, and a long list of reliability and performance improvements including a 70% reduction in peak load for the hot-path cron loop that runs every 30 seconds against the entire fleet.

• [Feature] Managed Kubernetes - Create production-grade Kubernetes clusters directly from the control panel. Choose single-node or 3-node HA control plane, pick instance plans and subnets for control plane and workers separately, and bring up the cluster with a bundled HAProxy load balancer for the Kubernetes API. Real-time progress streams to the cluster show page via WebSocket; downloaded kubeconfig points at the right private or public endpoint automatically.
• [Feature] Worker Node Pools - Each cluster has a default worker pool and supports unlimited additional pools, each with their own instance plan, labels, taints, autoscaling bounds, and drain settings. Useful for GPU nodes, memory-optimized workloads, or isolating tenants in a single cluster.
• [Feature] Cluster Autoscaler - Bundled cluster autoscaler binary speaks the Hypervisor API directly. Policy-driven scaling on CPU + memory pressure of pending pods, per-pool aware, and respecting each pool's min/max bounds. Manifest generated on demand from the cluster show page, customers grab the YAML and apply with kubectl apply -f -. Controller token refreshes on a rolling schedule so long-lived clusters never need a manual re-issue.
• [Feature] In-Cluster Cloud Controller Manager - Services of type LoadBalancer provision and tear down a real Hypervisor load balancer per service. Service annotations control listener port, backend mode (TCP / HTTP / per-port hybrid), session stickiness, multi-cert SNI, routing rules, and traffic split between subset endpoints.
• [Feature] Worker and Control Plane Rolling Upgrades - Upgrade Workers card on the Workers tab provisions new workers at the target version, drains old ones, repeats. Upgrade Control Plane card does the same for CPs via surge-replace strategy, etcd-quorum-safe at every step. Cluster card shows CP version and worker baseline as two distinct lines with a "mid-upgrade" badge when they diverge.
• [Feature] Retry Failed Cluster Create - A new "Retry create" button on the cluster page tears down partially provisioned artifacts and re-runs the bootstrap on the same row. Cluster name, slug, and identity certificates are preserved so any kubeconfig the user already downloaded stays valid. No more delete-and-recreate after a transient quota or capacity precondition fails.
• [Feature] Cluster Security Groups - Three auto-managed security groups per cluster (LB-only, CP-only, worker-only). Default rules expose the Kubernetes API on :443 via the LB and lock down direct access to CP nodes' :6443 from outside the cluster. Admins and users layer additional rules through a familiar Inbound / Outbound sub-tabbed interface.
• [Feature] Restricted Kubeconfig - Downloaded kubeconfig issued at cluster create exposes only worker nodes to kubectl get nodes. Control-plane VMs are hidden from end users in the Compute list, billing reports, monitoring tiles, and the cluster Nodes tab.
• [Feature] Master Backup Pipeline Redesign - Service-oriented orchestrator with pluggable storage drivers (Local, S3-compatible, Rsync over SSH, NFS), a singleton lock that survives long uploads, Grandfather-Father-Son retention, email + webhook notifications, and a configurable cron expression. Multiple destinations supported per install. Admin pages cover Destinations, Runs, Settings, and Scheduler Health.
• [Feature] Scheduled Task Health Tracking - Every scheduled task is observed via a unified health surface. Per-task tracking of last run, duration, exit code, and consecutive failures. Compact admin Scheduler page with a slide-in drawer per task, friendly task names, and a daily prune to keep the audit table compact. Dashboard tile shows healthy / degraded / failed scheduled-task counts at the top of every page.
• [Feature] System Health Dashboard Widget - Single compact strip on the admin dashboard showing four critical metrics at a glance: most recent successful master backup, scheduled-task health rollup, in-flight long-running tasks, and queue worker failed-job count. Replaces two separate tiles from earlier releases.
• [Feature] Application Timezone Setting - Pick any IANA timezone from a new dropdown under Admin > System > Settings > General. Applied app-wide on boot (Carbon, model date casts, scheduler firing times, direct PHP date functions). Default for customers signing up via self-registration and billing-API user creations, unless explicitly overridden. Existing users keep their own timezone selection.
• [Feature] Kubernetes Team Permissions - New kubernetes.* permission family with three tiers (view, manage, delete) granted through the existing team-member invitation flow. Predefined roles get sensible defaults from the migration. Custom roles need to be granted the new permissions explicitly.
• [Feature] Admin Destructive Controls for Clusters - Dedicated section for safe escape hatches when a cluster has gone wrong. Suspend locks out the customer while preserving forensics. Reset State clears stuck-operation flags. Force Cleanup bypasses normal teardown for clusters with zombie resources. Separate rate limits keep destructive (5/hour) and recovery (20/hour) actions distinct.
• [Feature] AWS-Style Node Drill-Down - Clicking a node on the cluster Nodes tab opens a side drawer with capacity gauges (CPU / RAM / disk), pod listing with pagination and search, taints section, and modern dark/light surface styling.
• [Feature] Cluster-Managed Resource Lockdown - Worker instances and the CP load balancer carry a clear "Cluster-managed" badge and a read-only banner in the user's Compute and Load Balancers lists. Direct power cycle, plan change, or LB rule edit is blocked at the controller. Manage them through the cluster page instead.
• [Feature] Live Load Balancer Filtering - User-side Load Balancers index now supports AJAX live filtering by name, status, and VPC. Useful for customers running dozens of LBs across multiple VPCs.
• [Feature] Cluster Activity Feed - User dashboard activity feed now translates Kubernetes audit-log actions into friendly sentences ("Created cluster prod-01", "Upgraded workers to 1.35.0") alongside the other resource types.
• [Feature] Pre-Flight Quota and Capacity Guards - Cluster create form rejects at submit time when load balancer quota is exhausted, when the chosen VPC has no NAT Gateway (needed for control-plane image pulls), or when the CP subnet is not private. Clear messages name the limit and point at the affected field instead of failing deep in the bootstrap chain.