One post tagged with "WHMCS"

Version v3.1.7 opens the platform's telemetry to your customers: Metrics Export adds Prometheus-format scrape endpoints on the user API, so a customer can point their own Prometheus or Grafana at the panel and chart their instances, managed databases and load balancers with the API token they already have. The second headline is the Import Doctor: virtual machines imported from Hyper-V, VMware or other platforms - which typically blue-screen on first boot because their disk drivers were never armed for our hardware - are now detected automatically and repaired with one click. Around them, a Fix Monitoring action repairs broken telemetry on managed services in one step, Kubernetes plan rotation picked up the fixes from its first weeks in the field, and the user panel's list pages were made consistent end to end.

Metrics Export​

• [Feature] Prometheus Scrape Endpoints - GET /api/metrics returns every instance, managed database and load balancer the account owns as one Prometheus text exposition, with per-resource endpoints alongside it for narrowly scoped scrape jobs. Authentication is the same bearer token used everywhere else on the user API - there is no separate credential to mint. Metric families are stable and customer-facing (hv_instance_*, hv_db_*, hv_lb_*), with resource id and name labels and no internal topology in the output. Instances include Kubernetes worker nodes and VPN gateway backing VMs.
• [Feature] Built for Scrapers, Not Browsers - The endpoints answer HTTP 200 always: a backing store being unreachable surfaces as hv_resource_up 0 on the affected resources, never a 5xx that turns into an error storm in the customer's Prometheus. Scrapes are rate limited per token at 30 per minute, the account-wide endpoint caches its rendered body for 10 seconds to absorb multi-target fan-out, and the whole call runs under a hard time budget so one slow backend cannot stall the response. Subuser tokens see exactly the resources their team permissions grant, nothing more.
• [Feature] Documentation and Examples - The user API documentation gained a Metrics Export section with the full endpoint table and a ready-to-paste scrape_config snippet, and a feature guide ships with this release. The pipeline was validated end to end against a real Prometheus and Grafana stack before release.

Imported Virtual Machines​

• [Feature] Import Doctor - Customers who convert a VHDX or VMDK, write it over their instance disk and boot typically hit INACCESSIBLE_BOOT_DEVICE: installing the virtio drivers inside Hyper-V or VMware stages them but never arms them for boot, because that hardware never presents a virtio disk. The platform now fingerprints every instance disk after each cold start, and when a foreign operating system appears, the instance page shows a banner on both the user and admin panels. One click repairs it offline: the staged virtio storage drivers are armed directly in the guest's registry, stale UEFI boot entries are reset, and the firmware type is matched to what the disk actually uses. Guests where the full repair is not possible fall back to compatible SATA and e1000 emulation so they boot regardless. Detection is automatic; repair only ever runs when someone asks for it, with the guest shut off.
• [Fix] UEFI Firmware on Debian Hypervisors - UEFI guest definitions probed only the Red Hat and legacy Ubuntu OVMF firmware layouts, so on Debian hypervisors (and newer Ubuntu, which drops the legacy names too) every UEFI define failed with a missing-file error until symlinks were made by hand. Firmware is now resolved from ordered CODE/VARS pairs covering all three layouts, requiring both halves of a pair so a partial install can never mix incompatible images.

Monitoring and Managed Databases​

• [Feature] Fix Monitoring - Managed databases and load balancers gained a Fix Monitoring action that repairs the telemetry pipeline in one step: the metrics agent configuration is re-rendered and re-installed, the metrics exporter login and its grants are re-created, and the agent is restarted. It is available on the metrics tab in both panels - once per resource per 24 hours on the user side, unthrottled for admins - and a failed attempt never consumes the user's daily budget. A new monitoring:fix console command batches the same repair across every active managed service, for fleet-wide rollout after an endpoint or credential change.
• [Fix] Customer-Created Read-Only Roles - On PostgreSQL, the customer admin account could create a role but not grant it anything useful: granting membership in the monitoring and read-all roles failed with a permission error, so a read-only or Grafana login ended up empty. On MySQL and MariaDB the grant capability had been removed outright. The admin account now carries the delegation rights it needs on both engines - scoped so the hardening that removed superuser-level capabilities stays in force - and the Fix Monitoring action applies the same correction to existing databases.
• [Fix] Honest Instance Memory Graphs - Instance memory usage was derived from the guest's free-memory figure, which page cache drains toward zero on any warm Linux guest, so memory graphs crept toward 100% regardless of real pressure. The calculation now uses the guest's available-memory figure, which counts reclaimable cache as free - the same number free -m shows in its available column. Guests with older drivers keep the previous behaviour.

Kubernetes​

• [Feature] Cluster API Endpoint by Name - New public clusters serve their API endpoint by their System DNS name (k8s-<name>-cp.<your-domain>) instead of a bare IP: the name is baked into the API server certificate at bootstrap, the cluster page shows it, and generated kubeconfigs use it. The name is pinned at creation and never rewritten, since it lives inside a signed certificate distributed to customers. Existing clusters are untouched and keep working by IP.
• [Feature] Webhook Subscriptions API - The user API gained full management of webhook subscriptions (/api/webhook-subscriptions): create, list, update, delete, and per-subscription delivery history. Deliveries are HMAC-signed with a per-subscription secret, endpoints must be HTTPS, and subscriptions can filter to a single cluster. Kubernetes lifecycle events are the first event source.
• [Improvement] Plan Changes Lead Into Rotation - After changing a pool's instance plan, nothing pointed at the rotation that actually resizes the workers, so pools sat reporting drift until someone found the unlabeled icon. The edit form now says what saving will and will not do, a plan-changing save flows directly into the rotation dialog (including the separate downsize confirmation where it applies), and the drift badge itself became a clickable "Rotate now" action. Declining the dialog simply leaves the badge as the reminder.
• [Fix] Rotation Waves Apply Labels and Taints - Workers added by a plan rotation joined without their pool's labels and taints and showed no role in kubectl get nodes. Each rotation wave now applies the pool's labels and taints after the new workers are ready and before the old ones drain - exactly the moment draining reschedules pods onto them - and the role label appears promptly instead of up to a day later.
• [Fix] Rotation Dispatch Self-Heals - A rotation is launched as a detached background process, and that launch can die silently. A rotation still pending after three minutes is now redispatched once by the periodic janitor - rotations are resume-safe by design, and a healthy run is never touched - and every detached launch now leaves a forensic log so a failed spawn is diagnosable rather than a mystery.
• [Fix] Pool Deletion Cannot Strand Workers - Deleting a node pool could remove the pool record while its workers were still protected by scale-down guards, leaving live worker VMs attached to a deleted pool where no cleanup process could reach them. Pool deletion now marks every member for collection with those guards bypassed - they exist to protect a pool that is staying - and refuses to remove the pool record if any live member could not be marked. The cleanup and drain paths also learned to resolve a deleted pool's configuration, so members marked before the deletion still drain correctly.
• [Fix] Placement on Heavily Committed Nodes - A node whose allocated memory exceeded its physical total crashed the placement query for its whole group with a database range error, aborting scale-ups that had healthy capacity elsewhere. Placement and the rotation capacity gate now both measure headroom against the node's effective memory ceiling - the operator-set overcommit limit where one is configured, physical memory otherwise - so a sanctioned overcommit node counts its real headroom and an over-allocated one simply sorts last instead of taking the group down. The placement query also gained the standard maintenance, lock and deployment gates it was missing.
• [Fix] Certificate Auto-Renewal Now Scheduled - The cluster certificate renewal service shipped fully built but nothing ever ran it. It now runs daily, renewing certificates inside a 30-day window ahead of expiry.

Networking​

• [Fix] Private DNS on Dual-Homed Instances - On instances with both a public IP and a VPC interface, the first lookup of a private DNS name could stall for seconds: the VPC link shipped without its search domains, so the resolver had no reason to route private-zone queries to the VPC resolver and raced it against the public nameserver. The deploy payload now carries the VPC's DNS zones as search domains. Already-deployed guests pick this up on their next network configuration rebuild.
• [Fix] Private DNS on VPC-Only Instances - VPC-only instances could lose private-zone resolution entirely: a fallback public nameserver on the same link could permanently win the resolver's affinity, and the VPC's own resolver was not even running until the VPC had at least one zone. The gateway resolver now always runs - from VPC creation, zones or not - and VPC guests use it exclusively on that link, with public resolution forwarded upstream through it.

User Panel​

• [Improvement] Consistent Filtering Everywhere - The service list pages were brought up to one standard: URL-shareable filter state, status and facet filters, and debounced server-side search. Managed database filters that the server always supported are now in the UI; VPN gateway and scaling group search actually filters instead of doing nothing; volume search no longer breaks the table; pagination keeps your filters instead of dropping them. Columns that were already on the wire but never rendered are now shown - private IPs for databases and VPN gateways, location and plan for instances, distribution and region for images.
• [Improvement] Capacity Errors Reworded for Customers - When a deploy or scale-up fails for a capacity reason - no memory, storage or IP headroom where the resource was requested - the customer now sees a clear "contact support" message instead of raw infrastructure wording that named things they cannot see or fix. Admins are emailed the precise original error (throttled per distinct error), see it unchanged in the admin panel, and it is always logged.
• [Improvement] AI Assistant Out of Beta - The AI assistant has run long enough in production to drop the beta label. The badge and the settings-page warning are gone; nothing about its configuration changes.

Billing and WHMCS​

• [Feature] Create-Time Add-Ons - External billing provisioning now accepts additional IPv4 addresses (up to 20) and additional disk (up to 5000 GB) at instance creation. The extra disk is placed on the same storage pool as the primary and the capacity check covers the combined footprint. The WHMCS module exposes both as configurable options; they apply at creation only, by design.
• [Improvement] Hardened WHMCS Module - The module now keeps its own service-to-instance link table (auto-migrated, self-healing from the custom fields it also auto-creates), so suspend, terminate and upgrade no longer depend on an admin having manually created a custom field. Re-running CreateAccount on a linked service refuses to mint a duplicate instance, and terminating a service whose instance is already gone converges cleanly instead of failing forever. The client-area overview page was redesigned to match the panel, with live data, copyable IPs, and one-click SSO into the panel.
• [Fix] Server Host Resolution - A WHMCS server saved with only the IP Address field filled (Hostname blank) could never load plans or hypervisor groups into product configuration. All module variants now resolve hostname-or-IP consistently, honour the configured port, tolerate a pasted URL, and fail with a named message when both fields are empty.
• [Improvement] TLS Posture Made Explicit - The billing API client does not verify the master's TLS certificate, because masters are routinely addressed by IP or carry self-signed certificates and verification would break provisioning on those installs. This trade-off is now documented rather than implicit: where possible, point WHMCS at a hostname with a publicly trusted certificate.

Platform and Admin​

• [Fix] Route Debt Sweep - An inventory pass over every registered route closed a set of long-standing gaps: the kubeconfig acknowledgement button now works, users can delete their object storage access keys (the endpoint existed but was never routed), several links that led to pages that do not exist now redirect to the real pages, and two admin route groups gated on permission slugs that could never be granted are now grantable. Dead controllers, models and page stubs were removed.
• [Fix] Volume Attach Validation - Attaching a volume now verifies the target instance belongs to the same account and derives the instance's hypervisor group correctly (the previous check read a field instances do not have, and failed for every attach). Cluster-managed workers are rejected as attach targets, and subusers now see their account's volumes and instances on the volume pages.
• [Fix] Web SSH Stability - Web SSH sessions on remote or CDN-fronted masters appeared to disconnect frequently: overlapping readiness polls could each open a connection against a one-time token, and a rejected duplicate would paint a disconnect overlay over the live terminal. Polling is now single-flight and a stale socket can never steal the display from a live one.
• [Fix] Encrypted Secret Storage - Columns storing encrypted provider secrets were widened; a secret over 23 characters could previously be truncated at rest.
• [Improvement] PowerDNS Onboarding - Adding a System DNS domain whose zone already exists in PowerDNS now adopts the zone instead of failing, the delegation checker installs its DNS tooling where missing, and a zone error clears automatically once resolved. The wizard's guidance on public-suffix domains was corrected.

Upgrade notes​

• Two migrations run on upgrade: the imported-OS state column and the encrypted secret column widening.
• Deploy the master before the hypervisor agents for the VPC DNS fixes; each side tolerates the other being old. The agent release carries the memory metric fix, the Debian UEFI firmware resolution and the Import Doctor tooling - the agent installs its guest-inspection packages during provisioning or update.
• The read-only role grant fix applies to newly provisioned databases automatically. For existing databases, run php artisan monitoring:fix --type=db once (or use the Fix Monitoring button per database) to roll it out.
• Metrics Export needs no setup: it uses the metrics backends you already configured per hypervisor group. See the new Metrics Export guide for the customer-facing details and a scrape configuration example.