4 posts tagged with "Security"

Version v3.1.5 closes out the 3.1 line. It adds no new product surface - it makes the surface added in v3.1.0 and v3.1.2 behave consistently. The admin REST API now matches the panel and its own documentation, load balancer routing and Proxmox firewall synchronization are more dependable, alert mail is delivered on its own queue, and every instance operation is fully localized. A platform review ran alongside this work; its outcomes are folded into the items below.

Upgrading is routine. There is nothing to reconfigure and no behavior to relearn.

• [Improvement] Admin REST API consistency - updating a user no longer requires resending a password, the acting administrator resolves the same way on every endpoint, and account credentials are excluded from responses. Load balancer security group rules are readable from the administrative API.
• [Improvement] Load balancer routing rules honor catch-all matches, applied after the specific rules and before the default backend, so precedence follows the order the rule list reads.
• [Improvement] Proxmox security group synchronization converges reliably, so firewall rule changes reach running VMs on every cycle, and an explicit deny takes precedence over a broader allow.
• [Improvement] Notification and alert mail - certificate lifecycle notices, autoscaler token renewals, the admin income digest, and invoice issuance now use the dedicated notifications queue rather than sharing one with live dashboard traffic.
• [Improvement] Full localization of instance operations, with a build-time guard so a new action cannot ship without its wording.
• [Improvement] Quota enforcement is identical from the panel and the API, and quota messages distinguish being at your limit from a brief collision with your own concurrent request.
• [Security] Tighter tenant boundaries on object storage keys and volume plans, stricter identity binding for cluster nodes, and payment capture bound to its originating transaction.
• [Security] Cluster join credentials are cleared on every terminal outcome, with kubernetes:scrub-cloudcfg provided for existing installs.
• [Operations] Managed MariaDB monitoring, scheduler health naming, and a single pinned PHP runtime for workers and cron.

Since v3.1.0​

If you are upgrading from the 3.0 line, the two releases between it and this one carry the feature work:

• v3.1.0 adds native Proxmox VE support. Point the platform at an existing PVE 8 or 9 node or cluster with a single API token and manage it beside your KVM fleet - deploys, VPC networking, security groups, backups, snapshots, live migration, consoles, HA, Docker, managed databases, load balancers, Kubernetes, and exact per-NIC bandwidth metering. Agentless, over the Proxmox API.
• v3.1.2 adds OAuth sign-in with Google, Microsoft and GitHub, enforceable admin OIDC single sign-on, private locations with a built-in request-access workflow, a per-node deployment readiness engine surfaced on the dashboard, the hypervisor list and each node's detail page, and a substantially tougher load balancer with static-IP deploys, captured error state, and long-lived TCP session support.

v3.1.5 builds directly on both: the Proxmox firewall and load balancer improvements below apply to the surfaces those releases introduced.

Kubernetes cluster credential hygiene​

Cluster join credentials are meant to be short-lived - they exist while a node is joining and are cleared once it settles. That clearing now runs on every terminal outcome, so a node that fails to join is cleaned up exactly like one that succeeds, and it runs from the periodic reconcilers as well as the join callback, covering a node that never reports back.

Because the behavior changed, v3.1.5 ships php artisan kubernetes:scrub-cloudcfg for existing installs. It is a dry run by default and requires an explicit --force to write, only touches clusters that are fully destroyed, holds a settle window on top of that, includes soft-deleted records, and logs everything it does. Operators upgrading from an earlier 3.1.x can run it once; new installs never accumulate this data.

Admin REST API​

Several /api/v1 endpoints behaved differently from the panel and from their own documentation. They now agree:

• Updating a user no longer requires a password. You can change a name, a role, or a quota without sending a credential, and an unchanged email address is accepted.
• The acting administrator resolves consistently across instance image listing, user update and delete, VPC enable and disable, and backup creation.
• Account secrets never appear in responses - API credentials and multi-factor state are excluded from every serialized user object, and the published examples match.

The API manifest, the OpenTofu provider, and the MCP server coverage gates remain in lockstep at 889 endpoints, verified in CI.

Tenant boundaries​

Object storage access keys resolve strictly within the owning account on every user-facing route, while the administrative surface keeps its intentionally cross-account view. Volume plans are enforced against the region that offers them, so a plan wired to one location cannot be provisioned into another and pricing stays aligned with your catalog. Cluster node identity is derived solely from the signed token a node presents when it joins.

None of this requires a schema change or any action on your part.

Localization​

Every user-facing string in the instance operation pipeline is now translatable, including task names and result messages assembled at runtime. A build-time guard reads the accepted actions directly from the service and fails the build if any of them lacks its wording, so the coverage cannot silently regress.

Version v3.1.2 is the trust and hardening release that follows the Proxmox debut in v3.1.0. It modernizes how people get into the panel - social sign-in for users and enforceable OIDC single sign-on for admins - and how you sell capacity, with private locations and a built-in request-access workflow. Operators get a per-node deployment readiness engine and a substantially tougher load balancer. Underneath, this cycle ran two full platform security sweeps plus dedicated audits of the backup system and managed databases, on both the master and the hypervisor agent.

• [Feature] Sign in with Google, Microsoft, or GitHub - users can register and log in through OAuth, link and unlink providers from their profile, and auto-link to an existing account only when the provider asserts a verified email.
• [Feature] Admin single sign-on (OIDC) - bind admin logins to your identity provider with strict subject binding and no just-in-time provisioning, optionally enforce SSO for all admin password logins, and keep a time-limited break-glass path for IdP outages.
• [Feature] Private locations with request access - lock any location to selected accounts. Locked regions stay visible in the catalog with a lock treatment, users request access in one click, and admins approve or deny from a dedicated queue with email notifications both ways.
• [Feature] Node deployment readiness - every hypervisor now carries a live readiness checklist (agent, storage, network, capacity, deploy gates) surfaced as a dashboard card, a fleet list badge, and a per-node checklist with failure-specific fix hints.
• [Feature] Load balancers on allocated static IPs, captured error state (full detail for admins, a subtle banner for users), and per-frontend idle timeouts - TCP frontends now default to one-hour timeouts with kernel keepalives, so SSH and database sessions through an LB no longer drop at 50 seconds.
• [Feature] Proxmox surface expansion - VM snapshots, instance tags, guest-agent IP discovery, and backup file-restore in the user API; node issues, scheduled backup jobs, and live migration in the admin API; and admin edits to resources, topology, boot order, and NICs now push live to running VMs.
• [Feature] Security group drop rules on KVM - rule actions are honored end to end, so explicit drop rules override broader accepts, matching the Proxmox behavior.
• [Improvement] Teams - instance password mails go to the account owner with every instance-manage member in CC. Admin task queue gains one-click pruning and clean deletion.
• [Security] Two platform-wide security sweeps, defense-in-depth guardrails for the AI assistant, a backup-system audit in three phases, and a managed-database hardening batch. Details below.

Sign in with Google, Microsoft, and GitHub​

The login and registration pages now offer OAuth sign-in for Google, Microsoft, and GitHub. Each provider is enabled individually in the admin settings with its own client credentials; nothing shows on the login page until a provider is configured and switched on.

The linking rules are deliberately conservative, because OAuth auto-linking is a classic account-takeover vector:

• An OAuth identity auto-links to an existing account only when the provider asserts the email as verified. Google must present a true email_verified claim, GitHub only ever returns primary-and-verified addresses, and Microsoft sign-ins are validated against the tenant-verified UPN with the known cross-tenant takeover patterns (nOAuth) explicitly rejected.
• Sign-ups that arrive without a usable verified email go through a complete-profile step instead of silently creating a half-formed account, and the account write is transactional so a double submit cannot orphan a user.
• Logged-in users manage linked providers from their profile: connect, view, and unlink, with relinking handled safely.

Admin single sign-on (OIDC)​

Admin access can now be delegated to your identity provider - Okta, Entra ID, Keycloak, or any OIDC-compliant IdP:

• Strict binding. An admin's IdP identity binds on sub (subject), never on mutable claims, and there is no just-in-time provisioning - only pre-existing admin accounts can bind. The first bind is forensically logged, and stale identities are deleted rather than left dangling.
• Enforcement. Once your IdP is verified, you can require SSO for all admin password logins. The enforcement policy carries a lockout interlock so you cannot switch it on in a state that would lock every admin out.
• Break-glass. For IdP outages, php artisan admin:sso-break-glass opens a time-limited bypass that expires on its own. It is a deliberate, logged, console-only action.
• Setup UI. A new Authentication settings tab covers both features, including an OIDC discovery test that validates your issuer before anything is enforced. HTTPS is required and JWT verification is always on.

Private locations and request access​

Locations (hypervisor groups) can now be restricted per account. The catalog stays honest about what exists:

• Locked regions render on every create surface - the deploy modal, Cloud Service, self-provisioning, VPC and Kubernetes pickers - with a frosted lock treatment and the region name still visible, instead of vanishing from the catalog.
• Users hit Request access on a locked location, confirm, and the request lands in a new admin queue with a navigation badge. Admins approve or deny inline; both outcomes notify the user by mail. Access states are tracked per account as available, requested, or locked.
• Admin user pages gain a Cloud Service tab consolidating the account's location grants, inline approve and deny, and the account's provisioning limits.
• A per-account cloud provisioning switch cleanly disables self-service provisioning for an account without touching its running services, and the billing-exemption logic was made consistent across every surface that renders a deploy button.

Access enforcement is server-side on every create path, across web, API, queue, and AI-assistant surfaces. The lock UI is presentation; the gate is in the services.

Node deployment readiness​

Answering "why is nothing deploying to this node" used to mean reading logs. Now every hypervisor - KVM and Proxmox - carries a readiness engine that evaluates the conditions a deploy actually requires: agent reachability, storage presence and free capacity, subnet availability, deploy flags, maintenance and lock state.

• The admin dashboard shows a fleet readiness card.
• The hypervisor list badges each node ready, pending, or blocked.
• The node detail page renders the full checklist, and every failed check carries a specific fix hint tied to the actual failure, not a generic message.
• Adding a Proxmox node now runs its first cluster reconcile synchronously, so a freshly linked node reports honest readiness immediately instead of waiting for the next cron pass.

The checks mirror the real deploy gates - a node the checklist calls ready is a node the scheduler will actually use.

Load balancer improvements​

• Static IP deploys. User load balancers can deploy onto allocated static IPs, so an LB's address can be planned, firewalled, and DNS'd before it exists.
• Error surfacing. LB provisioning and sync failures are captured as a last-error state: admins see the full detail on the LB page, users see a subtle banner that something is being worked on - operational detail stays internal.
• Long-lived TCP sessions. TCP-mode frontends previously inherited HTTP-tuned 50-second idle timeouts, which silently killed idle SSH, database, and message-queue connections through the LB. TCP frontends and backends now default to one-hour timeouts with kernel TCP keepalives on both sides, websocket tunnels get a matching post-upgrade timeout, and every frontend gains an optional idle timeout field (30 to 86400 seconds) in both the user and admin panels for workloads that need more or less.
• Kubernetes LB fixes. Service LBs honor the managed-loadbalancer-public-ip annotation, weighted routing-rule backends materialize correctly with collision-free ACL names, port 80 stays plaintext under global SSL mode, and NodePort backends are health-checked over TCP.
• Plan enforcement. Standalone LB deploys enforce the location's plan-group offering, closing a path where an LB could deploy from a plan the region does not sell.

Proxmox, continued​

v3.1.0 shipped the driver; v3.1.2 finishes the surfaces around it:

• User API: VM snapshots (list, create with optional RAM state, rollback, delete), instance tags, guest-agent IP discovery, and backup file-restore browse and download.
• Admin API: node issues (list, retry, resolve), PVE scheduled backup jobs, and live migration with precheck. Route binder failures return real 404s instead of leaking existence.
• Live VM edits. Admin changes to resources, CPU topology, boot order, and NICs push to the running VM where PVE allows it, with CPU flags, secure boot, and TPM handling brought to parity with KVM.

All new endpoints are covered by the API manifest and mirrored in the OpenTofu provider and MCP server coverage gates.

Reliability: Kubernetes, VPN gateways, VPC​

• Kubernetes: worker-pool scale-up crash fixed, long jobs no longer double-execute after 90-second queue redelivery, control-plane and worker plan pickers are scoped to the region's plan groups, node selection prefers the NAT-active hypervisor, and deploys survive recycled-IP ARP staleness and transient agent transport blips. Control-plane LB deploys from the queue were failing on an authentication-context assumption; provisioning gates now evaluate the acting user everywhere.
• VPN gateways: peer key pairs auto-generate as the UI always promised, and road-warrior clients receive the VPC's private DNS resolver.
• VPC on KVM: cross-node NAT egress now installs the correct default route on non-active nodes and repairs it in the periodic sync, the VPC bridge joins a firewalld zone so nftables cannot silently reject its traffic, and ICMP redirects are suppressed on VPC veths - closing a class of "works from one node, dead from another" reports.
• Node provisioning: fresh hypervisors install required CLIs rather than only upgrading existing ones, Debian contrib is enabled across both source layouts for ZFS, and half-merged /usr systems are repaired so kernel modules and ufw work on broken base images.

Managed database hardening​

The managed database service went through a dedicated audit. Highlights: six critical backup, restore, and HA defects fixed; incremental backup chain source pinning so a restore can never mix chains; encryption keys moved off process argv; a watchdog that rescues clusters stuck in configuring with init-phase visibility; callback token lifecycle hardening with a localhost guard; PostgreSQL cluster self-heal; replica resync credentials forwarded correctly; and the admin password revealed on the detail pages where operators actually need it.

Backup system audit​

A three-phase audit of the backup pipeline shipped on both sides:

• Master: failure alerting is throttled and queue-routed so it always sends, repeated failures auto-pause a plan instead of burning nightly cycles, prune notifications report what was actually pruned, backup sizes are captured from the agent callback, and remote restores gained a direct download path while a dead restore path was removed.
• Agent: a credential leak into backup artifacts was stopped, silently truncated backups are now detected and failed, backup and restore state files are no longer world-readable, and qcow restores verify the staged artifact and use tmp-then-rename so a partial download can never replace a disk.

Security sweeps​

Two platform-wide sweeps (2026-07-29 and 2026-07-30) ran during this cycle, with every finding remediated before release. The notable classes:

• Billing integrity: top-up capture is now bound to its originating transaction, closing a credit-fraud path; credit adds are validated; backup debits are atomic.
• Tenant scoping: SSH sessions, S3 access keys, Kubernetes certificate renewal, and VPC selection are all bound to the owning tenant; state-changing restore moved off GET.
• Auth: password-reset throttling, no exception reflection to clients, OAuth and email uniqueness guarantees, and the Microsoft cross-tenant (nOAuth) rejections described above.
• Secrets at rest and in transit: queue payloads carrying secrets are encrypted, failed-job rows are pruned, Kubernetes join credentials no longer travel through cloud-init user data, notification channel secrets are no longer serialized into events, and WireGuard AllowedIPs are validated before any privileged guest execution.
• AI assistant guardrails (three phases of defense in depth): streaming egress redaction of configured secrets, knowledge-base audience scoping that fails closed, untrusted-data framing around tool output with prompt-injection guards, and redaction of persisted tool calls and audit logs so the assistant's own storage cannot become the leak.
• Dependencies: dompdf bumped for CVE-2026-56722.

Version v2.2.5 turns the platform into a place you can ship code, not just run servers. The headline is Git Push-to-Deploy: customers connect GitHub, GitLab, Bitbucket, or Gitea, point the platform at a repository, and every push is built and rolled out to their instance with zero downtime - no Dockerfile required, thanks to automatic build packs. Pull requests get their own disposable preview environments, complete with status comments posted back to the PR. Around the headline, the release adds live master updates so operators can upgrade the platform from a banner in the admin panel and watch it happen, unicast VXLAN so VPC networking works in datacenters without multicast, per-interface VPC speed controls, a new bandwidth overage billing option that charges for extra traffic instead of cutting customers off, and a platform-wide security hardening pass.

• [Feature] Git Push-to-Deploy - Deploy applications directly from a Git repository to an instance through the Docker Manager. Pick a repository and branch, choose how it builds, and the platform clones, builds, and runs it. From then on, every push to the branch deploys automatically.
• [Feature] Four Git Providers - Connect GitHub through a guided GitHub App flow with repository and branch browsing built into the panel, or connect GitLab, Bitbucket, and Gitea. Private repositories are supported through access tokens or SSH deploy keys, including custom SSH ports and users for self-hosted servers.
• [Feature] Automatic Builds with Build Packs - Applications build with Nixpacks or Railpack, which detect the language and framework and produce a container image with no Dockerfile in the repository. A plain Dockerfile and a static-site build pack are also available. Builds run inside an isolated helper container, so no build tooling is ever installed onto the instance itself.
• [Feature] Zero-Downtime Rolling Deploys - A new deployment starts the new container, waits for it to come up healthy, switches traffic over, and only then removes the old one. Traffic is routed through a managed reverse proxy with automatic HTTPS certificates.
• [Feature] Pull-Request Preview Environments - Every pull request can get its own live preview deployment on its own URL. The preview is updated on each new commit, the deployment status is posted back to the pull request as a comment, and the environment is torn down when the PR closes. A previews list in the panel shows what is running and allows manual teardown.
• [Feature] Dedicated Build Hosts - Builds can be offloaded from the application instance to a separate build host. Each user gets their own deploy keypair, and the panel includes a guided onboarding flow with a connection test before the host is used.
• [Feature] Deploy Controls and Logs - Webhook deploys are signature-verified, a deployment can be skipped by putting [skip ci] or [skip cd] in the commit message, and full build and deploy logs are available in the panel with credentials automatically redacted.
• [Feature] Live Master Updates - When a new platform version is available, an update banner appears in the admin panel. One click starts the upgrade, and a live console streams the update progress in real time until it completes. The dashboard now also carries a version pill showing the running release.
• [Feature] Unicast VXLAN for VPC Networking - Each hypervisor group can now run its VPC overlay in unicast mode instead of multicast. Unicast mode works in datacenters and on networks where multicast is not available, which removes the most common blocker to enabling VPC networking. Existing groups keep multicast by default.
• [Feature] VPC Interface Speed Controls - Operators can set default inbound and outbound speed limits for VPC network interfaces per hypervisor group, and override them per instance plan. Customers see the effective limits on their instance's network details.
• [Feature] Bandwidth Overage Billing - Instance, managed database, load balancer, and VPN gateway plans gain a Charge Overage option. When a customer exhausts the plan's bandwidth allowance, traffic keeps flowing and the extra usage is billed per gigabyte at the rate set on the plan, charged through the hourly billing cycle. The existing cut-off behavior remains available for plans that prefer it.
• [Feature] Expanded Instance Charts - Instance monitoring gains disk I/O, disk usage, and network packet and error charts alongside the existing CPU, memory, and bandwidth graphs.

We're excited to announce Version v2.2.1 of the Hypervisor Control Panel! This release completely rebuilds the instance backup experience with support for remote backup destinations and adds GPU passthrough for compute-heavy workloads.

You can now back up your instances directly to Amazon S3, RustFS, Wasabi, Backblaze B2, DigitalOcean Spaces, SFTP servers, FTP servers, or any rclone-supported backend -- no need to mount backup storage locally on each hypervisor.

• [Feature] Remote Backup Destinations - Configure S3, SFTP, FTP, or rclone-based backup targets from the admin panel. Credentials are stored encrypted and pushed to hypervisors only when needed.
• [Feature] Streaming Backups - Full and incremental backups stream directly to the remote destination. Restores stream back the same way. No temporary disk space required.
• [Feature] Running-VM Backups Without Downtime - Backups of running VMs use external snapshots so the VM continues running normally while the backup completes. Changes made during the backup window are merged back seamlessly.
• [Feature] Efficient Incremental Backups - Incremental backups capture only the blocks that changed since the last backup, dramatically reducing backup size and duration.
• [Feature] Parallel Backups - Multiple backups can now run concurrently on the same hypervisor without conflict.
• [Feature] GPU Passthrough - PCIe GPU passthrough for compute-heavy workloads like ML training, rendering, and video encoding. Hypervisors automatically discover installed GPUs and show availability in the admin panel.
• [Feature] GPU Instance Plans - Instance plans can now specify a GPU count, enabling GPU-backed instance deployment through the standard creation flow.