Skip to main content

Kubernetes Image Baking

Overview​

Every Kubernetes node (control plane or worker) on the platform boots from a pre-baked VM image registered in the Images catalogue. Image baking is a one-time-per-version task that produces two images:

  • Control plane image -- has kubeadm + kubelet + containerd, plus pre-pulled kube-apiserver, kube-scheduler, kube-controller-manager, etcd, and the cloud-controller-manager container
  • Worker image -- has kubeadm + kubelet + containerd, plus the worker-side container images and the cluster-autoscaler binary

You bake the images outside the panel (inside a temporary VM you provision yourself), snapshot the disk, and register the snapshot in Media → Images with the correct purpose flag.

The cluster lifecycle then handles everything else: provisioning new nodes from the image, cloud-init seeding, kubeadm init/join, and CNI install.

One image per minor version

Kubernetes does not support cross-version skew beyond +/-1 minor. You bake separate images for 1.34, 1.35, 1.36, etc. Patch versions (.0, .1, .2) are baked at build time -- declare each in the Supported Versions admin catalogue.

Supported Base OS​

The build script auto-detects the OS family via /etc/os-release and branches on apt vs dnf. Pick whichever your fleet already uses:

  • Ubuntu 22.04 LTS or newer (recommended for new fleets)
  • Debian 12 or 13
  • RHEL 9+
  • CentOS Stream 9+
  • Rocky Linux 9+
  • AlmaLinux 9+
  • Fedora 39+

All families ship the same upstream kubeadm packages from pkgs.k8s.io.

CP vs Worker -- one image or two?​

Two strategies are supported:

Strategy A -- Two images (recommended)

ComponentCP imageWorker image
etcd container pre-pulledyesno
kube-apiserver / scheduler / controller-manager pre-pulledyesno
cluster-autoscaler binarynoyes
Disk size20 GB40 GB
Image purposeKubernetes Control PlaneKubernetes Worker

Smaller images, cleaner separation, faster CP boot.

Strategy B -- One combined image

If image-management overhead matters more than disk size, bake one combined hks-node-<ver> image and register it as both the control plane image and the worker image when registering the Kubernetes version. Trades ~200 MB extra per worker for half the build cost.

Before You Start​

You will need:

  • A throwaway VM running one of the supported base OSes
  • Root access on that VM
  • Internet access from inside the VM (the script downloads packages and pulls container images)
  • ~6 GB free disk space inside the VM during the build
  • Around 30--60 minutes per image (most of it package install + image pull)

You also need the registry references for the platform's two custom container images:

  • cloud-controller-manager-hypervisor -- the CCM that turns Service: type=LoadBalancer into a real managed LB
  • cluster-autoscaler-hypervisor -- the autoscaler that drives worker scale-up/down

These are built locally from the respective repositories and pushed to a registry your hypervisors can reach. The script defaults to ghcr.io/hypervisor-io/... -- override via CCM_REGISTRY and AUTOSCALER_REGISTRY env vars.

What Goes In The Image​

Required packages​

  • containerd (in the K8s version's compatibility window -- see Kubernetes version skew policy)
  • kubeadm, kubelet, kubectl (pinned to the target patch version)
  • kubernetes-cni
  • runc
  • cloud-init (with NoCloud and ConfigDrive datasources enabled)
  • qemu-guest-agent
  • openssh-server

Kernel + sysctl​

  • Modules pre-loaded: overlay, br_netfilter
  • sysctl tweaks:
    • net.bridge.bridge-nf-call-iptables=1
    • net.bridge.bridge-nf-call-ip6tables=1
    • net.ipv4.ip_forward=1
  • Swap disabled + masked (kubeadm preflight refuses to start with swap on)

Networking / utilities​

  • iptables / iptables-nft + nftables
  • socat, ipset, conntrack (or conntrack-tools on RHEL family)
  • ebtables (Debian/Ubuntu only -- folded into nftables on RHEL 9+)
  • jq, yq, curl, wget, openssl, ca-certificates, gnupg2
  • On Debian/Ubuntu: lsb-release, apt-transport-https

Pre-pulled container images​

Pulled once at build time so a brand-new node can bootstrap without internet on first boot:

  • registry.k8s.io/kube-apiserver:v<VERSION> (CP image)
  • registry.k8s.io/kube-controller-manager:v<VERSION> (CP image)
  • registry.k8s.io/kube-scheduler:v<VERSION> (CP image)
  • registry.k8s.io/kube-proxy:v<VERSION> (both)
  • registry.k8s.io/etcd:<ETCD_VERSION> (CP image)
  • registry.k8s.io/coredns/coredns:<COREDNS_VERSION> (both)
  • registry.k8s.io/pause:<PAUSE_VERSION> (both)
  • quay.io/cilium/cilium:v<CILIUM_VERSION> (both -- CNI default; swap to Calico/Flannel as needed)
  • quay.io/cilium/operator-generic:v<CILIUM_VERSION> (both)
  • registry.k8s.io/metrics-server/metrics-server:v<METRICS_VERSION> (both)
  • <your-registry>/cloud-controller-manager-hypervisor:<CCM_VERSION> (CP image)
  • <your-registry>/cluster-autoscaler-hypervisor:<CA_VERSION> (worker image)

To get the exact reference list for a given Kubernetes minor, run:

kubeadm config images list --kubernetes-version v1.34.2

What does NOT go in the image​

The cluster lifecycle injects these per-cluster via cloud-init:

  • Cluster certificates (kubeadm generates per cluster)
  • Kubeconfigs
  • CNI manifest YAMLs (master applies via kubectl apply after init)
  • Tenant CCM credentials (a per-cluster JWT lands in a Secret)
  • Cluster name, VPC config, Pod CIDR, Service CIDR

Do not pre-bake any of the above -- doing so will break per-cluster isolation.

OS-Specific Notes​

The build script applies these tweaks automatically. Listed here so you know what to expect.

Ubuntu / Debian​

Stock cloud images work as-is. No preconfiguration required.

RHEL / CentOS Stream / Rocky / AlmaLinux / Fedora​

  • SELinux is set to permissive. kubeadm preflight refuses to run under enforcing because kubelet's container processes cannot relabel host paths. /etc/selinux/config is rewritten so the change survives reboot.
  • firewalld is disabled (systemctl disable --now firewalld). kube-proxy manages its own iptables/nftables ruleset and firewalld racing against it produces broken NodePort / Service routing.
  • containerd is installed from the Docker CE repo (the containerd.io package), not the distro's own containerd. This matches what the kubeadm docs reference.
  • Kubernetes packages come from pkgs.k8s.io via a yum.repos.d/kubernetes.repo file with exclude= set so a stray dnf update cannot accidentally bump kubeadm/kubelet/kubectl.

For stock RHEL 9 (not CentOS/Rocky/Alma), make sure BaseOS and AppStream repos are enabled via subscription-manager before running the script.

Build Procedure​

1. Spin up a fresh VM​

Use one of the supported base OS cloud images. 4 vCPU / 4 GB RAM / 30 GB disk is comfortable for the build.

Boot, log in as root (or sudo to root).

2. Drop the build script onto the VM​

Save the build script (provided in the platform source tree at docs/kubernetes/scripts/build-k8s-image.sh) onto the VM. SCP, curl, or paste -- whatever works.

chmod +x build-k8s-image.sh

3. Run it​

# Worker image, K8s 1.34
./build-k8s-image.sh --version 1.34.2 --role worker --cni cilium

# Control plane image, K8s 1.35
./build-k8s-image.sh --version 1.35.0 --role cp --cni cilium

# Combined image (both roles in one)
./build-k8s-image.sh --version 1.36.0 --role combined --cni cilium

Flags:

  • --version <K8s patch> -- required, e.g. 1.34.2
  • --role cp|worker|combined -- required
  • --cni cilium|calico|flannel -- defaults to cilium

Environment overrides:

  • CCM_REGISTRY -- where to pull the cloud-controller-manager image (default ghcr.io/hypervisor-io)
  • CCM_VERSION -- tag (default latest)
  • AUTOSCALER_REGISTRY -- where to pull cluster-autoscaler-hypervisor (default ghcr.io/hypervisor-io)
  • AUTOSCALER_VERSION -- tag (default: per-minor mapping inside the script)

The script is idempotent -- re-run it if interrupted, it picks up where it left off.

4. Verify​

After the script reports Done, sanity-check inside the VM:

# kubelet binary present + correct version
kubelet --version

# kubeadm binary present + correct version
kubeadm version

# containerd running
systemctl is-active containerd

# kubelet NOT running (it'll fail without certs -- cloud-init re-enables after kubeadm)
systemctl is-enabled kubelet     # → should report 'disabled'

# Modules loaded
lsmod | grep -E 'overlay|br_netfilter'

# sysctls applied
sysctl net.bridge.bridge-nf-call-iptables net.ipv4.ip_forward

# Pre-pulled images present
crictl images | grep -E 'kube-apiserver|pause|coredns'

# Platform's CCM image cached locally (CP image only)
crictl images | grep cloud-controller-manager-hypervisor

# Platform's autoscaler image cached locally (worker image only)
crictl images | grep cluster-autoscaler-hypervisor

# qemu-guest-agent active
systemctl is-active qemu-guest-agent

# cloud-init NoCloud + ConfigDrive datasources enabled
cat /etc/cloud/cloud.cfg.d/*.cfg | grep -i datasource

Every check should pass. If anything fails, fix the script -- never patch the snapshot manually.

5. Shut down + snapshot​

shutdown -h now

Snapshot the VM's disk into your image catalogue using whatever tool your storage layer provides (Ceph snapshot, libvirt virsh snapshot-create-as, raw qemu-img convert, etc.).

6. Register in Images​

  1. Navigate to Media → Images in the admin panel
  2. Click Create Image
  3. Fill in:
    • Name -- e.g. HKS Control Plane 1.34.2 or HKS Worker 1.34.2
    • URL -- pointer to the snapshot (whatever URL format your storage backend accepts)
    • Default Interface -- usually virtio
    • Purpose -- Kubernetes Control Plane for the CP image, Kubernetes Worker for the worker image
    • Cloudinit -- on
    • Public -- off (admin-managed image)
    • Enabled -- on
  4. Save

Screenshot: Admin > Media > Images > Edit page showing the Purpose dropdown with Kubernetes Control Plane and Kubernetes Worker options highlighted.

7. Wire into a Supported Version​

  1. Navigate to Kubernetes → Supported Versions in the admin sidebar
  2. Click Register Version (or edit the existing row for that version)
  3. Pick the newly-registered images for Control Plane Image and Worker Image
  4. Save

Screenshot: Admin > Kubernetes > Supported Versions > Edit modal showing the Control Plane Image and Worker Image dropdowns populated with the freshly-baked images.

The version is now selectable from the user Create Cluster wizard.

Cloud-init Contract​

The image MUST honour the NoCloud cloud-init datasource. Master delivers user-data and meta-data to each node via a seed ISO attached during VM provision. The image's cloud-init runs:

  1. A runcmd: block containing kubeadm init or kubeadm join plus tokens and cert hashes
  2. A write_files: block dropping cluster metadata into /etc/hypervisor.io/cluster.json
  3. A systemd-units block enabling kubelet, containerd, and qemu-guest-agent

The image must not auto-start kubelet on first boot -- it would fail without certs. The build script disables it; cloud-init re-enables it after kubeadm has placed the certs.

Updating a Version​

Images are immutable per KubernetesSupportedVersion row. To roll out a new patch (e.g. 1.34.2 → 1.34.3):

  1. Re-run the build script with --version 1.34.3
  2. Snapshot the disk under a new tag (hks-cp-1.34.3, hks-worker-1.34.3)
  3. Register the two new images in Media → Images
  4. Register a new row in Kubernetes → Supported Versions pointing at the new images
  5. Existing clusters do NOT auto-upgrade -- users promote per cluster via the Workers tab (worker rolling upgrade) and CP rolling upgrade
  6. Mark the old version Deprecated once the new one is published; mark it EOL once no cluster references it

Troubleshooting​

Script halts during package install​

  • DNS / network -- the VM needs to reach pkgs.k8s.io, download.docker.com, the distro mirrors, and your container registry
  • On RHEL 9 stock: verify BaseOS and AppStream subscriptions are attached
  • On Debian/Ubuntu: stale apt lock from a previous unclean run -- dpkg --configure -a && apt update

crictl images shows missing pre-pulled containers​

  • Confirm the registry is reachable from the build VM
  • For custom registries: containerd may need credentials (/etc/containerd/config.toml [plugins."io.containerd.grpc.v1.cri".registry.configs])
  • Re-run the script -- the pull step is idempotent

Cluster create succeeds but worker stays NotReady​

Usually means CNI failed to install -- but that runs at cluster create time, not from the image. Confirm:

  • The image actually has the CNI plugin's container image pre-pulled (crictl images | grep cilium)
  • The Pod CIDR you picked at cluster-create time matches the CNI's expectations (Cilium accepts any; Flannel default is 10.244.0.0/16)

kubelet won't start on first boot​

Confirm:

  • systemctl is-enabled kubelet returns disabled on the freshly-snapshotted image
  • Cloud-init's runcmd: ran successfully -- check /var/log/cloud-init-output.log on the node

Future automation​

The current script is a single bash file you run by hand. The eventual roadmap is to convert it into a Packer template + Ansible role pair so image bakes become reproducible CI artifacts. Out of scope for now -- manual VM runs are acceptable until your image catalogue grows past ~10 entries.