From 106bcbd58b02c5d78fb3b51ce1fdb3591bf06af7 Mon Sep 17 00:00:00 2001 From: aleskandro Date: Wed, 12 Mar 2025 12:13:52 +0000 Subject: [PATCH] Update autoscaling from zero enhancement proposal with support for platform-aware autoscale from zero This commit updates the contract between the cluster-autoscaler Cluster API provider and the infrastructure provider's controllers that reconcile the Infrastructure Machine Template to support platform-aware autoscale from 0 in clusters consisting of nodes heterogeneous in CPU architecture and OS. With this commit, the infrastructure providers implementing controllers to reconcile the status of their Infrastructure Machine Templates for supporting autoscale from 0 will be able to fill the status.nodeInfo stanza with additional information about the nodes. The status.nodeInfo stanza has type corev1.NodeSystemInfo to reflect the same content, the rendered nodes' objects would store in their status field. The cluster-autoscaler can use that information to build the node template labels `kubernetes.io/arch` and `kubernetes.io/os` if that information is present. Suppose the pending pods that trigger the cluster autoscaler have a node selector or a requiredDuringSchedulingIgnoredDuringExecution node affinity concerning the architecture or operating system of the node where they can execute. In that case, the autoscaler will be able to filter the nodes groups options according to the architecture or operating system requested by the pod. The users could already provide this information to the cluster autoscaler through the labels capacity annotation. However, there is no similar capability to support future labels/taints through information set by the reconcilers of the status of Infrastructure Machine Templates. --- .../20210310-opt-in-autoscaling-from-zero.md | 23 ++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/proposals/20210310-opt-in-autoscaling-from-zero.md b/docs/proposals/20210310-opt-in-autoscaling-from-zero.md index 0366a7c9b663..fe5caa2b6326 100644 --- a/docs/proposals/20210310-opt-in-autoscaling-from-zero.md +++ b/docs/proposals/20210310-opt-in-autoscaling-from-zero.md @@ -107,8 +107,8 @@ node group. But, during a scale from zero situation (ie when a node group has ze autoscaler needs to acquire this information from the infrastructure provider. An optional status field is proposed on the Infrastructure Machine Template which will be populated -by infrastructure providers to contain the CPU, memory, and GPU capacities for machines described by that -template. The cluster autoscaler will then utilize this information by reading the appropriate +by infrastructure providers to contain the CPU, CPU architecture, memory, and GPU capacities for machines +described by that template. The cluster autoscaler will then utilize this information by reading the appropriate infrastructure reference from the resource it is scaling (MachineSet or MachineDeployment). A user may override the field in the associated infrastructure template by applying annotations to the @@ -160,6 +160,10 @@ the template. Internally, this field will be represented by a Go `map` type uti for the keys and `k8s.io/apimachinery/pkg/api/resource.Quantity` as the values (similar to how resource limits and requests are handled for pods). +Additionally, the status field could contain information about the node, such as the architecture and +operating system. This information is not required for the autoscaler to function, but it can be useful in +scenarios where the autoscaler needs to make decisions for clusters with heterogeneous node groups in architecture, OS, or both. + It is worth mentioning that the Infrastructure Machine Templates are not usually reconciled by themselves. Each infrastructure provider will be responsible for determining the best implementation for adding the status field based on the information available on their platform. @@ -175,6 +179,7 @@ const ( // DockerMachineTemplateStatus defines the observed state of a DockerMachineTemplate type DockerMachineTemplateStatus struct { Capacity corev1.ResourceList `json:"capacity,omitempty"` + NodeInfo *corev1.NodeSystemInfo `json:"nodeInfo,omitempty"` } // DockerMachineTemplate is the Schema for the dockermachinetemplates API. @@ -186,7 +191,7 @@ type DockerMachineTemplate struct { Status DockerMachineTemplateStatus `json:"status,omitempty"` } ``` -_Note: the `ResourceList` and `ResourceName` referenced are from k8s.io/api/core/v1`_ +_Note: the `ResourceList`, `ResourceName` and `NodeSystemInfo` referenced are from k8s.io/api/core/v1`_ When used as a manifest, it would look like this: @@ -204,8 +209,16 @@ status: memory: 500mb cpu: "1" nvidia.com/gpu: "1" + nodeInfo: + architecture: arm64 + operatingSystem: linux ``` +The information stored in the `status.nodeInfo` field is rendered as labels for the node object that is created within +the node group by the cluster autoscaler and fed into the cluster autoscaler's scheduler simulator `framework.NodeInfo` struct. +In particular, the `architecture` and `operatingSystem` fields are used to determine the simulated node's labels +`kubernetes.io/arch` and `kubernetes.io/os`. This logic will be implemented in the cluster autoscaler's ClusterAPI cloud provider code. + #### MachineSet and MachineDeployment Annotations In cases where a user needs to provide specific resource information for a @@ -229,6 +242,8 @@ metadata: capacity.cluster-autoscaler.kubernetes.io/memory: "500mb" capacity.cluster-autoscaler.kubernetes.io/cpu: "1" capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "100Gi" + node-info.cluster-autoscaler.kubernetes.io/cpu-architecture: "arm64" + node-info.cluster-autoscaler.kubernetes.io/os: "linux" ``` _Note: the annotations will be defined in the cluster autoscaler, not in cluster-api._ @@ -246,6 +261,8 @@ metadata: capacity.cluster-autoscaler.kubernetes.io/taints: "key1=value1:NoSchedule,key2=value2:NoExecute" ``` +If the `capacity.cluster-autoscaler.kubernetes.io/labels` annotation specifies a label that would otherwise be generated from the `capacity` or `node-info` annotations, the autoscaler will use the label defined in `capacity.cluster-autoscaler.kubernetes.io/labels`, overriding any labels produced by processing the other annotations. + ### Security Model This feature will require the service account associated with the cluster autoscaler to have