In today's article, I will show you how to deploy Cluster AutoScalar on AWS EKS cluster. Cluster Autoscaler is a tool that automatically adjusts the size of a Kubernetes cluster in response to workload demand. It works by periodically evaluating the resource utilization of the pods running on the cluster and comparing it to the target utilization set by the user. If the actual utilization exceeds the target, Cluster Autoscaler will spin up additional nodes in the cluster to accommodate the additional workload. Conversely, if the actual utilization falls below the target, Cluster Autoscaler will remove unnecessary nodes to save resources. By automatically scaling the cluster in response to workload demand, Cluster Autoscaler helps ensure that pods have the resources they need to function properly, while also minimizing resource waste.

Create a EKS Cluster

Before we start, we need to create a EKS cluster. If you already have a EKS cluster, you can skip this step. In this step we will create a simple EKS cluster using eksctl byusing the below config file.

cluster-oidc.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: demo ## your cluster name
  region: ap-southeast-1 ## your region

iam: 
  withOIDC: true ## enable OIDC
  serviceAccounts:
  - metadata:
      name: cluster-autoscaler ## service account name (you can use any name)
      namespace: kube-system ## namespace which you want to create the service account
    wellKnownPolicies: 
      autoScaler: true ## enable cluster autoscaler policy
    roleName: cluster-autoscaler ## role name (you can use any name)
    roleOnly: true ## create only role

managedNodeGroups:
  - name: demo-nodegroup
    instanceType: t3.large
    desiredCapacity: 1
    minSize: 1
    maxSize: 10
    privateNetworking: true ## enable private networking
    tags:
      k8s.io/cluster-autoscaler/enabled: "true" ## enable cluster autoscaler
      k8s.io/cluster-autoscaler/<your-cluster-name>: "owned" ## make sure to replace <your-cluster-name> with your cluster name

To create the cluster, run the below command. 

1
eksctl create cluster -f cluster-oidc.yaml

Warning:

  • The Cluster Autoscaler requires the following tags on your Auto-Scaling groups so that they can be auto-discovered ( EC2 -> Auto-Scaling groups -> `node-group-name` -> Tags )

    • 1
    2
    k8s.io/cluster-autoscaler/enabled: "true"
    k8s.io/cluster-autoscaler/<cluster-name>: "owned"

    If you used eksctl to create your node groups, these tags are automatically applied.

  • If you didn’t used `eksctl`, you must manually add the following tags to the node group's Auto-Scaling groups:

    • 1
    2
    k8s.io/cluster-autoscaler/enabled: true
    k8s.io/cluster-autoscaler/<CLUSTER-NAME>: owned

    Deploy Cluster Auto Scalar

    Now we can deploy cluster autoscaler using below manifest file.

    auto-scaler.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::<aws-acoount-id>:role/<role> ## role is equals to role name which you created in the previous step
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: cluster-autoscaler
    rules:
      - apiGroups: [""]
        resources: ["events", "endpoints"]
        verbs: ["create", "patch"]
      - apiGroups: [""]
        resources: ["pods/eviction"]
        verbs: ["create"]
      - apiGroups: [""]
        resources: ["pods/status"]
        verbs: ["update"]
      - apiGroups: [""]
        resources: ["endpoints"]
        resourceNames: ["cluster-autoscaler"]
        verbs: ["get", "update"]
      - apiGroups: [""]
        resources: ["nodes"]
        verbs: ["watch", "list", "get", "update"]
      - apiGroups: [""]
        resources: ["namespaces", "pods", "services", "replicationcontrollers", "persistentvolumeclaims", "persistentvolumes"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["extensions"]
        resources: ["replicasets", "daemonsets"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["policy"]
        resources: ["poddisruptionbudgets"]
        verbs: ["watch", "list"]
      - apiGroups: ["apps"]
        resources: ["statefulsets", "replicasets", "daemonsets"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["storage.k8s.io"]
        resources: ["storageclasses", "csinodes", "csidrivers", "csistoragecapacities"]
        verbs: ["watch", "list", "get"]
      - apiGroups: ["batch", "extensions"]
        resources: ["jobs"]
        verbs: ["get", "list", "watch", "patch"]
      - apiGroups: ["coordination.k8s.io"]
        resources: ["leases"]
        verbs: ["create"]
      - apiGroups: ["coordination.k8s.io"]
        resourceNames: ["cluster-autoscaler"]
        resources: ["leases"]
        verbs: ["get", "update"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
    rules:
      - apiGroups: [""]
        resources: ["configmaps"]
        verbs: ["create","list","watch"]
      - apiGroups: [""]
        resources: ["configmaps"]
        resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
        verbs: ["delete", "get", "update", "watch"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: cluster-autoscaler
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: cluster-autoscaler
    subjects:
      - kind: ServiceAccount
        name: cluster-autoscaler
        namespace: kube-system
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: cluster-autoscaler
    subjects:
      - kind: ServiceAccount
        name: cluster-autoscaler
        namespace: kube-system
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: cluster-autoscaler
      namespace: kube-system
      labels:
        app: cluster-autoscaler
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: cluster-autoscaler
      template:
        metadata:
          labels:
            app: cluster-autoscaler
        spec:
          serviceAccountName: cluster-autoscaler
          containers:
            - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.2
              name: cluster-autoscaler
              resources:
                limits:
                  cpu: 100m
                  memory: 600Mi
                requests:
                  cpu: 100m
                  memory: 600Mi
              command: 
                - ./cluster-autoscaler
                - --v=4
                - --stderrthreshold=info
                - --cloud-provider=aws
                - --skip-nodes-with-local-storage=false
                - --expander=least-waste
                - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<your-cluster-name> # Update cluster
                - --balance-similar-node-groups ## Ensures that there is enough available computing across all availability zones
                - --skip-nodes-with-system-pods=false ## Ensures that there are no problems with scaling to zero
              volumeMounts:
                - name: ssl-certs
                  mountPath: /etc/ssl/certs/ca-certificates.crt
                  readOnly: true
              imagePullPolicy: "Always"
          volumes:
            - name: ssl-certs
              hostPath:
                path: "/etc/ssl/certs/ca-bundle.crt"

    Apply the manifest file using below command.

    1
    kubectl apply -f cluster-autoscaler.yaml

    Now we have deployed cluster autoscaler. Let’s check the status of cluster autoscaler.

    1
    kubectl get pods -n kube-system

    Above command will give you the output as below.

    1
    2
    NAME                                       READY   STATUS    RESTARTS   AGE
    cluster-autoscaler-5f4f7f7f4f-5j4xg        1/1     Running   0          2m

    Deploy Sample Application

    Now we have deployed cluster autoscaler. Let’s deploy a sample application to test the cluster autoscaler. We will deploy a sample application using below command.

    1
    kubectl create deployment autoscaler-demo --image=nginx

    Now we have deployed a sample application. Let’s increase the number of the replicas of the application to 100 using below command.

    1
    kubectl scale deployment autoscaler-demo --replicas=100

    Now we have increased the number of replicas of the application to 100. Let’s check the status of the application using below command.

    1
    kubectl get pods

    Above command will give you the output as below.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    NAME                                READY   STATUS              RESTARTS   AGE
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     Running             0          2m
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     Running             0          2m
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     Running             0          2m
    ...
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     ContainerCreating   0          2m
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     Pending             0          2m
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     Pending             0          2m
    autoscaler-demo-5c8b9f4b7b-2j2xg    1/1     Running             0          2m

    As you can see, the number of replicas of the application is 100. Now let’s check the status of pending pods using below command.

    1
    kubectl describe pods autoscaler-demo-5c8b9f4b7b-2j2xg -n default

    In the output of the above command, it shows that, the pod was triggered by the cluster autoscaler. And also you can see the node count of the cluster is increased.

    Conclusion

    In this tutorial, we have learned how to deploy Cluster AutoScalar on AWS EKS cluster. You can obtain all the required script from this GitHub repository. If you have any issue regarding this tutorial, mention your issue in the comment section or reach me through my E-mail.

    Happy Coding