Running Kubernetes workloads is simple while your traffic is predictable. The real challenge begins when demand changes throughout the day. If the cluster has too few nodes, new Pods remain in the Pending state. If it has too many nodes, you continue paying for EC2 capacity that your applications do not need.

Kubernetes Cluster Autoscaler solves this problem by adjusting the desired capacity of the EC2 Auto Scaling Groups behind your EKS node groups. When Pods cannot be scheduled because the cluster has insufficient resources, it requests additional nodes. When nodes remain underutilized and their workloads can safely run elsewhere, it allows those nodes to be removed.

In this guide, we are going to build a secure and production-aware Cluster Autoscaler setup for Amazon EKS. We will use EKS Pod Identity instead of storing AWS credentials inside Kubernetes, install the upstream Helm chart, trigger a real scale-up event, verify scale-down, and troubleshoot the problems you are most likely to encounter.

Cluster Autoscaler, HPA, and Karpenter

These tools solve different scaling problems, so it is important not to confuse them.

Component What it scales Typical trigger
Horizontal Pod Autoscaler Number of application Pods CPU, memory, or custom metrics
Vertical Pod Autoscaler CPU and memory assigned to Pods Historical and current resource usage
Cluster Autoscaler Nodes inside existing node groups Unschedulable Pods or underutilized nodes
Karpenter EC2 capacity selected directly for workload requirements Unschedulable Pods and consolidation opportunities
EKS Auto Mode AWS-managed cluster compute and other EKS capabilities Workload and infrastructure demand

The Horizontal Pod Autoscaler and Cluster Autoscaler are commonly used together. HPA creates additional Pods when application demand increases. If those Pods no longer fit on the existing nodes, Cluster Autoscaler adds node capacity.

Cluster Autoscaler works with node groups and their EC2 Auto Scaling Groups. It changes the desired capacity but respects the minimum and maximum values already configured on each group. Karpenter and EKS Auto Mode use a different provisioning model and are worth considering when you need more flexible, workload-aware instance selection.

This tutorial is for an existing EKS cluster that uses EC2 managed or self-managed node groups. Do not install a second node autoscaler to control the same capacity.

Prerequisites

Before starting, make sure you have:

  • An existing Amazon EKS cluster with at least one EC2 node group
  • Permission to manage IAM roles, IAM policies, EKS add-ons, and Pod Identity associations
  • AWS CLI v2 installed and authenticated
  • kubectl configured for the target EKS cluster
  • Helm 3 installed
  • Permission to inspect and tag the node group’s EC2 Auto Scaling Group

Confirm the tools and current AWS identity:

1
2
3
4
aws --version
kubectl version --client
helm version
aws sts get-caller-identity

Configure the Working Environment

Set reusable variables for your environment. Replace the example cluster, Region, and node group names before continuing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
export AWS_REGION="eu-west-2"
export CLUSTER_NAME="codingtricks-eks"
export NODEGROUP_NAME="general-workers"
export CA_NAMESPACE="kube-system"
export CA_SERVICE_ACCOUNT="cluster-autoscaler"
export CA_POLICY_NAME="${CLUSTER_NAME}-cluster-autoscaler-policy"
export CA_ROLE_NAME="${CLUSTER_NAME}-cluster-autoscaler-role"

export AWS_ACCOUNT_ID="$(aws sts get-caller-identity \
  --query Account \
  --output text)"

aws eks update-kubeconfig \
  --region "$AWS_REGION" \
  --name "$CLUSTER_NAME"

Verify that you are connected to the intended cluster:

1
2
3
4
5
6
7
kubectl config current-context
kubectl get nodes -o wide

aws eks describe-cluster \
  --region "$AWS_REGION" \
  --name "$CLUSTER_NAME" \
  --query 'cluster.{Name:name,Version:version,Status:status}'

Never rely only on a shell variable before changing production infrastructure. Confirm the account, Region, cluster name, and current Kubernetes context.

Check the Node Group Scaling Boundaries

Cluster Autoscaler cannot grow a node group beyond its configured maximum size or reduce it below its minimum size.

1
2
3
4
5
aws eks describe-nodegroup \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --nodegroup-name "$NODEGROUP_NAME" \
  --query 'nodegroup.scalingConfig'

For a small lab, you might use a minimum of 1, desired capacity of 2, and maximum of 4:

1
2
3
4
5
aws eks update-nodegroup-config \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --nodegroup-name "$NODEGROUP_NAME" \
  --scaling-config minSize=1,maxSize=4,desiredSize=2

Do not copy these limits directly into production. Choose values from workload demand, Availability Zone requirements, IP address capacity, service quotas, disruption tolerance, and budget.

Wait until the update is complete:

1
2
3
4
aws eks wait nodegroup-active \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --nodegroup-name "$NODEGROUP_NAME"

Enable Auto-Discovery Tags

Cluster Autoscaler uses tags to discover the Auto Scaling Groups it is allowed to manage. First, retrieve the ASG name behind the managed node group:

1
2
3
4
5
6
7
8
export ASG_NAME="$(aws eks describe-nodegroup \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --nodegroup-name "$NODEGROUP_NAME" \
  --query 'nodegroup.resources.autoScalingGroups[0].name' \
  --output text)"

echo "$ASG_NAME"

Apply the two discovery tags:

1
2
3
4
5
aws autoscaling create-or-update-tags \
  --region "$AWS_REGION" \
  --tags \
    "ResourceId=${ASG_NAME},ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/enabled,Value=true,PropagateAtLaunch=false" \
    "ResourceId=${ASG_NAME},ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/${CLUSTER_NAME},Value=owned,PropagateAtLaunch=false"

Confirm them:

1
2
3
4
aws autoscaling describe-tags \
  --region "$AWS_REGION" \
  --filters "Name=auto-scaling-group,Values=${ASG_NAME}" \
  --query 'Tags[?contains(Key, `cluster-autoscaler`)].{Key:Key,Value:Value}'

Repeat this step for every node group that Cluster Autoscaler should control. The cluster-specific tag prevents an autoscaler in one EKS cluster from modifying an Auto Scaling Group that belongs to another cluster.

Install the EKS Pod Identity Agent

EKS Pod Identity lets the Cluster Autoscaler Pod receive temporary AWS credentials through its Kubernetes ServiceAccount. No static AWS access key is stored in a Secret, values file, or container environment variable.

Check whether the agent is already installed:

1
2
3
4
aws eks describe-addon \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name eks-pod-identity-agent

If AWS returns ResourceNotFoundException, create the add-on:

1
2
3
4
aws eks create-addon \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name eks-pod-identity-agent

Wait for it to become active and verify the DaemonSet Pods:

1
2
3
4
5
6
7
8
aws eks wait addon-active \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --addon-name eks-pod-identity-agent

kubectl get pods \
  --namespace kube-system \
  --selector app.kubernetes.io/name=eks-pod-identity-agent

EKS Auto Mode already includes Pod Identity functionality, so its agent does not need to be installed separately. For clusters with private nodes, ensure the nodes can reach the EKS Auth API, using the required interface endpoint when there is no suitable outbound route.

Create the Cluster Autoscaler IAM Policy

Cluster Autoscaler needs read access to inspect EC2 and Auto Scaling configuration. It also needs permission to change desired capacity and terminate an instance during scale-down.

Create cluster-autoscaler-policy.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowScopedScalingActions",
      "Effect": "Allow",
      "Action": [
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true",
          "aws:ResourceTag/k8s.io/cluster-autoscaler/codingtricks-eks": "owned"
        }
      }
    },
    {
      "Sid": "AllowDiscoveryReadActions",
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeScalingActivities",
        "autoscaling:DescribeTags",
        "ec2:DescribeImages",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:GetInstanceTypesFromInstanceRequirements",
        "eks:DescribeNodegroup"
      ],
      "Resource": "*"
    }
  ]
}

Replace codingtricks-eks in the IAM condition key with your actual cluster name. IAM policy JSON does not expand shell variables.

Create the policy:

1
2
3
4
5
aws iam create-policy \
  --policy-name "$CA_POLICY_NAME" \
  --policy-document file://cluster-autoscaler-policy.json

export CA_POLICY_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:policy/${CA_POLICY_NAME}"

The modifying actions use tag-based conditions, while read-only discovery calls use Resource: "*" because those APIs do not support the same resource-level restrictions. This prevents the role from scaling an unrelated ASG that does not carry both expected tags.

Create the Pod Identity IAM Role

Create cluster-autoscaler-trust-policy.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
      "Effect": "Allow",
      "Principal": {
        "Service": "pods.eks.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession"
      ]
    }
  ]
}

Create the role and attach the policy:

1
2
3
4
5
6
7
8
9
10
aws iam create-role \
  --role-name "$CA_ROLE_NAME" \
  --assume-role-policy-document file://cluster-autoscaler-trust-policy.json \
  --description "EKS Pod Identity role for Cluster Autoscaler"

aws iam attach-role-policy \
  --role-name "$CA_ROLE_NAME" \
  --policy-arn "$CA_POLICY_ARN"

export CA_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CA_ROLE_NAME}"

The trust policy is intentionally different from an IRSA trust policy. EKS Pod Identity uses the pods.eks.amazonaws.com service principal and does not require an OIDC provider or a role annotation on the ServiceAccount.

Create the ServiceAccount and Pod Identity Association

Create the ServiceAccount that the Helm release will reuse:

1
2
3
4
kubectl create serviceaccount "$CA_SERVICE_ACCOUNT" \
  --namespace "$CA_NAMESPACE" \
  --dry-run=client \
  --output yaml | kubectl apply -f -

Create the Pod Identity association:

1
2
3
4
5
6
aws eks create-pod-identity-association \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --namespace "$CA_NAMESPACE" \
  --service-account "$CA_SERVICE_ACCOUNT" \
  --role-arn "$CA_ROLE_ARN"

Confirm the association:

1
2
3
aws eks list-pod-identity-associations \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME"

There is no IAM role annotation to check with Pod Identity. The association is stored in EKS rather than inside the Kubernetes ServiceAccount object.

Select a Compatible Cluster Autoscaler Version

Cluster Autoscaler contains scheduler simulation logic, so its minor version must match the Kubernetes control-plane minor version. Cross-version compatibility is not tested or supported by the project.

Retrieve the EKS version:

1
2
3
4
5
6
7
export K8S_VERSION="$(aws eks describe-cluster \
  --region "$AWS_REGION" \
  --name "$CLUSTER_NAME" \
  --query 'cluster.version' \
  --output text)"

echo "EKS Kubernetes version: ${K8S_VERSION}"

Choose the latest supported Cluster Autoscaler patch release with the same minor version. For example, Kubernetes 1.34.x requires Cluster Autoscaler 1.34.x.

1
2
3
4
5
6
7
8
# Example for an EKS 1.34 cluster. At the time of writing, 1.34.5 is the
# latest upstream patch in this minor line. Confirm the release before use.
export CA_VERSION="1.34.5"

if [ "${CA_VERSION%.*}" != "$K8S_VERSION" ]; then
  echo "Version mismatch: EKS ${K8S_VERSION}, Cluster Autoscaler ${CA_VERSION}"
  exit 1
fi

Do not blindly use latest. Review the upstream compatibility table and release notes during every EKS upgrade.

Install Cluster Autoscaler with Helm

Add the official chart repository:

1
2
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm repo update

Create cluster-autoscaler-values.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
cloudProvider: aws
awsRegion: eu-west-2

fullnameOverride: cluster-autoscaler

autoDiscovery:
  clusterName: codingtricks-eks

rbac:
  create: true
  serviceAccount:
    create: false
    name: cluster-autoscaler

image:
  tag: v1.34.5

extraArgs:
  balance-similar-node-groups: "true"
  expander: least-waste
  skip-nodes-with-system-pods: "false"

podAnnotations:
  cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

priorityClassName: system-cluster-critical

resources:
  requests:
    cpu: 100m
    memory: 300Mi
  limits:
    cpu: 200m
    memory: 600Mi

Replace the Region, cluster name, and image tag with the values you verified earlier. Then install the chart:

1
2
3
4
5
6
helm upgrade --install cluster-autoscaler \
  autoscaler/cluster-autoscaler \
  --namespace "$CA_NAMESPACE" \
  --values cluster-autoscaler-values.yaml \
  --wait \
  --timeout 5m

The main values do the following:

  • autoDiscovery.clusterName discovers only ASGs carrying the matching tags.
  • fullnameOverride gives the Deployment a predictable name for the verification commands used below.
  • rbac.serviceAccount.create=false reuses the ServiceAccount associated with the IAM role.
  • image.tag pins Cluster Autoscaler to the Kubernetes-compatible minor version.
  • least-waste chooses the node group that leaves the least unused capacity after scale-up.
  • balance-similar-node-groups helps distribute capacity across equivalent node groups.
  • safe-to-evict=false prevents Cluster Autoscaler from evicting its own Pod during node consolidation.
  • system-cluster-critical gives the controller a high scheduling priority.

Setting skip-nodes-with-system-pods=false allows nodes containing movable kube-system Pods to be considered for scale-down. Test this carefully with your add-ons and PodDisruptionBudgets. We have intentionally not disabled the local-storage safety check; workloads that keep irreplaceable data on node-local storage need special attention.

Verify the Installation

Check the Helm release, Deployment, ServiceAccount, and Pod:

1
2
3
4
5
6
7
8
9
10
11
12
13
helm list --namespace "$CA_NAMESPACE"

kubectl get deployment cluster-autoscaler \
  --namespace "$CA_NAMESPACE"

kubectl get pods \
  --namespace "$CA_NAMESPACE" \
  --selector app.kubernetes.io/name=aws-cluster-autoscaler \
  --output wide

kubectl get serviceaccount "$CA_SERVICE_ACCOUNT" \
  --namespace "$CA_NAMESPACE" \
  --output yaml

Inspect the logs:

1
2
3
4
kubectl logs \
  --namespace "$CA_NAMESPACE" \
  deployment/cluster-autoscaler \
  --tail=100

Healthy logs should show that the AWS cloud provider initialized and one or more Auto Scaling Groups were discovered. You should not see AccessDenied, credential-provider, or repeated ASG discovery errors.

You can also inspect the status ConfigMap:

1
2
3
kubectl get configmap cluster-autoscaler-status \
  --namespace "$CA_NAMESPACE" \
  --output yaml

Test a Real Scale-Up Event

We need Pods with resource requests large enough to exceed current node capacity. Create inflate.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: registry.k8s.io/pause:3.10
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 500m
              memory: 512Mi

Apply the Deployment and increase the replica count:

1
2
kubectl apply -f inflate.yaml
kubectl scale deployment inflate --replicas=20

Adjust the replica count for your instance type and lab limits. Do not trigger a large test in a production cluster without a controlled change window and cost approval.

Watch the Pods:

1
2
3
kubectl get pods \
  --selector app=inflate \
  --watch

In a second terminal, watch the nodes:

1
kubectl get nodes --watch

Some Pods should initially remain Pending with an insufficient resource message:

1
2
3
4
5
6
kubectl get pods --selector app=inflate

kubectl describe pod "$(kubectl get pods \
  --selector app=inflate \
  --field-selector status.phase=Pending \
  --output jsonpath='{.items[0].metadata.name}')"

Follow the autoscaler’s decisions:

1
2
3
4
kubectl logs \
  --namespace "$CA_NAMESPACE" \
  deployment/cluster-autoscaler \
  --follow | grep -iE 'scale.up|expanding|unschedulable|node group'

Finally, confirm that the ASG desired capacity increased:

1
2
3
4
aws autoscaling describe-auto-scaling-groups \
  --region "$AWS_REGION" \
  --auto-scaling-group-names "$ASG_NAME" \
  --query 'AutoScalingGroups[0].{Min:MinSize,Desired:DesiredCapacity,Max:MaxSize,Instances:length(Instances)}'

The EC2 instance must launch, bootstrap, and join the cluster before the waiting Pods can run. The complete process normally takes longer than the autoscaler’s initial decision.

Test Scale-Down

Remove the test workload demand:

1
kubectl scale deployment inflate --replicas=0

Watch the autoscaler logs and node list:

1
2
3
4
kubectl logs \
  --namespace "$CA_NAMESPACE" \
  deployment/cluster-autoscaler \
  --follow | grep -iE 'scale.down|unneeded|removing|delete'
1
kubectl get nodes --watch

By default, a node normally needs to remain unneeded for a period before it becomes eligible for scale-down. Cluster Autoscaler will not reduce the group below its minimum size, and it will not remove a node when its remaining Pods cannot be safely moved.

After the test, delete the Deployment:

1
kubectl delete -f inflate.yaml

Common Problems and Troubleshooting

Cluster Autoscaler Shows AccessDenied

Confirm the Pod is using the expected ServiceAccount:

1
2
3
kubectl get deployment cluster-autoscaler \
  --namespace "$CA_NAMESPACE" \
  --output jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}'

Then verify:

  • The EKS Pod Identity Agent is running on the node.
  • The Pod Identity association uses the same cluster, namespace, and ServiceAccount.
  • The IAM role has the autoscaler policy attached.
  • The role trust policy allows pods.eks.amazonaws.com.
  • The node role permits eks-auth:AssumeRoleForPodIdentity.
  • Private nodes can reach the EKS Auth API.

Restart the Deployment after correcting the association:

1
2
kubectl rollout restart deployment cluster-autoscaler \
  --namespace "$CA_NAMESPACE"

The Node Group Is Not Discovered

Inspect the rendered Deployment arguments:

1
2
3
kubectl get deployment cluster-autoscaler \
  --namespace "$CA_NAMESPACE" \
  --output jsonpath='{.spec.template.spec.containers[0].command}'

Confirm that:

  • --cloud-provider=aws is present.
  • The auto-discovery argument includes both expected tag keys.
  • The ASG carries k8s.io/cluster-autoscaler/enabled=true.
  • The ASG carries k8s.io/cluster-autoscaler/<cluster-name>=owned.
  • The cluster name in Helm values matches the real EKS cluster name exactly.

A Pending Pod Does Not Trigger Scale-Up

First, read its scheduler events:

1
kubectl describe pod <pending-pod-name>

A new node helps only when a discovered node group can satisfy the Pod. Check for:

  • A resource request larger than any available instance type
  • Node selectors or required node affinity that no node group provides
  • Taints without matching tolerations
  • Topology or Availability Zone constraints
  • An unbound PersistentVolumeClaim
  • Host-port conflicts
  • A node group already at maximum size
  • EC2 capacity shortages, service quotas, or exhausted subnet IP addresses

For a node group that can scale from zero, publish its expected labels, taints, and extended resources as ASG node-template tags when the autoscaler cannot infer them from the launch template.

Nodes Scale Up but Pods Stay Pending

The scaling API call succeeded, but the scheduling requirement is still not satisfied. Compare the Pod requests and constraints with the new node:

1
2
3
kubectl describe pod <pending-pod-name>
kubectl describe node <new-node-name>
kubectl get events --sort-by=.lastTimestamp

Check the node’s allocatable CPU and memory after DaemonSets, labels, taints, EBS volume zone, ENI/IP capacity, and the EC2 instance type actually launched by a mixed instances policy.

Nodes Do Not Scale Down

Use the logs to find the exact blocker:

1
2
3
kubectl logs \
  --namespace "$CA_NAMESPACE" \
  deployment/cluster-autoscaler | grep -iE 'scale.down|unremovable|evict|pdb'

Common causes include:

  • The node group is already at its minimum size.
  • A PodDisruptionBudget does not allow another disruption.
  • A Pod has cluster-autoscaler.kubernetes.io/safe-to-evict: "false".
  • A Pod uses local storage and the relevant safety setting blocks eviction.
  • A system Pod cannot run elsewhere.
  • Affinity, anti-affinity, topology, or resource requests prevent rescheduling.
  • A bare Pod is not managed by a controller.

Do not weaken eviction protections only to make a node disappear. Fix the workload’s availability and scheduling design first.

Scale-Up Stops at a Specific Number of Nodes

Inspect the node group and ASG limits:

1
2
3
4
5
aws eks describe-nodegroup \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --nodegroup-name "$NODEGROUP_NAME" \
  --query 'nodegroup.scalingConfig'

If the group is at maxSize, Cluster Autoscaler is behaving correctly. Increasing the limit changes your cost and capacity risk, so treat it as an infrastructure change rather than a troubleshooting shortcut.

Production Best Practices

Match Versions During Every Upgrade

Upgrade Cluster Autoscaler as part of the EKS control-plane upgrade plan. Keep the Kubernetes and Cluster Autoscaler minor versions aligned, then test scale-up and scale-down before considering the upgrade complete.

Set Realistic Resource Requests

Cluster Autoscaler makes scheduling decisions from requests. Missing or severely underestimated requests create poor bin-packing decisions, while exaggerated requests can cause unnecessary nodes to launch. Use metrics and VPA recommendations to refine them.

Keep Node Groups Consistent

Nodes in one group should have equivalent scheduling properties. When using a mixed instances policy, select instance types with a similar CPU, memory, and GPU shape. Cluster Autoscaler simulates scheduling using the first instance type in the policy; smaller alternatives may still leave Pods pending, while larger alternatives can waste capacity.

Separate Spot and On-Demand Capacity

Spot and On-Demand nodes have different availability and scheduling characteristics. Separate them into different node groups, apply appropriate labels or taints, and use Pod tolerations and disruption controls. least-waste is a practical general-purpose expander, while the priority expander is useful when one capacity type should be attempted before another.

Protect Critical Workloads

Use multiple replicas, topology spread constraints, Pod anti-affinity where appropriate, and realistic PodDisruptionBudgets. Apply the safe-to-evict annotation only when eviction genuinely creates unacceptable risk because excessive protection can prevent cluster scale-down indefinitely.

Monitor the Autoscaler

At minimum, alert on:

  • Cluster Autoscaler Pod unavailable or repeatedly restarting
  • Unschedulable Pods persisting beyond the expected node launch time
  • Node groups remaining at maximum capacity
  • Repeated AWS API errors or throttling
  • Failed node launches or nodes that do not join the cluster
  • Nodes marked unneeded but never removed
  • Unexpected changes in desired capacity and EC2 cost

Collect Cluster Autoscaler logs centrally and retain Kubernetes events long enough to investigate incidents after they occur.

Plan Capacity Outside Kubernetes

Autoscaling cannot fix every infrastructure limit. Monitor subnet IP availability, EC2 quotas, Spot capacity, IAM failures, launch-template errors, bootstrap failures, and dependencies required by private nodes. Keep enough minimum capacity for critical system workloads and failure scenarios.

Tune Conservatively

Flags such as scan interval, scale-down delay, utilization threshold, and node provisioning timeout affect cost, availability, and AWS API usage. Start with upstream defaults, measure actual behaviour, and change one setting at a time. Aggressive scale-down can save money but also increase churn and application disruption.

Cluster Autoscaler or Karpenter?

Cluster Autoscaler remains a good choice when:

  • You already manage stable EKS node groups.
  • Your organization understands ASGs and managed node groups.
  • Capacity types and instance shapes are deliberately predefined.
  • You want a mature, portable Kubernetes autoscaling model.

Consider Karpenter or EKS Auto Mode when:

  • You need more flexible instance selection for individual workloads.
  • Faster, just-in-time provisioning is important.
  • You want automated consolidation and right-sizing across a broad EC2 catalog.
  • Maintaining many narrowly defined node groups has become operationally expensive.

Do not let two controllers independently manage the same nodes. Plan and test any migration so ownership of compute capacity is always clear.

Environment Cleanup

The following commands remove the tutorial workload and Cluster Autoscaler configuration. Confirm the account, Region, cluster, role, and policy names before running them.

Remove the test Deployment and Helm release:

1
2
3
4
kubectl delete deployment inflate --ignore-not-found

helm uninstall cluster-autoscaler \
  --namespace "$CA_NAMESPACE"

Find the Pod Identity association ID:

1
2
3
4
5
6
7
8
9
export ASSOCIATION_ID="$(aws eks list-pod-identity-associations \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --namespace "$CA_NAMESPACE" \
  --service-account "$CA_SERVICE_ACCOUNT" \
  --query 'associations[0].associationId' \
  --output text)"

echo "$ASSOCIATION_ID"

After confirming the value, delete the association and ServiceAccount:

1
2
3
4
5
6
7
8
aws eks delete-pod-identity-association \
  --region "$AWS_REGION" \
  --cluster-name "$CLUSTER_NAME" \
  --association-id "$ASSOCIATION_ID"

kubectl delete serviceaccount "$CA_SERVICE_ACCOUNT" \
  --namespace "$CA_NAMESPACE" \
  --ignore-not-found

Detach and delete the IAM resources only if no other workload uses them:

1
2
3
4
5
6
7
8
9
aws iam detach-role-policy \
  --role-name "$CA_ROLE_NAME" \
  --policy-arn "$CA_POLICY_ARN"

aws iam delete-role \
  --role-name "$CA_ROLE_NAME"

aws iam delete-policy \
  --policy-arn "$CA_POLICY_ARN"

Optionally remove the ASG discovery tags if the group will no longer be managed by Cluster Autoscaler:

1
2
3
4
5
aws autoscaling delete-tags \
  --region "$AWS_REGION" \
  --tags \
    "ResourceId=${ASG_NAME},ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/enabled" \
    "ResourceId=${ASG_NAME},ResourceType=auto-scaling-group,Key=k8s.io/cluster-autoscaler/${CLUSTER_NAME}"

Do not remove the EKS Pod Identity Agent when other workloads use Pod Identity. You may also want to restore the node group’s original minimum, desired, and maximum capacity after a lab test.

Conclusion

In this article, we learned how to configure Kubernetes Cluster Autoscaler on Amazon EKS using ASG auto-discovery, least-privilege IAM permissions, EKS Pod Identity, and the official Helm chart. We also created a controlled workload to verify real scale-up and scale-down behaviour, examined the most common failure modes, and reviewed the production decisions that affect security, availability, and cost.

Cluster Autoscaler works best when the surrounding platform is designed correctly: workloads have realistic resource requests, node groups expose consistent scheduling properties, disruption policies allow safe movement, and AWS capacity limits are monitored. With those foundations in place, it can keep your EKS cluster responsive without leaving unnecessary EC2 instances running.

Are you currently using Cluster Autoscaler, Karpenter, or EKS Auto Mode? Share your experience or any issue you faced in the comments.

Happy Coding