How to Set Up Cloud Controller Manager in AWS with Kubeadm

Cloud Controller Manager on AWS Kubeadm Cluster

In this guide, I have added detailed steps to set up Kubernetes cloud controller manager on an AWS kubeadm cluster.

The idea of this setup is to understand the AWS configurations involved in the Cloud Controller manager in a self-hosted kubernetes setup.

If you want to understand how cloud controller manager works, please go through the detailed kubernetes architecture.


This setup is performed in Ubuntu 22.04 Instance and the instance type is t2.medium.

You need a minimum of two nodes for this setup. (One controller and one worker node).

Also, ensure you have opened up all the required ports for Kubernetes on both nodes.

Let’s get started with the setup.

Step 1: Attach IAM Roles

Both controller and worker nodes need IAM roles with required permissions for the cloud controller manager to interact with the AWS APIs.

IAM Policy for the controller node

Create an IAM role with the following permissions and attach it to the controller node

    "Version": "2012-10-17",
    "Statement": [
            "Effect": "Allow",
            "Action": [
            "Resource": [

IAM Policy for the Worker Nodes

Create an IAM role with the following permissions and attach it to the worker nodes.

    "Version": "2012-10-17",
    "Statement": [
            "Effect": "Allow",
            "Action": [
            "Resource": "*"

Step 2: Set Hostname (On All Nodes)

Ensure each node hostname has its own private DNS address. If not, change the hostname using the following command. This is a requirement.

Run the following command on each node (controller and worker).

sudo hostnamectl set-hostname $(curl -s

Verify the hostname by executing the hostname command.

ubuntu@node01:~$ hostname

Step 3: Install System Utilities

Container Runtime (CRI-O), kubelet, Kubeadm, and kubectl are the important utilities that should be present in each node (controller and worker).

Install utilities using a shell script It installs kubeadm version 1.28


# Exit on error, undefined variable, or error in any pipeline
set -euxo pipefail

# Variables

# Disable swap and keep it off during reboot
sudo swapoff -a
(crontab -l 2>/dev/null; echo "@reboot /sbin/swapoff -a") | crontab - || true

# Update package list
sudo apt-get update -y

# Install CRI-O Runtime
# Load required modules and set sysctl params
echo -e "overlay\\nbr_netfilter" | sudo tee /etc/modules-load.d/crio.conf
sudo modprobe overlay
sudo modprobe br_netfilter

echo -e "net.bridge.bridge-nf-call-iptables=1\\nnet.ipv4.ip_forward=1\\nnet.bridge.bridge-nf-call-ip6tables=1" | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
sudo sysctl --system

# Add CRI-O to sources and install
echo "deb <$OS/> /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb <$VERSION/$OS/> /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

curl -L <$VERSION/$OS/Release.key> | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -
curl -L <$OS/Release.key> | sudo apt-key --keyring /etc/apt/trusted.gpg.d/libcontainers.gpg add -

sudo apt-get update
sudo apt-get install -y cri-o cri-o-runc

sudo systemctl daemon-reload
sudo systemctl enable crio --now

echo "CRI runtime installed successfully"

# Install Kubernetes components
sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl
curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg <>

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] <> kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list > /dev/null
sudo apt-get update -y
sudo apt-get install -y kubelet="$KUBERNETES_VERSION" kubectl="$KUBERNETES_VERSION" kubeadm="$KUBERNETES_VERSION"

# Install jq and set Kubelet node IP
sudo apt-get install -y jq
local_ip=$(ip --json addr show eth0 | jq -r '.[0].addr_info[] | select(.family == "inet") | .local')
echo "KUBELET_EXTRA_ARGS=--node-ip=$local_ip" | sudo tee /etc/default/kubelet > /dev/null

To run the script, provide execute permission.

chmod +x

Login as root and execute the script.


Step 4: Initialize Kubeadm Configuration (Only in Controller)

In this step, we will initialize the control plane with configurations required for the cloud controller manager.

Create configuration file kubeadm.config

kind: ClusterConfiguration
    bind-address: ""
    cloud-provider: external
clusterName: kubernetes
    bind-address: ""
    bind-address: ""
    cloud-provider: external
  podSubnet: ""
  serviceSubnet: ""
kind: InitConfiguration
    cloud-provider: external

In certSANs instead of choose your controller’s public IP and change the node registration name to your controller’s private DNS (hostname we set in step 2). Modify the pod and service subnet if required.

Initialize the configuration to bootstrap the kubeadm configuration.

kubeadm init --config=kubeadm.config

At the end of the initialization, a token will be generated by the controller to join the workers. This is required to create the controller configuration file.

It would like the following highlighted token. Note down the token and cert hash.

kubeadm join --token wthapw.xgfjvonbiidea4nr --discovery-token-ca-cert-hash sha256:8b2127f960d88432fb35fd7488a501ad189e1a4ab319158d0e01a1db7fec96d7

Step 5: Join the Worker Nodes (Only in Worker Nodes)

Create a configuration file in each worker node kubeadm-join-config.yaml.

kind: JoinConfiguration
    token: 30am0s.hleb5xs1dyz4ridc   
    apiServerEndpoint: ""
      - "sha256:8b2127f960d88432fb35fd7488a501ad189e1a4ab319158d0e01a1db7fec96d7" 
    cloud-provider: external

Change the token apiServerEndpoint, and caCertHashes values to the intended values that your control plane generated when it initialized. name should be the full hostname of the respective node.

Modify the node registration name to your worker node’s private DNS address.

To join the workers to the controller, use the following command.

kubeadm join --config kubeadm-join-config.yaml

Modify the intended values of the configuration file and run the kubeadm join command on each worker node to join with the controller node.

Step 6: Tag AWS Resources

Tagging is essential to configure the Cloud Controller Manager because it ensures which AWS resources are used by which cluster.

For example, if a cluster uses AWS Network Load Balancer and now the cluster is being destroyed, the specified NLB also be destroyed and not affect other resources. To tag the resource, the cluster-ID is required

To find the Cluster ID, use the following command.

kubectl config view

The output would be like this.

apiVersion: v1
- cluster:
    certificate-authority-data: DATA+OMITTED
  name: kubernetes
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
- name: kubernetes-admin
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

In the clusters section, the value of name is taken as the Cluster ID.

If the AWS resources are managed by one cluster, we can provide the tag key and the tag value as owned and if the resources are managed by multiple clusters, the tag key would be the same as and the tag value should be mentioned as shared.

Tags should be added to resourecs the controller and worker node consumes, such as VPC, Subnet, EC2 instance, Security Group, etc.

Here is an example.

Step 7: Configure the Cloud Controller Manager

Clone the AWS Cloud Controller repository to the controller plane node where you have the Kubectl access.

git clone

Navigate to the base directory. It has all the kubernetes manifests for the cloud controller manager and the Kustomize file.

cd cloud-provider-aws/examples/existing-cluster/base

Create the daemonset using the following command. -k is for Kustomize.

kubectl create -k .

To verify the daemonset is running properly, use the following command.

kubectl get daemonset -n kube-system

To ensure the CCM pod is running, use the following command.

kubectl get pods -n kube-system

Step 8: Provision Network Load Balancer

To test if the Cloud controller manager is working, we will deploy a sample Nginx deployment and expose it over service type LoadBlancer. The cloud controller manager should be able to create a Network load balancer that routes traffic to the nginx deployment.

Create a deployment file for the Nginx web server using the following YAML. Execute it directly.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
  name: nginx-deployment
  replicas: 3  
      app: nginx
        app: nginx
      - name: nginx-container
        image: nginx:latest
        - containerPort: 80

Create a service file to provision a load balancer nginx-service-nlb.yaml. Here the annotation nlb is very important. It tells the cloud controller manager to create a Network Load Balancer. By default, it deploys a classic load balancer.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
  name: nginx-service
  annotations: nlb
    app: nginx
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

To ensure the deployment is running properly, use the following command.

kubectl get pods

To ensure the service is running properly, use the following command.

kubectl get svc

You should see the load balancer endpoint in the output as highlighted below.

$ kubectl get svc nginx-service
NAME            TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)        AGE
nginx-service   LoadBalancer   80:30249/TCP   39s

The registration process takes a few minutes. If you check the load balancer after a while you should see the worker nodes registered as healthy targets. The service NodePort would be used as a health check port.

AWS network loadbalancer as Kubernetes service with worker nodes as registered targets.

Once the nodes are registered to the NLB, you should see the Nginx homepage if you visit the Load balancer URL as shown below.

Accessing Nginx deployment over NLB URL

The following image shows the CCM load balancer traffic workflow to pods.

Possible Errors & Troubleshooting

To find the logs of the CCM pod, first, you need to find the pod name and use the following command

kubectl logs -n kube-system aws-cloud-controller-manager-zwctc

Following are the errors I have faced in this setup

Error 1: Cloud provider could not be initialized

E1028 08:39:59.394822       1 tags.go:95] Tag "KubernetesCluster" nor "" not found; Kubernetes may behave unexpectedly.
F1028 08:39:59.394858       1 main.go:106] Cloud provider could not be initialized: could not init cloud provider "aws": AWS cloud failed to find ClusterID


Ensure all resources which are used by CCM (controller and worker) should be tagged with the correct cluster-ID.


Error 2: node has no providerID

E1028 13:25:07.544115       1 node_controller.go:277] Error getting instance metadata for node addresses: error fetching node by provider ID: Invalid format for AWS instance (), and error by node name: could not look up instance ID for node "ip-172-31-30-122": node has no providerID


Before starting the cluster configuration, ensure all the servers’ hostname is their private DNS address also keep in mind that the configuration files also use private DNS addresses.


root@ip-172-31-21-29:~# hostname

Error 3: couldn’t get current server API group list

E1030 06:04:54.829375    9140 memcache.go:265] couldn't get current server API group list: Get "<http://localhost:8080/api?timeout=32s>": dial tcp connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?


Ensure you have a copy of admin.conf file in your home directory, if not make a copy using cp command

cp /etc/kubernetes/admin.conf ~/.kube/config


root@ip-172-31-21-29:~# ls -l /etc/kubernetes/ ~/.kube/
total 36
-rw------- 1 root root 5648 Oct 30 05:17 admin.conf
-rw------- 1 root root 5676 Oct 30 05:48 controller-manager.conf
-rw------- 1 root root 2112 Oct 30 05:49 kubelet.conf
drwxr-xr-x 3 root root 4096 Oct 30 06:17 manifests
drwxr-xr-x 3 root root 4096 Oct 30 05:17 pki
-rw------- 1 root root 5624 Oct 30 05:17 scheduler.conf

total 12
drwxr-x--- 4 root root 4096 Oct 30 06:07 cache
-rw------- 1 root root 5648 Oct 30 06:07 config

Error 4: error execution phase preflight

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher


Usually, this error happens when you remove the worker node and try to join again. To resolve this issue, remove the kubelet.conf and ca.crt.

sudo rm -f /etc/kubernetes/kubelet.conf
sudo rm -f /etc/kubernetes/pki/ca.crt

Then join the worker node again using the following command.

kubeadm join --config kubeadm-join-config.yaml --ignore-preflight-errors=Port-10250


I have done a high-level setup and tested provisioning the Network load balancer.

Next, I will be testing storage provisioning using the cloud controller manager.

I have added the steps after several trials and errors due to less information in the documentation. If you face any error during the setup, do drop an comment and I will look into it. To explore more, you can refer to the official documentation.

Also, if you are getting started with the Kubeadm cluster for CKA certification, take a look at the detailed kubeadm setup guide.

  1. I have followed this doc its very helpfull for me but when i install the nginx ingress i have facing issue with loadbalancer like its created the NLB when i check the health for instance in target group its shown one node is healthy and another node is unhealthy can you please help on this

    1. Hi Subhash,

      This guide is focussed on understanding how the cloud controller manager works with Kubeadm cluster. You can check with the Kubeadm community on this issue.

  2. Your deployed the nginx application and also service as load balance but i need to deploy the nginc-ingress control for communicating different micro services i have followd your doc but i can’t able to deployed the nginx-ingress controller please help on this how to deploy the nginx ingress controller for kubeadm

Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like