How to Upgrade Kubernetes Cluster Using Kubeadm?

kubernetes kubeadm upgrade

In kubernetes tutorial, I have added step-by-step guides to upgrade the Kubernetes cluster using Kubeadm. It is one of the important tasks in the CKA Exam.

In another blog, I have covered the Kubeadm cluster setup. You can follow it to deploy a fresh cluser for testing the upgrade.

Kubeadm Kubernetes Cluster Upgrade

Note: In this example, I will be upgrading Kubernetes version 1.23 to 1.24. Please replace the version numbers as per your existing version and upgrade requirement.

We need to run the upgrades in the following order

  1. Control plane node
  2. Worker Nodes

Also, we need to upgrade the following components on both the control plane and the worker nodes.

  1. Kubeadm
  2. Kubelet
  3. Kubectl

Before we get started you should know the cluster and node names. Execute the following command to get the node names.

kubectl get nodes

Upgrade The Control Plane (Master Node)

Following are the high level steps for the control plane upgrade

  1. Login to control plane
  2. Check the existing version
  3. Decide on the upgrade version by running a kubeadm upgrade plan (Same version patch or new version)
  4. unhold kubeadm and Install required kubeadm version
  5. Apply Kubeadm upgrade
  6. Evict all workloads from control plane except daemonsets
  7. Upgrade kubectl and kubelet.
  8. Make the control plane schedulable (Uncordon)

Lets get started with the Upgrade.

Step 1: Check the existing Kubeadm version

Login to the control plane and can check the existing version using the following command.

kubeadm version -o json

Step 2: Decide on the upgrade version

You can get the list of available Kubeadm versions using the following command

sudo apt update
sudo apt-cache madison kubeadm | tac

Also you can run a kubeadm upgrde plan to get upgrade suggestions.

sudo kubeadm upgrade plan

My current version is 1.23.6. I will be upgrading it to one higher version, ie, 1.24.6-0.

Step 3: unhold kubeadm and Install the required version

During the Kubeadm cluster installation process, you would have hold the kubeadm installation to prevent upgrades.

Now we need to unhold kubeadm and install 1.24.6-0 using the following command.

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm=1.24.6-00 && \
sudo apt-mark hold kubeadm

Step 4: Apply Kubeadm upgrade

Apply the upgrade using the following command.

sudo kubeadm upgrade node

Now if you check the kubeadm version, you can see the upgraded version.

kubeadm version -o json

Step 6: Drain the Node to evict all workloads.

To apply the upgrade we nee to evict all the workloads except daemonsets using the following command.

kubectl drain master-node --ignore-daemonsets

If pods are using local storage, you will get the following error. This is mostly occur in vagrant setup. In CKA exam, you will not probably face this issue.

error: unable to drain node "master-node" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/dashboard-metrics-scraper-799d786dbf-vvhnx, kubernetes-dashboard/kubernetes-dashboard-fb8648fd9-79jmp, continuing command...
There are pending nodes to be drained:
 master-node
cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/dashboard-metrics-scraper-799d786dbf-vvhnx, kubernetes-dashboard/kubernetes-dashboard-fb8648fd9-79jmp

To rectify that issues, you need to add the --delete-local-data as well.

sudo kubectl drain master-node --ignore-daemonsets --delete-local-data

Step 7: Upgrade Kubelet and Kubectl.

Unhold and upgrade kubectl and kubelet using the following commands.

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet=1.24.6-00 kubectl=1.24.6-00 && \
sudo apt-mark hold kubelet kubectl

Restrat the services

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 7: Uncordon the Node and Verify the Node Status

Use the following command to uncordon the control plane node

kubectl uncordon master-node

Verify the node status and version using the following command.

kubectl get nodes

You can see the control plane upgraded to the new version and nodes running on the old version as shown below.

Upgrade Worker Nodes

Worker node upgrade steps are similar to control plane upgrade. However the internal upgrade process differ from control plane.

All the following steps should be executed on the Worker node. If you have multiple worker nodes, upgrade one at a time.

Step 1: Unhold Kubeadm and Install Required Version

Install the required version of Kubeadm on worker nodes. Replace 1.24.6-00 with you version.

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm=1.24.6-00 && \
sudo apt-mark hold kubeadm

Step 2: Upgrde Kubeadm

As you are running the following command on a worker node, it skips all the steps required for the control plane and update only kubelet configuration required for a worker node.

sudo kubeadm upgrade node

Step 3: Drain the Node

Evict all the pods on the worker node by draining it. Replace worker-node01 with your worker node name.

kubectl drain worker-node01 --ignore-daemonsets

Step 4: Upgrade Kubelet & Kubectl

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet=1.24.6-00 kubectl=1.24.6-00 && \
sudo apt-mark hold kubelet kubectl

Restart kubelet

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 5: Uncordon worker node

kubectl uncordon worker-node01

Verify Cluster Upgrade

Now that we have upgraded the control plane node and the worker nodes, lets verify if the cluster is upgraded and working as expected.

Get the node information

kubectl get nodes

You should see the control plane and worker nodes upgraded to the specified version.

From the master node, you can use any one of the following kubectl commands to check the status of all the cluster components.

kubectl get --raw='/readyz?verbose'
curl -k https://localhost:6443/livez?verbose

Also, check if all the kube-system control node pods are running.

kubectl get po -n kube-system

Conclusion

Kubeadm utility helps you upgrade the Kubernetes cluster control plane and worker nodes without downtime. Ensure you verify the cluster and application status after the upgrade.

Also, Kubeadm cluster upgrade is one of the important topics in Kubernets CKA certification. If you are preparing for CKA exam, you can check out CKA exam guide.

If you are preparation for Kubernetes certification, checkout the best Kubernetes certification guide that matches your skillset.

4 Shares:
Leave a Reply

Your email address will not be published. Required fields are marked *

You May Also Like