[Solved] ClusterInformation: connection is unauthorized: Unauthorized - Failed to create pod sandbox: rpc error

Issue: When working with Kubeadm cluster with Calico CNI plugin, the container fails to launch and if you describe the pod you may get the following error.

Warning  FailedCreatePodSandBox  16m                 kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_webserver-667ddc69b6-wq689_default_ffa6a237-7dc6-4bc4-9444-b0146a5b7f21_0(6dfe713911b0d60f98cf464a11928b041c885ff9dd3c59323ca5271be1df632b): error adding pod default_webserver-667ddc69b6-wq689 to CNI network "k8s-pod-network": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized

Solution

This error can be rectified by restarting the Calico pods running in the kube-system namespace.

Get the labels of calico pods using the following command.

kubectl get pods -n kube-system --show-labels

First delete all the calico pods using the following command.

kubectl delete pods -n kube-system -l k8s-app=calico-node

Once the calico pods are restarted, you you shouldn't see the error.

If you dont have enough CPU and memory available in the cluster, consider increasing the nodes so that calico pods will have enough resources to startup up. Otherwise the calico pods will crash again due to resource crunch.

Root Cause

This error primary happens in kubeadm due to temporary glitches. Restarting the pods will clear all the temporary problems. At least it did in my case.

Another reason could be the race condition in Calico configurations. This also gets rectified with calico pod restarts.

If pod restart doest solve the issue, then you can check the folllwing.

1. Not enough resources on node: If you a running a cluster with very less resources you might face this issue. When there is not resources for calico pods, it will create this calico issue. So ensure you have enough CPU memory resources available in the cluster.
2. Network Policies Blocking Access: check if there is any network policy blocking calico access to other components.
3. Incorrect Calico Configuration: A misconfiguration in calico could also lead to unauthorized errors. Check all the calico configurations.