Kubernetes monitoring With Sensu: Setting up Container Sidecar Agent

The rise of containerized infrastructure has caused us to rethink the way we build and deploy our applications. But, with all that speed and flexibility comes challenges, especially when it comes to maintaining visibility into your (often multi-generational) infrastructure.

In this post, I’ll discuss some of the challenges to monitoring containers and Kubernetes (the go-to for container orchestration), talk about some of the data sources and a few methods for monitoring Kubernetes, and walk you through a tutorial on monitoring containers with Sensu, a multi-cloud monitoring tool.  

Container monitoring: the challenges

In a containerized world, your applications are constantly moving, making it even more difficult to keep tabs on them. It’s also the norm to have distributed applications, which causes a whole new set of problems when it comes to observability. Said another way, this new reality of distributed, constantly moving infrastructure means there are many smaller pieces to monitor. Similarly, it’s important to keep track of the labels and annotations associated with pods and containers.

Monitoring Kubernetes: the data sources

When it comes to monitoring Kubernetes, there are essentially four data sources you’re plugging into your monitoring tool:

1. The Kubernetes hosts running the Kubelet. The most common way to get data out of these hosts is to use the Prometheus node exporter to scrape data from Kubernetes and expose system resource telemetry data on an HTTP endpoint.
2. The Kubelet metrics, which includes metrics for apiserver, kube-scheduler, and kube-controller-manager.
3. The Kubelet’s built-in cAdvisor, which collects, aggregates, processes, and exports metrics for your running containers.
4. kube-state-metrics, which gives you data at the cluster level, such as all the pods you have configured and their current state.

Monitoring Kubernetes: the sidecar pattern

Of these data sources, we tend to prefer kube-state-metrics, as it gives you most of what you need without overloading you with information. I’ll use that for my tutorial of container monitoring with Sensu, but wanted to note that Sensu can also scrape Prometheus metrics (but that’s a whole other post — for now, feel free to check out our Prometheus collector asset).

Using the sidecar pattern, you can deploy a Sensu agent alongside your application container. A sidecar approach makes it possible for each Kubernetes pod to host your application container alongside other containers running support processes, such as the Sensu agent. Since all containers running inside a pod all share the same network space, your applications can talk to Sensu as if they were running in the same container.

Tutorial

In this tutorial, we’ll install and configure sensuctl (Sensu’s command-line tool), deploy the Sensu backend using a Kubernetes deployment, and deploy Sensu agent sidecars. This tutorial assumes you have Kubernetes installed. If you don’t, take a look at setting up Minikube which will make things easier for the tutorial. 

1. Deploy the Sensu backend using a Kubernetes deployment

Download the sensu-kube-demo repo. Use this sensu-backend.yaml file from the repo and the following command to deploy Sensu:

kubectl create -f go/deploy/sensu-backend.yaml 

The go/deploy/ path reference is relative to the top directory of the repository checkout.

You will also want to establish a Kubernetes ingress policy and host DNS configuration that will allow you to communicate with the sensu-backend running inside of the Kubernetes node. For the rest of the tutorial, I will assume you have configured things such that sensu.local tcp port 80 maps to the sensu-backend pod’s TCP port 8080, the default port for the sensu-backend API service. The sensu-kube-demo repository includes an appropriate nginx ingress policy example.

2. [Optional] Install sensuctl on your workstation

sensuctl is a command-line tool for managing resources within Sensu. It works by calling Sensu’s HTTP API to create, read, update, and delete resources, events, and entities. The below instructions are for Ubuntu — check out our documentation for sensuctl installation on CentOS, Windows, and macOS.

To install sensuctl on Ubuntu:

1. Add the Sensu repository

   curl -s https://packagecloud.io/install/repositories/sensu/stable/script.deb.sh | sudo bash

2. Install the sensu-go-cli package

   sudo apt-get install sensu-go-cli

In order to use sensuctl from your workstation to communicate to the containerized sensu-backend, you’ll need to make sure to configure your Kubernetes ingress controller and network DNS settings to allow access to the sensu-backend API outside of the Kubernetes cluster.

To save time, for this tutorial you can alternatively use the sensuctl provided in the sensu-backend container. In general you’ll want to use sensuctl installed natively in your workstation environment for ease of operation.

3a. Configure sensuctl to use the built-in admin user on your workstation

If you are using the sensuctl configured on your workstation, enter the following to configure sensuctl:

sensuctl configure -n \
--username 'admin'\
--password 'P@assw0rd!'\
--namespace default\
--url 'http://sensu.local' 

Please note you’ll need to adjust the URL to match your Kubernetes cluster ingest configuration and DNS configuration.

3b. Configure sensuctl to use the built-in admin user inside the cluster

Open up a new terminal window and start a shell inside the sensu-backend pod:

kubectl exec -it sensu-backend-<replace> -- /bin/sh

You can discover the correct pod name with:

kubectl get pods -l app=sensu-backend

sensuctl configure -n \
--username 'admin'\
--password 'P@assw0rd!'\
--namespace default\
--url 'http://localhost:8080' 

Here you can use the localhost URL as if the sensu-backend were running locally, because your sensuctl is operating in the same pod, sharing the pod network with the sensu-backend container.

4. Create the Sensu namespace for the dummy application

All the agents used in this tutorial operate in a Sensu namespace named demo.  We’ll need to create the namespace now using sensuctl:

sensuctl namespace create demo

We can optionally reconfigure sensuctl to use the demo namespace as default:

sensuctl configure -n \
--username 'admin'\
--password 'P@assw0rd!'\
--namespace demo\

5. Deploy Sensu agent sidecars

Using the aforementioned sidecar pattern and the dummy.sensu.yaml file, enter the following to deploy Sensu agent sidecars for two example app instances using a Kubernetes deployment:

kubectl apply -f go/deploy/dummy.sensu.yaml

If you have set the default namespace to demo, use sensuctl entity list to see the agent containers, otherwise, use sensuctl entity list –namespace demo to set the namespace explicitly. Each agent is running in a separate pod as a sidecar to support the dummy application as per the deployment configuration in the dummy.sensu.yaml file. From here you can use sensuctl to create Sensu checks to examine the running dummy service in each pod by using the dummy subscription, or a specific pod using the specific agent associated with each agent. 

For more on container monitoring with Sensu — including using the Sensu web UI to view events and setting up workflows to Slack and InfluxDB — check out our interactive tutorial. Thanks for reading, and happy monitoring!