Month: December 2015

It is important to get visibility in to status and health of docker environments as the deployments grow larger. In this tutorial, we will look into few options for monitoring Docker containers.

Monitoring container using cAdvisor

cAdvisor is a tool created by google for their own container tool and later they added support for Docker containers. cAdvisor stands for container advisor. It helps you to gain insights on how much resource is being used by Docker containers and also helps you understand the performance characteristics on the running containers. cAdvisor has a GUI and it also exposes api’s to get obtain the data programmatically.

cAdvisor is well suited for monitoring your running containers and its resource usage. There is an official cAdvisor image in docker hub which comes configured with cAdvisor. Using that image is the best way to get started with its functionalities.

In this section we will look in to how to deploy a cAdviosor container to monitor out Docker containers.

Launching a cAdvisor container:

You can launch a cAdvisor container using google’s official image “google/cadvisor”. Launch a cAdvisor container using the following command.

sudo docker run \

--volume=/:/rootfs:ro \

--volume=/var/run:/var/run:rw \

--volume=/sys:/sys:ro \

--volume=/var/lib/docker/:/var/lib/docker:ro \

--publish=8080:8080 \

--detach=true \

--name=cadvisor \

google/cadvisor:latest

Accessing UI:

cAdvisor GUI can be accessed on the host port 8080. Point your browser to the IP of the host running cAdvisor followed by port 8080 http://<host IP>:8080.

Also read: Getting started with docker machine

Docker Host Monitoring Using Sensu

Sensu is an open source monitoring framework for self-hosted and centralized metric service.

Setting up Sensu server:

In this section we will learn how to set up a Sensu server using a Docker container. To deploy a sensu server, you can make use of a prebuild sensu server Docker image hiroakis/docker-sensu-server. This container will deploy sensu server, uchiwa web interface, rabbit-mq server, redis and sensu api. There is no dedicated functionality to monitor Docker, however, using plugin system you can configure status checks and container metrics.

Before deploying sensu server, you must create a check that has to be loaded in to the server. Follow the instructions below to deploy the sensu server container.

1. Create a directory named sensu and cd in to it.

mkdir sensu && cd sensu

2. Create a file name check-docker.json and copy the following content to it.

{

"checks": {

"load_docker_metrics": {

"type": "metric",

"command": "load-docker-metrics.sh",

"subscribers": [

"docker"

],

"interval": 10

}

}

}

The above check shows that all sensu client will have a script named load-docker-metrics.sh. We will deploy this script in all sensu clients (agents).

3. From the same directory you have the check-docker.json file, run the following docker command to deploy the sensu server.

$ sudo docker run -d --name sensu-server         \

-p 3000:3000                                 \

-p 4567:4567                                 \

-p 5671:5671                                 \

-p 15672:15672                               \

-v $PWD/check-docker.json:/etc/sensu/conf.d/check-docker.json  hiroakis/docker-sensu-server

4. Now, you will be able to launch the uchiwa dashboard at http://host-ip:3000.

Setting up Sensu client:

Now we have our sensu server up and running. Next step is to deploy sensu clients (agents) on hosts running Docker docker containers.

Follow the instructions below to configure sensu client on nodes running Docker containers.

1. Create a directory name sensu-client and cd in to the directory.

$ mkdir  sensu-client && cd sensu-client 

2. While deploying sensu server, we created a check saying that all sensu agents will be running a script named load-docker-metrics.sh. Create a file name load-docker-metrics.sh and copy the following script on to it. This script uses native Docker API to get the list of running containers, images etc.

#!/bin/bash

set -e 

# Get the containers count

containers_running=$(echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
    | tail -n +5           \

    | python -m json.tool  \

    | grep \"Id\"          \

    | wc -l) 

# Get the count of all the containers

total_containers=$(echo -e "GET /containers/json?all=1 HTTP/1.0\r\n" | nc -U /var/run/docker.sock \

 | tail -n +5           \

 | python -m json.tool \

 | grep \"Id\"          \

 | wc -l)

# Count all images

images_count=$(echo -e "GET /images/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock         \

 | tail -n +5           \

 | python -m json.tool \

 | grep \"Id\"          \

 | wc -l)

echo "docker.HOST_NAME.containers_running ${containers_running}"

echo "docker.HOST_NAME.total_containers ${total_containers}"

echo "docker.HOST_NAME.images_count ${images_count}"

if [ ${containers_running} -lt 3 ]; then

    exit 1;

fi 

In the above script, replace HOST_NAME with the hostname of the server in which you are deploying the sensu client.

3. Make the script executable using the following command.

$ chmod 700 load-docker-metrics.sh 

4. From the directory you have load-docker-metrics script, execute the following docker command.

Note: In the following command, replace the following values.

SENSU_SERVER_IP: ip of host running sensu server

RABIT_MQ_USER : sensu (default value)

RABIT_MQ_PASSWORD : password (default value)

CLIENT_NAME CLIENT_IP : ip address of host running sensu client.

$ sudo docker run -d --name sensu-client --privileged  \

  -v $PWD/load-docker-metrics.sh:/etc/sensu/plugins/load-docker-    metrics.sh  \

  -v /var/run/docker.sock:/var/run/docker.sock   \

usman/sensu-client SENSU_SERVER_IP RABIT_MQ_USER \ RABIT_MQ_PASSWORD CLIENT_NAME CLIENT_IP 

5. Once the sensu client container is launched, the host will get registered to the sensu server in few seconds. Access the sensu uchiva dashboard and you can see the registered client with the checks configured.

6. If you click on the registered client and see the status of keepalive, all the values will be shown as 0. It means your Docker host is running without any critical issues. If it turns to any non-zero number, it means that the Docker daemon is not running as expected.

You might like: List of devops blogs and resources

Docker stats command

Docker client has a native functionality to inspect the resource consumption of Docker containers. You need to specifically mention the names of the containers with “docker stats” command to look in to its stats. If you haven’t specified the CPU usage for a container, the stats command will show the total memory available in the host machine. It doesn’t mean that a container has that much usable resources.

Execute the following command to see the stats of a container.

Syntax: docker stats <container name or id>

$ sudo docker stats tender_kowalevski 

To get the more information about container statistics, you can use docker stats api as given below.

Syntax: echo -e "GET /containers/<container_name>/stats HTTP/1.0\r\n" | nc -U /var/run/docker.sock 

$ echo -e "GET /containers/tender_kowalevski/stats HTTP/1.0\r\n" | nc -U /var/run/docker.sock

The response for above rest request is shown below.

 {

   "read":"2015-05-13T10:07:33.885214393Z",

   "network":{

      "rx_bytes":648,

      "rx_packets":8,

      "rx_errors":0,

      "rx_dropped":0,

      "tx_bytes":648,

      "tx_packets":8,

      "tx_errors":0,

      "tx_dropped":0

   },

   "cpu_stats":{

      "cpu_usage":{

         "total_usage":13763940722,

         "percpu_usage":[

            13763940722

         ],

         "usage_in_kernelmode":20000000,

         "usage_in_usermode":90000000

      },

      "system_cpu_usage":101902770000000,

      "throttling_data":{

         "periods":0,

         "throttled_periods":0,

         "throttled_time":0

      }

   },

   "memory_stats":{

      "usage":9994240,

      "max_usage":13570048,

      "stats":{

         "active_anon":9986048,

         "active_file":0,

         "cache":73728,

         "hierarchical_memory_limit":18446744073709551615,

         "inactive_anon":0,

         "inactive_file":0,

         "mapped_file":0,

         "pgfault":7903,

         "pgmajfault":0,

         "pgpgin":3641,

         },

      "failcnt":0,

      "limit":1040683008

   }

}