It is important to get visibility in to status and health of docker environments as the deployments grow larger. In this tutorial, we will look into few options for monitoring Docker containers.
Monitoring container using cAdvisor
cAdvisor is a tool created by google for their own container tool and later they added support for Docker containers. cAdvisor stands for container advisor. It helps you to gain insights on how much resource is being used by Docker containers and also helps you understand the performance characteristics on the running containers. cAdvisor has a GUI and it also exposes api’s to get obtain the data programmatically.
cAdvisor is well suited for monitoring your running containers and its resource usage. There is an official cAdvisor image in docker hub which comes configured with cAdvisor. Using that image is the best way to get started with its functionalities.
In this section we will look in to how to deploy a cAdviosor container to monitor out Docker containers.
Launching a cAdvisor container:
You can launch a cAdvisor container using google’s official image “google/cadvisor”. Launch a cAdvisor container using the following command.
sudo docker run \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ google/cadvisor:latest
Accessing UI:
cAdvisor GUI can be accessed on the host port 8080. Point your browser to the IP of the host running cAdvisor followed by port 8080 http://<host IP>:8080.
Also read: Getting started with docker machine
Docker Host Monitoring Using Sensu
Sensu is an open source monitoring framework for self-hosted and centralized metric service.
Setting up Sensu server:
In this section we will learn how to set up a Sensu server using a Docker container. To deploy a sensu server, you can make use of a prebuild sensu server Docker image hiroakis/docker-sensu-server. This container will deploy sensu server, uchiwa web interface, rabbit-mq server, redis and sensu api. There is no dedicated functionality to monitor Docker, however, using plugin system you can configure status checks and container metrics.
Before deploying sensu server, you must create a check that has to be loaded in to the server. Follow the instructions below to deploy the sensu server container.
- Create a directory named sensu and cd in to it.
mkdir sensu && cd sensu
- Create a file name check-docker.json and copy the following content to it.
{
"checks": {
"load_docker_metrics": {
"type": "metric",
"command": "load-docker-metrics.sh",
"subscribers": [
"docker"
],
"interval": 10
}
}
}
The above check shows that all sensu client will have a script named load-docker-metrics.sh. We will deploy this script in all sensu clients (agents).
- From the same directory you have the check-docker.json file, run the following docker command to deploy the sensu server.
$ sudo docker run -d --name sensu-server \ -p 3000:3000 \ -p 4567:4567 \ -p 5671:5671 \ -p 15672:15672 \ -v $PWD/check-docker.json:/etc/sensu/conf.d/check-docker.json hiroakis/docker-sensu-server
- Now, you will be able to launch the uchiwa dashboard at http://host-ip:3000.
Setting up Sensu client:
Now we have our sensu server up and running. Next step is to deploy sensu clients (agents) on hosts running Docker docker containers.
Follow the instructions below to configure sensu client on nodes running Docker containers.
- Create a directory name sensu-client and cd in to the directory.
$ mkdir sensu-client && cd sensu-client
- While deploying sensu server, we created a check saying that all sensu agents will be running a script named load-docker-metrics.sh. Create a file name load-docker-metrics.sh and copy the following script on to it. This script uses native Docker API to get the list of running containers, images etc.
#!/bin/bash
set -e
# Get the containers count
containers_running=$(echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
| tail -n +5 \
| python -m json.tool \
| grep \"Id\" \
| wc -l)
# Get the count of all the containers
total_containers=$(echo -e "GET /containers/json?all=1 HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
| tail -n +5 \
| python -m json.tool \
| grep \"Id\" \
| wc -l)
# Count all images
images_count=$(echo -e "GET /images/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock \
| tail -n +5 \
| python -m json.tool \
| grep \"Id\" \
| wc -l)
echo "docker.HOST_NAME.containers_running ${containers_running}"
echo "docker.HOST_NAME.total_containers ${total_containers}"
echo "docker.HOST_NAME.images_count ${images_count}"
if [ ${containers_running} -lt 3 ]; then
exit 1;
fi
In the above script, replace HOST_NAME with the hostname of the server in which you are deploying the sensu client.
- Make the script executable using the following command.
$ chmod 700 load-docker-metrics.sh
- From the directory you have load-docker-metrics script, execute the following docker command.
Note: In the following command, replace the following values.
SENSU_SERVER_IP: ip of host running sensu server
RABIT_MQ_USER : sensu (default value)
RABIT_MQ_PASSWORD : password (default value)
CLIENT_NAME CLIENT_IP : ip address of host running sensu client.
$ sudo docker run -d --name sensu-client --privileged \ -v $PWD/load-docker-metrics.sh:/etc/sensu/plugins/load-docker- metrics.sh \ -v /var/run/docker.sock:/var/run/docker.sock \ usman/sensu-client SENSU_SERVER_IP RABIT_MQ_USER \ RABIT_MQ_PASSWORD CLIENT_NAME CLIENT_IP
- Once the sensu client container is launched, the host will get registered to the sensu server in few seconds. Access the sensu uchiva dashboard and you can see the registered client with the checks configured.
- If you click on the registered client and see the status of keepalive, all the values will be shown as 0. It means your Docker host is running without any critical issues. If it turns to any non-zero number, it means that the Docker daemon is not running as expected.
You might like: List of devops blogs and resources
Docker stats command
Docker client has a native functionality to inspect the resource consumption of Docker containers. You need to specifically mention the names of the containers with “docker stats” command to look in to its stats. If you haven’t specified the CPU usage for a container, the stats command will show the total memory available in the host machine. It doesn’t mean that a container has that much usable resources.
Execute the following command to see the stats of a container.
Syntax: docker stats <container name or id>
$ sudo docker stats tender_kowalevski
To get the more information about container statistics, you can use docker stats api as given below.
Syntax: echo -e "GET /containers/<container_name>/stats HTTP/1.0\r\n" | nc -U /var/run/docker.sock
$ echo -e "GET /containers/tender_kowalevski/stats HTTP/1.0\r\n" | nc -U /var/run/docker.sock
The response for above rest request is shown below.
{
"read":"2015-05-13T10:07:33.885214393Z",
"network":{
"rx_bytes":648,
"rx_packets":8,
"rx_errors":0,
"rx_dropped":0,
"tx_bytes":648,
"tx_packets":8,
"tx_errors":0,
"tx_dropped":0
},
"cpu_stats":{
"cpu_usage":{
"total_usage":13763940722,
"percpu_usage":[
13763940722
],
"usage_in_kernelmode":20000000,
"usage_in_usermode":90000000
},
"system_cpu_usage":101902770000000,
"throttling_data":{
"periods":0,
"throttled_periods":0,
"throttled_time":0
}
},
"memory_stats":{
"usage":9994240,
"max_usage":13570048,
"stats":{
"active_anon":9986048,
"active_file":0,
"cache":73728,
"hierarchical_memory_limit":18446744073709551615,
"inactive_anon":0,
"inactive_file":0,
"mapped_file":0,
"pgfault":7903,
"pgmajfault":0,
"pgpgin":3641,
},
"failcnt":0,
"limit":1040683008
}
}
2 comments
Hi,
I tried to use the REST GET command as you described in order to retrieve the stats of a specific container and I receive the following reply:
HTTP/1.0 200 OK
Server: Docker/1.10.3 (linux)
Date: Sun, 22 Jan 2017 15:39:56 GMT
Content-Type: text/plain; charset=utf-8
Can you please help me to retreive the stats of this container?
Thanks,
Avraham
I’d like to recommend using Dynatrace as a Docker monitoring solution. As highly dynamic environments like Docker grow rapidly, monitoring solutions must become smarter. Dynatrace uses artificial intelligence to auto-detect issues and provide root cause analysis in real time. You can read more about it at https://www.dynatrace.com/technologies/cloud-and-microservices/docker-monitoring/