Cloud Based Docker Container Monitoring Using Datadog

Cloud Based Docker Container Monitoring

2016 has been the year of Docker containers, as it has been the latest and the most advanced technology the IT industry is talking about. With Docker, companies are speeding up the production and version releases without disturbing the programming and deployment environment.

Docker Container Monitoring

Containers resolve several operational issues, but they have their problems, too. They come and go very quickly; hence, Docker monitoring is extremely crucial. You cannot just create containers without monitoring their performance and results. It is just like driving on a crowded road without following the rules. Docker monitoring is must, to ensure that you find and fix issues before they turn into a nightmare. Also, the infrastructure monitoring is a must to know the impact of processes on CPU usage, memory, and other hardware.

Though Docker containers have eased the process of developing and running the applications, Docker users and organizations need a platform from which they can perform monitoring for their Docker-based applications and their distribution. These requirements are exceeded by Datadog, the leading cloud-based monitoring platform for cloud applications.

What is Datadog and its relation with Docker?

Datadog is the world’s leading provider of SaaS-based monitoring services for cloud
applications. It has been used by many top IT companies to streamline Docker monitoring.
Now, Docker and Datadog work together to enhance cloud-based application development
monitoring. Datadog can be integrated with Docker with the help of Docker API to enable the engineering teams to monitor their docker applications with minimal configuration.

Also, Datadog has an excellent Docker integration plugin which provides an enhanced
monitoring dashboard for container-based monitoring.

Getting Started with Datadog

Follow the steps given below to set up docker monitoring using Datadog.

Step 1: Head over to and sign up for a 14-day free trial.


datadog free trial


Step 2: On the next page, select docker from the list of applications.




Step 3: In the next page, select docker from the list . Once selected, you will get the docker agent installation script on the same page as shown below.


Screen Shot 2016-08-20 at 5.21.08 PM_mini


Step 4:  Copy the docker agent installation command and run in on your docker host.  An example command is shown below.

docker run -d --name dd-agent -v /var/run/docker.sock:/var/run/docker.sock:ro -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=53c7f2a43aaf96b327a392ef846e33ec datadog/docker-dd-agent:latest

Note: API key is a unique key for your account. Make sure you use the agent installation script from your account.

Step 5: Once the agent docker container agent is deployed, it will register all your docker metrics on Datadog dashboard as shown below.

Screen Shot 2016-08-20 at 5.27.08 PM_mini

Step 6: Install the docker integration plugin using the option in the integration dashboard. Once installed, click dashboards option in the left pane and select docker under “Integration Dashboard”. There you can see all the information about your containers (Memory, CPU, Network, IO etc) running on your docker host.

Screen Shot 2016-08-20 at 5.38.33 PM_mini

Now, whenever you deploy a new container on to the host, the data about that container will be sent to Datadog with all the containers details and metrics such as memory, CPU etc.

Why docker monitoring is required and how Datadog helps with it?

To optimize the performance of a docker-based application, its monitoring is required. Without optimal docker container monitoring, evaluating the performance of an application is not possible. You need insights of an application to understand its performance matrix.

Datadog has made monitoring easier and faster by providing a platform that allows the users to quickly gather application insights and performance matrix across dockers, their engines, and hosts. By evaluating this matrix, the users can work with the related teams to address the performance issues.

What are the features of Datadog’s Cloud-based monitoring?

  1. Collects matrix in just a few seconds without putting any impact on the containers
  2. Allows building real-time dashboards that are not limited to just display a summary; it
    generates high-resolution graphs in real-time
  3. Upon detecting critical performance issues, Datadog generates alerts, which can be
    received via an e-mail
  4. Displays the location of the alerts so that the user can quickly locate the problem area
  5. It provides the user with full API access

Facts about Datadog Adoption by companies using Docker

  1. Since 2015, above 30% companies who use Docker have adopted Datadog as a
    platform for monitoring the performance of Docker applications, finding issues, and
    generating problem alerts
  2. By logging the rapid growth, Datadog covers the 10% of the hosts that they monitor
  3. The surprising fact is the bigger companies are heartily adopting Datadog for docker container monitoring.
  4. Many companies that have started using Docker also adopt Datadog within 30 to 60

For more information about Datadog, and their integrations, or 14-day free trial, click here.

Cloud Based Docker Container Monitoring

How To Setup a Zookeeper Cluster – Beginners Guide

setup a zookeeper cluster

Zookeeper is a distributed coordination tool which can be used for various distributed workloads. In this article, we have explained the necessary steps to install and configure zookeeper cluster with a definite quorum.

Setup a Zookeeper Cluster


1. Three VM’s. (Forms a quorum). For high availability cluster, you can go with any odd number above 3. For example, if you set up 5 servers, the cluster can handle two failed nodes and so on.

2. VM’s with inbound connections on ports 2888, 3888 and 2181. If IPtables is enabled, make sure you enable the specified ports because zookeeper communication happens through these ports.

Note: If you are using AWS or any other cloud provider, apart from server level inbound connections, make sure you enable the security groups or endpoints for the zookeeper ports.

Install and Configure Zookeeper

You need to perform the following steps in all the three VM’s.

1. Update your server.

 sudo yum -y update

2. Install Java if is not installed.

 sudo yum  -y install java-1.7.0-openjdk

3. Download zookeeper. b If you wish to choose another version, you can get the download links from here.


4. Untar the application to /opt folder

 sudo tar -xf zookeeper-3.5.2-alpha.tar.gz -C /opt/

5. Rename the zookeeper app directory

 cd /opt
sudo mv zookeeper-* zookeeper

6. Create a zoo.cfg file in /opt/zookeeper/conf directory with the configurations shown below.


In the above code, server 1, 2 and 3 represent our three zookeeper servers. You need to replace the Zookeeper_IP with relevant IP or resolvable hostnames.

7. Create a zookeeper directory in the lib folder. Thar will be zookeepers data directory as mentioned in the zoo.cfg file.

 sudo mkdir /var/lib/zookeeper

8. Create a file name myid in /var/lib/zookeeper/ directory .

 sudo touch /var/lib/zookeeper/myid

9. Each zookeeper server should have a unique number in the myid file. For example, server 1 will have value 1, server 2 will have value 2 and so on.

server 1

 sudo sh -c "echo '1' > /var/lib/zookeeper/myid"

server 2

 sudo sh -c "echo '2' > /var/lib/zookeeper/myid"

server 1

 sudo sh -c "echo '3' > /var/lib/zookeeper/myid"

Configuring Zookeeper as a Service

Zookeeper can be started and stopped using the scripts. But it is good to run it as a service to manage it in an elegant way.

1. Open zkServer.zh file for editing.

 sudo vi /opt/zookeeper/bin/

2. Add the following below the shebang “#!/usr/bin/env bash” to add zookeeper to the system start up.

# description: Zookeeper Start Stop Restart
# processname: zookeeper
# chkconfig: 244 30 80

3. Find a line which says “# use POSTIX interface, symlink is followed automatically”. Replace the existing variables after that line with the following.

ZOOSH=`readlink $0`
ZOOBIN=`dirname $ZOOSH`

4. Now, create a symlink for the zookeeper service.

sudo ln -s /opt/zookeeper/bin/ /etc/init.d/zookeeper

5. Add zookeeper to the boot menu.

 sudo chkconfig zookeeper on

6. Now, Restart all the servers.

 sudo  init 6

7. Once restarted, you can manage zookeeper servers using the following commands.

sudo service zookeeper status
sudo service zookeeper stop
sudo service zookeeper start
sudo service zookeeper restart

8. When you check the status, it should produce an output like the following.

ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader

out of three servers, one will be in leader mode and other two will be in follower mode.

setup a zookeeper cluster