How To Setup an Elasticsearch Cluster – Beginners Guide

In part I, we learned the basic concepts of elasticsearch. In this tutorial, we will learn how to set up an elasticsearch cluster with client, master and a data node.

Setup an Elasticsearch Cluster

For this setup to work, as a prerequisite, you need three virtual machines with enough memory. This tutorial is based on ubuntu server 14.04. You can set up an ubuntu server using vagrant, or on any cloud provider.

Do the following before we start configuring the server for elasticsearch.

1. Create three ubuntu 14.04 VM’s with 1GB RAM each.
2. Update all the servers using the following command.

3. Change the hostnames to es-client-01, es-master-01 and es-data-01 to match the client, master and data node roles.

4. Edit /etc/hosts file of all the nodes and make entries for all the nodes for the hostnames as shown below. Change the IP addresses with the IP addresses of your VM’s.

The above configuration is very important because we will be using the hostname for the nodes to communicate each other.

Setting up Client Node (es-client-01)

Now we have the base VM. Let’s start with elastic search configuration.

Install Latest Java

Elasticsearch needs java runtime as its core is java. You can install the latest java version by executing the following commands.

1. Add the official oracle java repository.

2. Now, refresh the package list.

3. You can now install java using the following command.

4. Once installed, verify the installation by checking java version.

Recommended: Elasticsearch Administration Course (Learn Using Free Trial)

Install Elasticsearch

1. Download the elasticsearch installation file

Note: At the time of writing, the release of elasticsearch is 2.2.0

2. Install the downloaded package.

Note: If you have downloaded any version other than 2.2.0, change the package name accordingly.

3. Start the elasticsearch service.

4. Our node es-client-01 has elasticsearch service running and we will consider as the client node. Also, you need to set elasticsearch to start automatically on bootup. Use the following command to do that.

5. Verify the elasticsearch service by sending a HTTP request to port 9200. By default elasticsearch run on port 9200.

READ  How To Install Latest Sonatype Nexus 3 on Linux

You would see a JSON response, which looks like the following.

The above output shows the name of the node, cluster name, and few other details.

If you do not specify a node name in the configuration, elasticsearch assigns a random name on every restart.

All the elasticsearch configurations are present in elasticsearch.yml file, which is located in /etc/elasticsearch folder.

6. Now, the elasticsearch.yml file has to be edited for the configuring the node as a client node. Open the elasticsearch.ym file located in /etc/elasticsearch directory and change the configurations as follows.

The configuration file has many sections like cluster, node, paths etc.

Note: Refer this config file for all the configurations explained below.

Under the cluster section, change the cluster name parameter.

Under node section, change the node name parameter and add other parameters as shown below.

Under network section, change the “network.host” parameter with the IP address of your client node.

Under discover section add the following.

The above parameters disable the multicast and send unicast message to the specified hosts. As we have already made the hosts entry for all the hostnames, the unicast messages will go the respective nodes.

4. Save the file and restart the elasticsearch service for changes.

5. Now, we need to make some system level changes. Open /etc/security/limits.conf file to change the file limits that can be used. Bu default it is 1024 for ubuntu. You can check this by running “unlimited -n” command.

Add the following lines at the end of the file.

6. Open /etc/pam.d/common-session file and add the following line.

7. Open /etc/pam.d/common-session-noninteractive and add the following.

READ  Packer Tutorial For Beginners - Automate AMI Creation

8. It is recommended to have the heap size as half as the RAM. This tutorial is based on 1 GB RAM VM. So we will configure 512 MB swap space.

You need to set an environment variable for elasticsearch heap size. You can do this by editing the /etc/environment file. The file should look like the following.

Once edited, you should reboot the server.

Setting Up Master and Data Node

Follow all the steps we used to setup the client node for the master and data node. Only while configuring the elasticsearch.yml file just uses the data given below. All the other steps are same for all the nodes.

For master node (elasticsearch.yml)

Under node section of the elasticsearch.yml file, add the following. Refer this file.

Under network section, change the “network.host” parameter. Change the IP address accordingly.

For data node (elasticsearch.yml)

Under node section, add the following. refer this for configurations.

Under network section, replace the data nodes IP address as you did for the client and master nodes.

Once you configure all the three nodes, restart the elasticsearch service on all the three nodes.

Now you will have working elasticsearch cluster.

Installing elasticsearch GUI plugin

Once you setup an elasticsearch cluster, you can view the cluster status on the client node(es-client-01) using the following command.

But the but is not that easy to comprehend. So you can make use of the elasticsearch head plugin to view the cluster details in the browser UI.

We will install this plugin on our client node. To install the plugin, navigate to “/usr/share/elasticsearch/bin” directory and execute the following command.

Restart the elasticsearch service for the plugin to work.

Now, if you access http://<IP>:9200/_plugin/head/ in your browser, you will be able to see all the cluster details.

Wrapping Up

In this tutorial, I have explained all the steps to setup a three-node elasticsearch cluster. In the next article, I will cover more on indexing strategies for elasticsearch.

Also, you can take a look at the devopscube vagrant repository for setting up the three node cluster. Elasticsearch vagrant cluster setup

Free DevOps Resources

Get DevOps news, tutorials and resources in your inbox. A perfect way If you want to get started with devops. Like you, we dont like spam.

7 comments

  1. Hanukido Priori Reply

    Thank you very much! The best ES installation and configuration tutorial I’ve seen on web. I’m not actually familiar with linux, but it seems it’s enough to follow the guide!!!!

  2. sam Reply

    Thank you soo much. Lets say i have two host and i would like to make it has automated installation/upgrade, how you can suggest me to make it automated?

    • devopscube Post authorReply

      Hi Sam,

      You can use configuration management tools like chef, Ansibe etc for automating the whole confiuration.

  3. Jacob Reply

    Great post thank you.

    I have a clustering question…. Is it possible to create an ES cluster using nodes in AWS but where the nodes are in different locations say one in Europe, one in China, and one in US. Could these nodes in different locations be clustered together ?

    Thanks

  4. Vinod H Y Reply

    from logstash, how to direct the output to elasticsearch cluster?

Leave a Reply

Your email address will not be published. Required fields are marked *