How to setup ELK stack on Amazon EC2 – Getting Started Guide

ELK is the most popular log aggreration and analysis stack. ELK stands for Elasticsearch, Logstash and Kibana.

Elasticsearch – It is a NoSQL, analytics and search engine.

Logstash – It is a log aggregation tool.

Kibana – It is a dashboarding tool.

ELK stack on Amazon EC2

This guide will walk you through setting up a test ELK stack on amazon ec2 Ubuntu 14.04  instance. Follow the steps given below to have a working ELK stack.

Note: I assume that you have a ubuntu 14.04 server up and running in a public subnet.

1. Login and update the server.

sudo apt-get update

2. ELK stack need openjdk. Install it using the following command.

sudo apt-get install openjdk-7-jre-headless

3. Chef the jdk installion by checking its verison.

java -version

Installing Elasticsearch

4. Add the elasticsearch repository using the following command.

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

5. Elasticsearch key has to be added to the list using the following command.

echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

6. Update the server

sudo apt-get update

7. Install elasticsearch using the following command.

sudo apt-get install elasticsearch

8. Execute the following command to start the elasticsearch server.

sudo service elasticsearch start

9. Test the elasticsearch service using curl

curl localhost:9200

The output should look like the following.

{
  "status" : 200,
  "name" : "Merlin",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.3",
    "build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
    "build_timestamp" : "2015-10-15T09:14:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Installing and setting up Logstash

1. Add logstash to the list.

echo "deb http://packages.elasticsearch.org/logstash/1.5/debian stable main" | sudo tee -a /etc/apt/sources.list

2. Update the server.

sudo apt-get update

3. Install logstash.

sudo apt-get install logstash

4. Update the server.

sudo apt-get update

5. Start and check the status of logstash using the following commands.

sudo service logstash start
sudo service logstash status

Since it is a single host installation, we will direct all our logs to logstash using a configuration file.

6. create a file named /etc/logstash/conf.d/logstash-syslog.conf and copy the following contents on to the file.

    input {
    file {
    type => "syslog"
    path => [ "/var/log/messages", "/var/log/*.log" ]
    }
    }
    output {
    stdout {
    codec => rubydebug
    }
    elasticsearch {
    host => "localhost" # If you are running elasticsearch in different instance, use #the prive ip instead of localhost.
    }
    }

7. Now, restart the logstash server to make necessary changes.

sudo service logstash restart

Installing and configuring kibana

1. Download the kibana source file.

wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

2. Extract the source content

tar -xzf kibana

3. Create a kibana directory under /opt

sudo mkdir -p /opt/kibana

4. Move the extracted folder contents to the /opt/kibana directory.

sudo mv kibana-4.1.1-linux-x64/* /opt/kibana

5. Create a service file for startup using the following commands.

cd /etc/init.d && sudo wget https://raw.githubusercontent.com/akabdog/scripts/master/kibana4_init -O kibana4
sudo chmod +x /etc/init.d/kibana4
sudo update-rc.d kibana4 defaults 96 9

6. Start the kibana dashboard service.

sudo service kibana4 start

The above command would start kibana service on port 5601.

Now you can access the kibana dashboard on 5601 port with the public ip of your instance.