How to setup ELK stack on Amazon EC2 – Getting Started Guide

ELK stack on amazon ec2

ELK is the most popular log aggreration and analysis stack. ELK stands for Elasticsearch, Logstash and Kibana.

Elasticsearch – It is a NoSQL, analytics and search engine.

Logstash – It is a log aggregation tool.

Kibana – It is a dashboarding tool.

ELK stack on Amazon EC2

This guide will walk you through setting up a test ELK stack on amazon ec2 Ubuntu 14.04  instance. Follow the steps given below to have a working ELK stack.

Note: I assume that you have a ubuntu 14.04 server up and running in a public subnet.

1. Login and update the server.

sudo apt-get update

2. ELK stack need openjdk. Install it using the following command.

sudo apt-get install openjdk-7-jre-headless

3. Chef the jdk installion by checking its verison.

java -version

Installing Elasticsearch

4. Add the elasticsearch repository using the following command.

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

5. Elasticsearch key has to be added to the list using the following command.

echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

6. Update the server

sudo apt-get update

7. Install elasticsearch using the following command.

sudo apt-get install elasticsearch

8. Execute the following command to start the elasticsearch server.

sudo service elasticsearch start

9. Test the elasticsearch service using curl

curl localhost:9200

The output should look like the following.

{
  "status" : 200,
  "name" : "Merlin",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.3",
    "build_hash" : "05d4530971ef0ea46d0f4fa6ee64dbc8df659682",
    "build_timestamp" : "2015-10-15T09:14:17Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Installting and setting up Logstash

1. Add logstash to the list.

echo "deb http://packages.elasticsearch.org/logstash/1.5/debian stable main" | sudo tee -a /etc/apt/sources.list

2. Update the server.

sudo apt-get update

3. Install logstash.

sudo apt-get install logstash

4. Update the server.

sudo apt-get update

5. Start and check the status of logstash using the following commands.

sudo service logstash start
sudo service logstash status

Since it is a single host instaltion, we will direct all our logs to logstach using a configuration file.

READ  List of Best Cloud Computing Certifications

6. create a file named /etc/logstash/conf.d/logstash-syslog.conf and copy the following contents on to the file.

    input {
    file {
    type => "syslog"
    path => [ "/var/log/messages", "/var/log/*.log" ]
    }
    }
    output {
    stdout {
    codec => rubydebug
    }
    elasticsearch {
    host => "localhost" # If you are running elasticsearch in different instance, use #the prive ip instead of localhost.
    }
    }

7. Now, restart the logstash server to make necessary changes.

sudo service logstash restart

Installing and configuring kibana

1. Download the kibana source file.

wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

2. Extract the source content

tar -xzf kibana

3. Create a kibana directory under /opt

sudo mkdir -p /opt/kibana

4. Move the extracted folder contents to the /opt/kibana directory.

sudo mv kibana-4.1.1-linux-x64/* /opt/kibana

5. Create a service file for startup using the following commands.

cd /etc/init.d && sudo wget https://raw.githubusercontent.com/akabdog/scripts/master/kibana4_init -O kibana4
sudo chmod +x /etc/init.d/kibana4
sudo update-rc.d kibana4 defaults 96 9

6. Start the kibana dashboard service.

sudo service kibana4 start

The above command would start kibana service on port 5601.

Now you can access the kibana dashboard on 5601 port with the public ip of your instance.

Free DevOps Resources

Get DevOps news, tutorials and resources in your inbox. A perfect way If you want to get started with devops. Like you, we dont like spam.

3 comments

  1. bjuggs Reply

    Great tutorial, thank you. I was unable to get `sudo service kibana4 start` to work. “kibana4: unrecognized service.” Any idea what might cause this? Thanks again!

  2. Vignesh Reply

    Great Tutorial. I installed ELK in a fresh AWS instance. And I followed the same steps that is provided above. Installed everything successfully. I started all the ELK services (i.e Elasticsearch, Logstash ,Kibana) too without any errors. But when I try to run the output in the browser, (http://localhost:5601 or http://IPAddress:5601 – Kibana UI) or (http://localhost:5601 or http://IPAddress:9200 – Elasticsearch), it says The output cannot be displayed. I am not sure if there is any error. Any help would be appreciated.

Leave a Reply

Your email address will not be published. Required fields are marked *