Elastic Search, Logstash and Kibana

Overview

Authors: Samuel Ambroj Pérez, Kajorn Pathomkeerati, [1] [2]

Installation of ELK in one single machine (Debian 8)

Installation of Elasticsearch

Connect to the first machine (on the left side) that has been provided to you:

ssh gks@141.52.X.X

This machine is going to be used for a while, so do not connect yet to the second VM.

The gks user has sudo rights because is included in the sudoers file, so from your gks user, execute:

sudo -i bash

Update and upgrade all the packages:

# apt-get update
# apt-get upgrade

Change the timezone (optional):

# dpkg-reconfigure tzdata

Installation of aptitude, curl, openjdk (open source java) and chkconfig:

# apt install -y aptitude curl openjdk-7-jdk chkconfig

Download and install the Public Signing Key:

# wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Save the repository definition:

# echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list

Update in order to make the repository ready to use and install Elasticsearch:

# aptitude update && aptitude install elasticsearch

Start Elasticsearch:

# /etc/init.d/elasticsearch start

Check the status (two options):

# systemctl status elasticsearch
# /etc/init.d/elasticsearch status

Modify chkconfig in order to start ES when booting:

# chkconfig elasticsearch on

Elasticsearch is now installed. Congratulations! Let's continue with the installation of Logstash and finally Kibana.

Installation of Logstash

Download the .deb file:

# wget -q https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.4-1_all.deb 

Install it:

# dpkg -i logstash_1.5.4-1_all.deb

It would be enough, but when preparing the tutorial we saw the following WARNING:

WARN -- Concurrent: [DEPRECATED] Java 7 is deprecated, please use Java 8.
Java 7 support is only best effort, it may not work. It will be removed in next release (1.0).

So, we install the Oracle java, version 8.

Installation of Oracle Java 8

Download the tarball:

# wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u60-b27/jdk-8u60-linux-x64.tar.gz

Create the folder /opt/jdk and move the tarball there:

# mkdir /opt/jdk
# mv jdk-8u60-linux-x64.tar.gz /opt/jdk/

Extract the tarball there:

# cd /opt/jdk
# tar -xzvf jdk-8u60-linux-x64.tar.gz

Update alternatives:

# update-alternatives --install /usr/bin/java java /opt/jdk/jdk1.8.0_60/bin/java 100
# update-alternatives --install /usr/bin/javac javac /opt/jdk/jdk1.8.0_60/bin/javac 100

Display the priorities and the version of Java:

# update-alternatives --display java
# java -version

java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.6) (7u79-2.5.6-1~deb8u1) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

It is not pointing to the Oracle version, so we increase the value from 100 to 10000:

# update-alternatives --install /usr/bin/java java /opt/jdk/jdk1.8.0_60/bin/java 10000
# update-alternatives --install /usr/bin/javac javac /opt/jdk/jdk1.8.0_60/bin/javac 10000
# java -version

java version "1.8.0_60" Java(TM) SE Runtime Environment (build 1.8.0_60-b27) Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)

Installation of Kibana 4

Download Kibana4:

# wget https://download.elastic.co/kibana/kibana/kibana-4.1.1-linux-x64.tar.gz

Extract the file:

# tar xvf kibana-4.1.1-linux-x64.tar.gz

Move it and change to a shorter name:

# mv kibana-4.1.1-linux-x64/ /opt/
# mv /opt/kibana-4.1.1-linux-x64/ /opt/kibana4

Launch Kibana:

# cd /opt/kibana4/
# ./bin/kibana > /dev/null &

Check that Kibana is working in a browser:

http://141.52.X.X:5601

The access is not secured. In order to make it more secure we are going to install a reverse nginx proxy in the next section.

Installation and configuration of an nginx reverse proxy (Debian 8)

Kibana comes with a plugin called Shield which allows you to easily protect this data with a username and password, while simplifying your architecture. Advanced security features like encryption, role-based access control, IP filtering, and auditing are also available when you need them.

It is necessary a Gold or Platinum subscription to use it. We do not have it for the tutorial, so we secure our Kibana server using an nginx reverse proxy.

The steps are the following.

Install nginx and apache2-utils (see man aptitude in case you do not know the -y option)

# aptitude install -y nginx apache2-utils

Create the folder where the certificate and public key for your server will be saved:

# mkdir /etc/nginx/ssl

Get the certificate and public key:

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/nginx.key -out /etc/nginx/ssl/nginx.crt

Create the user admin (you can choose any other) which will be allowed to access to your Kibana server (choose a password)

# htpasswd -c /etc/nginx/htpasswd.users admin     

Change the file /etc/nginx/sites-enabled/default to the following one:

# cat /etc/nginx/sites-enabled/default 

server {
    listen 80;
    return 301 https://$host$request_uri;
}
server {
listen 443 ssl;

server_name <your-server>;  

ssl on;
ssl_protocols  SSLv2 TLSv1; # Remove SSLv3 because of security hole!
ssl_ciphers  ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers   on;

ssl_certificate /etc/nginx/ssl/nginx.crt;
ssl_certificate_key /etc/nginx/ssl/nginx.key;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.users;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
  

Restart the nginx service:

# /etc/init.d/nginx restart

In a browser:

141.52.X.X

Accept the certificate and access using your username and password.

Exercise 1: Daily value of Yahoo stock market (using csv plugin)

Due to the fact that we cannot use real data with sensitive information, we are going to see a first example using the publicly available data at Yahoo Finance web page. We have selected the daily evolution of the Yahoo index (YHOO) from the beginning (April 1st, 1996) to today, but you can choose another one.

For the Yahoo index, you can get the data from here:

# wget -O yahoo-stock.csv 'http://ichart.finance.yahoo.com/table.csv?s=YHOO&c=1962'

Take a look at this file:

# vi yahoo-stock.csv

We would like to use Logstash in order to parse the data and send it to Elasticsearch. There is a filter plugin called csv, so take a look at it:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html

Please, try to think about the plausible logstash configuration file before checking the following solution (save this file in /etc/logstash/conf.d/<filename>.conf)

input {  
  file {
    path => "/root/csvkit-tutorial/yahoo-stock.csv"
    sincedb_path => "/dev/null"
    start_position => "beginning"    
    type => "stock-market"
  }
}
filter {  
  csv {
      separator => ","
      columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
  }
  mutate {convert => ["High", "float"]}
  mutate {convert => ["Open", "float"]}
  mutate {convert => ["Low", "float"]}
  mutate {convert => ["Close", "float"]}
  mutate {convert => ["Volume", "float"]}
  mutate {convert => ["Adj Close", "float"]}
}
output {  
    elasticsearch {
        action => "index"
        host => "localhost"
        index => "stock"
        workers => 1
    }
}
 

Execute logstash:

# /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d/stock-market.conf  > /var/log/logstash/execution-log 2>&1 &

Check if the data has been imported to Elasticsearch:

# curl 'localhost:9200/_cat/indices?v'

You will see yellow status for the stock index because we are working with one single host and the number of replicas is distinct from zero.

To solve it, set the property index.number_of_replicas equal to zero in the file /etc/elasticsearch/elasticsearch.yml:

index.number_of_replicas: 0

It is necessary to delete the indices in Elasticsearch and to kill the current logstash process (ps aux | grep logstash).

In order to delete the indices with curl:

# curl -XDELETE 'localhost:9200/stock?pretty'

Run logstash again and check if the data has been imported again.

Now, it is time to play with Kibana (in a browser):

141.52.X.X

We will show you how to do it. When you are ready, call us.