Elasticsearch, Fluentd and Kibana: Open source log search and visualization scheme

Elasticsearch, Fluentd and Kibana: Open source log search and visualization scheme

Offers: Zstack community

Objective

The combination of Elasticsearch, Fluentd and Kibana (EFK) enables the collection, indexing, searching, and visualization of log data. The combination is an alternative to commercial software Splunk: Splunk is free at the start, but charges are required if there is more data.

This article describes how to build your log solution with this combination.

Pre-conditions

• A cloud host with Ubuntu 14.04 installed
• Users with sudo permissions on the host

Install Java for elasticsearch installation configuration

Elasticsearch requires Java, so install Java first.

sudo apt-get updatesudo apt-get install openjdk-7-jre-headless --yes

Check to see if Java was installed successfully:

java -version

The output should look like this:

java version "1.7.0_55"OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1)OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Get Elasticsearch

Next, download Elasticsearch's Deb installation package and install it.

sudo wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.2.debsudo dpkg -i elasticsearch-1.2.2.deb

Elasticsearch Safety Reinforcement

By the 1.2 version, the dynamic scripting feature of Elasticsearch is turned on by default. Because this article will set the Kibana dashboard to be accessible from the public network, it is best to turn off this feature for security reasons. Enter the /etc/elasticsearch/elasticsearch.yml file and add the following line at the end:

script.disable_dynamic: true

Start Elasticsearch

Run the following command to run Elasticsearch:

sudo service elasticsearch start

Kibana installation configuration Get Kibana

Go to your user home directory:

cd ~

Enter the following command to download the Kibana:

curl -L https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz | tar xzf -sudo cp -r kibana-3.1.0 /usr/share/

Configure Kibana

We need kibana to communicate with the Elasticsearch using port 80 instead of the default 9200 port, so we need to change the Kibana configuration file config.js .

Open with the editor /usr/share/kibana-3.1.0/config.js and find the following line:

elasticsearch: "http://"+window.location.hostname+":9200",

Replace it with the following line:

elasticsearch: "http://"+window.location.hostname+":80",

Installing the configuration nginx (proxy server)

We use Nginx as a proxy server, allowing authenticated users to access Kibana's dashboards from the public network.

First, install Nginx:

sudo apt-get install nginx --yes

Kibana's own nginx.conf has been better written, we just need to make a little change.

First, download the installation configuration file:

wget https://assets.digitalocean.com/articles/fluentd/nginx.confsudo cp nginx.conf /etc/nginx/sites-available/default

Note: This configuration file originates from this GitHub repository.

Then, open in the editor /etc/nginx/sites-available/default , make the following changes (mainly server_name, Access_log, location three parts):

## 针对Elasticsearch + Kibana的Nginx代理配置## 此处将为仪表盘设置密码保护。# 你也可以考虑为所有路径设置密码。# # 浏览器在首次触发访问该路径的ajax请求时，# 会弹出用户名/密码的输入框。## 如果你使用本功能，请将config.js配置为 http://FQDN:80/# 取代原来的 http://FQDN:9200#server { listen                *:80 ; server_name           localhost; access_log            /var/log/nginx/kibana.log; location / {   root  /usr/share/kibana-3.1.0;   index  index.html  index.htm; }

Finally, restart Nginx:

$ sudo service nginx restart

Now, to open the server's IP address or domain name in the browser, you should be able to see the Kibana dashboard:

Installation configuration for FLUENTD

Finally, the installation of the FLUENTD. We will use FLUETD's package version td-agent, which is maintained by treasure data.

Installing FLUENTD from the Td-agent package

Install FLUENTD using the following command:

wget http://packages.treasuredata.com/2/ubuntu/trusty/pool/contrib/t/td-agent/td-agent_2.0.4-0_amd64.debsudo dpkg -i td-agent_2.0.4-0_amd64.deb

Installing plugins

We need to install several plugins as follows:

1. Out_elasticsearch: The plug-in will fluentd the data stream to Elasticsearch.
2. Outrecordreformer: The plugin can process data into more formats.

Enter the following command to install the plug-in (the first apt-get is used to install Out_elasticsearch, which will need to be installed on the server make libcurl ):

sudo apt-get install make libcurl4-gnutls-dev --yessudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearchsudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer

Then, configure FLUENTD to listen to the syslog and send it to Elasticsearch. Open with the editor /etc/td-agent/td-agent.conf and add the following at the beginning of the file:

<source> type syslog port 5140 tag  system</source><match system.*.*> type record_reformer tag elasticsearch facility ${tag_parts[1]} severity ${tag_parts[2]}</match><match elasticsearch> type copy <store>   type stdout </store> <store> type elasticsearch logstash_format true flush_interval 5s #debug </store></match>

Start Fluentd

Enter the following command to start Fluentd:

sudo service td-agent start

Rsyslog traffic is forwarded to FLUENTD

Ubuntu 14.04 comes with RSYSLOGD. We need to configure it to forward syslog events to Fluentd listening ports (this article uses 5140 ports).

Open with editor /etc/rsyslog.conf (requires sudo permission), add the following to the file header:

*.* @127.0.0.1:5140

Save exit, restart RSYSLOGD:

sudo service rsyslog restart

Custom Kibana Dashboards

Kibana The default dashboard shows only the most common content, so consider customizing. Two methods are shown below.

Method 1: Use templates (template)

The FLUENTD team provides another Kibana configuration file that will work better than the default configuration of Kibana if the template is used to customize it. Run the following command to obtain the configuration file:

wget -O default.json https://assets.digitalocean.com/articles/fluentd/default.jsonsudo cp default.json /usr/share/kibana-3.1.0/app/dashboards/default.json

Note: Source files are from this github gist.

Now, to refresh the page in the browser, you should see that Kibana now displays a rectangular chart based on the Syslog severity level (severity) and program module (facility) and shows the most recent log content in a table.

Method 2: Manually configure

Access the Kibana Dashboard home page in the browser:

Select the bottom blank dashboard:i ' m comfortable configuring on My own(blank template):

On this page, click on the + add arow button on the right, and the configuration interface for adding a new row will pop up (a "row" can contain one or more "plates"). Enter a name, then click the Create Row button, then click Save. Now you can see the new line.

After the blank line is created, Kibana will remind the Add panel to emptyrow (add a plate in a blank line) to the left of the page. Clicking this button will bring up the configuration screen of the new section. Select histogram(rectangular chart) in the drop-down menu. A rectangular chart is a time chart, see Kibana documentation.

The configuration of the rectangle diagram involves many parameters, but we can simply pull down to the bottom of the page and click the Save button. So, the new plate is created.

Extended Reading

For more information on Kibana configuration, refer to the Kibana documentation page.

This article originates from Digitalocean Community. English Original: Elasticsearch, Fluentd, and Kibana:open Source Log Search and visualization by Kiyoto Tamura

This article is submitted by FLUENTD.

Translation: Lazycai

Elasticsearch, Fluentd and Kibana: Open source log search and visualization scheme