Elasticsearch, Fluentd and Kibana: Open source log search and visualization scheme
Offers: Zstack community
Objective
The combination of Elasticsearch, Fluentd and Kibana (EFK) enables the collection, indexing, searching, and visualization of log data. The combination is an alternative to commercial software Splunk: Splunk is free at the start, but charges are required if there is more data.
This article describes how to build your log solution with this combination.
Pre-conditions
- A cloud host with Ubuntu 14.04 installed
- Users with sudo permissions on the host
Install Java for elasticsearch installation configuration
Elasticsearch requires Java, so install Java first.
sudo apt-get updatesudo apt-get install openjdk-7-jre-headless --yes
Check to see if Java was installed successfully:
java -version
The output should look like this:
java version "1.7.0_55"OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1)OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)
Get Elasticsearch
Next, download Elasticsearch's Deb installation package and install it.
sudo wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.2.debsudo dpkg -i elasticsearch-1.2.2.deb
Elasticsearch Safety Reinforcement
By the 1.2 version, the dynamic scripting feature of Elasticsearch is turned on by default. Because this article will set the Kibana dashboard to be accessible from the public network, it is best to turn off this feature for security reasons. Enter the /etc/elasticsearch/elasticsearch.yml
file and add the following line at the end:
script.disable_dynamic: true
Start Elasticsearch
Run the following command to run Elasticsearch:
sudo service elasticsearch start
Kibana installation configuration Get Kibana
Go to your user home directory:
cd ~
Enter the following command to download the Kibana:
curl -L https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz | tar xzf -sudo cp -r kibana-3.1.0 /usr/share/
Configure Kibana
We need kibana to communicate with the Elasticsearch using port 80 instead of the default 9200 port, so we need to change the Kibana configuration file config.js
.
Open with the editor /usr/share/kibana-3.1.0/config.js
and find the following line:
elasticsearch: "http://"+window.location.hostname+":9200",
Replace it with the following line:
elasticsearch: "http://"+window.location.hostname+":80",
Installing the configuration nginx (proxy server)
We use Nginx as a proxy server, allowing authenticated users to access Kibana's dashboards from the public network.
First, install Nginx:
sudo apt-get install nginx --yes
Kibana's own nginx.conf has been better written, we just need to make a little change.
First, download the installation configuration file:
wget https://assets.digitalocean.com/articles/fluentd/nginx.confsudo cp nginx.conf /etc/nginx/sites-available/default
Note: This configuration file originates from this GitHub repository.
Then, open in the editor /etc/nginx/sites-available/default
, make the following changes (mainly server_name, Access_log, location three parts):
## 针对Elasticsearch + Kibana的Nginx代理配置## 此处将为仪表盘设置密码保护。# 你也可以考虑为所有路径设置密码。# # 浏览器在首次触发访问该路径的ajax请求时,# 会弹出用户名/密码的输入框。## 如果你使用本功能,请将config.js配置为 http://FQDN:80/# 取代原来的 http://FQDN:9200#server { listen *:80 ; server_name localhost; access_log /var/log/nginx/kibana.log; location / { root /usr/share/kibana-3.1.0; index index.html index.htm; }
Finally, restart Nginx:
$ sudo service nginx restart
Now, to open the server's IP address or domain name in the browser, you should be able to see the Kibana dashboard:
Installation configuration for FLUENTD
Finally, the installation of the FLUENTD. We will use FLUETD's package version td-agent, which is maintained by treasure data.
Installing FLUENTD from the Td-agent package
Install FLUENTD using the following command:
wget http://packages.treasuredata.com/2/ubuntu/trusty/pool/contrib/t/td-agent/td-agent_2.0.4-0_amd64.debsudo dpkg -i td-agent_2.0.4-0_amd64.deb
Installing plugins
We need to install several plugins as follows:
- Out_elasticsearch: The plug-in will fluentd the data stream to Elasticsearch.
- Outrecordreformer: The plugin can process data into more formats.
Enter the following command to install the plug-in (the first apt-get is used to install Out_elasticsearch, which will need to be installed on the server make
libcurl
):
sudo apt-get install make libcurl4-gnutls-dev --yessudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearchsudo /opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-reformer
Then, configure FLUENTD to listen to the syslog and send it to Elasticsearch. Open with the editor /etc/td-agent/td-agent.conf
and add the following at the beginning of the file:
<source> type syslog port 5140 tag system</source><match system.*.*> type record_reformer tag elasticsearch facility ${tag_parts[1]} severity ${tag_parts[2]}</match><match elasticsearch> type copy <store> type stdout </store> <store> type elasticsearch logstash_format true flush_interval 5s #debug </store></match>
Start Fluentd
Enter the following command to start Fluentd:
sudo service td-agent start
Rsyslog traffic is forwarded to FLUENTD
Ubuntu 14.04 comes with RSYSLOGD. We need to configure it to forward syslog events to Fluentd listening ports (this article uses 5140 ports).
Open with editor /etc/rsyslog.conf
(requires sudo
permission), add the following to the file header:
*.* @127.0.0.1:5140
Save exit, restart RSYSLOGD:
sudo service rsyslog restart
Custom Kibana Dashboards
Kibana The default dashboard shows only the most common content, so consider customizing. Two methods are shown below.
Method 1: Use templates (template)
The FLUENTD team provides another Kibana configuration file that will work better than the default configuration of Kibana if the template is used to customize it. Run the following command to obtain the configuration file:
wget -O default.json https://assets.digitalocean.com/articles/fluentd/default.jsonsudo cp default.json /usr/share/kibana-3.1.0/app/dashboards/default.json
Note: Source files are from this github gist.
Now, to refresh the page in the browser, you should see that Kibana now displays a rectangular chart based on the Syslog severity level (severity) and program module (facility) and shows the most recent log content in a table.
Method 2: Manually configure
Access the Kibana Dashboard home page in the browser:
Select the bottom blank dashboard:i ' m comfortable configuring on My own(blank template):
On this page, click on the + add arow button on the right, and the configuration interface for adding a new row will pop up (a "row" can contain one or more "plates"). Enter a name, then click the Create Row button, then click Save. Now you can see the new line.
After the blank line is created, Kibana will remind the Add panel to emptyrow (add a plate in a blank line) to the left of the page. Clicking this button will bring up the configuration screen of the new section. Select histogram(rectangular chart) in the drop-down menu. A rectangular chart is a time chart, see Kibana documentation.
The configuration of the rectangle diagram involves many parameters, but we can simply pull down to the bottom of the page and click the Save button. So, the new plate is created.
Extended Reading
For more information on Kibana configuration, refer to the Kibana documentation page.
This article originates from Digitalocean Community. English Original: Elasticsearch, Fluentd, and Kibana:open Source Log Search and visualization by Kiyoto Tamura
This article is submitted by FLUENTD.
Translation: Lazycai
Elasticsearch, Fluentd and Kibana: Open source log search and visualization scheme