ELKstack Log Analysis Platform

ELKstack Log Analysis Platform

ELKstack is a combination of three open source software: Elasticsearch, Logstash, and Kibana. Currently, all are under the Elastic. co company name.
ELK is a common open-source log monitoring and analysis system, including a distributed index and search service Elasticsearch, A logstash tool for managing logs and events, and a data visualization service Kibana.
Logstash_1.5.3 collects, processes, and stores logs.
Elasticsearch-1.7.2 is responsible for Log retrieval and analysis
Kibana-4.1.2-linux-x64.tar.gz is responsible for log Visualization
Jdk-1.7.0_03 java environment
Redis-2.4.14 DB
 
I. Basic Environment
1. Roles, ip addresses, versions, and kernels
ServerA 10.1.10.185 3.2.0-4-amd64 7.8 java elasticsearch redis kibana logstash (agent indexer)
ClientB 10.1.10.117 3.2.0-4-amd64 7.8 java logstash (agent)
 
2. Install the basic package
Apt-get-y install curl wget lrzsz axel
 
Ii. Install redis server
1. Installation Package
Apt-get-y install redis-server
 
2. Create a redis storage directory
Mkdir/opt/redis-p
 
3. Permissions
Chown redis/opt/redis/-R
 
4. Configuration
1) backup configuration
Cp/etc/redis. conf/etc/redis. conf. bak
2) modify the configuration
Sed-I's! ^ Bind .*! Bind 10.1.10.185! G'/etc/redis. conf
Sed-I's! ^ Dir .*! Dir/opt/redis! G'/etc/redis. conf
 
5. Restart the service
/Etc/init. d/redis-server restart
 
6. View processes and ports
1) view Processes
Ps-ef | grep redis
Redis 23193 1 0? 00:00:00/usr/bin/redis-server/etc/redis. conf
2) view the port
Netstat-tupnl | grep redis
Tcp 0 0 10.1.10.185: 6379 0.0.0.0: * LISTEN 25188/redis-server
 
7. Check whether the instance is started (the instance is started by default)
Ll/etc/rc2.d/| grep redis
Lrwxrwxrwx 1 root 22 Sep 20 S02redis-server-> ../init. d/redis-server
 
3. Install the java environment
1. Installation Package
Apt-get-y install openjdk-7-jdk
 
2. view the version
Java-version
Java version "1.7.0 _ 03"
OpenJDK Runtime Environment (IcedTea7 2.1.7) (7u3-2.1.7-1)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
 
4. Install elasticsearch
1. Download elasticsearch
Wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb
 
2. Install elasticsearch
Dpkg-I elasticsearch-1.7.2.deb
Selecting previusly unselected package elasticsearch.
(Reading database... 30240 files and directories currently installed .)
Unpacking elasticsearch (from elasticsearch-1.7.2.deb )...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Setting up elasticsearch (1.7.2 )...
 
3. Configuration
1) backup configuration
Cp/etc/elasticsearch. yml/etc/elasticsearch. yml. bak
2) modify the configuration
Echo "network. bind_host: 10.1.10.185">/etc/elasticsearch. yml
 
4. Start the elasticsearch Service
/Etc/init. d/elasticsearch start
 
5. View processes and ports
1) view Processes
Ps-ef | grep java
106 22835 1 63? 00:00:03/usr/lib/jvm/java-7-openjdk-amd64 // bin/java-Xms256m-Xmx1g-Djava. awt. headless = true-XX: + UseParNewGC-XX: + UseConcMarkSweepGC-XX: bytes = 75-XX: + bytes-XX: + HeapDumpOnOutOfMemoryError-XX: + DisableExplicitGC-Dfile. encoding = UTF-8-Delasticsearch-Des. pidfile =/var/run/elasticsearch. pid-Des. path. home =/usr/share/elasticsearch-cp:/usr/share/elasticsearch/lib/elasticsearch-1.7.2.jar:/usr/share/elasticsearch/lib /*: /usr/share/elasticsearch/lib/sigar/*-Des. default. config =/etc/elasticsearch. yml-Des. default. path. home =/usr/share/elasticsearch-Des. default. path. logs =/var/log/elasticsearch-Des. default. path. data =/var/lib/elasticsearch-Des. default. path. work =/tmp/elasticsearch-Des. default. path. conf =/etc/elasticsearch org. elasticsearch. bootstrap. elasticsearch
2) view the port
Netstat-tupnl | grep java
Tcp6 0 0 10.1.10.185: 9200: * LISTEN 22835/java
Tcp6 0 0 10.1.10.185: 9300: * LISTEN 22835/java
Udp6 0 0: 54328: * 22835/java
 
6. Test
Curl-x get http: // 10.1.10.185: 9200
{
"Status": 200,
"Name": "Ned Leeds ",
"Cluster_name": "elasticsearch ",
"Version ":{
"Number": "1.7.2 ",
"Build_hash": "e43676b1385b8125d647f593f7202acbd816e8ec ",
"Build_timestamp": "2015-09-14T09: 49: 53Z ",
"Build_snapshot": false,
"Inclue_version": "4.10.4"
},
"Tagline": "You Know, for Search"
}
 
7. Add to start
Update-rc.d elasticsearch defaults
Update-rc.d: using dependency based boot sequencing
 
5. Install logstash
1. Download logstash
Wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb
 
2. Install logstash
Dpkg-I logstash_1.5.3-1_all.deb
(Reading database... 30338 files and directories currently installed .)
Unpacking logstash (from logstash_1.5.3-1_all.deb )...
Setting up logstash (. 5.3-1 )...
 
3. configuration (this configuration file is not available by default)
1) Configure logstash_agent
Cat/etc/logstash/conf. d/logstash_agent.conf
Input {
File {
Type => "messages"
Path => ["/var/log/messages"]
}
File {
Type => "elasticsearch"
Path => ['/var/log/elasticsearch. log *']
}
}
Output {
Redis {
Host => "10.1.10.185"
Data_type => "list"
Key => "logstash: redis"
}
}
2) Configure logstash_indexer
Cat/etc/logstash/conf. d/logstash_indexer.conf
Input {
Redis {
Host => "10.1.10.185"
Data_type => "list"
Key => "logstash: redis"
Type => "redis-input"
Port => "6379"
}
}
Output {
Elasticsearch {
Host => "10.1.10.185"
}
}
 
4. Start the service
/Etc/init. d/logstash start
Logstash started.
 
5. Use jps-mlv or ps-ef to view the process
Ps-ef | grep logst
Logstash 22932 1 16 00:00:01 pts/0/usr/bin/java-XX: + UseParNewGC-XX: + UseConcMarkSweepGC-Djava. awt. headless = true-XX: CMSInitiatingOccupancyFraction = 75-XX: + UseCMSInitiatingOccupancyOnly-Djava. io. tmpdir =/var/lib/logstash-Xmx500m-Xss2048k-Djffi. boot. library. path =/opt/logstash/vendor/jruby/lib/jni-XX: + UseParNewGC-XX: + UseConcMarkSweepGC-Djava. awt. headless = true-XX: CMSInitiatingOccupancyFraction = 75-XX: + UseCMSInitiatingOccupancyOnly-Djava. io. tmpdir =/var/lib/logstash-Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby. jar-classpath:-Djruby. home =/opt/logstash/vendor/jruby-Djruby. lib =/opt/logstash/vendor/jruby/lib-Djruby. script = jruby-Djruby. shell =/bin/sh org. jruby. main -- 1.9/opt/logstash/lib/bootstrap/environment. rb logstash/runner. rb agent-f/etc/logstash/conf. d-l/var/log/logstash. log
 
6. Set startup
Update-rc.d: logstash ults
Update-rc.d: using dependency based boot sequencing
 
6. Install kibana (front-end web)
1. Download
Axel-n https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
 
2. decompress the package to the specified directory.
Tar zxvf kibana-4.1.2-linux-x64.tar.gz-C/opt
 
3. Create a log directory
Mkdir-p/opt/kibanalog
 
4. Configuration
1) backup configuration
Cp/opt/kibana-4.1.2-linux-x64/config/kibana. yml/opt/kibana-4.1.2-linux-x64/config/kibana. yml. bak
2) modify the configuration
Sed-I's! ^ Elasticsearch_url :.*! Elasticsearch_url: "http: // 10.1.10.185: 9200 "! G'/opt/kibana-4.1.2-linux-x64/config/kibana. yml
Sed-I's! ^ Host :.*! Host: "10.1.10.185 "! G'/opt/kibana-4.1.2-linux-x64/config/kibana. yml
 
5. Start the service
Cd/opt/kibanalog & nohup/opt/kibana-4.1.2-linux-x64/bin/kibana &
 
6. View processes and ports
1) view Processes
Ps aux | grep kibana
Root 22982 5.4 20.1 612576 47716 pts/0 Sl/opt/kibana-4.1.2-linux-x64/bin /.. /node/bin/node/opt/kibana-4.1.2-linux-x64/bin /.. /src/bin/kibana. js
2) view the port
Netstat-tupnl | grep 5601
Tcp 0 0 10.1.10.185: 5601 0.0.0.0: * LISTEN 22982/node
 
7. Access http: // 10.1.10.185: 5601 on windows

For more details, please continue to read the highlights on the next page:

1 2 Next Page