Original address: http://www.cnblogs.com/saintaxl/p/3946667.html
In short, his specific workflow is to Logstash agent to monitor and filter the log, the filtered log content to Redis (here Redis only processing queues do not store), Logstash Index collects the logs together to the full-text search service Elasticsearch can use Elasticsearch to customize the search by Kibana to combine custom search for page presentation
- Ruby Run Kibana must
- RubyGems Installing Ruby extensions must
- The Bundler function is similar to Yum
- The JDK running the Java program must
- Redis is used to process log queues
- Logstash collecting and filtering logs
- ElasticSearch Full-Text Search service (Logstash integrates one)
Kibana page Show
First to Logstash Index server above, Logstash is divided into index and aget, agent is responsible for monitoring, filtering logs, index is responsible for collecting logs and the log to Elasticsearch to do the search in addition Logstash collection method is divided into Standalone and centralized.
Standalone is all functions on a server, spontaneous self-collection, centralized is collected centrally, a server receives all shipper (personal understanding is Logstash agent) log.
Actually logstash itself does not divide what shipper and collector, just is the configuration file is different, we this time in the centralized way to test
There are two servers here.
192.168.124.128 Logstash INDEX,ELASTICSEARCH,KIBANA,JDK
192.168.124.132 Logstash AGENT,REDIS,JDK
Preparatory work
Installation: OpenSSL
Uninstalling older versions
Apt-get Remove Opensslapt-get autoremove OpenSSL
Download the latest version
wget http://www.openssl.org/source/openssl-1.0.1i.tar.gz
TAR-ZXVF openssl-1.0.1i.tar.gzcd/opt/openssl-1.0.1i./config--prefix=/usr/local/sslmake & make Install
Establishing a soft connection
Ln-s/usr/local/ssl/bin/openssl/usr/bin/opensslln-s/usr/local/ssl/include/openssl/usr/include/openssl
Refresh Dynamic Configuration
Vim/etc/ld.so.conf
Insert a row at the end of a text
/usr/local/ssl/libldconfig-v
Test
OpenSSL version-a
Installing the Pcre Library
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.33.tar.gz
TAR-ZXVF pcre-8.33.tar.gzcd pcre-8.33./configure--prefix=/usr/local/pcre-8.33make & make Install
Installing zlib
wget http://zlib.net/zlib-1.2.8.tar.gz
TAR-ZXVF zlib-1.2.8.tar.gzcd zlib-1.2.8./configure--prefix=/usr/local/zlib-1.2.8make & make Install
Installing Nginx
wget http://nginx.org/download/nginx-1.6.1.tar.gz
TAR-ZXVF nginx-1.6.1.tar.gzcd nginx-1.6.1./configure--prefix=/usr/local/nginx--with-http_ssl_module-- With-openssl=/opt/openssl-1.0.1i--with-pcre=/opt/pcre-8.33--with-zlib=/opt/zlib-1.2.8
nginx Command
Start:/usr/local/nginx/sbin/nginx Restart:/usr/local/nginx/sbin/nginx–s Reload stop:/usr/local/nginx/sbin/nginx-s Stop to view the main process: NETSTAT-NTLP check whether it started successfully: Netstat-ano|grep 80
Installing Ruby running Kibana must be
sudo apt-get update wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz./configure--prefix=/usr/ Local/rubymake && make Install
Environment settings
Vi/etc/environment
Add Ruby's path to the environment variable and save the/etc/environment, as follows:
Path= "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/ruby/bin"
After modifying the environment variable file, you need to make the modification take effect with the source command as follows:
$ source/etc/environment
To check if the installation was successful, you can enter the following command to test:
$ruby –v
After confirming that the installation is successful and adding a command link through a command, I am not sure what the purpose of creating these links is, and it should be a convention to follow Ruby's "convention is greater than configuration" principle. (Keyboardota)
$ sudo ln-s/usr/local/ruby/bin/ruby/usr/local/bin/ruby$ sudo ln-s/usr/local/ruby/bin/gem/usr/bin/gem
Or:
Apt-get Install Ruby-full
Installing the RubyGems Ruby extension must
wget http://production.cf.rubygems.org/rubygems/rubygems-2.4.1.tgz
TAR-ZXVF RUBYGEMS-2.4.1.TGZCD Rubygems-2.4.1ruby setup.rb
Installing Redis to handle log queues
wget http://download.redis.io/releases/redis-2.8.13.tar.gz
Install the Elasticsearch full-text Search service (Logstash integrates one)
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.2.tar.gz
TAR-ZXVF elasticsearch-1.3.2.tar.gzcd elasticsearch-1.3.2 Start:
/usr/local/elasticsearch-1.3.2/bin/elasticsearch-d Access
http://localhost:9200
Install: Logstash Collect, filter logs
wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
TAR-ZXVF logstash-1.4.2.tar.gz
Start
Nohup/usr/local/logstash-1.4.2/bin/logstash-f/usr/local/logstash-1.4.2/agent.conf &
Nohup/usr/local/logstash-1.4.2/bin/logstash-f/usr/local/logstash-1.4.2/indexer.conf &
Vim/usr/local/logstash-1.4.2/agent.conf
Input { file { Path = = ["/var/log/*.log", "/var/log/messages", "/var/log/syslog", "/var/log/denyhosts", "/ VAR/LOG/DMESG ","/var/log/faillog ","/var/log/aptitude "] start_position = = beginning } file { Type = "Nginx-access" path = "/var/log/nginx/access.log" }}output { redis{ host = = " 192.168.124.128 " data_type =" List " key =" Logstash "} }
Vim/usr/local/logstash-1.4.2/indexer.conf
Input { Redis { host = "192.168.124.128" data_type = "List" key = "Logstash" }}output { Elasticsearch { host = "192.168.124.132" #指定elasticsearch服务位置 }}
Installing Kibana
wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
TAR-ZXVF Kibana-3.1.0.tar.gzvim/usr/local/kibana-3.1.0/config.js
Search for the "Elasticsearch" parameter and modify it to suit your environment:
Elasticsearch: "http://192.168.124.132:9200",
You can also modify the Default_route parameter to open the Logstash dashboard instead of the Kibana Welcome page by default:
Default_route : '/dashboard/file/logstash.json ',
Download the configuration template
wget https://raw.github.com/elasticsearch/kibana/master/sample/nginx.conf
Modify Nginx Configuration
Vim/usr/local/nginx/conf/nginx.conf
Add server node
# # Nginx proxy for Elasticsearch + Kibana # In this setup, we is password protecting the saving of DASHBO Ards. wish to extend the password protection to all paths. # # Even though these paths is being called as the result of an AJAX request, the # Browser would prompt for a usern Ame/password on the first request # # IF You use this, you'll want to point config.js at Http://FQDN:80/instead of # http://FQDN:9200 # server {listen *:80; server_name localhost; Access_log/usr/local/nginx/logs/kibana.access.log; Location/{root/usr/local/kibana-3.1.0; Index index.html index.htm; } location ~ ^/_aliases$ {Proxy_pass http://127.0.0.1:9200; Proxy_read_timeout 90; } location ~ ^/.*/_aliases$ {Proxy_pass http://127.0.0.1:9200; Proxy_read_timeout 90; } location ~ ^/_nodes$ {Proxy_pass http://127.0.0.1:9,200; Proxy_read_timeout 90; } location ~ ^/.*/_search$ {Proxy_pass http://127.0.0.1:9200; Proxy_read_timeout 90; } location ~ ^/.*/_mapping {Proxy_pass http://127.0.0.1:9200; Proxy_read_timeout 90; } # Password protected end points Location ~ ^/kibana-int/dashboard/.*$ {Proxy_pass http://127.0.0.1:920 0; Proxy_read_timeout 90; Limit_except GET {Proxy_pass http://127.0.0.1:9200; Auth_basic "Restricted"; AUTH_BASIC_USER_FILE/USR/LOCAL/NGINX/KIBANA.MYHOST.ORG.HTPASSWD; }} Location ~ ^/kibana-int/temp.*$ {Proxy_pass http://127.0.0.1:9200; Proxy_read_timeout 90; Limit_except GET {Proxy_pass http://127.0.0.1:9200; Auth_basic "Restricted"; AUTH_BASIC_USER_FILE/USR/LOCAL/NGINX/KIBANA.MYHOST.ORG.HTPASSWD; } } }
If a firewall needs to release these ports:
- Port (for the Web interface)
- Port 5544 (to receive remote syslog messages)
- Port 6379 (for the Redis broker)
- Port 9200 (so the Web interface can access Elasticsearch)
Install Kibana and Logstash under Ubuntu