Elasticsearch, Logstash and Kibana Windows environment Setup (i)

Source: Internet
Author: User
Tags install node kibana logstash filebeat
Tag: sql bin enter dcl ica master redis friendly lin

I. Overview
ELK official website https://www.elastic.co
ELK consists of three parts: Elasticsearch, Logstash and Kibana;

Elasticsearch is an open source distributed search engine. Its features are: distributed, zero configuration, automatic discovery, automatic index sharding, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc.
Logstash is a completely open source tool that can collect, analyze, and store your logs for later use
Kibana is an open source and free tool that can provide log analysis friendly Web interface for Logstash and ElasticSearch, which can help you aggregate, analyze and search important data logs.
Common platform architecture

ELK = Elasticsearch + Logstash + Kibana
EFK = Elasticsearch + Flume + Kibana

Simple architecture:
ELK + Redis
or
(Logstash / Flume)-> (Kafka / Redis)-> ES-> Web

It can be simpler:
Client (Filebeat)-> LogServer (Logstash-> Elasticsearch-> Kibana)
(You can directly output to ES without using Logstash and Filebeat)

Complex architecture:
Data collection layer [offline / real-time data, MySQL / ES / Logstash / Flume]
-> Data transmission layer [Kafka / Redis]
-> Data cache layer [Storm-> Kafka]
-> Compute storage layer [offline computing (Kylin / Spark SQL / Impala / MR), real time computing (Storm / Spark Streaming / Flink / Druid), storage (ES / MySQL / Hbase / HDFS / Kudu)]
-> Application layer [Web / REST API]

Second, download and install ELK
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.3.tar.gz --no-check-certificate
wget https://artifacts.elastic.co/downloads/logstash/logstash-5.4.3.tar.gz --no-check-certificate
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.3-windows-x86.zip --no-check-certificate

Note that the jdk8 environment is required


Run elasticsearch distributed log analysis and retrieval on windows
https://www.elastic.co/downloads

Download elasticsearch Logstash Kibana in the interface

1. Elasticsearch download, unzip zip, enter the bin file elasticsearch.bat
Visit http: // localhost: 9200

Successful start



2. Since elasticsearch is just string and other file information, you need to install the elasticsearch-head plugin
            In the current version, since 6.4.0 is used to build the environment, you need to use tools such as node to install the plugin

            step one. Install node will not repeat them here

            Step two. Install npm install -g grunt-cli install grunt

                         Use grunt --version to view the version information of grunt installation

 

3. Modify the elasticsearch / conf folder Modify the elasticsearch.xml file,

 

Enter the config directory under the installation directory and modify the elasticsearch.yml file. Add at the end of the file to

http.cors.enabled: true

http.cors.allow-origin: "*"

node.master: true

node.data: true

Other operations

Then remove the comment of network.host: 192.168.0.1 and change to network.host: 0.0.0.0, remove the comment of cluster.name; node.name; http.port (that is, remove #)

  Double-click elasticsearch.bat to restart es



4. Download the elasticsearch-head package

https://github.com/mobz/elasticsearch-head download the head plugin, choose to download zip

5. Unzip to the specified folder, G: \ elasticsearch-6.6.2 \ elasticsearch-head-master enter this folder, modify G: \ elasticsearch-6.6.2 \ elasticsearch-head-master \ Gruntfile.js in the corresponding Location plus hostname: '*'
6. After the modification is complete, use the cmd command to enter the elasticsearch-head-master folder, execute the command npm install and then execute npm run start to run the head
Successful operation Successful interface



Access interface

The above is the elasticsearch installation and viewing tutorial

Test elasticsearch-head

  1. Use cmd request

curl -XGET http://127.0.0.1:9200/_cluster/health?pretty
Request result
 2. Use elasticsearch-head to make requests
 Request result
☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆☆ 3.5 ☆☆

To start the logstash service, you need to modify the conf to add logstash.conf and paste it in the newly created file
input {
     file {
         type => "nginx_access"
         path => "D: \ nginx \ logs \ access.log"
     }
}
output {
     elasticsearch {
         hosts => ["192.168.10.105:9200"]
         index => "access-% {+ YYYY.MM.dd}"
     }
     stdout {
         codec => json_lines
     }
}

Go to the bin folder and execute
   Command 1 logstash.bat agent --f ../config/logstash.conf

    Command 2 logstash.bat -f ../config/logstash.conf
Start logstash. If an error is reported, use "% CLASSPATH%" in logstash.bat in quotation marks, just execute

Startup error, cannot be regarded as continuous service

     reason => "Expected one of #, input, filter, output at line 1, column 1 (byte1) after"}
Is a file format problem.

The default Windows file is utf8. Change the file format to: ansi format. The program can run normally bin \ logstash -f ./config/test.conf


Start Kibana service
Go to the kibana-6.4.0-windows-x86_64 / bin folder and double-click Kibana.bat

When you first start, you need to wait about one minute, the cmd command window does not respond







Elasticsearsh set Chinese word breaker
Use the installation command elasticsearch-plugin.bat install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.4.0/elasticsearch-analysis-ik-6.4.0.zip

Errors will be reported when using windows installation, it is better to download directly, and then unzip the file information to the elasticsearch / plugs / ik folder

Verification can be requested in kibana or in elasticsearsh

GET _search {"query": {"match": {"message": "中华"}}}







Set elasticsearsh as a windows system service
     Execute elasticsearch-service.bat install in bin

Set logstash Kibana as a windows service
Install using nssm

   nssm (official website address) https://nssm.cc/ci/nssm-2.24-101-g897c7ad.zip

use

Go to ELK_WORK \ nssm-2.24-101-g897c7ad \ win64> nssm install logstash / Kibana

After entering, the execution box will pop up

NSSM usage (same for Linux service)

To manage a service:

        nssm start <servicename>

        nssm stop <servicename>

        nssm restart <servicename>

        nssm status <servicename>

        nssm statuscode <servicename>

        nssm rotate <servicename>

        nssm processes <servicename>

 

 

to sum up,

Start the ELK log collection, analysis and query function in windows,

Three services will be installed in the service

Elasticsearch service access address 192.168.10.105:9200

Service name Elasticsearch 6.4.0 (elasticsearch-service-x64)



logstash service access address 192.168.10.105:9600

Kibana service Visit address: http: // localhost: 5601

Setting up Elasticsearch, Logstash and Kibana Windows environment (1)
Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.