Enterprise-elk log Analysis for Linux

First, Introduction

1. Core composition

ELK Consists of three parts: Elasticsearch,Logstash and Kibana ;

Elasticsearch is an open source distributed search engine, it features: distributed, 0 configuration, automatic discovery, Index auto-shard, index copy mechanism, RESTful style interface, multi-data source, automatic search load, etc.

Logstash is a fully open source tool that collects, analyzes, and stores your logs for later use

Kibana is an open source and free tool that provides log analytics friendly Web interface for Logstash and ElasticSearch to help you summarize, analyze, and search for important data logs.

2. Four components

Logstash: Logstash server side used to collect logs;

Elasticsearch: Store all kinds of logs;

Kibana: The Web interface is used as a search and visualization log;

Logstash forwarder: Logstash client is used to send logs to Logstash server via Lumberjack Network protocol;

3. Elk Work Flow

Deploy Logstash on all services that need to collect logs, as Logstash agent (Logstash shipper) to monitor and filter the collection logs, send filtered content to Redis, and Logstash Indexer collect the logs together to the full-text search service Elasticsearch, you can use Elasticsearch to customize the search by Kibana to combine custom search for page presentation.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/74/25/wKioL1YWGBnxpAwyAAGFWRpT6X8070.jpg "title=" work Flying.png "alt=" Wkiol1ywgbnxpawyaagfwrpt6x8070.jpg "/>

4, Elk's Help manual

Elk Official Website: https://www.elastic.co/

Elk Official Documents: https://www.elastic.co/guide/index.html

Elk Chinese Manual: http://kibana.logstash.es/content/elasticsearch/monitor/logging.html

Comments

Elk has two mounting options

(1) Integrated environment: Logstash has an integration package that includes its full set of three components, which is the installation of an integration package.

(2) Independent environment: three components are individually installed, run, and perform their respective duties. (more commonly used)

Two. Installation configuration

1. Environment configuration

server1:172.25.29.1 Master

server2:172.25.29.2 data

server3:172.25.29.3 data

2. Installation

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/eadd5c5b28ba462c23df0082e8796b37.png "style=" float: none; "Title=" screenshot from 2017-11-07 09-44-53.png "alt=" Eadd5c5b28ba462c23df0082e8796b37.png "/>

Installing the Java Environment

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/b99c82804f65301409d1e9083dc4a4df.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-02-04.png "alt=" B99c82804f65301409d1e9083dc4a4df.png "/>

3. Configuration Server1

650) this.width=650; "src=" Https://s5.51cto.com/oss/201711/07/14bbb830023be24f556bc065832f8da7.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-07-35.png "alt=" 14bbb830023be24f556bc065832f8da7.png "/>

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/8723fd5de21591ef0eeb3fd0773953a4.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-05-50.png "alt=" 8723fd5de21591ef0eeb3fd0773953a4.png "/>

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/05c576cb5d79c83cf2ddd7743dfa8d5c.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-06-11.png "alt=" 05c576cb5d79c83cf2ddd7743dfa8d5c.png "/>

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/918098c3a60656d77a1a639f80d40104.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-06-35.png "alt=" 918098c3a60656d77a1a639f80d40104.png "/>

650) this.width=650; "src=" Https://s5.51cto.com/oss/201711/07/dcdbea9dbc5b3651ddb6fa5349bba73b.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-06-54.png "alt=" Dcdbea9dbc5b3651ddb6fa5349bba73b.png "/>

4. Test is normal

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/2c7203832fa9a5c731055728b6ddd213.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-07-50.png "alt=" 2c7203832fa9a5c731055728b6ddd213.png "/>

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/66799606294a10d0353b615696bc5ea2.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-08-18.png "alt=" 66799606294a10d0353b615696bc5ea2.png "/>

Three. Install head plug-in, Web Access Management node, add data Storage node

1. Installing plugins

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/2cb1ec71f3850d532ae553a2ded79de1.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-27-11.png "alt=" 2cb1ec71f3850d532ae553a2ded79de1.png "/>

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/f1ca6ab25d501c754dc1f75f323cb927.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-29-17.png "alt=" F1ca6ab25d501c754dc1f75f323cb927.png "/>

To see if the plugin was successfully installed

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/28c5b92d507a764754e118bcd3463920.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-29-24.png "alt=" 28c5b92d507a764754e118bcd3463920.png "/>

2. Web page by accessing the management node 172.25.29.1:9200/_plugin/head/

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/2e4cce9da8955fa209078ede38bc1544.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-30-39.png "alt=" 2e4cce9da8955fa209078ede38bc1544.png "/>

Test acquisition

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/5330ba3251e7778cc355384f6ddb338c.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-31-53.png "alt=" 5330ba3251e7778cc355384f6ddb338c.png "/>

3. Add a second host Server2

Install the Configuration service, note the Discovery option to open Server1 and Server2, and modify the correct host name

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/fb0d7b4420c84810e430ab6bca46f053.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-34-05.png "alt=" Fb0d7b4420c84810e430ab6bca46f053.png "/>

650) this.width=650; "src=" Https://s4.51cto.com/oss/201711/07/564e78578636912ea311802167ec8eea.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-32-32.png "alt=" 564e78578636912ea311802167ec8eea.png "/>

650) this.width=650; "src=" Https://s2.51cto.com/oss/201711/07/e1903fadded705349cad8f303d6395b5.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-35-50.png "alt=" E1903fadded705349cad8f303d6395b5.png "/>

650) this.width=650; "src=" Https://s1.51cto.com/oss/201711/07/88aa213005bd23b652c55a7f9e5e6577.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-36-32.png "alt=" 88aa213005bd23b652c55a7f9e5e6577.png "/>

Server1 and Server2 are going to change.

650) this.width=650; "src=" Https://s3.51cto.com/oss/201711/07/03208b391c48e0456669adc7c78b82ab.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-36-59.png "alt=" 03208b391c48e0456669adc7c78b82ab.png "/>

650) this.width=650; "src=" Https://s3.51cto.com/oss/201711/07/396570ade29bcb17c75023c4dd9130d9.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-37-24.png "alt=" 396570ade29bcb17c75023c4dd9130d9.png "/>

4. Access Master node, Server1 and server display normal

650) this.width=650; "src=" Https://s3.51cto.com/oss/201711/07/e11b5190428f6da447c005cd75313716.png "style=" float: none; "Title=" screenshot from 2017-11-07 10-37-34.png "alt=" E11b5190428f6da447c005cd75313716.png "/>

Enterprise-elk log Analysis for Linux