Logstash+elasticsearch+kibana-based Log Collection Analysis Scheme (Windows)

PartyCase BackJingTypically, the logs are stored on different devices that are scattered. If you manage hundreds of dozens of of servers, you are also using the traditional method of logging in to each machine in turn. This is not feeling very cumbersome and inefficient. Open Source Real-time log analyticsELKthe platform can perfectly solve the problem of log collection and log retrieval and analysis,elk means Elasticsearch .,Logstashand theKiabanaThree of open source tools. Because elk can be deployed across platforms, it is ideal for applications with multi-platform deployments.
Two Environment Preparation1. Installing the JDK1.8 Environment
2.Download Elk Package

• Logstash:https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.zip
• Elasticsearch:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.zip
• Kibana:https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-windows-x86.zip
  

Unzip the downloaded software separately, Elasticsearch,logstash,kibana can be placed under a unified folder

three deployments    1.ConfigurationLogstash

in the lower bin directory of the Logstash folder Create the configuration file logstash.conf, as follows:

  
 

   
  

  
input {
   
  

  
 # 以文件作为来源
   
  

  
 file {
   
  

  
 # 日志文件路径
   
  

  
 path => "F:\test\dp.log"
   
  

  
 }
   
  

  
}
   
  

  
filter {
   
  

  
 #定义数据的格式，正则解析日志（根据实际需要对日志日志过滤、收集）
   
  

  
 grok {
   
  

  
     match => { "message" => "%{IPV4:clientIP}|%{GREEDYDATA:request}|%{NUMBER:duration}"} 
   
  

  
 }
   
  

  
 #根据需要对数据的类型转换
   
  

  
 mutate { convert => { "duration" => "integer" }}
   
  

  
} 
   
  

  
# 定义输出 
   
  

  
output {
   
  

  
 elasticsearch {
   
  

  
 hosts => ["localhost:9200"] #Elasticsearch 默认端口
   
  

  
 }
   
  

  
}　　
  
 

Create the Run.bat in the bin directory and write the script:

 
   
  

  
    
   
logstash.bat -f logstash.conf
 
   
  

Perform Run.bat startup Logstash.

2. Configure Elasticsearch

Elasticsearch.bat can be started.

After launching the browser to access 127.0.0.1:9200, the following JSON indicates success.

3. ConfigurationKibana

KibanaStart from Filekibana.ymlRead properties. Default settings configuration Kibana runlocalhost:5601。 To change the host or port number, or to connect to a elasticsearch running on another machine, you need to updatekibana.ymlFile.

Kibana.bat start Kibana.

Four-Test    1. CREATE INDEX        opening http://localhost:5601/with a browser prompts you to create INDEX, which you can create by time. On the Discover tab you'll see you just in DPthe content entered in the. log.
2. Retrieve the log for quick Search and locate. 3.Log Analysiscreate a new visualize and select line (other views are available). Then select the data source.
x-axis selection time, and the y-axis are the maximum time-consuming and average time-consuming for the access interface respectively. Create a dashboard view that allows you to place related icons in a single view for easy analysis.  

Logstash+elasticsearch+kibana-based Log Collection Analysis Scheme (Windows)