Spring Mvc+elk build log platform from start

Build a distributed log system from scratch, mainly on spring MVC with the Elk Suite (some of the work has been done by different colleagues because of the division of labor, I just developed it in an already configured environment), including the following technical points:

• Spring MVC
• Logback
• Logstash
• Elasticsearch
• Kibana
• Redis

Looking at the overall architecture diagram, this kind of architecture is very easy to solve the problem of logging, querying and analyzing difficult problems in the current distributed system.

 

Operating system, IDE environment:

• Eclipse
• Windows

1: Build the Spring MVC Project
The Dynamic Web project created by Eclipse comes with an empty structure, without any configuration, and if we want to run a Hello World project, we need to do some configuration, such as creating program files such as View,controller and so on.
Spring Tool suite helps us solve this problem by providing a project template for spring MVC, which comes with a "Hello World" bootable app page, which makes it easy to install the Spring tool suit as a plug-in form in eclipse. Once installed, you can create it.
It is important to note that different versions of the Spring tool suite will have different menus at the time of creation, and my current menu is located at:

The first thing to do is to select the spring tag:

 

Then, under the File menu, look for:

Once created, we can run directly under Tomcat without any other action, which is much more convenient than creating a dynamic Web project, but there are drawbacks to projects created with this template: If you like some new dependency packages, Then you need to manually go to the Pom file to update the version number for what you want, and may introduce some third-party packages that you may not be able to use temporarily. is a slightly modified project completion diagram, which is a standard MAVEN structure of spring MVC.

 

2:redis Installation
Because I am a Windows environment, I need to download the Windows version of Redis as appropriate:
Windows Edition: Https://github.com/mythz/redis-windows
Download it and unzip it and select a version:

Config file I only modified one: Bind, which is used to bind a fixed IP, why should it be shown to bind an IP? I'm going to show you the problem later.
Start server: Execute in redis/bin directory: Redis-server.exe redis.windows.conf to start

Start client: Execute in redis/bin directory: Redis-cli.exe-h 127.0.0.1-p 6379, in this window you can test Redis with some Redis commands, such as Get,set, keys, and so on.

3:elk Installation

On this site you can find the latest version of Elk: Https://www.elastic.co/downloads, will elasticsearch,logstash,kibana all three downloads down.

• Configure Elasticsearch

Most of the configuration is used by default, just for the sake of good identification I modified the cluster.name and Node.name, detailed parameters to study the document. Then run Elasticsearch.bat directly in the bin directory to start up.

Open the Http://127.0.0.1:9200/, you can see the following information to indicate that the boot is normal.

There are many plugins that can be installed to help us view the monitoring Elasticsearch, where the head is installed first, the command line enters the Elasticsearch directory, and then the plugin install mobz/ Elasticsearch-head can be installed.

Open http://127.0.0.1:9200/_plugin/head/after successful installation

 

• Configure Logstash

First look at the architecture of the Logstash and with other elk, the data source of this article is Redis, does not involve the filter, the final log output to Elasticsearch.

 

Here we only configure input and output, it is important to note that different versions of the Logstash in the configuration will be slightly different, you are interested in further comparison.

Input {    Redis {        data_type = "List"        key = "Logstash"        host = "127.0.0.1"        port = 6379< C6/>threads = 5        codec = "JSON"    }}filter {}output {    Elasticsearch {        hosts = = [" 127.0.0.1:9200 "]        index =" logstash-%{type}-%{+yyyy. MM.DD} "        Document_type ="%{type} "        workers = 1        flush_size =        Idle_flush_time = 1< C16/>template_overwrite = True    }    stdout{}}

Then execute logstash-f etc/logstash.d/in the Logstash directory to start

• Configure Kinbana
  • Elasticesearch.url points to the previously configured Elasticsearch address.
  • Kinbna.index, this is used to store some information about the Kibana itself.

• Integrated Logback

A log entry is required, Logback-classic is introduced, and in order to pass the log to Redis, you need to configure a logback-redis-appender, depending on the following:

<!--Logging--<dependency> <groupId>org.slf4j</groupId> <artifa Ctid>slf4j-api</artifactid> <version>${org.slf4j-version}</version> </dependency > <dependency> <groupId>ch.qos.logback</groupId> &LT;ARTIFACTID&GT;LOGB        Ack-classic</artifactid> <version>${logback.version}</version> </dependency>            <!--Logstash Begin--<dependency> <groupId>com.cwbase</groupId> <artifactId>logback-redis-appender</artifactId> <version>1.1.3</version> &L                    T;exclusions> <exclusion> <groupId>redis.clients</groupId>        <artifactId>jedis</artifactId> </exclusion> </exclusions> </dependency>

The configuration Logback.xml,key needs to match the key configured in the Logstash configuration file.

<appender name= "LOGSTASH" class= "Com.cwbase.logback.RedisAppender" >        <source>logstashdemo</ source>        <type>dev</type>        
Logging is recorded in HomeController, and slf4j is perfectly integrated with logback, so we do not need to do any transformation class configuration to achieve logging.

As mentioned earlier, when you configure Redis, you set the Bind property to a fixed IP. If you do not specify, there will be a problem connecting to Redis in Logstash, and the reasons for this need to be further confirmed later.

4: Run Web site, view logs
When the Redis,elasticsearch,logstash service is working properly, start spring MVC, and the log recorded by logger can be easily viewed in Kibana.
To test whether Logback has sent logs to Redis, you can use the Redis command to see if the configured Logstash key is included, and Llen to determine if the log is incrementing normally.
 
If it is normal, then find the Kibana page, the first open will prompt the creation of index rules, create a good after you can see the log has been collected into the elasticsearch.

 
Spring Mvc+elk build log platform from start