. NET under the construction of log system--log4net+kafka+elk

. NET down-log system construction--log4net+kafka+elk preface

Our company's program log is a way of using log4net to record file logs (the simple use of log4net can be seen in my other blog), but as our team grew larger and bigger, the project grew and our users grew more and more. Slowly the system has exposed a lot of problems, this time our log system can not meet our requirements. The main problems are as follows:

• As our traffic increases, our log files increase dramatically.
• A lot of messy file log, it is difficult for us to debug the program
• Logging of files consumes resources from our application servers, resulting in a decrease in the ability of our application servers to process user requests
• Our logs are distributed on multiple application servers, when the program encountered problems, our programmers need to find operations and maintenance personnel to log, as the team is getting bigger, more and more problems, so that the programmers queued to find operations to log, solve the problem of the rapid decline!

At first, when the user volume is small, the above problems can be tolerated. But any kind of small problem will be magnified when the user volume of access is large. Finally in a few waves of promotional activities, it is very tragic that we have to work late at night to pay for our previous attention to these issues. So, at the end of the campaign, as our programmers got a bit of a breather, I decided to build our own log system to improve the way we log records. According to the above problem analysis of our log system needs to have the following requirements:

• Log writes are highly efficient and cannot have too much impact on the application server
• To centralize logs on a single server (or group)
• Provide a convenient search and analysis of the visual page (this is the most important, can not stand every day to find operations to log, get a pile of files to analyze the day! )

At first I wanted to use log4net Adoappender to write our logs into the database, and then we developed a corresponding function to query and analyze our logs. However, given the performance problems of writing relational databases, we gave up, but there is an alternative, which is to write to MONGO, which solves the improved performance. But we also need to develop a function to query the analysis. This time from the Internet to find a lot of solutions:

//方案1:这是我们现有的方案，优点:简单 缺点:效率低，不易查询分析，难以排错...service-->log4net-->文件              //方案2:优点:简单、效率高、有一定的查询分析功能 缺点:增加mongodb，增加一定复杂性，查询分析功能弱，需要投入开发精力和时间service-->log4net-->Mongo-->开发一个功能查询分析             //方案3:优点:性能很高，查询分析及其方便,不需要开发投入 缺点：提高了系统复杂度，需要进行大量的测试以保证其稳定性，运维需要对这些组件进行维护监控...service-->log4net-->kafka-->logstash-->elasticsearch-->kibana搜索展示               //其它方案service-->log4net-->文件-->filebeat-->logstash-->elstaicsearch-->kibanaservice-->log4net-->文件-->filebeat-->elstaicsearch-->kibanaservice-->log4net-->文件-->logstash-->elstaicsearch-->kibana

Finally and after the team exchange decided to adopt the combination of Scenario 2 and Scenario 3, I added a log4net for MONGO Appender (which is also available on this appender,nuget), and our team developed a feature that supports simple query search. I'm synchronizing to build Scenario 3. About Scenario 2 is not much introduced, very simple. A major mention of programme 3.

I. ELKB INTRODUCTION

• Elastic Search: As you can see from the name, Elastic search is used for searching, providing data and corresponding configuration information (what field is the data type, which fields can be retrieved, etc.), and then you are free to use the API to search for your data.
• Logstash:. Log files are basically each line, each of which has a variety of information, the function of the software is to parse each log into various fields.
• Kibana: Provides a set of web interfaces for interacting with Elastic search so that we do not use the API to retrieve data, we can enter keywords directly in Kibana, Kibana will present the returned data to us, of course, There are many beautiful data visualization charts to choose from.
• Beats: Installed on each server that needs to collect logs, send the logs to logstash for processing, so Beats is a "porter" and will carry your logs to the Log collection server. Beats are divided into many kinds, each of which collects specific information. Commonly used is filebeat, listen to file changes, transfer the contents of the file. The general log system is sufficient to use filebeat.

Two. Kafka Introduction 2.1 Introduction

Kafka is a high-throughput distributed publish-subscribe messaging system that handles all the action flow data in a consumer-scale website. This kind of action (web browsing, search and other user actions) is a key factor in many social functions on modern networks. This data is usually resolved by processing logs and log aggregations due to throughput requirements.

2.2 Applicable scenarios

• Messaging
  For some conventional messaging systems, Kafka is a good choice; partitons/replication and fault tolerance can make the Kafka have good scalability and performance advantages. But so far, we should be aware that Kafka does not provide "transactional "" Message transmission guarantee (message acknowledgement mechanism) "message packet" and other enterprise-class features; Kafka can only be used as a "regular" message system, to some extent, has not ensured that the message is sent and received absolutely reliable (for example, the message resend, message sent lost, etc.)

• WEBSIT Activity Tracking
  Kafka can be the best tool for "Site activity tracking" and can send information such as Web page/user actions to Kafka. And real-time monitoring, or offline statistical analysis, etc.

• Log Aggregation
  The Kafka feature determines that it is well suited as a "log collection center", application can send the operation log "bulk" "asynchronously" to the Kafka cluster instead of being stored locally or in db; Kafka can submit messages in batches/compressed messages, etc. For the producer end, there is almost no performance overhead. At this point consumer can make other systematic storage and analysis systems such as Hadoop.

Iii. Log4net+elk+kafka Log System 3.1. Introduction

From the above we can learn that we can add a log4net kafkaappender log producer through this Appender write logs to Kafka, due to Kafka bulk commit, compression characteristics, so the cost of our application server performance is very small. The log consumer uses Logstash to subscribe to the messages in Kafka, which are sent to Elasticsearch and displayed to us via Kibana. At the same time we can also through the Kibana to our log statistical analysis and so on. Just to solve some of our problems. The whole process is roughly like this:

About Log4net for Kafka Appender, I wrote one myself, there are ready-made packages on nuget, and we need to find them on nuget.

3.2. Build

Simple introduction of the building, the construction process using Docker.

3.2.1 Docker Installation Kafka

//下载//下载zookeeperdocker pull wurstmeister/zookeeper//下载kafkadocker pull wurstmeister/kafka:2.11-0.11.0.3

//启动//启动zookeeperdocker run -d --name zookeeper --publish 2181:2181 --volume /etc/localtime:/etc/localtime wurstmeister/zookeeper//启动kafkadocker run -d --name kafka --publish 9092:9092 \--link zookeeper \--env KAFKA_ZOOKEEPER_CONNECT=192.168.121.205:2181 \--env KAFKA_ADVERTISED_HOST_NAME=192.168.121.205 \--env KAFKA_ADVERTISED_PORT=9092  \--volume /etc/localtime:/etc/localtime \wurstmeister/kafka:2.11-0.11.0.3

//测试//创建topicbin/kafka-topics.sh --create --zookeeper 192.168.121.205:2181 --replication-factor 1 --partitions 1 --topic mykafka//查看topicbin/kafka-topics.sh --list --zookeeper 192.168.121.205:2181//创建生产者bin/kafka-console-producer.sh --broker-list 192.168.121.205:9092 --topic mykafka //创建消费者bin/kafka-console-consumer.sh --zookeeper 192.168.121.205:2181 --topic mykafka --from-beginning

3.2.2 Docker Installation Elk

//1.下载elkdocker pull sebp/elk

//2.启动elk//Elasticsearch至少需要单独2G的内存//增加了一个volume绑定，以免重启container以后ES的数据丢失docker run -d -p 5044:5044 -p 127.0.0.1:5601:5601 -p 127.0.0.1:9200:9200 -p 127.0.0.1:9300:9300 -v /var/data/elk:/var/lib/elasticsearch --name=elk sebp/elk

//若启动过程出错一般是因为elasticsearch用户拥有的内存权限太小，至少需要262144切换到root用户执行命令：sysctl -w vm.max_map_count=262144查看结果：sysctl -a|grep vm.max_map_count显示：vm.max_map_count = 262144

上述方法修改之后，如果重启虚拟机将失效，所以：解决办法：在   /etc/sysctl.conf文件最后添加一行vm.max_map_count=262144即可永久修改

Access after successful start: http://your-host:5601 See Kibana page indicates successful installation

Configuration uses

//进入容器docker exec -it <container-name> /bin/bash

//执行命令/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'/* 注意：如果看到这样的报错信息 Logstash could not be started because there is already another instance using the configured data directory.  If you wish to run multiple instances, you must change the "path.data" setting. 请执行命令：service logstash stop 然后在执行就可以了。*/

Test

When the command is executed successfully, see: Successfully started Logstash API endpoint {:p ort=>9600} after the information, enter: This is a dummy entry then enter, simulate a log to test.
Open the browser, enter: Http://your-host:9200/_search?pretty, you will see the log content we just entered.

3.2.3 Logstash-kafka Configuration Instance

This is a configuration file for my test.

input {        kafka{                //此处注意:logstash5.x版本以前kafka插件配置的是zookeeper地址，5.x以后配置的是kafka实例地址                bootstrap_servers =>["192.168.121.205:9092"]                client_id => "test" group_id => "test"                consumer_threads => 5                decorate_events => true                topics => "logstash"        }}filter{        json{                source => "message"        }}output {        elasticsearch {                hosts => ["192.168.121.205"]                index=> "hslog_2"                codec => "json"        }}

Configuration file Start Logstash mode

/opt/logstash/bin/logstash -f "配置文件地址"

Conclusion

     as above, our log system is basically built, and of course there are a lot of questions about the use of Kafka,logstash,elstaicsearch,kibana, and some of the problems we use, Let's try to build it ourselves. Of course, there is no best solution, it is recommended to combine the reality of your company and system to find and choose solutions. If you can solve the problem with a simple solution, don't use complex solutions. Because the complex solution solves the problem, it also brings us other problems. Just like our solution, which solves our problem at the time, but also adds to the complexity of our system, for example: each of these components has a problem that will cause our log system to be unavailable ..., in addition, 工欲善其事 its prerequisite, although we solved the problem of the device, but want to " Good for us "there is a long way to go, because the fundamental, log records do not record, where the record, record what level of the log, or we choose to record. Log records no specification, disorderly, blind, how to standardize the log record is the next big problem we want to solve! Welcome to leave a message to discuss these questions!