Introduction to the kids log system

Introduction to the kids log system

As the business and system scale grows, the number of machines in Our backend is also increasing, bringing about many new problems, such as unified system release and upgrade, monitoring, and so on. We believe that everyone has their own ideas and practices for solving these problems. This blog post does not intend to expand on these problems, but starts from one of the points-logs. No matter how a system is implemented, logs are indispensable, as are background systems. Logs play a very important role in business monitoring based on logs or in the case of an emergency by searching logs to locate problems. In the single-host era, logs can be simply written to a local disk. When the log file is increased, you need to consider issues such as log splitting, log classification, and even log cleanup. There are many existing tools to solve these problems, such as log4j and log4cpp. However, when multiple machines are deployed, the local disk writing method cannot meet the business requirements. For example, if a business problem occurs, you need to restore the scene in time to find the crux of the problem. In the past, you may first find the log flow, but how do you check multiple machines? Search for an ssh server? This is a very unrealistic approach, inefficient and time-consuming. At this time, you may need to collect logs from multiple machines for search. kids is a reference.

Kids is a distributed log system developed by zhihu team. Its abbreviation is "kids is a data stream", which is somewhat GNU. At the beginning, there were three ideas in Technology Selection: one was to manually write a transmission tool to Send Logs to a fixed server, and the other was to collect logs in combination with the pubsub message queue, the third is the existing distributed log system. At the beginning, we prefer to use the second solution, such as kafka and rabbitmq. Because the background technology stack is dominated by C ++, I am unwilling to continue looking for it. Later, I accidentally saw facebook's open-source scribe, but this was not updated for a long time. Second, it depends on thrift, it is not a lightweight thing. Later, I accidentally saw an article on zhihu architecture on infoq. The article introduced the log system kids developed by their team, which is also open-source, so I cloned it from github and used it. First, the configuration is very simple and clear, and you can try it out.

Since the gcc of the online system does not support C ++ 11 and only one test machine is supported, I switched to the system to use libstdc ++ for static compilation. Because there is no strict performance test, and in fact there is no idea of performance testing, we only use kids for some services with a small number of logs. The daily log volume is more than 100 kb, the kids process on each machine occupies more than 50 MB of memory, and even if the Kids crash is dropped, the actual service is not affected. Kids logs are not written to logs in real time, but are buffered into the memory for a period of time before being written. You can set the flush frequency in the configuration file to ensure that the data in the memory is written to the file in time. Later, you can study its implementation mechanism to see if there is any room for optimization. This is probably the case.

Kids features real-time subscription, distributed collection, message persistence, multithreading, Redis protocol, and no third-party dependency. Kids can be configured as an agent or server on each server. The agent directly accepts messages from the application. After collecting the messages, you can call the next agent or directly call the central server. When subscribing to logs, you can obtain logs from the server or some agents on the central node.

Usage + DQpGaWxlU3RvcmWjrM7EvP605rSio6y24NPD09ogU2VydmVyILXEs9a + usage "brush: java;">Store file {path/to/logs/[topic]/[date]; name [time]. log; rotate 1 hour; # time units support hour and min}

With this configuration, kids stores logs in the form of/topic/date/time. log and splits files hourly. In addition, you can add flush 5s or other configurations to the file storage configuration block to ensure that the buffer data in the memory is written to the file in time. This may be helpful when the processing volume is small but the key log type is critical, but you do not need the flush option most of the time, or use the subscription mode to obtain real-time data.

NetworkStore, network storage, forward logs to a specified host through the network.

store network { host kidsserver; port 3388;}

NullStore, empty storage. In this case, kids does not store specified logs in any way. It can be used in combination with other storage methods to filter specified logs.

store null { topic kids.ignoretopic;}

Hybrid ConfigurationThere are also three methods: PriorityStore, BufferStore, and MultipleStore:

PriorityStore, logs are stored in the order in the configuration file. If a storage method is successfully stored, the log will not be further stored.

store priority { store null { topic kids.ignoretopic1; topic kids.ignoretopic2; } store network { host kidsserver; port 3388; } store file { path /path/to/logs/[topic]/[date]; name [time].log; rotate 1hour; }}

After this configuration is used, kids ignores the storage of kids. ignoretopic1 and kids. ignoretopic2. For other topics, kids are forwarded to kidsserver: 3388. If the forwarding fails, kids are saved to the specified file.

BufferStore uses Primary and Secondary storage. When a Primary storage exception occurs, logs are cached in the Secondary storage. After Primary is restored, the cache in Secondary is restored to Primary.

store buffer { store network primary { # primary host kidsserver; port 3388; } store file secondary { # secondary path /data/kidsbuf; rotate 5min; }}

MultipleStore and MultipleStore store logs in multiple locations. By defining the success type, determine the storage success and Failure success values: any or all.

store multiple { success any;# master store buffer { store network primary { host kidsserver; port 3388; } store file secondary { path /tmp/kidsbuf; rotate 10min; } }# backup store buffer { store network primary { host kidsbackup; port 3388; } store file secondary { path /tmp/kidsbuf2; rotate 10min; } }}