Distributed collaborative design based on zookeeper for log system

The most recent time in the design and implementation of the log system, the role of zookeeper in the entire log system system is important-it is used to coordinate the various distributed components and provide the necessary configuration information and metadata. This article mainly shares the Zookeeper usage scenarios. This is mainly related to the use of zookeeper in the log system, but in fact it is also very important in our message bus and search module.

Log meta data

The log type and log fields Here are collectively referred to as log metadata. The purpose of our construction of log system is mainly for: Log search, log analysis. We rely heavily on--elasticsearch for these two chunks (about what is ElasticSearch, here is not much to do with the introduction).

About Log Fields

The field definition of a log, in Elasticsearch, is the schema of one of the mapping in an index. Elasticsearch is a supposedly Schema Free distributed full-text retrieval system. Here it is necessary to understand correctly Schema Free that it does not require schema, nor does it require that you have a clear schema, and that you can do full-text search. However, many advanced functions like aggregation, analysis, etc. are based on a clear schema. Therefore, from the point of view of statistical analysis, it is necessary for us to make explicit field definitions for the log.

Collaboration of log and search modules

Log metadata is the basic information of the log system, and we manage it on the Web console and synchronize it to zookeeper for use by other modules, such as search modules. Because the type of log mentioned above, the field is actually the Mapping Type equivalent mapping relationship with Elasticsearch, so the search module will rely heavily on log metadata. In addition, to ensure that new log metadata (usually a new log type is created) is synchronized to Elasticsearch as soon as possible, we use the Zookeeper event push mechanism (or pub/sub mechanism) to get real-time information about the changes in log metadata. Once a new log metadata is generated, our search module is immediately notified of the event, which gets the latest log metadata and then creates a new mapping for it in the Elasticsearch indices. Prepare the logs for this type of log deposit Elasticsearch.

The benefits of this approach are at least three points:

• Real-time: The search module can first sense the creation of a new log type
• Low coupling: The log module on the platform and the search module do not have strong coupling due to the dependence of information; they are decoupled through zookeeper.
• Mapping type controllability: Elasticsearch has a very good feature, that is, when you save a document to a Mapping type, if there is a field in the document that Mapping undefined, Elasticsearch will automatically add the field definition for you. We think that this mechanism will make the log fields become uncontrolled. Therefore, we ensure that the schema is controllable by using the unified log metadata plus the same parsing behavior later on.

Log capture

From the previous article, you should be able to find the log collector selection, for a variety of reasons (with the message bus integration, customization, support from zookeeper to obtain configuration, etc.), we chose the flume-ng. It provides the ability to acquire the acquisition configuration from the zookeeper in the latest stable version (1.6.0). As a result, the cost of log acquisition on operations is greatly reduced. Because of the configuration, operators do not need to manually modify the configuration file on the target server to complete various configurations, only need to type the following startup command:

sudo bin/flume-ng---z127.0.0.1:2181-p /component/log-name mysql-slowquery-30-Dflume.root.logger=DEBUG,console

The only thing that needs to change in the above command is the following:

• Zookeeper Server (cluster) information
• The flume path of the type of log that will be collected
• The Znode name of the flume configuration of the log that will be collected, as in the example abovemysql-slowquery-30

In fact, some of the parameter items that were originally needed to manually modify the configuration file will be configured in the provided console, but web-based form filling is obviously much easier than the command line on the server.

Our practice here is to disassemble the flume configuration file, make a fixed part of it into a template, and make it a variable part of the form. Before committing, the template is merged with the configuration information through the template engine into a full configuration and pushed into the zookeeper.

Some of the parameters that need to be configured are:

• Path location of the destination server where the log file resides
• Format of the log file name
• Whether the log is single-line or multiline mode
• Secret of the message source (Message bus section)
• The name of the message slot (Message bus section)
• Token of the message flow (part of the message bus)
• ....

Log parsing

Also in the previous article I mentioned that our choice in log parsing is morphline. Morphline is an ETL Framework in the Hadoop ecosystem. Morphline also has a configuration file for defining a series of commands. And it also has a fixed part and a variable part, because the parsing mainly applies the morphline Grok command, so for this command, the variable part is mainly:

• Parsing dictionaries
• Parsed Regular Expressions

Our approach is similar to the Log Capture module, which makes the fixed section of the Morphline configuration file a fixed template, then configures the variable part on the console and eventually merges it into the zookeeper.

Log parsing service (belonging to the following 后台服务 ), at startup, according to its own log type, from a specific node in the zookeeper to find the configuration of the log type Morphline, to be taken down and saved in the local file system, Then build the Mrophline object (because Morphline currently only provides a method based on the file object, so there is a step to save to the local file and then construct the Morphline object based on the file) for parsing.

Background services

Log parsing This is just to provide metadata for the parsing task, the real parsing is done by the parsing task in the background, we will like these allow all tasks in the background in general to boil down to the background service .

Background services follow: 任务组 任务 工作线程 ----hierarchical organization. According to the service type of business (that is, the same set of processing logic), divided into different task groups (such as: ETL Group, log Index group, etc.), there will be at least one task under different task groups, For example, there will be many tasks under the ETL Task group (for example: Nginx Access log parsing task, MySQL slow query log parsing task, etc.), and there is at least one worker thread under one task.

Task Meta Data

There is a backend service module in the console that is dedicated to managing task objects and their metadata.

The worker threads that perform the tasks themselves are stateless, and when they are started they go to zookeeper to download the metadata they need to perform the task.

Hot standby mechanism

In general, we will equip each task with not less than one worker thread for the availability of the service. Of course, no less than one is counted not just based on a JVM process running a background service or a host node, but for a cluster of multiple nodes.

This is achieved through a configuration:

worker.minimumNumPerTask=2

Its meaning is: the minimum number of worker threads to start for each task. If a JVM process of two background services is started on a host node, then this task will correspond to 4 worker threads.

Normally, each task has only one worker thread in the active state at the same time, and the other preemption fails to switch itself to standby mode, which is available as a standby.

How is this mechanism implemented? This is due to the temporary order node provided by zookeeper.

When multiple worker threads compete for a task, they first create a child path under the path of the task and register their own host information. Note that the type of child path created here is different from the path type of the other zookeeper usage scenario (the other path is usually persistent), which is temporary and sequential . These two properties are important:

• Temporary: When a worker thread is hung, its local zookeeper session is invalidated, and the temporary node disappears after its session is invalidated.
• Order: It can arbitrate the order of the client that created the path and append the identity to the new path

After each worker thread creates a temporary sequential path, zookeeper appends the path with a number incremented from 1 in the order in which they were created, because of the order . After each worker is created, the individual numbers are numbered, and then they are judged in a sequential order, who is the smallest who will get the job execution opportunity and become an active worker, while other preemption failures will default to the standby mode. These preemption-failed worker threads are registered as child node change watcher for their preemption task. The temporary attributes come in handy when the worker thread in active mode loses the session, and the standby will be notified, and then they will again judge if their number is the smallest, If so, you can replace the previous worker thread as active. If a new worker thread is added, the change notification will also be triggered, but this will not affect normal logic.

There are, of course, some problems to be perfected:

• An active thread may cause a zookeeper error if a short-term session loss occurs due to network latency
• Frequent changes to child nodes may result in broadcast storms

Dynamic execution of new tasks

Each task group will have a watcher to monitor whether a new task has been created (such as a new log type being committed). If there are new tasks, new workers are created in the pool of threads to which they belong to perform new tasks. For now, this watcher by default only focuses on the new task, and for the task is removed or the task's metadata changes, watcher has no response mechanism for the time being. This is also part of the follow-up needs to be considered and perfected.

Background Service Configuration Templating

This mechanism has been applied separately 日志采集 , 日志解析 modules, and is here to simplify the configuration of background service startup.

Summarize

In conclusion, the zookeeper topological diagram of the whole design is as follows:

From the above analysis, we can see that we have configured the various variable parameters to the zookeeper to be the configuration center of the whole distributed system in series, and our control console has degenerated into a "configuration system" in a sense--the configuration is interface and persistent. At the same time, we also use the real-time event push mechanism of zookeeper for distributed coordination.

Distributed collaborative design based on zookeeper for log system