Flume-based Log collection system (ii) Improvement and optimization

In the flume-based log collection system (a) architecture and design, we detail the architecture design of the flume-based log collection system and why it is designed. In this section, we will describe the problems encountered in the actual deployment and use process, the functional improvements to flume, and the optimizations that are made to the system.

1 Summary of Flume issues

The main problems encountered during the use of Flume are as follows:

A. Channel "acclimatized": the use of fixed-size memorychannel at log peaks often reported that the queue size is not enough, the use of FileChannel also cause IO busy problem;

B. Performance issues with Hdfssink: use Hdfssink to write logs to HDFs, which is slower at peak times;

C. System Management issues: Configuration upgrade, module restart, etc.;

2 flume feature improvements and optimization points

As you can see from the above questions, there are some requirements that the native flume cannot satisfy, so we have added a lot of features to the flume based on open source, modified some bugs, and made some tuning. Some of the key aspects will be explained below.

2.1 Add Zabbix Monitor Service

On the one hand, flume itself provides HTTP, ganglia monitoring services, while we are currently mainly using Zabbix to monitor. As a result, we have added the Zabbix monitoring module to Flume and seamlessly converged with SA's monitoring services.

On the other hand, purify the flume metrics. Send only the metrics we need to Zabbix to avoid stress Zabbix server. At present, we are most concerned about whether Flume can timely send the application end of the log written to HDFs, the corresponding concern metrics:

• Source: The number of event received and the number of event processing
• Number of Channel:channel in the event of congestion
• Sink: Number of event already processed

2.2 Add auto-create INDEX function for Hdfssink

First, our Hdfssink writes to Hadoop files with Lzo compressed storage. Hdfssink can read a list of the encoding classes provided in the Hadoop configuration file, and then configure the way to get the compression encoding used, we currently use Lzo to compress the data. With Lzo compression instead of bz2 compression, it is based on the following test data:

td>544

event Size (Byte)	sink.batch-size	hdfs.batchsize	compressed format	Total data size (g)	time-consuming (s)	average events/s	compressed Size (g)
544	300	10000	bz2	9.1	2448	6833	1.36
	10000	lzo	9.1	612	27333	3.4 9

Second, our Hdfssink added the ability to automatically create an index after creating a Lzo file. Hadoop provides the ability to create indexes on Lzo so that compressed files are segmented so that hadoop jobs can process data files in parallel. The hdfssink itself is lzo compressed, but the Lzo file is not indexed, and we added the build index function after the close file.

  /** * Rename Bucketpath file from the. tmp to permanent location.              */private void Renamebucket () throws IOException, interruptedexception {if (bucketpath.equals (TargetPath)) {        Return        } final Path Srcpath = new Path (Bucketpath);        Final path Dstpath = new Path (TargetPath);  Callwithtimeout (New callrunner<object> () {@Override public Object call () throws Exception {if (filesystem.exists (Srcpath)) {//could block Log.info ("renaming" + Srcpath + "                     To "+ Dstpath); Filesystem.rename (Srcpath, Dstpath); Could block//index the Dstpath lzo file if (CodeC! = null && ". Lzo". Equals (Codec.getdefaultextension ())) {Lzoindexer lzoindexer = new Lzoindexer (New Configurat                              Ion ());                      Lzoindexer.index (Dstpath);              }                }  return null; }    });}

2.3 Increasing the Hdfssink switch

We add switches in Hdfssink and Dualchannel, and when the switch is on, Hdfssink no longer writes data to HDFs, and the data is written only to FileChannel in Dualchannel. This policy prevents the normal shutdown of HDFs from being maintained.

2.4 Increase Dualchannel

The flume itself offers Memorychannel and FileChannel. Memorychannel processing is fast, but the cache size is limited and not persisted; FileChannel is just the opposite. We want to take advantage of both, in the sink processing speed is fast enough, the channel does not cache too many logs, the use of Memorychannel, when the sink processing speed, and the need for channel to cache the application sent over the log, Using FileChannel, we developed the Dualchannel to intelligently switch between two channel.

Its specific logic is as follows:

/*** * Puttomemchannel indicate put event to Memchannel or FileChannel * Takefrommemchannel indicate take event from MEMCH Annel or FileChannel * */private atomicboolean puttomemchannel = new Atomicboolean (true);p rivate Atomicboolean takeFromMe              Mchannel = new Atomicboolean (true), void DoPut (event event) {if (Switchon && puttomemchannel.get ()) {              Write data to Memchannel Memtransaction.put (event);              if (Memchannel.isfull () | | filechannel.getqueuesize () >) {Puttomemchannel.set (false);        }} else {//to FileChannel Write Data filetransaction.put (event);    }}event Dotake () {event event = null;        if (Takefrommemchannel.get ()) {//Fetch data from Memchannel event = Memtransaction.take ();        if (event = = null) {Takefrommemchannel.set (false);        }} else {//Fetch data from FileChannel event = Filetransaction.take ();       if (event = = null) {     Takefrommemchannel.set (TRUE);        Puttomemchannel.set (TRUE); }} return event;}

2.5 Increase Nullchannel

Flume provides Nullsink, which can discard unwanted logs directly through Nullsink, without storing them. However, source needs to store events in the channel first, and nullsink the events out and throws them away. To improve performance, we moved this step into the channel, so we developed the Nullchannel.

2.6 Increase Kafkasink

To support the provision of real-time data streams to storm, we have added Kafkasink to write real-time data streams to Kafka. Its basic logic is as follows:

public class Kafkasink extends Abstractsink implements configurable {private String zkconnect;        Private Integer zktimeout;        Private Integer batchsize;        Private Integer queuesize;        Private String Serializerclass;        Private String Producertype;        Private String Topicprefix;        Private producer<string, string> Producer;  public void Configure (context context) {//Read configuration, and check configuration} @Override public synchronized void Start () {//Initialize producer} @Override public synchronized void Stop () {//off PROD Ucer} @Override Public Status process () throws Eventdeliveryexception {Status status = S Tatus.            Ready;            Channel channel = Getchannel ();            Transaction tx = Channel.gettransaction ();                    try {tx.begin (); Store the logs by category Map<string, list<string>> toPic2eventlist = new hashmap<string, list<string>> ();                    Take the batchsize size log from the channel, get the category from the header, generate the topic, and store it in the map above;//Send the data in the map via producer to Kafka            Tx.commit ();                    } catch (Exception e) {tx.rollback ();            throw new Eventdeliveryexception (e);            } finally {tx.close ();        } return status; }}

2.7 Fix and scribe compatibility issues

scribed when sending a packet to flume via Scribesource, a packet larger than 4096 bytes will first send a dummy packet to check the response of the server, while Flume Scribesource for logentry.size () = 0 of the package returns Try_later, at which point scribed is considered to be faulted and disconnected. This loops over and over again, unable to actually send the data. Now in the Scribesource thrift interface, the case of size 0 returns OK, ensuring subsequent normal data transmission.

3. Flume System Tuning Experience Summary 3.1 basic parameter tuning experience

• The default serializer in Hdfssink will add a line break at the end of each line, and we have a newline character in the log itself, which will result in a blank line after each log, and modify the configuration to not automatically add line breaks;

lc.sinks.sink_hdfs.serializer.appendNewline = false

• Adjust the capacity of Memorychannel, make the best use of memorychannel fast processing ability;

• Adjust the batchsize of hdfssink, increase the throughput and reduce the flush times of HDFs;

• Appropriately adjust the calltimeout of hdfssink to avoid unnecessary time-out errors;

3.2 Hdfssink getting the filename optimization

The path parameter of the Hdfssink indicates where the log is written to HDFs, which can refer to the formatted parameter and write the log to a dynamic directory. This facilitates the management of logs. For example, we can write the logs to the category category directory and store them by day and by the hour:

lc.sinks.sink_hdfs.hdfs.path = /user/hive/work/orglog.db/%{category}/dt=%Y%m%d/hour=%H

When processing each event in HDFSS ink, the HDFs path and filename to which this event should be written are obtained based on the configuration, and the default fetch method is to replace the variable in the configuration with a regular expression to get the real path and filename. This is a lengthy process because it is an action to be done for each event. Through our test, 200,000 logs, this operation takes time 6-8s around.

Since our current path and filename have fixed patterns, they can be obtained by concatenation of strings. The latter is dozens of times times faster than regular matches. The way to splice a string of characters, 200,000 log operations only need hundreds of milliseconds.

b/m/s Optimization of 3.3 hdfssink

In our initial design, all the logs were written to HDFs via a channel and a hdfssink. Let's take a look at what's wrong with this.

First, let's take a look at the logic of Hdfssink in sending data:

//从Channel中取batchSize大小的eventsfor (txnEventCount = 0; txnEventCount < batchSize; txnEventCount++) {    //对每条日志根据category append到相应的bucketWriter上；    bucketWriter.append(event);｝for (BucketWriter bucketWriter : writers) {    //然后对每一个bucketWriter调用相应的flush方法将数据flush到Hdfs上    bucketWriter.flush();｝

Let's say we have 100 category,batchsize size set to 200,000 in our system. For every 200,000 data, 100 files need to be append or flush.

Secondly, for our journal, basically conforms to the 80/20 principle. That is, 20% of the category produces 80% of the log volume of the system. This way, for most logs, each 200,000 may contain only a few logs, and it needs to be flush to hdfs once.

The above situation causes Hdfssink to write HDFs very poorly. is the single channel of the case per hour of the send volume and write HDFs time trend graph.

In view of this practical application scenario, we classify the log into big, middle and small, which can effectively avoid small log with the large log with frequent flush, improve the effect is obvious. Is the amount of each hour of the big queue after the queue and the time trend graph to write HDFs.

4 future development

Currently, the Flume log collection system provides a highly available, highly reliable, scalable distributed service that has effectively supported the log data collection efforts of the American mission.

Follow-up, we will continue to study in the following areas:

• Log Management system: Graphical display and control log collection system;

• Follow up community development: Follow up the progress of Flume 1.5 while giving back to the community;

Flume-based Log collection system (ii) Improvement and optimization