How to quickly output logs to a disk

The log subsystem is a very important component for both client and server services.

The log system output destination can be disk, tty, or network.

My log system can be output to TTY, and different log levels can have different colors. In this way, the log is very eye-catching, of course, this article focuses on how to quickly write the log content to the disk.

In fact, how to quickly write the log content to the disk, online articles have been full of resources, the real quality is not much, this article may also be a dog's tail. However, my log subsystem can reach an output rate of 106 Mb/s.

Before giving a detailed introduction to my log system, we recommend that you take a look at the implementation of muduo log in "LINUX multi-thread Server programming: using the muduo C ++ network library, the idea and implementation of muduo's log are both very beautiful. There are also related ppt files on the Internet. There are a lot of dry goods here.

How can the log system quickly write log content to the disk? The key is to write logs sequentially, that is, the size of each log written must be a multiple of 4 K or 4 K.

Given that the language description is not my strong point, the following describes the key code and then details it.

Struct log_t {
Int file;
Pthread_mutex_t mutex;
Int buf_cursor;
Char Buf [2*1024];
Struct iovec _ [2];
};

Int log_write (log_t * log, char * Buf, int Len)
{
Int ret = 0;

Pthread_mutex_lock (& log-> mutex );
Do {
Size = Log-> buf_cursor + Len;
If (size <sizeof (log-> BUF )){
Memcpy (log-> BUF + Log-> buf_cursor, Buf, Len );
Log-> buf_cursor = size;
Break;
}

Log-> iovec _ [0]. iov_base = Log-> Buf;
Log-> iovec _ [0]. iov_len = Log-> buf_cursor;
Log-> iovec _ [1]. iov_base = Buf;
Log-> iovec _ [1]. iov_len = Len;
Ret = (INT) writev (log-> file, log-> iovec _, 2 );
If (is_neq (Ret, size )){
Ret =-2;
}

Log-> cursor + = size;
Log-> buf_cursor = 0;
} While (0 );
Pthread_mutex_unlock (& log-> mutex );

Return ret;
}
As shown in the code above, log has a 4 K Buf. If the output content can be put into the Buf, copy the content and exit, otherwise, the writev function is called to write the content into the log file.

The idea is similar to the muduo log of Chen Shuo danniu, which is of course not so tall. The key to the method is to reduce the lock granularity and Merge multiple writes into one write to write the log content to the disk in sequence.

The log system has two differences: synchronous and asynchronous. The above implementation method is essentially a synchronous method.

The so-called asynchronous output log is dedicated to starting a log thread, which can have a log queue, and other threads as producers output content to the queue, the log thread extracts the log content from the queue as a consumer and writes it to the disk.

I am now working in a company in xierqi. Recently I have implemented an asynchronous log system for the company. The key process is of course the above section. However, the above method alone cannot achieve fast asynchronous log output. In addition to the log thread queue, I also added a log Buf (struct log_buf {int cursor; char Buf [1*1024*1024] ;};). The idea is the same as that of synchronization. First, put the log Content in this log_buf. When log_buf overflows, put log_buf into the queue.

Because the copyright of the Code is owned by the company, the code will not be pasted here. The final output speed of asynchronous log can reach 140 Mb/s.

Recently, there have been a lot of unpleasant things, and there is a sense of departure, but we have not reached the homeless point, I will not learn a very high-profile man to beg for work.

However, if you think your company can provide me with a suitable space for development, I am very welcome, I qq3028042655.