Access Log IO Performance optimizations

In high concurrency scenarios, disk IO is often the bottleneck of performance, and access logs involve frequent write operations, so this part needs to be optimized as much as possible, or it will slow down the overall performance of the system. There are two ways to improve write performance for file records and database records.

L Avoid frequent opening and closing of files. The general procedure for writing a log to a file is to open - write - Close, but it is certainly a performance problem in scenarios where you need to write to the log frequently, because cost is required for each turn-off. So you have to think about whether there are other better ways, even if the problem occurs in the number of open close, then solve the problem from here, you can open the log after the first time not to close the operation, keep open state, the next write will not need to open again to write directly. And because the actual process of access log production is very large, it is certainly not possible to write data to a file, you may write a file per day or by the file size every 50M Write a file, so in the actual design will involve the operation of the replacement file, The original file stream needs to be closed when replacing. Through the above measures to avoid the file frequently open close operation, but it also has some shortcomings, the file stream does not shut down will always occupy the operating system resources, and if not timely shutdown may occur during the process of the operation of the exception caused by improper processing of the file stream could cause a memory leak.

L Add buffers: the role of adding buffers is simply to reduce the number of times a file is actually written to disk, typically writing a file to write to disk media for each write operation, while the buffer mode writes the data to be written to the memory. When the buffer memory reaches a certain level to write to the disk, so the file operation to add a buffer is not every write is a storage medium, buffer for IO operation is a very important concept, buffer specific implementation can refer to my previous related chapters, of course JDK the relevant buffer classes have been provided and we do not have to make the wheels again. The addition of buffers undoubtedly improves the performance of the operation, but it also has the disadvantage that it cannot guarantee that all data will be successfully logged to the storage medium, which may cause the buffer to be unable to write to the file when the system is unexpected, and only the buffer data is lost.

L Use "pool" technology to optimize the connection to avoid each time you create a connection, the pool technology is more for the database as a storage-side scenario, that is, we are most familiar with the connection pool,JDBC every time the creation of a connection requires a large cost of overhead, If each write re-establish the connection this is unbearable for the system, so you can create a connection pool when the system is initialized, the pool contains a lot of established connections, each use only to dominant out, using the end of the connection is not closed, but to put the connection back into the pool. The benefits of this approach are obvious, the performance of the operation is greatly improved, and if there are any drawbacks, it is estimated that the connection pool needs to occupy some operating system resources, even if it is not used. But for today's machines this cost is basically negligible.

l  Optimization Lock competition, first of all, in a high concurrency scenario, the log write is sure to be multi-threaded, multi-threaded use in improving the performance of the system is no doubt, second, unexpectedly in the file stream write operation is multi-threaded, that inevitably involves the problem of lock competition, Because if a protected file without a lock can be written in a messy and incorrect way, the thread must try to compete for a write lock before writing, only the thread that successfully acquires the lock can write, and once it is written, the lock is released; Finally, the competition with JDK JVM jdk The accompanying package is the first attempt to spin-fetch, A pending operation after several failures.

The

l  considers non-blocking mode, so-called non-blocking (nio ) means io cpu This mode is event-driven, Different events are handled by one or several threads, in practice nio Patterns are proven to be a powerful tool for performance improvements in some scenarios. Its only drawback is that it makes your program more complex.

With asynchronous io, asynchronous io(AIO) is a new IO provided after JDK1.7 mode, which is mainly for CPU optimization, is CPU -level optimization attempt, in the actual use of the system needs to do performance test comparison and then decide whether to do AIO Retrofit.

Like to study Java classmate can make a friend, the following is my number:

Access Log IO Performance optimizations