Talking about the lock-free design of storage software

Storage software for disk design does not need to consider the performance impact of competitive locks. Disk storage software has a performance bottleneck in disk, and disk Jitter introduces a significant performance drain. Therefore, the design of traditional storage software is not particularly concerned with the efficiency of the processor. Once the performance tuning of a storage virtualization software, the lock competition has done a lot of optimization, and finally did not achieve performance gains, the reason is that storage virtualization performance bottleneck is the disk, not the efficiency of the use of the processor. Because of this, in the disk-oriented software, many of the use of single-threaded, single-queue processing, to some extent, can also avoid the problem of disk jitter introduced by concurrency.

in the face ofNVMe SSDdesign of the storage software, all of this is changing. Compared with traditional disks,NVMe SSDhas a very highIoRead and write performance, there is no access to the traditional disk, jitter problems. In order to playNVMe SSDperformance, in both software and hardware need to adopt multi-queue technology, through the multi-queue way to fully playNVMe SSDthe performance. UsuallyNVMe SSDController can provide128aIoCommit and end queues,NVMethe drive will be utilized based on the processor's number of coresSsdsThe controller provides the hardware capability to achieveIoconcurrent processing. At the drive level, you can bind the processor toIoqueues, reachingIothe effect of concurrent processing. In this case, the business thread must be multi-threading and concurrently exploiting the processor toNVMe SSDMulti-queue is really used. In many cases, business software models tend to be divided into computational units and storage unit two parts. The computational unit is often the mode of multithreading concurrency, and the storage unit is a single-threaded mode on a traditional disk,NVMe SSDwill become multithreaded concurrent processing. This software processing model can be described as follows:

650) this.width=650; "src=" http://s5.51cto.com/wyfs02/M00/88/5B/wKioL1fzEbWS8PWEAABu8ybAMCE359.jpg "style=" float: none; "title=" 1.jpg "alt=" Wkiol1fzebws8pweaabu8ybamce359.jpg "/>

In the software model above, if Compute and io units are partitioned by reasonable resource segmentation,and the Compute and io units are in the same execution context, so the efficiency of the entire software will become very high. In many cases, however, compute and storage can be two software modules, or compute and storage resources cannot be effectively partitioned, in which case the compute and storage threads need to communicate through Message Queuing. This message communication can be abstracted into a producer - consumer model. Compute units are producers;IO units are consumers.

producers - consumers need to synchronize during the implementation process. If a "producer-consumer" queue is implemented with a traditional locking mechanism , the performance bottleneck of the storage system will be shifted to the lock competition point. Lock competition on the one hand, the processor is in a long wait or sleep scheduling state, on the other hand, due to a large number of cache invalid operation makes the CPU memory efficiency greatly reduced. In general, lock contention will lead to a reduction in CPU usage efficiency, so that the processor's IPC metrics become very low.

So in the software model above, how to improve processor efficiency, give full play to the NVMe ssd performance? This requires consideration of the use of a lock-free design. In the multi-threaded concurrent processing scenario, it is unavoidable to have data interaction between threads, in order to improve the efficiency of the cpu cas Dummy node, so there is a problem of dynamically allocating memory. To avoid dynamic memory allocation, you can use a fixed-length, lock-free queue lock-free ring io number of flow control. Therefore, according to the characteristics of the storage system, it is necessary to design a lock-free algorithm that conforms to its own characteristics. In a recent storage project, we created a lock-free producer - consumer and lock-free flow control algorithm, using this algorithm, we can find that io cpu The more cores, io performance higher. Is the performance of the actual test with the io The graph of the number of threads:

650) this.width=650; "src=" http://s2.51cto.com/wyfs02/M02/88/5F/wKiom1fzEbawst9tAABhZsobkbY774.jpg "style=" float: none; "title=" 2.jpg "alt=" Wkiom1fzebawst9taabhzsobkby774.jpg "/>

650) this.width=650; "src=" http://s2.51cto.com/wyfs02/M01/88/5B/wKioL1fzEbaR5LLSAABeOlUoD0c274.jpg "style=" float: none; "title=" 3.jpg "alt=" Wkiol1fzebar5llsaabeoluod0c274.jpg "/>

we can see that when After the number of IO threads has been increased, the random read performance has not changed significantly, and some performance degradation has occurred. On this test platform, there is a total of four CPU cores , through Hyper-threading way, to achieve a maximum of CPU cores. From the experimental results above, hyper-threading contributes little to IO performance. Random Read performance peaks after the number of IO Thread reaches 8 . This is due to the peak performance of the back-end NVMe SSD disk.

NVMe Storage System design is facing the problem of CPU performance bottleneck, how to improve CPU 's IO processing efficiency is a problem that must be considered in high performance storage system design. The lock-free design is an effective means to improve the CPU efficiency, and a good lock-free design needs to be considered with the system software design as a whole.

This article is from the "Save the Way" blog, make sure to keep this source http://alanwu.blog.51cto.com/3652632/1858675

Talking about the lock-free design of storage software