Simulates the cyclic caching implemented by the Linux kernel Kfifo

Http://www.cnblogs.com/wangguchangqing/p/6070286.html

Want to implement a loop buffer (Circular buffer), search for some information is based on the implementation of circular queue. Use a variable to hold the length of the data in the buffer or empty a space to determine if the buffer is full. It is really ingenious to see the implementation of the cyclic buffer queue for analyzing the Linux kernel by chance kfifo . kfifoThe main features are as follows:

• The size of the buffer space is guaranteed to be a power of 2, not an upward rounding of 2.
• The position of the input (in) and the output (out) is saved with an unsigned integer, and the values of in and out are not modulo in the input and output, which naturally overflows, and can guarantee that in-out the result is the length of the data stored in the buffer, which is the best embodiment kfifo of the implementation technique;
• Use memory Barrier technology to achieve a single consumer and single-producer-to kfifo -non-lock concurrent access, multiple consumers, producers of concurrent access or need to lock.

The main three parts of this paper are as follows:

• With regard to the power of 2, it is judged that the recurses of 2 and the power of rounding up to 2.
• kfifoimplementation and brief analysis of Linux kernel
• Based on kfifo the implemented loop buffer, and perform some tests

The memory barrier of this article does not make too much analysis, you can refer to Wikimemory Barrier. In addition, the integers involved in this article are defaulted to unsigned integers, and no one by one instructions are made.

1. Power of 2

• Determine if a number is a power of 2
  kfifoTo ensure that the size of its cache space is a power of 2, if it is not then rounded up to 2 of the power of the second. The method of judging the power of 2 is also very ingenious. If an integer n is a power of 2, then the binary mode must be, whereas the binary mode of 1000... n-1 is 0111... , that is, each bits of N and n-1 are different, for example: 8 (1000) and 7 (0111); n is not a power of 2, Then the binary of N and N-1 must have the same bit of 1, for example: 7 (0111) and 6 (0110). This can be based on n & (n-1) the results to determine whether the integer n is the power of 2, the implementation of the following:

  /*判断n是否是2的幂若n为2的次幂,   则 n & (n-1) == 0，也就是n和n-1的各个位都不相同。例如 8(1000)和7(0111）若n不是2的次幂, 则 n & (n-1) != 0,也就是n和n-1的各个位肯定有相同的，例如7(0111)和6(0110)*/static inline bool is_power_of_2(uint32_t n){return (n != 0 && ((n & (n - 1)) == 0));}

• Take a number up to the power of 2
  If the set buffer size is not a power of 2, the rounded up is a power of 2, for example: set to 5, then up to 8. The above mentioned that the integer n is the power of 2, then the second binary mode is 100... , so if the positive k is not the power of N, just find its highest significant bit 1 location (starting from 1) POS, and then 1 << pos you can take k up to the power of 2. The implementation is as follows:

  static inline uint32_t roundup_power_of_2(uint32_t a){if (a == 0) return 0;uint32_t position = 0;for (int i = a; i != 0; i >>= 1) position++;return static_cast<uint32_t>(1 << position);}

2. Linux implementation Kfifo and analysis

The implementation techniques in the Linux kernel are kfifo mainly focused on the method of putting the data put and the method of fetching the data get . The code is as follows:

Unsignedint __kfifo_put (struct Kfifo *fifo,UnsignedChar *buffer,Unsignedint len) {Unsignedint l; len = min (len, fifo->size-fifo->In + fifo->Out);/* * Ensure that we are sample the Fifo->out index-before-we * Start putting bytes into the Kfifo. */SMP_MB ();/* First put the data starting from Fifo->in to buffer end */L = min (len, fifo->size-(fifo->In & (Fifo->size-1))); memcpy (Fifo->buffer + (fifo->In & (Fifo->size-1)), buffer, L);/* Then put the rest (if any) at the beginning of the buffer */memcpy (fifo->buffer, buffer + L, len-l);/* * Ensure that we add the bytes to the kfifo-before-* We update the fifo->in index. */SMP_WMB (); Fifo->In + = Len;return Len; }Unsignedint __kfifo_get (struct Kfifo *fifo,UnsignedChar *buffer,Unsignedint len) {Unsignedint l; len = min (len, fifo->In-fifo->out); /* * Ensure that we sample the Fifo->in index-before-we * Start removing bytes from the KF Ifo. */SMP_RMB (); /* first get the data from Fifo->out until the end of the buffer */L = min (len, fifo->si Ze-(fifo->out & (Fifo->size-1)); memcpy (buffer, Fifo->buffer + (Fifo->out & (Fifo->size-1)), L); Span class= "Hljs-comment" >/* then get the rest (if any) from the beginning of the buffer */memcpy (buffer + L, Fifo->bu Ffer, len-l); /* * Ensure that we remove the bytes from the kfifo-before-* We update the fifo->out index . */SMP_MB (); Fifo->out + + len; return Len; 

putReturns the length of the data actually saved to the buffer, get returning the actual length of the data taken. In the above code, you need to be aware of the two operations at the time of writing and fetching min . About kfifo The analysis, has a lot of information, can also refer to Ogle's ingenious Kfifo.

The KFIFO implemented by the Linux kernel has the following features:

• Using Memory barrier Memories Barrier
• When initializing buffer space, ensure that the size of the buffer is 2 times the power of
• Use an unsigned integer to hold in and out (the pointer to the input and output), and do not do a modulo operation when the data is put out, allowing it to overflow naturally.

Advantages:

1. Achieve lock-free concurrent access for single-consumer and single-producer. Multi-consumer and multi-producer time still need to lock.
2. Use and operations in & (size-1) instead of modulo operations

3. Do not do a modulo operation when updating the value of in or out, but let it overflow automatically . This should be the KFIFO to achieve the most fork, take advantage of the value after the overflow to participate in the operation, and can guarantee the correct results. The overflow operation guarantees the following points:

   • In-out is the length of the data in the buffer
   • Size-in + out for free space in buffer
   • in = = The buffer is empty when out
   • Size = = (in-out) When the buffer is full

3. Imitation kfifoThe implemented cyclic buffering

It mainly simulates the arithmetic of unsigned overflow, and does not use the memory barrier to realize the no-lock concurrent access of single producer and single consumer. The code for initializing and entering the output is as follows:

struct kfifo{uint8_t *buffer; uint32_tIn Input pointer uint32_tOut Output pointer uint32_tSize The buffer size, which must be2 Power Kfifo (uint32_t _size) {if (!is_power_of_2 (_size)) _size = Roundup_power_of_2 (_size); Buffer = new Uint8_t[_size];in =0;out =0;size = _size; }//Returns the data in the actual write buffer uint32_t put (const uint8_t *Data, uint32_t len) {//current buffer free space Len =Min (Len,Size-In +Out); CurrentIn position to the length of the end of buffer auto L =Min (Len,Size-(In & (Size-1))); First copy the data to the [End of In,buffer] memcpy (buffer + (In & (Size-1)),Data, L); Copy the remaining data (if any) to [start of buffer,...] memcpy (buffer,Data + L, len-l);In + = Len; Directly add, do not do modulo operation. When overflow, start again from the beginning of the bufferreturn Len; }//Returns the actual read data length uint32_t get (uint8_t *data, uint32_t len) The data length in the {//Buffer len = min (len, in -out );//read data from [Out,buffer end] to auto l = min (len, size-( Out & (size- 1)): memcpy (data, buffer + (out& (size- 1)), l);//from [buffer sta RT,...] Read Data memcpy (+ L, buffer, len-l); Out + = Len;//Direct Plus, good modulo operation. After overflow, restart the return Len from the beginning of the buffer;             

When initializing buffer space, verify that it is a size power of 2, and if not, take the rounding up to the power of 2. The following is an analysis of the processing of pointers and processes when putting out data in out , and how to ensure that the existing in - out data lengths are still in the buffer after overflow.

Detailed put and Get methods

When you put data into a buffer, you need two parameters: the data pointer to put and the length of the data data expected to be put len , the return value is the length of the data actually stored in the buffer (less than when there is not enough space in the buffer len ). Below is a detailed explanation of the put effect of each statement under.

• putThe first sentence in the function is len = min(len,size - in + out) to calculate the actual size of the data written to the buffer. If the data you want to write is len larger than the free space in the buffer size - in + out , only fills the buffer space.

Because it is a loop buffer, its free space has two parts: from in to the end of the buffer space->[in,buffer end] and the starting position of the buffer space to Out->[buffer start,out].

• auto l = min(len, size - (in & (size - 1)));This is the judgment [In,buffer end] Whether this part of the space is enough to write the data
• memcpy(buffer + (in & (size - 1)), data, l);Writes data to this part of the [In,buffer end] Space
• memcpy(buffer, data + l, len - l);If the data is not finished, write the data to [buffer start,out] This part of the space.
• in += lenUpdate in, do not do modulo operation, let it overflow naturally.

getSimilar to put, first of all to determine whether there is enough data to take out, the first time to fetch data from out to the end of buffer, if not enough from the beginning of the buffer to take, the last update out is not to do modulo operation, let it overflow. Look at the statement above put to explain, there is no more to say.

Unsigned overflow operation

kfifoThe reason for the simplicity of as follows is due to its in and out overflow operations. Here is an explanation of how the data length in the buffer is guaranteed to be still in the case of overflow in - out . First look at the diagram:

• Buffer is empty
  

• After put a heap of data
  

• Get a bunch of data after
  

• The length of the put's data is longer than in the end of the buffer, with a portion from the put to the start of the buffer
  

The analysis of the KFIFO from the Linux kernel data structure is kfifo also very detailed.

The first three cases can be clearly seen in the in - out buffer of the existing data length, but the last discovery in the run to the front of the out, this time in - out is not supposed to be negative, how can be data length? This is exactly kfifo the point, in and out are unsigned integers, then in < out is a in - out negative number, the negative numbers as unsigned, the value is still the length of the data in the buffer. This is basically consistent with the in-overflow situation, which is put together here to say.

Here, a 8-bit unsigned integer is used to hold in and out for easy overflow. This assumes that out = 100,in = 255,size = 256, as

/*--------------------------------------    |                  |   | |--------------------------------------out =100in =250 the data already in the buffer is:In-out =150, free space is: size-(in-out) = 106 PUT10 data into a buffer --------------------------------------| | | | --------------------------------------in out this time in + 10 = 260 overflow changed to in = 4; this is in-out = 4-100 = -96, still overflow -966 binary as ' 0xa0 ', convert it directly to signed number ' 0xa0 = 160 ', After the data before the put is 150,put10, the data in the buffer is just 160, just for overflow calculation results. */ 

The premise of this operation is that size must be a power of 2. If size = 257, then the above operation will not succeed.

Test example

The above descriptions are based on operation derivation, which is validated in conjunction with the code in this article.
The test code is as follows: Set the space size to 128,in and out to be 8-bit unsigned integers

IntMain(){uint8_t output[512] = {0};uint8_t data[256] = {0};for (int i =0; I <256; i++) Data[i] = i;KfifoFifo(128); Fifo.put (data,100); Fifo.get (Output,50); Fifo.put (data,30);Auto C = fifo.put (data +10,92);cout <<"Empty:" << fifo.isempty () <<Endlcout <<"Left Space:" << fifo.left () <<Endlcout <<"Length:" << fifo.length () <<Endluint8_t A = fifo.size-fifo.in + fifo.out;uint8_t B = fifo.in-fifo.out;cout <<"=======================================" <<Endl Fifo.get (Output,128);cout <<"Empty:" << fifo.isempty () <<Endlcout <<"Left Space:" << fifo.left () <<Endlcout <<"Length:" << fifo.length () <<Endlcout <<  "======================================" < < ENDL; fifo.put (output, 100); cout <<  "Empty:" << fifo.isempty () << ENDL; auto d = static_cast<uint8_t > (Fifo.left ()); auto e = static_cast<uint8_t > (Fifo.length ()); printf ( "left space:%d\n", D); printf ( "length:%d\n", e); GetChar (); return 0;}         

Execution Result:

• The first output is the state that fills the buffer
• The second output is the state in which the buffer is emptied
• The third is in overflow situation, specifically to look at:
  When the second output takes the buffer empty, in = out = 178. Next, put 100 data into the buffer, this time in += 100 overflow, overflow after in = 22. Look at the output: put before the buffer is empty, put100 data, the buffer free space is 28, the data length is 100, is correct.

Simulates the cyclic caching implemented by the Linux kernel Kfifo