Numerical compression storage method varint

When writing network communication, we often need to store some data in byte [] and then send it out. The value is the data member that we often process. The less I/O and bandwidth, the more important it is to compress the transmitted data. Next we will talk about a digital storage-based method that can save the value storage space in most cases.

Varint is a compact numeric representation method. It uses one or more bytes to represent a number. The smaller the value, the fewer bytes are used. This reduces the number of bytes used to indicate numbers. For example, for int32 numbers, four bytes are generally required. However, varint is used. For small int32 numbers, one byte can be used. Of course, everything is both good and bad. varint notation is used, and a large number is represented by five bytes. From the statistical point of view, generally, not all messages contain a large number of numbers. Therefore, in most cases, after varint is used, a smaller number of bytes can be used to represent numerical information. The following describes varint in detail.

The highest bit of each byte in varint has a special meaning. If this bit is 1, it indicates that the subsequent byte is also part of the number. If this bit is 0, it ends. The other 7 bits are used to represent numbers. Therefore, numbers smaller than 128 can be expressed in a byte. A number greater than 128, such as 300, is expressed in two bytes: 1010 1100 0000 0010

Because the negative number is 1 at a high position, when using this compression process, the negative number must be converted to a positive number.CodeConverts int to uint.

Private Static int zag (uint ziggedvalue) {int value = (INT) ziggedvalue; Return (-(Value & 0x01) ^ (value> 1 )&~ (1 <31);} Private Static uint Zig (INT value) {return (uint) (value <1) ^ (value> 31 ));}
 

Perform the following operations to encode a uint:

 
Private Static arraysegment <byte> writeuint32variant (uint value) {byte [] DATA = new byte [5]; int COUNT = 0; do {data [count] = (byte) (Value & 0x7f) | 0x80); count ++;} while (value >>= 7 )! = 0); Data [count-1] & = 0x7f; return New arraysegment <byte> (data, 0, count );}
 

Data [count] = (byte) (Value & 0x7f) | 0x80); returns the first 7 digits. | 0x80 indicates that the following bytes are also part of the number.

While (value >>= 7 )! = 0) if the right shift is 7 digits, if it is not zero, the above work will continue.

Data [count-1] & = 0x7f sets the highest bit of the last byte to 0;

The next step is the uint decoding process.

Private Static uint readuint32variant (arraysegment <byte> data) {uint value = data. array [0]; If (Value & 0x80) = 0) return value; Value & = 0x7f; uint chunk = data. array [1]; value | = (chunk & 0x7f) <7; If (chunk & 0x80) = 0) return value; chunk = data. array [2]; value | = (chunk & 0x7f) <14; If (chunk & 0x80) = 0) return value; chunk = data. array [3]; value | = (chunk & 0x7f) <21; If (chunk & 0x80) = 0) return value; chunk = data. array [4]; value | = chunk <28; If (chunk & 0xf0) = 0) return value; throw new overflowexception ("readuint32variant error! ");}
 

(Value & 0x80) = 0 indicates that the maximum bit is 0, indicating that the byte is no longer a part of the value.

(Chunk & 0xf0) = 0 chunk has only four digits. If not, this byte is not part of the numerical storage.

Test the encoding effect.

 
Arraysegment <byte> DATA = writeuint32variant (Zig (0); console. writeline (data. count); Data = writeuint32variant (Zig (567); console. writeline (data. count); Data = writeuint32variant (Zig (10000); console. writeline (data. count); Data = writeuint32variant (Zig (-100000); console. writeline (data. count );
 

1 byte, 2 byte, 3 byte, 3 byte

In fact, some people may ask why int16 is not used for storage based on the situation. If int16 is used, it means that it is very troublesome to convert int32.ProgramAll need to be adjusted. If varint is used for processing, the best scalability and bandwidth utilization can be achieved.