GF Frame gbinary-binary data manipulation module

This is a creation in Article, where the information may have evolved or changed.

Article Source: http://gf.johng.cn/500342

The GF Framework provides a stand-alone binary data manipulation package gbinary, which is used primarily for conversion between various data types and []byte binary types, and for precise bitwise processing of integer data. It is commonly used for data encoding/decoding during network communication, and for encoding/decoding during data file operation.

The list of methods used for binary data structure conversion processing is as follows:

Func Encode (vs. ... interface{}) ([]byte, error) func encodeint (i int) []bytefunc EncodeInt8 (i int8) []bytefunc EncodeInt16 ( I int16) []bytefunc EncodeInt32 (i int32) []bytefunc EncodeInt64 (i int64) []bytefunc encodeuint (i uint) []bytefunc Encodeui Nt8 (i uint8) []bytefunc EncodeUint16 (i uint16) []bytefunc EncodeUint32 (i uint32) []bytefunc EncodeUint64 (i uint64) []byte Func Encodebool (b bool) []bytefunc EncodeFloat32 (f float32) []bytefunc EncodeFloat64 (f float64) []bytefunc encodestring ( s string) []bytefunc Decode (b []byte, vs ... interface{}) errorfunc Decodetoint (b []byte) IntFunc DecodeToInt8 (b []byte) in T8func DecodeToInt16 (b []byte) Int16func DecodeToInt32 (b []byte) Int32func DecodeToInt64 (b []byte) Int64func Decodetouint (b []byte) Uintfunc DecodeToUint8 (b []byte) Uint8func DecodeToUint16 (b []byte) Uint16func DecodeToUint32 (b [ ]byte) Uint32func DecodeToUint64 (b []byte) Uint64func Decodetobool (b []byte) Boolfunc DecodeToFloat32 (b []byte) Float32func DecodeToFloat64 (b []byte) Float64func DecOdetostring (b []byte) string 

The list of methods that support bitwise processing is as follows:

func EncodeBits(bits []Bit, i int, l int) []Bitfunc EncodeBitsWithUint(bits []Bit, ui uint, l int) []Bitfunc EncodeBitsToBytes(bits []Bit) []bytefunc DecodeBits(bits []Bit) uintfunc DecodeBitsToUint(bits []Bit) uintfunc DecodeBytesToBits(bs []byte) []Bit

The bit type represents a binary number (0 or 1), which is defined as follows:

type Bit int8

Binary Operation Example

Let's take a look at a more complete binary operation example, which basically demonstrates the vast majority of binary conversion operations.

Gitee.com/johng/gf/blob/master/geg/encoding/gbinary/binary.go

Package Mainimport ("FMT" "Gitee.com/johng/gf/g/os/glog" "Gitee.com/johng/gf/g/encoding/gbinary") func main () { Use Gbinary. Encoded binary packaging of the base data type if buffer, err: = Gbinary. Encode (18, 300, 1.01); Err! = Nil {glog. Error (Err)} else {fmt. PRINTLN (buffer)}//Use Gbinary. Decode to the shaping binary unpacking, note that the second and subsequent parameters are the pointer address of the shaping variable determined by the word length, the type of word length determination,///For example: INT8/16/32/64, UINT8/16/32/64, FLOAT32/64// Here the 1.01 defaults to the Float64 type (under 64-bit system) if buffer, err: = Gbinary. Encode (18, 300, 1.01); Err! = Nil {glog. Error (Err)} else {var i1 int8 var i2 int16 var f3 float64 If Err: = Gbinary. Decode (buffer, &I1, &I2, &F3); Err! = Nil {glog. Error (Err)} else {fmt. Println (I1, I2, F3)}}//encode/parse int, automatically identify variable length FMT. Println (gbinary. Decodetoint (gbinary. Encodeint (1))) FMT. Println (gbinary. Decodetoint (gbinary. Encodeint ()) fmt. Println (gbinary. Decodetoint (gbinary. Encodeint (70000)) FMT. Println (GbinarY.decodetoint (gbinary. Encodeint (2000000000)) fmt. Println (gbinary. Decodetoint (gbinary. Encodeint (500000000000)))//encode/parse uint, automatically identify variable length FMT. Println (gbinary. Decodetouint (gbinary. Encodeuint (1))) FMT. Println (gbinary. Decodetouint (gbinary. Encodeuint ()) fmt. Println (gbinary. Decodetouint (gbinary. Encodeuint (70000)) FMT. Println (gbinary. Decodetouint (gbinary. Encodeuint (2000000000)) fmt. Println (gbinary. Decodetouint (gbinary. Encodeuint (500000000000)))//encode/parse INT8/16/32/64 FMT. Println (gbinary. DecodeToInt8 (gbinary. EncodeInt8 (int8 ())) Fmt. Println (gbinary. DecodeToInt16 (gbinary. EncodeInt16 (Int16 ())) Fmt. Println (gbinary. DecodeToInt32 (gbinary. EncodeInt32 (Int32 ())) Fmt. Println (gbinary. DecodeToInt64 (gbinary. EncodeInt64 (Int64 (100)))//encode/parse UINT8/16/32/64 FMT. Println (gbinary. DecodeToUint8 (gbinary. EncodeUint8 (Uint8 ())) Fmt. Println (gbinary. DecodeToUint16 (gbinary. EncodeUint16 (UInt16 ())) Fmt. Println (gbinary. DecodeToUint32 (gbinary. EncodeuiNt32 (UInt32 ())) Fmt. Println (gbinary. DecodeToUint64 (gbinary. EncodeUint64 (UInt64 (100)))//encode/parse string FMT. Println (gbinary. Decodetostring (gbinary. Encodestring ("I ' M string!"))}

The results of the above program execution are:

[18 44 1 41 92 143 194 245 40 240 63]18 300 1.0113007000020000000005000000000001300700002000000000500000000000100100100100100100100100I'm string!

1. Coding
   Gbinary. The Encode method is a very powerful and flexible way to convert all the basic types into binary types ([]byte). In Gbinary. Within the Encode method, the variable is automatically evaluated for length, with the minimum binary length to hold the binary value of the variable. For example, for a variable of type int with a value of 1, gbinary. The encode will only be stored with 1 bytes, and a variable with an int value of 300 will be stored using 2 bytes, minimizing the storage space for binary results. Therefore, in the resolution of the time to pay great attention to []byte length, it is recommended to determine the length of the variable, in the binary encoding/decoding, as far as possible to use the fixed-length basic type, such as INT8/16/32/64 to store variables, so that the resolution can also be used in the corresponding variable form to parse, Errors are not easy to produce.
   The Gbinary package also offers a range of gbinary. Encode* method that converts the base data type to binary. Among them, Gbinary. Encodeint/gbinary. Encodeuint also automatically recognizes the variable value size internally, returning an indeterminate length of []byte value, length range 1/2/4/8.
2. Decoding
   In binary type parsing operations, the length of the binary ([]byte length) is very important, only given the correct length to perform the correct parsing, so gbinary. The variable length given by the Decode method must be a variable of the length type, for example: INT8/16/32/64, UINT8/16/32/64, FLOAT32/64, and if the variable type of the given second variable address corresponds to Int/uint, the length cannot be determined, Therefore, the parsing will fail.
   In addition, the Gbinary package also offers a range of gbinary. A decodeto* method that converts a binary to a specific data type. Among them, Gbinary. Decodetoint/gbinary. The Decodetouint method automatically identifies the binary length, and the supported binary parameter length range is 1-8.

Bitwise operations Processing Examples

Gbinary BITS-related operations simplify the complexity of the underlying bits operations, providing the possibility for accurate data bitwise processing.

1. Batch sensor state Data escalation example
   For example, for IoT projects, sensor devices are a relatively common hardware device, and the following example shows the status information that the gateway reports to the platform about the sensors it manages. Since there are only 4 sensor states (0: offline, 1: On, 2: OFF, 3: Standby), it corresponds to 2 bits (2 bit), so 1byte (8 bit) can represent the status of 4 sensor devices. Look at the following example, to report the platform 100 sensors are turned on, the state order of escalation is the sensor in the Gateway Port sequence (index starting from 0):

   Gitee.com/johng/gf/blob/master/geg/encoding/gbinary/bits1.go

   package mainimport (    "fmt"    "gitee.com/johng/gf/g/encoding/gbinary")func main() {    // 传感器状态，0:已下线, 1:开启, 2:关闭， 3:待机    count  := 100    status := 1    // 网关编码    bits := make([]gbinary.Bit, 0)    for i := 0; i < count; i++ {        bits = gbinary.EncodeBits(bits, uint(status), 2)    }    buffer := gbinary.EncodeBitsToBytes(bits)    fmt.Println("buffer length:", len(buffer))    /* 上报过程忽略，这里只展示编码/解码示例 */    // 平台解码    alivecount := 0    sensorbits := gbinary.DecodeBytesToBits(buffer)    for i := 0; i < len(sensorbits); i += 2 {        if gbinary.DecodeBits(sensorbits[i:i+2]) == 1 {            alivecount++        }    }    fmt.Println("alived sensor:", alivecount)}

   After execution, the output is:

   buffer length: 25alived sensor: 100

   As you can see, it only takes 25byte to escalate the status data of 100 sensors, and in this example we decode the platform and count the number of sensors that have been opened to 100.

2. GKVDB database meta Data structure Operation example
   Let's take a look at a practical example. Gkvdb is a high performance Key-value embedded database based on the DRH algorithm developed by the same author of GF framework, where the stored data structure of metadata is as follows:

   [键名哈希64(64bit,8byte) 键名长度(8bit,1byte) 键值长度(24bit,3byte) 数据文件偏移量(40bit,5byte)](变长)

   We use a single piece of metadata to demonstrate the encoding/decoding operation.

   Gitee.com/johng/gf/blob/master/geg/encoding/gbinary/bits2.go

   package mainimport ("FMT" "Gitee.com/johng/gf/g/encoding/gbinary") func main () {//Meta metadata file data node Structure: [Key name Hash (64bit,8byte) Key name Length (8bit,1byte) key value Length (24bit,3byte) data file offset (40bit,5byte)] (variable length) hash: = 521369841259754125 kle N: = Vlen: = 35535 Offset: = 80000000//encoded bits: = Make ([]gbinary. Bit, 0) bits = gbinary. Encodebits (Bits, hash, up) bits = gbinary. Encodebits (Bits, Klen, 8) bits = Gbinary. Encodebits (Bits, vlen,) bits = gbinary. Encodebits (Bits, offset,) buffer: = Gbinary. Encodebitstobytes (BITS) fmt. PRINTLN ("Meta length:", len (buffer))/* file storage and data query process ignored, here only show the metadata encoding/decoding example///decoding metabits: = Gbinary. Decodebytestobits (buffer) fmt. Println ("Hash:", gbinary. Decodebits (metabits[0:64]) fmt. Println ("Klen:", Gbinary. Decodebits (metabits[64:72]) fmt. Println ("Vlen:", Gbinary. Decodebits (metabits[72:96]) fmt. Println ("offset:", gbinary. Decodebits (metabits[96:136])} 

   After running, the output is:

   meta length: 17hash  : 521369841259754125klen  : 12vlen  : 35535offset: 80000000