Python uses struct to process binary (pack and unpack usage)

Sometimes it is necessary to use Python to process binary data, for example, access to files, socket operation. At this point, you can use the Python struct module to do this. Structs can be used to manipulate structures in C.

The three most important functions in a struct module are pack (), unpack (), calcsize ()

Pack (FMT, V1, v2, ...) encapsulates data in a given format (FMT) as a string (actually a byte stream similar to a C struct)

Unpack (FMT, String) parses a byte stream string in the given format (FMT), returning the parsed tuple

CalcSize (FMT) calculates how many bytes of memory a given format (FMT) occupies

The supported formats in a struct are the following table:

Format C Type Python byte count

x Pad Byte No value 1

c Char string of length 1 1

b signed Char integer 1

B unsigned char integer 1

? _bool Bool 1

H Short Integer 2

H unsigned short integer 2

I int integer 4

I unsigned int integer or Long 4

L Long Integer 4

L unsigned long Long 4

Q Long Long 8

Q unsigned long long 8

F Float Float 4

D Double float 8

S char[] string 1

P char[] String 1

P void * Long

Note 1.Q and q are only interesting when the machine supports 64-bit operation

Note 2. You can have a number in front of each format, indicating the number of

Note the 3.S format represents a length of string, and 4s represents a string of length 4, but p represents a Pascal string

Note 4. P is used to convert a pointer whose length is related to the machine word size

Note 5. The last one can be used to represent a pointer type, accounting for 4 bytes

In order to exchange data with structs in C, it is also necessary to consider that some C or C + + compilers use byte alignment, usually 32-bit systems in 4 bytes, and therefore structs are converted according to the local machine byte order. You can change the alignment by using the first character in the format. defined as follows:

Character Byte order Size and alignment

@ Native Native enough 4 bytes

= Native Standard by original number of bytes

< Little-endian standard by original number of bytes

> Big-endian standard by original number of bytes

! Network (= Big-endian)

Standard by original number of bytes

The use method is placed in the first position of the FMT, just like ' @5s6sif '

Example 1:

The structure is as follows:

struct header{    unsigned short id;    CHAR[4] tag;    unsigned int version;    unsigned int count;}

Through SOCKET.RECV received a structure of the above data, the existence of the string s, now need to parse it out, you can use the unpack () function:

Import Structid, tag, version, Count = Struct.unpack ("! H4s2i ", s)

In the format string above,! Indicates that we want to use network byte order resolution because our data is received from the network, and it is the network byte order when it is transmitted over the network. The following H represents a unsigned short id,4s that represents a 4-byte long string, 2I indicates that there are two unsigned int types of data.

Through a unpack, now ID, tag, version, Count has saved our information.

Also, it is convenient to pack local data into a struct format:

SS = Struct.pack ("! H4s2i ", ID, tag, version, count);

The pack function converts the ID, tag, version, and count to the struct in the specified format Header,ss is now a string (actually a byte stream similar to the C struct) that can be sent out by Socket.send (ss).

Example 2:

Import structa=12.34# changes A to binary bytes=struct.pack (' I ', a)

At this point, Bytes is a string literal, which is the same as the binary storage of a byte in bytes.

Again, the existing binary data bytes, which is actually a string, translates it into a Python data type:

#注意, unpack returns a tuple!!

A,=struct.unpack (' i ', bytes)

If it is composed of multiple data, you can:

A= ' Hello ' b= ' world! ' C=2d=45.123bytes=struct.pack (' 5s6sif ', a,b,c,d)

At this point the bytes is the binary form of the data, you can write directly to the file such as Binfile.write (bytes)

Then, when we need to, we can read it again, Bytes=binfile.read ()

Then decode the python variable by struct.unpack ():

A,b,c,d=struct.unpack (' 5s6sif ', bytes)

' 5s6sif ' is called FMT, which is a formatted string, consisting of numbers plus characters, 5s representing a 5-character string, 2i, representing 2 integers, and so on, the following are the available characters and types, and the CType representation can correspond to type one by one in Python.

Note: Problems encountered while processing binary files

When we work with binary files, we need to use the following methods:

Binfile=open (filepath, ' RB ')    #读二进制文件binfile =open (filepath, ' WB ')   #写二进制文件

So what's the difference between the results and Binfile=open (filepath, ' R ')?

The difference is two places:

First, if you touch ' 0x1A ' when using ' R ', it will be considered as the end of the file, which is EOF. There is no problem with ' RB '. That is, if you use binary writing to read the text again, if there is ' 0X1A ' in it, only a portion of the file will be read. Using ' RB ' will always read the end of the file.

Second, for the string x= ' abc\ndef ', we can use Len (x) to get its length to 7,\n what we call a newline character, which is actually ' 0X0A '. When we write with ' W ' as text, the ' 0X0A ' is automatically changed to two characters ' 0X0D ', ' 0X0A ', that is, the length of the file actually becomes 8 in the Windows platform. When read with the ' R ' text, it is automatically converted to the original newline character. If you replace it with a ' WB ' binary, it will keep one character intact and read as is. So if you write it in text and read it in binary mode, consider the extra byte. ' 0X0D ' is also called carriage return. Linux does not change. Because Linux uses only ' 0X0A ' to represent line breaks.

