Analysis of the struct module in Python

Recently in the study of Python network programming this piece, in writing a simple socket communication code, encountered the use of the struct module, was not very clear at that time this has and effect, and then consulted about the relevant information probably understand, here to do a brief summary.

People who know the C language must know the role of struct structure in C, which defines a structure that contains different types of data (Int,char,bool, etc.) to facilitate the processing of a structure object. In the network communication, most of the data transmitted is the binary stream (binary data) exists. When passing strings, you don't have to worry about too many problems, and when you pass basic data such as int, char, you need a mechanism to package certain struct types into a binary stream string and then transfer it over the network. The receiving end should also be able to unpack the original structure data by some mechanism. The struct module in Python provides a mechanism for converting a Python primitive type value to a C struct type represented in a Python string format (this module performs conversions Between Python values and C structs represented as Python strings.). The Stuct module provides a very simple number of functions, which are written in a few examples below.

1. Basic Pack and unpack

The struct provides packaging and unpacking of data in format specifier (Packing and unpacking). For example:

123456789101112

importstructimportbinasciivalues =(1, ‘abc‘, 2.7)s =struct.Struct(‘I3sf‘)packed_data =s.pack(*values)unpacked_data =s.unpack(packed_data)print‘Original values:‘, valuesprint‘Format string :‘, s.formatprint ‘Uses :‘, s.size, ‘bytes‘print‘Packed Value :‘, binascii.hexlify(packed_data)print‘Unpacked Type :‘, type(unpacked_data), ‘ Value:‘, unpacked_data

Output:

Original values: (1, ' abc ', 2.7)
Format STRING:I3SF
Uses:12 bytes
Packed VALUE:0100000061626300CDCC2C40
Unpacked Type: <type ' tuple ' > Value: (1, ' abc ', 2.700000047683716)

Code, first defines a tuple of data, containing int, string, float three data types, and then defines the struct object, and has the format ' i3sf ', I means int,3s represents three character length of the string, F for float. Finally, packaging and unpacking is done through the pack and unpack of the struct. Through the output can be found that the value of the pack after the conversion to a binary byte string, and unpack can convert the byte string back to a tuple, but it is worth noting that the accuracy of float has changed, which is determined by some such as the operating system and other objective factors. The number of bytes that are consumed after packaging is very similar to the struct in C. Define format to refer to the comparison table provided by the official API:

2. Byte order

On the other hand, the post-packaged byte order is determined by the operating system by default, and of course the struct module also provides a custom byte order function that specifies the specific byte order of big-endian storage, small-end storage, and so on, which is important for the byte order of the underlying communication. Different byte order and storage methods also result in different byte sizes. Adding a specific symbol in front of the format string can represent different ways of storing bytes sequentially, for example, using small-end storage s = struct. The Struct ('<I3SF ') is available. The Official API library also provides a corresponding control list:

3. Using buffer, use Pack_into and Unpack_from methods

The scenarios in which binary packaging data are used are mostly performance-demanding environments. And the pack method mentioned above is to re-create a memory space for return after the input data, that is, each time we pack in memory allocated the corresponding memory resources, which is sometimes a great performance waste. The struct module also provides methods for Pack_into () and Unpack_from () to solve the problem of populating a byte with a buffer that has been allocated well in advance, instead of producing a new object to store the bytes each time.

123456789101112

importstructimportbinasciiimportctypesvalues =(1, ‘abc‘, 2.7)s =struct.Struct(‘I3sf‘)prebuffer =ctypes.create_string_buffer(s.size)print‘Before :‘,binascii.hexlify(prebuffer)s.pack_into(prebuffer,0,*values)print‘After pack:‘,binascii.hexlify(prebuffer)unpacked =s.unpack_from(prebuffer,0)print‘After unpack:‘,unpacked

Output:

before:000000000000000000000000
After PACK:0100000061626300CDCC2C40
After unpack: (1, ' abc ', 2.700000047683716)
In contrast to packaging using the pack method, the Pack_into method has been operating on the Prebuffer object without generating unnecessary memory waste. It is also important to note that both the Pack_into and Unpack_from methods operate on a string buffer object and provide the offset parameter, allowing the user to make the appropriate processing more flexible by specifying the corresponding offset. For example, we can pack multiple objects into a buffer and then unpack by specifying different offset:

12345678910111213141516

importstructimportbinasciiimportctypesvalues1 =(1, ‘abc‘, 2.7)values2 =(‘defg‘,101)s1 =struct.Struct(‘I3sf‘)s2 = struct.Struct(‘4sI‘)prebuffer =ctypes.create_string_buffer(s1.size+s2.size)print‘Before :‘,binascii.hexlify(prebuffer)s1.pack_into(prebuffer,0,*values1)s2.pack_into(prebuffer,s1.size,*values2)print‘After pack:‘,binascii.hexlify(prebuffer)prints1.unpack_from(prebuffer,0)prints2.unpack_from(prebuffer,s1.size)

Output:

before:0000000000000000000000000000000000000000
After pack:0100000061626300cdcc2c406465666765000000
(1, ' abc ', 2.700000047683716)
(' DEFG ', 101)

Analysis of the struct module in Python