Serialization and compression of python

When to use serialization?

For example, I have a very complex data structure (similar to a dictionary, key and value are basic objects of python) and I want to store them in a database. What are you going to do, split one item and store it separately? Answer: no.

Now we can use serialization to directly convert the data structure into byte strings, which are stored in the database. Directly retrieve thisByte stringTo restore the string.

Common modules are marshal and cPickle.

They all use the dump, dumps, load, and loads methods. According to some professional tests, the speed of marshal is faster than that of cPickle. cPickle is written by C, which is also super fast. If a long string exists, cPickle is faster.

At the same time, cPickle can be used with gzip to compress the converted data to the shortest byte string, saving space and storage.

 

Import cPickle, gzip
Def save (filename, * objects ):
Fil = gzip. open (filename, 'wb ')
For obj in objects:
CPickle. dump (obj, fil, proto = 2)
Fil. close ()
Def load (filename ):
Fil = gzip. open (filename, 'rb ')
While True:
Try:
Yeild cPickle. load (fil)
Failed t EOFError: break
Fil. close ()

For example, to process the list of 45000 strings, after cPickle uses protocol 0 in dump, the size of the disk file generated is 972KB, and Protocol 2 only needs kb. If both gzip and Protocol 2 are used, the generated file is 268KB, but the compression of Protocol 0 is actually the best. After testing, the files generated by Protocol 0 and gzip are 252KB.