Pickle and cPickle: serialization of Python objects, picklecpickle

Pickle and cPickle: serialization of Python objects, picklecpickle

Objective: to serialize a Python object
Availability: pickle must be at least version 1.4 and cPickle version 1.5 or later.

pickleThe module implements an algorithm to convert any Python object into a series of bytes (byets ). This process is also calledserializingObject. It indicates that the object's byte stream can be transmitted or stored. After reconstruction, a new object with the same features (the same characteristics) will be created.

cPickleThe same algorithm is implemented using C instead of Python. This is several times faster than Python implementation, but it does not allow users to derive subclass from Pickle. CPickle is a better choice if subclass is irrelevant to your use.

Warning this document describes that pickle does not provide security assurance. Be careful if you use pickle in multi-thread communication or data storage. Do not trust data that cannot be identified as secure.

Import

As usual, try to import cPickle and assign it an alias "pickle ". If the Import fails for some reason, the next step is to implement the pickle module in native Python. If cPickle is available, it can provide you with a faster execution; otherwise, it can only be a lightweight execution (the portable implementation ).

try:   import cPickle as pickleexcept:   import pickle

Encoding and decoding

In the first example, encode a data structure into a string and print the string to the console. Use a data structure that contains all native types. All types of instances can be pickled. Usepickle.dumps()Create a string that represents the object value.

try:    import cPickle as pickleexcept:    import pickleimport pprintdata = [ { 'a':'A', 'b':2, 'c':3.0 } ]print 'DATA:',pprint.pprint(data)data_string = pickle.dumps(data)print 'PICKLE:', data_string

By default, pickle only consists of ASCII characters. You can also use a more efficient binary format, because it is easier to understand during printing. All examples on this page use ASCII output.

$ python pickle_string.pyDATA:[{'a': 'A', 'b': 2, 'c': 3.0}]PICKLE: (lp1(dp2S'a'S'A'sS'c'F3sS'b'I2sa.

After data is serialized, you can write them into files, sockets, pipelines, and so on. You can also read from the file and unpickled it to construct a new object with the same value.

try:    import cPickle as pickleexcept:    import pickleimport pprintdata1 = [ { 'a':'A', 'b':2, 'c':3.0 } ]print 'BEFORE:',pprint.pprint(data1)data1_string = pickle.dumps(data1)data2 = pickle.loads(data1_string)print 'AFTER:',pprint.pprint(data2)print 'SAME?:', (data1 is data2)print 'EQUAL?:', (data1 == data2)

As you can see, the newly constructed object is the same as the original object, but not the same object. This is not surprising.

$ python pickle_unpickle.pyBEFORE:[{'a': 'A', 'b': 2, 'c': 3.0}]AFTER:[{'a': 'A', 'b': 2, 'c': 3.0}]SAME?: FalseEQUAL?: True

Work with stream

Divisiondumps()Andloads()In addition, pickle also provides a pair of conversion functions used in file-like streams. You can write objects to a stream and read them from the stream. In this process, there is no need to write a few objects in advance and how large they are.

Try: import cPickle as pickleexcept: import pickleimport pprintfrom StringIO import StringIOclass SimpleObject (object): def _ init _ (self, name): self. name = name l = list (name) l. reverse () self. name_backwards = ''. join (l) returndata = [] data. append (SimpleObject ('pickle') data. append (SimpleObject ('cpickle') data. append (SimpleObject ('last') # Use StringIO to simulate a file out_s = StringIO () # Write the stream for o in data: print 'writing: % s (% s) '% (o. name, o. name_backwards) pickle. dump (o, out_s) out_s.flush () # create a readable stream in_s = StringIO (out_s.getvalue () # Read data while True: try: o = pickle. load (in_s) failed t EOFError: break else: print 'read: % s (% s) '% (o. name, o. name_backwards)

In this example, the SringIO buffer is used to simulate a stream. Therefore, when a readable stream is created, we play with it. You can also use pickles to store objects for formatting in a simple database,shelveIt is easier to work.

$ python pickle_stream.pyWRITING: pickle (elkcip)WRITING: cPickle (elkciPc)WRITING: last (tsal)READ: pickle (elkcip)READ: cPickle (elkciPc)READ: last (tsal)

In addition to data storage, pickles is also very popular in inter-process communication. For exampleos.fork()Andos.pipe()Worker processes can be created to read job instructions from one pipeline (pipe) and then write the results to another pipeline. The worker pool and core code that sends and receives responses to jobs can be reused because jobs and responses do not belong to a specific class. If you use an MTS queue or a socket (sockets), do not forget to flush it after dumping all objects and pushing data by connecting to the end ). If you want to write your own worker pool manager, seemultiprocessing.