Example of using data object persistence enclosure pickle in Python

In Python, you can use the Pickle module to convert an object to a file, save it on disk, and then read and restore it when needed. The specific usage is as follows:
Pickle is a common serialization tool in the Python library that allows you to export a memory object as a text or binary format as a string, or to write to a document. Later, you can revert from a character or document to a Memory object. The new version of Python was re-implemented in C, called Cpickle, with higher performance. The following code demonstrates the common interface usage of the Pickle Library, which is very simple:

Import Cpickle as pickle# dumps and loads# dump the memory object into a string, or load the string as a Memory object Def test_dumps_and_loads (): t = {' name ': [' v1 ', ' V2 '} print t o = pickle.dumps (t) print o print ' len o: ', Len (o) p = pickle.loads (o) Print P # about Highest_protocol parameters , pickle supports 3 kinds of protocol,0, 1, 2:# http://stackoverflow.com/questions/23582489/python-pickle-protocol-choice# 0:ascii protocol, compatible with older versions of python# 1:binary format, compatible with older versions of python# 2:binary format,python2.3, better support New-sytle classdef Test_ Dumps_and_loads_highest_protocol (): print ' Highest_protocol: ', pickle. Highest_protocol t = {' name ': [' v1 ', ' v2 ']} print t o = pickle.dumps (t, Pickle. Highest_protocol) print ' Len o: ', Len (o) p = pickle.loads (o) print p# new-style classdef test_new_sytle_class (): Clas  s TT (object): Def __init__ (self, ARG, **kwargs): Super (TT, self). __init__ () Self.arg = arg Self.kwargs = Kwargs def test (self): print self.arg print Self.kwargs # ASCII Protocol t = TT (' Test ', A=1, b=2) O1 = Pi Ckle.dumps (T) print O1 print ' O1 len: ', Len (O1) p = pickle.loads (O1) p.test () # Highest_protocol for New-style class support better, higher performance O2 = Pickle.dumps (t, Pickle. Highest_protocol) print ' O2 len: ', Len (O2) p = pickle.loads (O2) P.test () # Dump and load# serialize the memory object and dump it directly into the file or object that supports the file interface # for dump, you need to support the write interface, accept a string as input parameter, for example: stringio# for load, need to support read interface, accept int input parameter, support ReadLine interface, no input parameters, such as stringio# use file     , ASCII encoded DEF test_dump_and_load_with_file (): t = {' name ': [' v1 ', ' v2 ']} # ASCII format with open (' Test.txt ', ' W ') as FP: Pickle.dump (T, FP) with open (' Test.txt ', ' R ') as Fp:p = Pickle.load (FP) print p# using file, binary encoded DEF test_dump_and_l Oad_with_file_highest_protocol (): t = {' name ': [' v1 ', ' v2 ']} with open (' Test.bin ', ' WB ') as Fp:pickle.dump (T, FP, PI Ckle. Highest_protocol) with open (' Test.bin ', ' RB ') as Fp:p = Pickle.load (FP) print p# using Stringio, binary encoded def Test_dump_and _load_with_stringio (): import Stringio t = {' name ': [' v1 ', ' v2 ']} fp = Stringio.stringio () pickle.dump (t, FP, Pickle.H IgheSt_protocol) fp.seek (0) p = pickle.load (FP) print P fp.close () # Use a custom class # here to demonstrate the user-defined class, as long as the write, read, ReadLine interface is implemented, # can be used as a D      UMP, load file parameter Def test_dump_and_load_with_user_def_class (): Import Stringio class FF (object): Def __init__ (self): Self.buf = Stringio.stringio () def write (self, s): Self.buf.write (s) print ' Len: ', Len (s) def read (sel      F, N): Return Self.buf.read (n) def readline (self): return Self.buf.readline () def seek (self, POS, mod=0): Return Self.buf.seek (POS, MoD) def close (self): self.buf.close () fp = FF () t = {' name ': [' v1 ', ' v2 ']} pick Le.dump (t, FP, Pickle. Highest_protocol) fp.seek (0) p = pickle.load (FP) print P fp.close () # pickler/unpickler# Pickler (file, PROTOCOL). Dump (o BJ) is equivalent to Pickle.dump (obj, file[, Protocol]) # Unpickler (file). Load () equivalent to Pickle.load (file) # Pickler/unpickler encapsulation is better, It is convenient to replace FileDef Test_pickler_unpickler (): t = {' name ': [' v1 ', ' v2 ']} f = file (' Test.bin ', ' WB ') pick = pickle. Pickler (f, pickle. HIghest_protocol) pick.dump (t) f.close () F = file (' Test.bin ', ' rb ') Unpick = pickle. Unpickler (f) p = unpick.load () print P f.close ()

Pickle.dump (obj, file[, Protocol])
This is the method of persisting the object, with the meanings of the arguments:

• Obj: The persisted object to persist;
• File: An object that owns the write () method, and the write () method can receive a string as a parameter. This object can be a file object that is opened in write mode or an Stringio object, or other custom object that satisfies the condition.
• Protocol: This is an optional parameter, default is 0, if set to 1 or True, the persisted object is saved in a highly compressed binary format, otherwise it is saved in ASCII format.

How do I restore an object after it is persisted? The Pickle module also provides the appropriate method, as follows:

Pickle.load (file)
There is only one parameter file, which corresponds to the file parameter in the Dump method above. This file must be a read () method that can receive an integer argument and a ReadLine () method that does not receive any parameters, and the return value of both methods should be a string. This can be an open file object that is read, a Stringio object, or any other object that satisfies the criteria.

The following is a basic use case:

#-*-Coding:utf-8-*-import pickle# can also do this: # import Cpickle as Pickleobj = {"A": 1, "B": 2, "C": 3}# save obj persisted to file tmp.t XT Pickle.dump (obj, open ("Tmp.txt", "W")) # do something else ... # reads and restores obj objects from tmp.txt obj2 = pickle.load (Open ("Tmp.txt  "R")) print obj2#-*-coding:utf-8-*-Import pickle# can also do this: # import cpickle as pickle obj = {"A": 1, "B": 2, "C": 3} # Persist obj to file tmp.txt pickle.dump (obj, open ("Tmp.txt", "W")) # do something else ... # reads and restores obj objects from tmp.txt = Pi Ckle.load (Open ("Tmp.txt", "R")) print Obj2

However, in practical applications, we may have some improvements, such as replacing Pickle with Cpickle, which is a C language implementation version of the latter, with faster speeds, and sometimes the third parameter is set to True at dump to improve the compression ratio. Take another look at the following example:

#-*-Coding:utf-8-*-import cpickle as Pickleimport randomimport osimport timelength = 1024x768 * 10240def main (): D = {} A = [] for I in range (LENGTH): A.append (random.randint (0, 255)) d["a"] = a print "dumping ..." T1 = Time.time () pickle.dump (d  , open ("Tmp1.dat", "WB"), True) print "DUMP1:%.3fs"% (Time.time ()-T1) T1 = Time.time () pickle.dump (d, open ("Tmp2.dat", "W")) print "DUMP2:%.3fs"% (Time.time ()-T1) S1 = Os.stat ("Tmp1.dat"). St_size s2 = os.stat ("Tmp2.dat"). St_size print "  %d,%d,%.2f%% "% (S1, S2, 100.0 * s1/s2) print" Loading ... "T1 = time.time () obj1 = pickle.load (Open (" Tmp1.dat "," RB "))  Print "Load1:%.3fs"% (Time.time ()-T1) T1 = time.time () obj2 = pickle.load (Open ("Tmp2.dat", "R")) print "Load2:%.3fs" % (Time.time ()-T1) If __name__ = = "__main__": Main () #-*-coding:utf-8-*-import cpickle as Pickleimport Randomimport OS import Time LENGTH = 1024x768 * 10240 def main (): D = {} A = [] for I in range (LENGTH): A.append (random.randint (0, 255)) d ["a"] = a print "DumpinG ... "T1 = Time.time () pickle.dump (d, open (" Tmp1.dat "," WB "), True) print" DUMP1:%.3fs "% (Time.time ()-T1) T1 = time. Time () Pickle.dump (d, open ("Tmp2.dat", "W")) print "DUMP2:%.3fs"% (Time.time ()-T1) S1 = Os.stat ("Tmp1.dat"). St_size s 2 = Os.stat ("Tmp2.dat"). St_size print "%d,%d,%.2f%%"% (S1, S2, 100.0 * s1/s2) print "Loading ..." T1 = Time.time () Obj1 = pickle.load (Open ("Tmp1.dat", "RB")) print "Load1:%.3fs"% (Time.time ()-T1) T1 = time.time () Obj2 = Pickle.load ( Open ("Tmp2.dat", "R")) print "load2:%.3fs"% (Time.time ()-T1) If __name__ = = "__main__": Main ()

The results on my computer are as follows:

dumping...dump1:1.297sdump2:4.750s20992503, 68894198, 30.47%loading...load1:2.797sload2:10.125s

As you can see, when you specify protocol as True for dump, the size of the compressed file is only 30% of the original file, and it takes less time both at dump and load. Therefore, it is generally advisable to set this value to True.

In addition, the Pickle module provides the dumps and loads two methods, similar to the dump and load methods above, except that you do not need to enter the file parameter, the input and output are string objects, and some scenarios may be more convenient to use these two methods.

