Python uses the re-modular regular precompiled and pickle scheme

Project on-line requirements have speech and nickname filtering requirements, the client is using Python script, Python script directly using the RE module for regular matching, the first practice is to open the game, each frame compiled 2 regular, but operational requirements inside 100+ a slightly more complex regular style, A compilation on the PC requires 80ms, causing the client to start the time lag.

The solution, of course, is to save the regular results compiled by the RE module, and then load directly on the line, but note that re.compile () returns the _sre. Although the Sre_pattern object can be saved using pickle, it is only a false impression, in fact it just saves the regular content you need to compile, and then instantiates it back to re-call Re.compile to recompile, which is not helpful for increasing the loading speed.

Looking at the source code of the Python re module can be found, actually re.compile finally call to the Sre_compile compile, and this sre_compile can be clearly divided into prev_compile and Post_compile ( Name is I up), to python2.7 re source code for example, __version__ is 2.2.1, I saw the next python3.5 re, __version__ is unchanged. Here is the code for Sre_compile.compile

1 defCompile (p, flags=0):2     #Internal:convert Pattern list to internal format3 4     ifisstring (p):5         ImportSre_parse6Pattern =P7p =Sre_parse.parse (p, flags)8     Else:9Pattern =NoneTen  OneCode =_code (p, flags) A  -     #Print Code -  the     #XXX: <fl> get rid of this limitation! -     ifP.pattern.groups > 100: -         RaiseAssertionerror ( -             "Sorry, but this version is only supports named groups" +             ) -  +     #map in either direction AGroupindex =p.pattern.groupdict atIndexgroup = [None] *p.pattern.groups -      forKiinchGroupindex.items (): -Indexgroup[i] =k -  -     return_sre.compile ( -Pattern, Flags |p.pattern.flags, code, inP.pattern.groups-1, - Groupindex, Indexgroup to)

With #print code as the demarcation, the above part is Prev_compile, and then Post_compile, the pre-preprocessing generated p and code will eventually be transmitted to the lower _sre.compile interface as a parameter, p is by sre_ Parse.parse () call generation, the result is a list, I think is to carry out sre_compile opcode, through opcode to generate code, and generate code calls is the most time-consuming, and most importantly, This part of P and code are Python lists, which can be saved to a file via pickle, which actually solves the problem of re precompilation regular time-consuming, but since Post_compile will eventually use your regular expression, Regular content must still be retained, and there is a one by one corresponding relationship, that is, you generate p, code, need to correspond to your pattern.

The final solution to our game is to build the precompiled result into a. py file, and then import the Post_compile directly into the script, following the specific code

1 ImportSYS2Sys.path.append (".. /.. /script/data" )3 4 defPrev_compile (p, flags =0):5     ImportSre_compile, Sre_parse6     ifsre_compile.isstring (p):7p =Sre_parse.parse (p, flags)8 9Code =Sre_compile._code (p, flags)Ten     returnp, Code One  A defPost_compile (pattern, p, code, flags =0): -     Import_sre -     ifP.pattern.groups > 100: the         RaiseAssertionerror ("Sorry, but this version is only supports named groups" ) -  -Groupindex =p.pattern.groupdict -Indexgroup = [None] *p.pattern.groups +      forKiinchGroupindex.items (): -Indexgroup[i] =k +  A     return_sre.compile ( atPattern, Flags |p.pattern.flags, code, -P.pattern.groups-1, - Groupindex, Indexgroup -         ) -  - defprecompile (): in     ImportCpickle as Pickle -     ImportFilterprop toName_p = [] +Name_code = [] -      forSinchFilterprop.namefilter: thes = S.decode ("Utf-8" ) *P, Code =Prev_compile (s) $ name_p.append (P)Panax Notoginseng name_code.append (code) -  theWord_p = [] +Word_code = [] A      forSinchFilterprop.wordfilter: thes = S.decode ("Utf-8" ) +P, Code =Prev_compile (s) - word_p.append (P) $ word_code.append (code) $  -With file (".. /.. /script/precompile.py","W") as F: -name_p_s =repr (Pickle.dumps (name_p)) thename_code_s =repr (Pickle.dumps (name_code)) -word_p_s =repr (Pickle.dumps (word_p))Wuyiword_code_s =repr (Pickle.dumps (word_code)) the         Print>> F,"#-*-coding:utf-8-*-" -         Print>> F,"precompile = [" Wu         Print>> F, name_p_s,"," -         Print>> F, name_code_s,"," About         Print>> F, word_p_s,"," $         Print>>F, word_code_s -         Print>> F,"]" -  - if __name__=="__main__": APrecompile ()

Filterprop is the specific regular configuration, including the Wordfilter and Namefilter, the above code eventually generated a precompile.py file, including the prev_compile generated p, code, After the client initiates the use of this part of the structure, call Post_compile to generate the _sre. The Sre_pattern object, which greatly improves the efficiency, is quite elegant for the need to precompile a large number of regular-style requirements.

Resources:

Http://stackoverflow.com/questions/4037339/is-there-a-way-to-really-pickle-compiled-regular-expressions-in-python

Python uses the re-modular regular precompiled and pickle scheme