Regular expression principles and optimization notes under Python

A little bit of learning about regular expressions in the most recent time. The selected textbook is "Mastering regular Expressions", the so-called "proficient regular expression". After reading it again, the feeling of the regular expression of the strong and superb. The first three chapters are the introduction and foreshadowing of the basic rules of regular expressions. Chapter Seven is the application to the specific language.        And the core part is 456 of these three chapters. The fourth chapter is about the essence of the whole regular expression, that is, the retrospective thought of the traditional engine NFA. The fifth chapter is a few examples of the understanding of the idea of backtracking. The sixth chapter is the study of efficiency.        The root cause is also the extension and research of the retrospective thought.        This article is a summary of the documentation I've combined with the official Python re module and the book. Among the official documents: http://docs.python.org/3.3/library/re.html
Since I'm all connected and used on Python, the questions that follow are basically presented in Python, so I'm not involved in any of the other regular genres in this book. According to the book, Python and Perl style is similar, belonging to the traditional NFA engine, that is, "expression-dominated", using backtracking mechanism, matching to stop (Sequential Sensitivity, unlike POSIX NFA, which matches the longest left-most result).
For the backtracking section, and when it comes to matching, the position of the engine is always placed between the character and the character, not the character itself. For example ^ corresponds to the "blank" position that precedes the first character.Introduction to Basic rulesescape symbols in PythonInterference

In Python, command lines, scripts, and so on, the escape symbols are processed, and the string is conflicted with the engine of the regular expression. That is, the string ' \ n ' in Python is considered a newline symbol, so that when passed into the RE module, it is no longer a literal two symbol, but a newline character. So, when we pass to the regular engine, we have to let the engine simply think of as a ' \ ' and a ' n ', so we need to add the escape character to be ' \\n ', for this case, Python uses the Raw_input method, preceded by the string R, So that the escape character in the string is no longer special (that is, not handled in Python, and all is lost to the regular engine to handle), then the line break is R ' \ n '

Basic Characters

.        # In Normal mode, matches any character except the line break. (Specify Dotall tag to match all characters)

quantifier Qualifier

*        # matches 0 or more of the preceding objects. Don't overlook the 0 of the situation here. +        # matches 1 or more of the preceding objects. The point here is that there is at least one. ?        # match 0 or 1 of the preceding objects. {m}      # matches the preceding object m times {m,n}    # matches the previous object at least m times, up to N times. 

Anchor Point character

^        # matches the beginning position of the string, under the multiline tag, can match any \ n position after the $        # Match string end position, Under the multiline tag, you can match any position before \ n

Escape symbols inside the regular engine

\m M is a number, called a reverse reference, that refers to a matching object in front of the captured type brackets. The number is the corresponding bracket order. \a matches only the beginning of the string \b can understand the symbol of an anchor point, which matches the boundary of the word.      This is where word is defined as consecutive letters, numbers, and underscores. To be exact, \b's position is at the junction of \w and \w, and of course the string begins to end and between \w. \b and \b correspond, and themselves match the null character, but their position is in a non-"Border"the case. such as R'py\b'Can match'python', but does not match'Py,','py.'\d Match number \d matches a non-numeric \s when no Unicode and locale tags are specified, equivalent to [\t\n\r\f\v] (note \ t is preceded by a space that also matches a space) \s is opposite \s \w unspecified Unicode and lo When Cale is marked, it is equivalent to [a-za-z0-9_]\w and \w opposite \z only match the end of the string other Python-supported transfer symbols are also supported, as in the previous'\ t'

Character Set

[]
In particular, this character set only matches one character at a time (neither empty nor more than one!), so some quantifier qualifiers in front of it lose their original meaning here.
In addition, the'-' symbol is placed between two characters, representing all characters between ASCII characters, such as [0-9], representing 0 to 9.
When placed at the beginning or end of a character set, or escaped by ' \ ' , it simply means '-' this symbol
Finally, when you use ' ^ 'at the beginning, the exclusion type character group is used.

related content of parenthesesNormal type brackets

(...)    Normal capture type brackets, which can be referenced by \number.

Extended Brackets

(? ailmsx) a re. Ai Re. I#Ignore CaseL Re. Lm Re. Ms Re. S#dot match includes line breakx re. X#you can write multiple lines of an expressionsuch as: Re_lx= Re.compile (r'(? is) \d+$') RE_LX= Re.compile (r'\d+', Re. I|re. S#these two compile expressions are equivalent(?:......)#non-capturing brackets, which do not record capture, save space(? P<name&gt, ...)#This captured parenthesis can be called using name instead of relying on numbers. Use (? P=name) call. (?#...) #注释型括号, this parenthesis is completely ignored(?=...)#Positive Lookahead Assertion if the following is in parentheses, the match succeeds(?! ...)#Negative Lookahead Assertion if the following is not in parentheses, the match succeeds(? <= ...)#Positive Lookbehind Assertion if the front is in parentheses, the match succeeds(?<!...)#Negative Lookbehind Assertion if the front is not in parentheses, the match succeeds #The above four types of assertions, themselves do not match the content, just tell the regular engine whether to start matching or stop.  #In addition, in the latter two latter assertions, a fixed-length assertion must be asserted. (? (id/name) yes-pattern|no-pattern)#If there is a group specified by ID or name, it will match yes-pattern, otherwise the no-pattern will be matched, usually no-pattern can be omitted. 

Match priority/ignore precedence symbolIn quantifier qualifiers, the default is the match priority, that is, the regular engine matches as many characters as possible if the condition is met (Greedy RulesWhen you add '? ' to these symbols, the regular engine becomes a ignore priority, and the regular engine matchesas little as possibleThe character.

Like '??' Matches the absence, then the case of 1 objects. and {m,n}? It is a priority to match M-objects, not more N objects.

Related Advanced knowledge

Python is a Perl style and belongs to the traditional NFA engine, which is relative to the POSIX NFA and DFA engines. So most of the discussion is for the traditional NFA

sequential problems in the traditional NFA

The NFA is an expression-driven engine, and the traditional NFA engine stops immediately when it finds the first match: The engine is stopped after it has been matched.
POSIX NFA does not stop immediately and will seek the longest results in all possible matches. This is also a bug that does not appear in the traditional NFA, but is exposed in the latter.
In a point, theNFA scientific name is "uncertain type has a poor automaton", the DFA scientific name is "deterministic type has poor automata"
The non-deterministic and deterministic is for the characters in the target text to be matched, in the NFA, each character in a match even if it is detected, it is not sure whether he really passed, because the NFA will appear backtracking ! Even more than one or two times. See the following example for a legend. In the DFA, because it is the target text-dominated , all object characters are detected only once, to the end of the text, too, but just. This is the reason for the "OK" argument.

backtracking/Standby statusStandby StatusWhen an optional branch appears, other options are stored as an alternate state. When the current match fails, the engine goes back to the nearest standby state. In the case of matching, the match priority is consistent with ignoring precedence in a sense, but only in order. When there are multiple matches, the two approaches are likely to be different, but when there is no match, the situation is consistent, that is, it is necessary to try all possible.Two essentials of backtracking mechanism

1. When a regular engine chooses to attempt or skips an attempt , matching the precedence quantifier and ignoring the precedence quantifier controls its behavior.
2. When a match fails, backtracking needs to be returned to the previous standby state, with the principle that LIFO (the post-generated state is first traced back to)

Typical examples of backtracking:As you can see, the traditional NFA to D-Point is the end of the match. And in the shadow of POSIX NFA's matching process, you need to find all the results,and take the longest result back in these results.。

As a comparison, here is the path that the engine passes when the target text does not match:
For example, we see that the POSIX NFA and the traditional NFA match paths are consistent .

The above example raises the thought of a match, and many times we should try to avoid using '. * ' because it can always match to the bottom or end of the line, wasting resources.
Since we are only looking for data between quotes, it is often possible to do the work with an excluded array .
In this example, using the ' [^ ' ']* ' to replace the '. * ' function is obvious, we only match the non-quoted content , then the first quotation mark can exit the control of the * number.

Curing Group ThinkingThe idea of curing groups is very important,However, Python does not support。 Use (...) If an alternate state is created when matching in parentheses, the parentheses are immediatelyThe engine's thrown away .(So it can't be traced!) ）。 Give a typical example such as:

' \w+: '

The process in which the expression is matched is the first to match all of the \w characters, if there is no ': 'at the end of the string, that is, the match does not find the colon, which triggers the backtracking mechanism, He will force the previous \w+ to release the character and try to match the ': ' In the returned character .
But here's the problem: \w does not contain colons , and obviously does not match success anyway , but according to backtracking mechanism, the engine still has to bite the bullet forward, this is the waste of resources.
So we need to avoid this backtracking, the way to do this is to cure the previous match, do not store the standby state ! , the engine will have to end the matching process because there is no standby status available. Greatly reduce the number of backtracking!

python simulation Curing process

Although not supported in Python, the book provides the use of forward assertions to simulate the curing process.

(?=(...)) \1

itself, the result in the assertion expression does not save the standby state , and he does not match the specific characters, but by subtly adding a capturing bracket to the inverse reference of the result, it achieves the effect of the curing group! The corresponding example is:

' (? = (\w+)) \ 1: '

Multi-Select structure

Multi-select structure in the traditional NFA, neither match nor neglect priority. But in order . So there are the following ways to use

1. In cases where the results are guaranteed to be correct , priority should be given to matching the more likely results. Put the most likely branch in front of you as much as possible .
2. Multi-select structures cannot be abused because the cache records the corresponding number of alternate states when matching to a multi-select structure. For example: both [abcdef] and ' a|b|c|d|e|f ' are two expressions that can accomplish one of your purposes, but try to choose a character array , because the latter will create 6 standby states per comparison , wasting resources.

some ideas and tips for optimization
The law of BalanceA good regular expression should seek the following balance:

1. Matches only the desired text, excluding the unwanted text. (Good at using non-capturing parentheses to conserve resources)
2. Must be easy to control and understand. Avoid being written in the heavenly book.
3. With the NFA engine, the efficiency must be guaranteed ( if matched, the matching result must be returned quickly, and if it does not match, the matching failure should be reported in the shortest possible time.) )

Handle Unexpected matches

In the process, we are always accustomed to using the asterisk and other non-mandatory quantifiers (in fact, a bad habit),
Such results may result in the matching expressions we use that do not have to match the characters, as shown in the following example:

' [0-9]? [^*]*\d*'    # Just for example, no practical meaning. 

This is the case, and when the target text is "ideal", there may not be a problem, but if the data itself is problematic. The result of this equation is completely unpredictable.
The reason is that no part of him is necessary! It matches any content that is successful ...
understanding and assumptions about the dataIn fact, when processing a lot of data, our operation data is not the same situation,sometimes it's regular ., then we can omit the case of complex expressions,but in turn, when the source is messy,You need to think a little more and deal with every possible situation accordingly.optimizations that are generally present in the enginecompiling the cacheWhen you reuse compiled objects, you should compile them using the Re.compile () method before you use them, so that you do not have to recompile each time you call later. Save time. In particular, when regular matches are called repeatedly in the loop body.Anchor Point OptimizationWith the optimization of some engines, the anchor points should be highlighted separately. Compared with ^a|^b, its efficiency is not as good as ^ (a|b), the system will also handle the end of line anchor optimization. So when writing the relevant regular, use the anchor point if possible.quantifier OptimizationOptimization in the engine, will be the same as. * Such a measure of the uniform treatment, rather than according to the traditional backtracking rules, so, theoretically speaking ' (?:.) * ' and '. * ' are equivalent, but when specific to the engine implementation, the '. * ' is optimized. The speed has a difference.eliminate unnecessary brackets and character groupsDoes this have in PythonUnknown。 Only in the supported engine, will be the same as [.] into \., because the latter is obviously more efficient (character group processing causes additional overhead)These are some of the optimization of the engine belt, nature is actually beyond our control, but after some understanding, some of our later processing and use of a lot of help. Additional Tips and additional contentExcessive backtracking problemseliminate finger-level matchingThe shape is as follows:

(\w+) *

The expression in this case, what is the problem when matching long text, if the text match fails (remember, if it fails, the description is backall the possibilities), imagine that the * number is back in a state, the inside of the + number includes the restall States, after the verification has failed, go back outside, * numberback to the penultimate standby state, and then into the parentheses, the + number will go back one side than the previous round of 1 differenceStandby StatusWhen the string is very long,there will be exponential backtracking totals.。 The system will be ' stuck '. Even when there is a match, this match is hidden in the middle of the total number of backtracking, it will also cause the situation of card death. Therefore, when using the NFA engine, you must pay attention to this problem!

We use the following ideas to avoid this problem:
occupy a priority quantifier (use forward assertion plus reverse reference impersonation in Python)
The reason is simple, since the large number of backtracking is caused by the storage of the standby state, then we directly let the engine abandon these states. In the final analysis is to get rid of (regex*) * this form.

Import= re.compile (R'(? = (\w+)) \1*\d')

Efficiency test Code

When you test the efficiency of an expression, you can compare the time required with the following code. In the two possible outcomes, the preferred one.

ImportReImporttimere_lx1= Re.compile (r'Your_re_1') re_lx2= Re.compile (r'your_re_2') StartTime=time.time () repeat_time= 100 forIinchRange (repeat_time): s='Test Text'*10000result=Re_lx1.search (s) time1= Time.time ()-StartTimePrint(time1) StartTime=time.time () forIinchRange (repeat_time): s='Test Text'*10000result=Re_lx2.search (s) time2= Time.time ()-StartTimePrint(time2)

equivalent conversion of quantifierNow let's look at the efficiency of the curly brace quantifier.1, when the brace-decorated object is similar to a character array or \d thisNon-deterministicCharacters, using braces is more efficient than repeating overlapping objects. That is: \d{5} is better than \d\d\d\d\dtested in Python the latter is superior to the former. will be much faster.2, but when a character is determined by a repeating character, it is more efficient to simply repeat the overlay object. This is because the engine will be optimized for purely string internals (although we don't know how the specific optimizations are done) AAAAA better than a{5} Overall said ' \d ' must be slower than ' 1 'I use the Python3 in the RE module, tested, do not use quantifiers will be fast.
In summary, the overall use of quantifiers in Python is not as simple as listed! (different from the book!) ）
the utilization of Anchor point optimization

The following example assumes that the matching content appears at the end of the string object, and the first expression below is faster than the second expression , because the former has the advantage of an anchor point.

RE_LX1 = Re.compile (R'\d{5}$')    = Re.compile (R'\d{5}  ')    # The former is fast and has an anchor point optimization

the use of excluded arrays

Continue, assuming we want to match 5 digits in a string, there are two expressions to choose from:
After analysis, we found that \w is included in the \d, when the use of matching priority, the preceding \w will contain numbers, the reason to match the success, or to determine the failure, is the back of the \d force the preceding quantifier to return some characters.
Knowing this, we should try to avoid backtracking, a natural idea is not to let the preceding match the first quantifier involves \d

RE_LX1 = Re.compile (R'^\w+ (\d{5})'= re.compile (R'^[^\d]+\d{5}  ')    # better than the above expression

In general, when we have no time to delve into the module code, we can only get the final composite expected expression by trying and repeating the changes.

Common-sense optimization measures However, when we try to modify them with the possible effects of ascension, it is possiblecounterproductive，because some of the slow backtracking we seem to have within the regular engine will be optimized.，
The "trickery" changes may turn off or avoid these optimizations, so the results may disappoint us. Here are some of the things that are mentioned in the bookCommon sense ofOptimization measures:

Avoid recompiling (creating objects outside of the loop) use non-capturing parentheses (saves the capture time and the number of States in backtracking) using the anchor symbol to extract text and anchor points without abusing the character set. Extracting them from the possible multi-select branching structure extracts the speed. The most likely match expression is placed in front of the multiple selection branch

a very useful core formula' opening normal* (Special normal*) * closing '

This formulaespecially for the normal text that matches in two special demarcation sections (which may not be a character), special is the case where the demarcation section may be confused with the normal part. Like the next three points to avoid the occurrence of an endless match of this formula.

1. The beginning of the special part and the normal part cannot coincide. It is guaranteed that these two parts will not match the same content under any circumstances, otherwise the path of the engine cannot be determined if the match cannot be traversed in all cases.
2. The normal part must match at least one character
3. The special part must be fixed-length

As an example:

[^\\"]+(\\.[ ^\\"# matches text within two quotes, but does not include escaped quotes

Regular expression principles and optimization notes under Python