The efficiency of regular expression: greed, non-greed and backtracking

Some students want to filter the content between, that is so write regular and program.

1 $str = preg_replace ('%<script>.+?</script>%i ', ', ', $str);/Not greedy

It seems that there is no problem, but it is not. If

1 $str = ' <script<script>alert (document.cookie) </script>>alert (document.cookie) </script> ' ;

Then after the above procedure processing, the result is

1 $str = ' <script<script>alert (document.cookie) </script>>alert (document.cookie) </script> ' ;

2 $str = preg_replace ('%<script>.+?</script>%i ', ', ', $str);/Not greedy

3 Print_r ($STR);

4//$STR output is <script>alert (document.cookie) </script>

Still not up to the effect he wanted. The above is not greedy, but also some called inertia. Its logo is not greedy labeled as a measure of the number of characters behind add? , such as +?, *?、?? (More special, in the future blog, I will write) and so on. That is, the identity is not greedy, if not written? is greed. Like what

1 $str = ' <script<script>alert (document.cookie) </script>>alert (document.cookie) </script> ' ;

2 $str = preg_replace ('%<script>.+</script>%i ', ', ', $str);/Not greedy

3 Print_r ($STR);

4//$STR output for <script only these, as if not quite appropriate, ha, you know how to rewrite the regular?

The above is an introduction to the difference of greed, not greed. Below, talk about the backtracking problem caused by greed, not greed. Let's look at a small example first.
The regular expression is w* (d+), and the string is cfc456n, so what is the number of this regular match??

If your answer is 456, then, congratulations, the answer is wrong, the result is not 456, but 6, you know why?

cfc4n to explain, when the regular engine with regular w* (d+) to match the string cfc456n, the first w* to match the string cfc456n, first of all, w* will match the string cfc456n all the characters, and then give d+ to match the rest of the string, and the rest is gone, at this time, W* rules will be reluctant to spit out a character, to d+ to match, at the same time, before spit out the characters, record a point, this point, is used for backtracking points, and then d+ to match N, found not to match the success, will again ask w* again spit a character, w* will first record a backtracking point, then spit out a character. At this time, the result of w* matching only cfc45, has spit out 6n, d+ again to match 6, found matching success, will inform the engine, matching success, directly displayed. So, (d+) The result is 6, not 456.

When the regular expression above is changed to w*? (d+) (note that this is not greedy) and that the string is still cfc456n, so what is the regular match of $??
A classmate answer: The result is 456.
Well, yes, right, is 456,cfc4n weak and weak ask, why is 456?
I'm here to explain why it's 456.
Regular expression has a rule, is the quantifier priority match, so w*? will go first to match the string cfc456, due to w*? Is it greedy, the regular engine uses an expression w+? Match only one string at a time, then give control to the back d+ to match the next character, while recording a point, For when the match is unsuccessful, return here and match again, that is, the backtracking point. Since W is the quantifier is *,* 0 to countless times, so, first is 0 times, that is w*? match the null, record the backtracking point, give control to d+,d+ to match the cfc456n's first character C, and then, the match fails, so, then the control over to the w*? to match cfc456n's C. , w*? Match c success, because it is greedy, so he only matches one character at a time, record the backtracking point, and then give control to d+ match F, then, d+ match F again failed, and then control to w*?,w*? Match C again, record the backtracking point (then w*?) The result of the match is CFC), And then give control to d+,d+ to match 4, match successfully, then, because the quantifier is +, is 1 to countless times, so, then match, and then match 5, success, and then, and then match 6, success, and then, continue to match the operation, the next character is n, matching failure, then, d+ will control the right hand over. Since there is no regular expression behind the d+, the entire regular expression is declared matched and the result is cfc456, where the first group of results is 456. Dear classmate, you understand just the result of the topic, why is 456?

Well, do you know from the above example the greedy, not greedy match principle? Do you understand when you need to use greed, not greed to deal with your string?
Brother Bird's article is about
expressions, Programs for

1 $reg = "/<script>.*?</script>/is";

2 $str = "<script>********</script>"; Length greater than 100014

3 $ret = Preg_repalce ($reg, "", $str); Returns null

The reason for this is that there is too much backtracking until it causes the stack to run out of space.

Let's look at an example.
String

1 $str = ' <script>123456</script> ';

The regular expression is

1 $strRegex 1 = '%<script>.+</script>% ';

2 $strRegex 2 = '%<script>.+?</script>% ';

3 $strRegex 3 = '%<script> (?:(?! </script>).) +</script>% ';

These three regular, each will cause a few backtracking??

The answer to see the next chapter PHP regular expression efficiency: backtracking and solidification grouping

In the above, we talked about a bit about PHP (NFA PCRE) Regular expression matching priority classifier, ignoring the matching principle of the priority classifier. So what is your answer to the question left above?
Let's look at the problem first.

String

View Source code
< id= "Highlighter_297259_clipboard" title= "Copy to clipboard classid=" clsid:d27cdb6e-ae6d-11cf-96b8-444553540000 "width=" "Height=" codebase= "http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,0,0" Type= "Application/x-shockwave-flash" >
Print Help
1 $str = ' <script>123456</script> ';

The regular expression is

View Source code
< id= "Highlighter_598167_clipboard" title= "Copy to clipboard classid=" clsid:d27cdb6e-ae6d-11cf-96b8-444553540000 "width=" "Height=" codebase= "http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,0,0" Type= "Application/x-shockwave-flash" >
Print Help
1 $strRegex 1 = '%<script>.+</script>% ';

2 $strRegex 2 = '%<script>.+?</script>% ';

3 $strRegex 3 = '%<script> (?:(?! </script>).) +</script>% ';

These three regular, each will cause a few backtracking??

Answer:

View Source code
< id= "Highlighter_886386_clipboard" title= "Copy to clipboard classid=" clsid:d27cdb6e-ae6d-11cf-96b8-444553540000 "width=" "Height=" codebase= "http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,0,0" Type= "Application/x-shockwave-flash" >
Print Help
1 $strRegex 1 = '%<script>.+</script>% '; 9 times, remember to distinguish the escape symbol.

2 $strRegex 2 = '%<script>.+?</script>% '; 5 times

3 $strRegex 3 = '%<script> (?:(?! </script>).)  +</script>% '; 7 times

For the first greedy match matching rule, the 9 times backtracking is the regular "" to the string "" match, the composition of backtracking, the number of backtracking, exactly is the length of the string.
The second non-greedy matching rule, backtracking 5 times, is the backtracking of the regular ". +?" Match to the string "123456". The number of backtracking, minus the minimum number of string lengths. which is 6-1 = 5 times. If the regular expression is ". *?" then the number of backtracking is 6 times.
The third is a 0-wide assertion, or a glance. (not to mention.) ）
In the NFA regular engine, backtracking is his soul, so, whether greedy, not greedy, look around and so on there will certainly be backtracking, which we can not avoid (the word is not very accurate), but we could reduce the number of backtracking, or to protect a part of the matching rules do not backtrack.

As for the bird brother mentioned in the last blog about a large number of backtracking problems caused by the non greed, we can know that backtracking is indeed the culprit of wasting resources, then, can we not let it go backwards?
The answer is yes, NFA engine, there is a concept, called the solidification group. Quote the concepts on the book

Specifically, the use of "(? ...)" The match does not differ from the normal match, but if the match is made to this structure (that is, after the closing parenthesis), all the standby states in the structure are discarded. That is, at the end of a cured packet match, the text that it has matched has been solid into a unit and can only be retained or discarded as a whole. Fallback states that have not been tried in the subexpression in parentheses are no longer available, so backtracking will never be able to select the state (at least, the state in which the lock (locked in) is in) when the structure match completes.

So what's the use of curing groups? Let me give you an example. (I can't find the right example, I have to borrow the example from the book)
For example, to deal with a batch of data, the original format is 123.456, and later because of floating point number display problems, some of the data format into 123.456000000789 of this, the requirement to keep only the decimal point after 2-3 digits, but the last one can not be 0, this is how to write it? (Consider the number following the decimal point directly below), after writing the positive, we will also use this to match the data, the original data to replace the matching results.
First, we can write this regular ". dd[1-9]?d*" immediately, and the PHP code is

1 $str = Preg_replace ('. ( Dd[1-9]?  d* ', ' \1 ', $str); Group1 that match the result to the reverse reference

Obviously, this type of writing, for some data format 123.456 of this format, in vain to deal with, in order to improve efficiency, we have to deal with this regular. From the 123.456 string compared to the other, we found that there is doubt 123.456 the data behind the number, so, in vain to deal with it. That's good to do, we change this is a bit, the following quantifier * changed to +, so for 123.45 decimal point after 1, 2 digits, will not be processed in vain, and, for more than three digits, processing normal. Its PHP code is

1 $str = Preg_replace ('. ( Dd[1-9]? D+ ', ' \1 ', $str);

Well, is this really OK?? Are you sure? In the previous blog, we learned the principle of matching, so we also analyze this regular matching process.
String "123.456", the regular expression is ". (Dd[1-9]?) D+ "Let's take a look at
First (no 123 points before the decimal point), "." Match ".", match succeeded, put control to the next "D", "D" Match "4" success, give control to second "D", this "D" match "5" success, then, give control to "[1-9]?", because quantifiers are "?", Regular expressions Follow "quantifier preference" and, here is "?", leaving a backtracking point. Then match "6" success, and then the control to "d+", "d+" found no characters behind, the most follow the "LIFO" rule, back to the last backtracking point, to match, at this time, "[1-9]?" will return the matching character "6", "[1-9]?" Match "6" success. The match was completed. Everybody finds out "(Dd[1-9]?) "The result of the match is really" 45 ", not what we want" 456 "," 6 "by" d+ "match. So, what should we do? Can you make "[1-9]?" matches once successful, without backtracking? This is what we said above "curing group", PHP (Preg_replace function) used in the regular engine to support the curing group, we according to the curing group, you can change the code to the following way

1 $str = Preg_replace ('. ( DD (? >[1-9]?) D+ ', ' \1 ', $str);

In this case, the string "123.456" is not compliant and will not be matched. Then we can meet our demands.

From the above example, know the role of curing group, then for the Bird Brother blog written on the Not greedy retrospective question, can we also transform it, so that it does not backtrack?
Let's see the answer to Brother Bird.

1/<script>[^<]*</script>/is

Brother Bird is very good at writing. Excluding "<" All characters are compliant, and the middle part does not backtrack, high efficiency. However, if there is a character "<" in the middle (the following code)

1 <script>

2 If a < b

3 </script>

That bird brother's this is not match, can not achieve the function we want.
Then we can implement this requirement according to the solidification group, look around (0 wide assertion), and finally, the regular and PHP code examples given by cfc4n are as follows

1 $reg = '%<script> (? >[^<]*) (?> (?!) </?script>) <[^<]*) *</script>%is ';

2 $str = Str_pad ("<script>", 111111, "*"); 100000 character length greater than PHP backtracking limit

3 $str. = ' If a < b;    If B > c;</script> '; Add a few test characters including < >

4 $ret = preg_replace ($reg, "OK", $str);

5 Print_r ($ret); Print result OK, proof match correct

6 Var_dump (Preg_last_error ()); Last match error. Its output is int (0)

Hey, classmate, do you understand?

The above is a piece of cfc4n, if there is a mistake, welcome to point out.