first, talk about the limitations of the Shell scripting language itself
As an interpretative scripting language, it is inherently inefficient. Although the other commands it invokes may be good in efficiency.
The execution of the shell script is performed sequentially, not in parallel. This is largely a waste of the resources that might be available on the system.
The shell creates a new process every time it executes a command, and if the script writer is not aware of it, it is a waste of system resources to write a script incorrectly.
Secondly, we can improve the efficiency of the script as far as possible through our experienced coding in the limitations of the Shell scripting language.
1, for example, I want to do a cyclic processing of data, may be a simple process of data, it will make it easier to think of the shell in the loop like this:
Copy Code code as follows:
Sum=0
For ((i=0;i<100000;i++))
Todo
sum=$ (($sum + $i))
Done
Echo $sum
We can use time this script to test the three execution times of the 100,000 cycles:
Real 0m2.115s
User 0m1.975s
SYS 0m0.138s
Real 0m2.493s
User 0m2.173s
SYS 0m0.254s
Real 0m2.085s
User 0m1.886s
SYS 0m0.195s
The average time is 2.2s, if you know the loop in the awk command, that's better, let's test it. Cycles with data size three times are time-consuming:
Copy Code code as follows:
awk ' begin{
sum=0;
for (i=0;i<100000;i++)
Sum=sum+i;
print sum;
}'
Real 0m0.023s
User 0m0.018s
SYS 0m0.005s
Real 0m0.020s
User 0m0.018s
SYS 0m0.002s
Real 0m0.021s
User 0m0.019s
SYS 0m0.003s
You dare not imagine that the average time is only 0.022s, the efficiency of the pure cycle is already two-bit higher than the shell order of magnitude. In fact you run millions of cycles you will find that the shell has been more difficult, tens is even harder. So you should be aware that your program uses awk as much as possible to do cyclic operations.
2, about the regular, often write the shell of the students understand its importance, but you can really use it efficiently?
Here's an example: Now I have a 1694617-line log file Action.log, which is similar in content:
2012_02_07 00:00:04 1977575701 183.10.69.47 login 500004 1977575701 old/***/port/***.php?...
I now want to get a string between the port and/or, I can do this:
Awk-f '/' {print $} ' < 7action.log >/dev/null
But you don't want to know how efficient it is:
Real 0m12.296s
User 0m12.033s
SYS 0m0.262s
Believe me, I won't want to see the cursor Flash for 12 seconds. But if you do this:
awk ' {print $} ' < 7action.log | Awk-f '/' {print $} ' >/dev/null
The efficiency of this sentence is three times respectively:
Real 0m3.691s
User 0m5.219s
SYS 0m0.630s
Real 0m3.660s
User 0m5.169s
SYS 0m0.618s
Real 0m3.660s
User 0m5.150s
SYS 0m0.612s
The average time is about 3.6 seconds, the efficiency is about 4 times times the gap, although not as a hundredfold gap, but enough to make 4 hours into 1 hours. I think you know the difference.
In fact, this regular instance you can try to speculate about other situations, because the regular each run is a need to start string matching, and the default separator will be faster by the field of distinction. So, after we know some data laws, we can try to shorten the string that we're going to do a complex regular match, which is a very obvious efficiency boost based on the scale of your data reduction, and there's a relatively simple regular match, with only one single character "\" You can imagine if the regular expression is like this:
$7!~/\.jpg$/&&$7~/\. [s]?html|\.php|\.xml|\/$/&& ($9==200| | $9==304) &&$1!~/^103\.108|^224\.215|^127\.0|^122\.110\.5/
I think you can imagine the great significance of a target matching string from 500 characters down to 50 characters!
PS: In addition to the detailed regular optimization please see a blog post after this date.
3. Repeat the shell's redirection and piping. I will not give an example of this item, just to say my personal understanding.
Weeks are well known, many programs or languages have a more prominent efficiency bottleneck is Io,shell is no exception (the individual so consider). Therefore, it is recommended to use redirects as little as possible for input and output operations or to create temporary files for subsequent use, and, of course, if you have to do this, then do it, I'm just talking about a process as much as possible.
We can use the pipeline provided by the Shell to implement data transfer between commands. If the data for the continuous filtering of the time, as far as possible to the one-time filtering more commands in the front, this reason all understand? Reduce the size of data delivery.
Finally, I would like to say that the pipeline is also as little as possible, although the pipeline than normal with the same direction IO faster than a few orders of magnitude, but it is also necessary to consume additional resources, well design your code to reduce the cost of it. Like sort | Uniq command, you can use Sort-u to implement it completely.
4, say again the sequence execution of the shell script program. This optimization depends on whether your system load reaches the limit, and if your system has a higher line of command execution load, you don't need to do a parallel transformation of the shell script program. Here's an example, if you want to emulate this optimization, make sure your system has load space. For example, now there is a program:
Supportdatacommand1
Supportdatacommand2
Supportdatacommand3
Supportdatacommand4
Supportdatacommand5
Supportdatacommand6
Need13datacommand
Need24datacommand
Need56datacommand
The main idea is that there are 6 commands to provide data at the front, followed by 3 commands that need data, the first command that requires data requires data 13, the second requires 24, and the third requires 56. But normally the shell executes these commands sequentially, from Supportdatacommand1 to Need56datacommand. Does this process look like you have a lot of pain? Obviously can do this piece better, the egg pain program can be so modified:
Copy Code code as follows:
Supportdatacommand1 &
Supportdatacommand2 &
Supportdatacommand3 &
Supportdatacommand4 &
Supportdatacommand5 &
Supportdatacommand6 &
#2012 -02-22 PS: the loop here to determine whether the background command is completed is a problem, pidnum to the end of the ring is still 1 will not get 0 value, the specific solution to see the appendix, because there are explanations, it is not in this # added and modified.
While True
Todo
Sleep 10s
Pidnum= ' Jobs-p | Wc-l '
If [$pidnum-le 0]
Then
echo "Run Over"
Break
Fi
Done
Need13datacommand &
Need24datacommand &
Need56datacommand &
Wait
...
Can resemble the above modification. After this transformation, the feeling of egg pain is much more relieved. But still feel not very carefree, well, we can be a little more fun (I mean the program ...) ), you can do something like this:
Copy Code code as follows:
For ((i=0;i<1;i++));d o
{
Command1
Command2
}&
Done
For ((i=0;i<1;i++));d o
{
command3&
command4&
}&
Done
For ((i=0;i<1;i++));d o
{
Command5 &
Command6 &
If 5 6 completed ...
Command7
}&
Done
Such a transformation would allow the forward and backward commands to be executed together in a for loop, so that the three for loops are actually executed in parallel. And then for the command inside the For loop you can also transform or embed 2 of this parallel for loop in the same way that transformation 1 is possible, and the key is to see your imagination. Well? Oh, no, the key is to see what kind of a base-friend relationship is between these commands. It's OK to put a room on the connection, and you don't have to worry about the rest. Hey ~ ~
In fact, this optimization really needs to look at the system load.
5, about the shell command understanding. This item depends on experience, because there seems to be no relevant books to see, if anyone knows there, please recommend to me, I will often thank AH.
For example: Sed-n ' 45,50p ' and Sed-n ' 51q;45,50p ', which is also read from 45 to 50 rows, the latter is also, but the latter to 51 lines to perform the exit SED command, to avoid subsequent operations read. If the target file is large, the rest of you understand.
There are also sed ' s/foo/bar/g ' and sed '/foo/s/foo/bar/g '
SED supports the use of regular matching and substitution, in order to consider the need for string substitution, not to add address to improve speed. example, by adding a judgment logic, "prior matching" instead of "direct replacement", because SED retains the previous regular match environment, does not produce redundant regular match, so the latter has higher efficiency. This two-point optimization of the SED command is also mentioned in the SED instructions.
There are similar sort if the numbers are as far as possible with the-n option; There are also statistics on the number of rows, if each row of data in the case of the same number of bytes can be used to check the file size and then divided by the size of each row of the number of trips, and avoid direct use of wc-l such commands; and find out the data, Don't go directly to the-EXEC option, if the data size is good, but if you find out thousands of data or more, you will be crazy, no, the system will be crazy, because each row of data will produce a new process, you can find ... | Xargs ... and ... (If you also know similar to the efficiency of the situation please tell me to make progress together!) )
iii. better choices about optimization
A better way to improve the efficiency of shell scripting is to ... It is...... It is...... Well, just try to use the shell as little as possible (don't hit me!!!) Here are some official Debian statistics on the efficiency of the various languages on the Linux system, we are in C + + as the baseline (System specification: x64 Ubuntu™intel®q6600®quad-core):
The view methods of these graphs, such as the first diagram Java and C + + program Efficiency comparison diagram, a total of three parts, is time, memory, code comparison, if it is C++/java, that is, C + + to do the comparison of the numerator, Java to do comparison of the denominator, if the bar on the side Description of the program on the other side of the use of time or memory or code more, how much more specific to see how much longer long strip. Each section has more than one strip of graphics, each representing the tests that are performed when the program handles different aspects of the task. For example, the first, C + + and Java in the environment in most cases time is similar, even Java-server has a slight advantage, the memory of C + + has a great advantage, can use much less than Java content to do the same thing, but the encoding of C + + is a little bit more. The following figure is similar.
I can see from the above diagram that C + + in time and space for Python, Perl, PHP has an absolute overwhelming advantage, but relatively high coding. With Java than only the advantage of memory usage. But this one is mainly for the shell, but, again, the Debian official website does not include the shell script in the statistical scope of efficiency comparisons!!! Still, we know that Python, Perl, and PHP are known to have a clear advantage over the shell in terms of efficiency, so if you're not satisfied with your shell script after all the optimizations you've provided, you can try a different language.
But we are often not so easy to discard such a convenient and simple way of processing data, there can be a compromise method, you first use time to test the time of each shell script command time, for particularly time-consuming, particularly intolerable command efficiency using C + + program processing, Let your shell script call this C + + program for local data processing, so the compromise seems to be acceptable?
Four, finally said this is not dare to call comprehensive or detailed article, is my shell to learn and practice some of the experience, I hope to have expert advice. I also hope to help new students to enter this field. After a new experience to add it.
Thank the authors of this article for their posts.
2012-02-22 PS: Cyclic detection of whether the background command is the end of the judgment modified:
There are two solutions for the moment (no explanation, no clear reason):
1,
Copy Code code as follows:
Sleep 8 &
Sleep &
While True
Todo
Echo ' Jobs-p | Wc-l '
Jobs-l >> Res
Sleep 4
Done
2, check the remaining number of statements changed into Jobs-l |grep-v "done" |wc-l
The first solution is to perform more jobs, which can be explained in order to eliminate the end result, but this explanation is not workable, because the loop is always executed and has been performed many times in Echo, not once.
The second option is to filter out Jobs ' final output and do this statement. To get around the problem is the expected result.
Personal feel bash interpreter optimized out of jobs query command without background command, if it is optimized it should have an empty return, WC can still get 0 results ah. So this problem can not find a specific reason, if you know please tell me, thank you very much ... Here first thanks to just do Shell group Eric Silent Bandits GS Three people, thank you very much for your help.
Here are two methods that are not good, just wonder why not, and how to explain the line. Later knew to use wait order to solve all, delay so much time still use unwise method.
Reprint Please specify: The San Jiang Xiao du kind Oh!
