Ten habits of UNIX experts-overcome bad UNIX usage patterns

Level: Intermediate
By Michael Stutz
From: IBM developerworks China
Link: http://www.ibm.com/developerworks/cn/aix/library/au-badunixhabits.html

When you often use a system, it is often in a fixed usage mode. Sometimes, you do not get into the habit of doing things in the best way possible. Sometimes, your bad habits may even lead to confusion. One of the best ways to correct such defects is to consciously adopt the good habits of resisting these bad habits. This article presents 10 Unix Command Line habits worth using-a good habit of helping you overcome many common usage quirks and improve the efficiency of command line work in this process. These 10 good habits are listed below, and a more detailed description is given later.

Ten good habits to use are:

1. Create a directory tree in a single command.
2. Change the path. Do not move the archive.
3. Combines commands with control operators.
4. Exercise caution when referencing variables.
5. Use escape sequences to manage long input.
6. Group commands in the list.
7. InfindExternal Usexargs.
8. Know whengrepCount should be executed-When should I bypass.
9. Matches some fields in the output, not just the rows.
10. StopcatUse pipelines.

Listing 1 demonstrates one of the most common UNIX bad habits: defining a directory tree at a time.

~ $ mkdir tmp ~ $ cd tmp ~/tmp $ mkdir a ~/tmp $ cd a ~/tmp/a $ mkdir b ~/tmp/a $ cd b ~/tmp/a/b/ $ mkdir c ~/tmp/a/b/ $ cd c ~/tmp/a/b/c $

UsemkdirOf-pIt is much easier to create all parent directories and Their subdirectories in a single command. However, even for administrators who know this option, they are still bound to gradually creating sub-directories at each level when creating sub-directories on the command line. It is worth taking the time to consciously develop this good habit:

~ $ mkdir -p tmp/a/b/c

You can use this option to create the entire complex directory tree (which is ideal for use in scripts), instead of simply creating a simple hierarchy. For example:

~ $ mkdir -p project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat/a}

In the past, the only excuse for defining directories separately was yours.mkdirThe implementation does not support this option, but it is no longer the case in most systems. IBM, Aix,mkdir, GNUmkdirThis option is now available for other systems that comply with the single UNIX specification.

You can usemkdirhierScript (see references), which executes the same functionmkdirPackaging:

~ $ mkdirhier project/{lib/ext,bin,src,doc/{html,info,pdf},demo/stat/a}

Another bad usage mode is to move the. tar archive file to a directory, because this directory is exactly the directory where you want to extract the. tar file. In fact, you do not need to do this. You can decompress any. tar archive file to any directory as you like-this is-COption. When extracting an archive file, use-COption to specify the directory in which the file is to be decompressed:

~ $ tar xvf -C tmp/a/b/c newarc.tar.gz

Instead of moving an archive file to the location where you want to decompress it, switch to the directory, and then decompress it.-CThis is especially true when the archive file is located somewhere else.

You may already know that in most shells, you can combine commands on a single command line by placing a semicolon (;) between commands. This Semicolon is ShellControl Operators, Although it is useful for concatenating discrete commands on a single command line, it does not apply to all situations. For example, if you use a semicolon to combine two commands, the correct execution of the second command is completely dependent on the successful completion of the first command. If the first command does not exit as expected, the second command will still run -- the result will cause failure. Instead, you should use more appropriate control operators (some operators described in this article ). As long as your shell supports them, it is worth developing the habit of using them.

Use&&Control Operators to combine two commands to facilitateOnly whenThe first command returns the zero exit status before running the second command. In other words, if the first command runs successfully, the second command runs. If the first command fails, the second command does not run at all. For example:

~ $ cd tmp/a/b/c && tar xvf ~/archive.tar

In this example, the archived content is extracted ~ /Tmp/A/B/C directory, unless the directory does not exist. If the directory does not existtarThe command does not run, so it does not extract any content.

Similarly,||The control operator separates two commands and runs the second command only when the first command returns a non-zero exit state. In other words, if the first commandSuccessfulThe second command does not run. If the first command fails, the second commandYesRun. This operator is usually used to test whether a given directory exists. If the directory does not exist, create it:

~ $ cd tmp/a/b/c || mkdir -p tmp/a/b/c

You can also combine the control operators described in this section. Each operator affects the execution of the final command:

~ $ cd tmp/a/b/c || mkdir -p tmp/a/b/c && tar xvf -C tmp/a/b/c ~/archive.tar

Always use shell extensions and variable names with caution. Generally, it is best to include variable calls in double quotation marks unless you have enough reasons for not doing so. Similarly, if you use a variable name directly after a letter or number, make sure that the variable name is included in square brackets, to separate it from the surrounding region. Otherwise, shell interprets the trailing text as part of the variable name -- and may return a null value. Listing 8 provides examples of variables for reference and non-reference and their impact.

~ $ ls tmp/ a b ~ $ VAR="tmp/*" ~ $ echo $VAR tmp/a tmp/b ~ $ echo "$VAR" tmp/* ~ $ echo $VARa ~ $ echo "$VARa" ~ $ echo "${VAR}a" tmp/*a ~ $ echo ${VAR}a tmp/a ~ $

You may have seen a code example that uses a backslash (/) to extend a long line to the next line, in addition, you know that most shells treat the content you typed on subsequent rows connected by backslash as a single long line. However, you may not use this function in the command line as usual. If your terminal cannot properly process multi-line loopback, or your command line is smaller than usual (for example, when there is a long path in the prompt), The backslash is particularly useful. The backslash is also useful for understanding the meaning of long input lines, as shown in the following example:

~ $ cd tmp/a/b/c || / > mkdir -p tmp/a/b/c && / > tar xvf -C tmp/a/b/c ~/archive.tar

Alternatively, you can use the following configurations:

~ $ cd tmp/a/b/c / > || / > mkdir -p tmp/a/b/c / > && / > tar xvf -C tmp/a/b/c ~/archive.tar

However, when you divide an input row into multiple rows, shell always treats it as a single continuous row because it always deletes all backslash and extra spaces.

Note:In most shells, when you press the up arrow, the entire multi-line input is repainted to a single long input line.

Most shells have methods to group commands in the list so that you can pass their aggregate output down to a pipe or redirect any part or all of the streams to the same place. Generally, you can run a command list in a subshell or a command list in the current shell.

Use parentheses to include the command list in a single group. In this way, the command will be run in a new subshell, And you can redirect or collect the output of the entire group of commands, as shown in the following example:

~ $ ( cd tmp/a/b/c/ || mkdir -p tmp/a/b/c && / > VAR=$PWD; cd ~; tar xvf -C $VAR archive.tar ) / > | mailx admin -S "Archive contents"

In this example, the archive content is extracted to the tmp/A/B/C/directory, and the output of the group command (including the list of extracted files) is sent to the address by mail.admin.

When you redefine the environment variables in the command list, and you do not want to apply those definitions to the current shell, subshell is preferred.

Enclose the command list in braces ({})CurrentRun in shell. Make sure that there are spaces between the brackets and the actual commands. Otherwise, shell may not be able to correctly interpret the brackets. In addition, make sure that the last command in the list ends with a semicolon, as shown in the following example:

~ $ { cp ${VAR}a . && chown -R guest.guest a && / > tar cvf newarchive.tar a; } | mailx admin -S "New archive"

UsexargsTool as a filter to make full usefindCommand selection output.findRunning usually provides a list of files that match certain conditions. This list is passedxargsThe latter then uses the file list as a parameter to run some other useful commands, as shown in the following example:

~ $ find some-file-criteria some-file-path | / > xargs some-great-command-that-needs-filename-arguments

However, do notxargsOnlyfindIs one of the underutilized tools. When you get into the habit of using it, you will want to perform all the experiments, including the following usage.

In the simplest form of calling,xargsLike a filter, it accepts a list (each member is on a separate row) as input. This tool places those members on rows separated by a single space:

~ $ xargs a b c Control-D a b c ~ $

You can sendxargsTo output the output of any tool for the file name, so as to obtain the parameter list for some other tools that accept the file name as a parameter, as shown in the following example:

~/tmp $ ls -1 | xargs December_Report.pdf README a archive.tar mkdirhier.sh ~/tmp $ ls -1 | xargs file December_Report.pdf: PDF document, version 1.3 README: ASCII text a: directory archive.tar: POSIX tar archive mkdirhier.sh: Bourne shell script text executable ~/tmp $

xargsThe command is not only used to pass the file name. You can also use it whenever you need to filter text into a single row:

~/tmp $ ls -l | xargs -rw-r--r-- 7 joe joe 12043 Jan 27 20:36 December_Report.pdf -rw-r--r-- 1 / root root 238 Dec 03 08:19 README drwxr-xr-x 38 joe joe 354082 Nov 02 / 16:07 a -rw-r--r-- 3 joe joe 5096 Dec 14 14:26 archive.tar -rwxr-xr-x 1 / joe joe 3239 Sep 30 12:40 mkdirhier.sh ~/tmp $

Technically speaking, usexargsThere is little trouble. By default, the end string of a file is an underscore (_). If this character is sent as a single input parameter, all subsequent content is ignored. To prevent this situation, you can use-eIt completely disables the end string without parameters.

Avoid addinggrepSendwc -lTo count the number of output rows.grepOf-cThis option provides a count for the rows that match a specific pattern, and is generally compared towcFaster, as shown in the following example:

~ $ time grep and tmp/a/longfile.txt | wc -l 2811 real 0m0.097s user 0m0.006s sys 0m0.032s ~ $ time grep -c and tmp/a/longfile.txt 2811 real 0m0.013s user 0m0.006s sys 0m0.005s ~ $

In addition to the speed factor,-cOption is a good way to execute the count. For multiple files-cOptionalgrepReturns the individual count of each file, one count per line, andwcProvides the total number of combinations of all files.

However, regardless of the speed, this example shows another common error to be avoided. These counting methods only provideNumber of rows that contain the matching mode-- If it is the result you want to search for, there is no problem. However, when a row has multiple instances in a specific mode, these methods cannot provide you with actual matchingInstance quantity. In the final analysis, to count instances, you still need to usewcTo count. First, use-oOption (If your version supports it) to rungrepCommand. This optionOnlyOutput the matching mode. Each line has one mode without losing the trip itself. However, you cannot-cYou must usewc -lTo count rows, as shown in the following example:

~ $ grep -o and tmp/a/longfile.txt | wc -l 3402 ~ $

In this examplewcCompared with the second callgrepInsert a virtual mode (for examplegrep -c.

If you only want to matchSpecific fieldsIn, suchawkAnd other tools are bettergrep.

The following simplified example shows how to list only files modified on January 1, December.

~/tmp $ ls -l /tmp/a/b/c | grep Dec -rw-r--r-- 7 joe joe 12043 Jan 27 20:36 December_Report.pdf -rw-r--r-- 1 root root 238 Dec 03 08:19 README -rw-r--r-- 3 joe joe 5096 Dec 14 14:26 archive.tar ~/tmp $

In this example,grepFilters rows and outputs the modified Date and nameDecAll files. Therefore, files such as december_report.pdf match, even if they have not been modified since January. This may not be the expected result. To match the pattern in a specific field, it is best to useawk, A relational operator matches the exact field, as shown in the following example:

~/tmp $ ls -l | awk '$6 == "Dec"' -rw-r--r-- 3 joe joe 5096 Dec 14 14:26 archive.tar -rw-r--r-- 1 root root 238 Dec 03 08:19 README ~/tmp $

How to UseawkFor more information, see references.

grepA common basic usage error of iscatOutputgrepTo search for the content of a single file. This is absolutely unnecessary. It is a waste of time, becausegrepSuch a tool accepts the file name as a parameter. You do not need to usecat, As shown in the following example:

~ $ time cat tmp/a/longfile.txt | grep and 2811 real 0m0.015s user 0m0.003s sys 0m0.013s ~ $ time grep and tmp/a/longfile.txt 2811 real 0m0.010s user 0m0.006s sys 0m0.004s ~ $

This error exists in many tools. Most tools accept standard input with hyphens (-) as a parameter, even ifcatTo distributestdinAnd the parameters are usually invalid. Only when you use one of the multiple filtering optionscatIt is really necessary to execute the connection before the pipeline.

It is best to check any bad usage mode in your command line habits. Poor usage modes may speed up and lead to unexpected errors. This article introduces 10 new habits that can help you get rid of many of the most common usage errors. Developing these habits is a positive step to enhance your Unix Command Line skills.

Learning

• "Use free software within commercial Unix" (developerworks, November February 2006) is a short getting started book on the use of many open-source software on commercial dedicated UNIX implementations.

• "Working in bash shell" (developerworks, May 2006) provides an introductory tutorial on the popular bash shell.
• "Gawk Getting Started: awk language basics" (developerworks, November September 2006) describes how to use the awk language to manipulate text.
• "Hone your skills in building Regular Expression Patterns" (developerworks, July 2006) describes how to usegrep.
• Visit the developerworks AIX and Unix area to get the resources you need to improve your skills.
• Are you new to Aix and UNIX? For more information, visit the "getting started with AIX and Unix" page.
• Browse the technical bookstore to learn about these technical topics and other related books.

Obtain products and technologies

• To obtainmkdirhierYou can download a version from Haskell compiler.

Discussion

• Podcast: Listen to the podcast and stay in sync with IBM technical experts.

• Visit the developerworks blog to join the developerworks community.