Dialog Unix: Hello, Shell

Dialogue Unix: Hello, shell!

One of the most peculiar and prominent features of UNIX systems is itsCommand LineFunction. You only need to enter a small amount of text that contains a certain logical relationship, you can use the command line to combine a limited Unix utility group into an unlimited amount of real-time available data conversion.

For example, to find a list of unique file names in the folder hierarchy under the current working directory, you can enter the following command at a shell prompt:

find . -type f -print | sort | uniq

This command line combines three different utilities:

• findPerforms a deep search on the specified directory. In this example.OrPoint(Representing the current working directory. Here,-type fTellfindOnly search for text files.

• sortAs the name suggests, the list is processed and a new list is generated in alphabetical order.

• uniq(Read as "unique"), scan the list, and compare Adjacent Elements in the list to remove any repeated items. For example, assume that you have the following list:

  Listing 1. List Example
  

  GrouchoGrouchoChicoChicoGrouchoHarpoZeppoZeppo

  uniqThe list can be reduced:

  List 2. uniq command
  

  GrouchoChicoGrouchoHarpoZeppo

  However, if you first sort the initial list of Marx Brothers (rename the names that appear multiple times during continuous running), rununiqThe following results are generated:

  Listing 3. Running uniq
  

  ChicoGrouchoHarpoZeppo

To learn morefind,sortAnduniqFor more extended features, seemanPage.

Back to Top

Input data, output data, and all data

Independent usefindAlways use the content of the file system as the input data. HoweversortAnduniqYou needStandard Input Device(Stdin) request data input. In most cases, you use the keyboard as stdin: for example, you need to enter the data rows to be sorted.

By default,findInStandard output device(Stdout, usually your terminal window.sortAnduniqOutput to stdout.

To describe stdin and stdout, you can enter the following text in the terminal window (assuming the percentage sign (%) For your shell prompt ):

Listing 4. stdin and stdout

% sortmustachehornhatControl-D

sortRead the three lines of text you entered from stdin, sort it, and write the result to stdout.Figure 1As shown insortAnd most Unix Command Line Utilities.

Figure 1. Typical Unix Command Line Utilities read from stdin and write to stdout

Some utilities, suchfindDoes not read content from stdin. They read the data to be processed from system resources (such as a file system or system kernel) and then write the results to stdout. Intuitive viewingfindSee the followingFigure 2.

Figure 2. Some utilities read data from system resources and write the results to stdout

In addition to stdin and stdout, Unix Commands also output generated error messages to a special exit for diagnosis, which is generally not mandatory. This outlet is calledStandard Error Device(Generally referred to as stderr ).Figure 3Shows the simple command line for running the utility.

Figure 3. Unix Command generation error and output to special channel, that is, standard error device

For exampleFigure 3As shown in, most Unix Commands read the input from the terminal, send the result to the terminal, and print the error to the terminal. By default, unless otherwise specified, your terminal is both the data source of stdin and the output target of stdout and stderr.

Back to Top

Data Transmission

However, you can change the source of stdin and the target of stdout and stderr. You can force stdin to read from text files, devices (such as probes connected to a computer), or network connections. Similarly, you can send output results to a file, device, or network connection. In UNIX, all resources are regarded as files, so a certain source or target is easily accepted or generated as another source or target.

Changing the source and target of process data is calledRedirection. You can redirect stdin to read data from a file or other sources. You can also redirect stdout and stderr to write data to a location other than the terminal window. In many cases, such as the initialfindYou can also redirect utilities to receive and generate required data from other tools. This isMPs queue(|. You can generate a process link through pipelines in the command to send the data of a command to the next command, similar to transmitting water from the water heater to the sink through a copper tube.

Figure 4As shown infind . -type f -print | sort | uniqCommand.

Figure 4. Three utilities linked through pipelines

findThe command stdout becomesuniqStdin, and thenuniqStdout becomessortStdin. Finally,sortPrint the result to its standard output device, that is, the connected terminal window. The stderr of these commands is not redirected, so all three utilities print the error message to the terminal. (Error messages from the three utilities will be mixed together, but they must be in the correct order .)

If necessary, you can further expand the pipelineuniqTo another utility. You only need to use another pipeline to expand the conversion. For example, you can add| lessTo uselessPagination of output results, or you can add| wc -lTo count the number of unique file names. (wcIsWord CountAbbreviation,wcCan be used to count characters, words, and rows .)

You can also use>To save the output results of the entire command sequence to a file (this will overwrite the existing file content ). You can use>>To output the resultAdditionalAfter the existing file (if the file does not exist, create a new file ).

Another useful redirection is<.Figure 5Shows how to redirect stdin to read from a file. CommandsortReads the word list from a specified file and sorts words alphabetically.

Figure 5. Redirect standard input to Read File Content

You often need to capture stdout and stderr. For example, if you are running a large data mining task, you may need to check the intermediate output during execution and any possible errors. You can use some variants of the redirection syntax to implement this function:|&,>&,>>&You can perform pipeline, creation, and additional functions for stdout and stderr respectively.Figure 6Shows how to merge stdout and stderr into a single output stream.

Figure 6. Merge standard output and standard error Devices

Back to Top

Z shell Introduction

Including the Bourne shell (bash) And Korn shell (kshMost of the modern Unix shells in the kernel support the redirection function mentioned here, although the specific syntax may be slightly different in these shells. (View your shell document for details ).

Most operators in redirection have been used for at least 25 years in all UNIX shells. However, most shells do not provide new features or adopt new methods to apply redirection. For example, most shells can only redirect input to a single file, so you must useteeAnd other tools to output to multiple targets. (Similar to tee ),teeOnly one or two outputs are supported .) Here is a usagebashAs an example of shell (command line interpreter:

Listing 5. Bash example

bash$ lstellmebash$ cat tellmeecho Your current login, working directory, and system are...whoamipwdsystemnamebash$ bash < tellme |& tee logYour current login and working directory are...strike/home/strikebash: systemname: command not foundbash$ lstellme logbash$ cat logYour current login and working directory arestrike/home/strikebash: systemname: command not found

Although Unix shell has a high specificity and usually uses a keyboard for interaction, some shells suchbashAnd so on. (In fact, stdin is also a file .) In the previous section, the phrasebash < commandsTellbashRun the command list found in the Tellme file. Phrase|&tee logSetbashStdout and stderrto are redirectedteeUtility, which prints its stdin to stdoutAndFile log.

However, if you want to usebashTo process multiple files, what should I do?cat file1 file2 file3 | bashIs a feasible method, which may be the only method, becausebashDoes not supportbash < file1 < file2 < file3.

And,bashThe output cannot be redirected to multiple targets. For example, you canbashEnter commands in the command linebighairyscript > ~/log | mail -s "Important stuff" team.

However, in a relatively new shell such as Z shell (zsh; SeeReferences), You can process multiple inputs and outputs in the same command line. For example, you can use the following command to save stdout to a file named log and send it to yourself by email:

Listing 6. Z Shell

zsh% bash < tellme > log | mail -s "Who you are" 'whoami'bash: line 4: systemname: command not foundzsh% <logYour current login, working directory, and system are...strike/home/strike

(Phrase“whoami”Run commandswhoamiInsert the result of the command to the location of the phrase. It is similar to running a short shell command before running other parts of the command line .)

Now we analyze the previous command from left to right.bashCommand to create the file log and send the stdout of the command found in the Tellme file to yourself. Because stderr fails>Or pipeline, so the error message is printed to stdout. Command<logIs another Z shell shortcut, which correspondscatSame. (Therefore, the command> fileEquivalentcat > file.)

Z shell can also process multiple input redirects. Z shell command linecat < file1 < file2 < file3Equivalentcat file1 file2 file3. Obviously, the original syntax is more complex than the latter. In general, multiple stdout redirects are more commonly used. However, if the utility you want to run does not accept multiple input parameters, you can use multiple input redirection of Z shell.

Z shell also has other new features, including betterGlobbing(Wildcard matching), advanced mode matching, and extended commands automatically complete the system, reducing your character input in the command line. The subsequent two articles in this series will further explore Z shell.

Back to Top

Shell skills

With some powerful command line combinations, You can significantly improve your work efficiency. These commands can work in all the shells, not justzsh.

• UsetarCreate a complete copy of any directory including symbolic links:

  tar cf - /path/to/original | / (mkdir -p /path/to/copy; cd /path/to/copy; tar xvf -)

  FirsttarCommand to archive the directory/path/to/original and write the archive file to stdout, create (c(-) Indicates stdout. The command in parentheses is a subshell: The command in subshell does not affect the current shell environment.mkdir -pCreate a specified directory, including any intermediate directory to be created;cdCommand to switch to the new directory. SecondtarCommand to read and expand the archive file from stdin, expand (x.

• You can useless -O file .-OOption will copy stdin to the specifiedfile . For example:

  sort /etc/aliases | less -Osorted

• If the directory contains thousands of files, your shell (includingzsh, Depending on the number of files and their names) You may not be able to use wildcard matching to list all files, because the command line usually has a certain number of characters. Therefore, similar to the following shell script:

  foreach i (*)...end

  Execution may fail. (When the length of the allowed command line is exceeded, you may seeLine length exceededMessages .) If this type of error occurs, you can use pipelines.xargsUtility.xargsCommand to read data from the MPs queue and run the specified command for reading content from each line.

  For example, if you want to find all webpages on the server that reference www.example.com, you can use the following command line:

  % find / -name '*html' -print / | xargs grep -l 'www.example.com' / | less -Opages

  xargsReceive fromfindFile Name and run it repeatedlygrep -lTo process each file, regardless of the number of files. (grep -lAfter a matching item is found, print the file name and stop further matching in the file .)lessYou can paging the results and save the list on the specified page of the file. The command returns a list of file names containing the string "www.example.com.