Getting Started with Linux-7 Linux pipelines, redirects, and text processing

Linux pipelines, redirects, and text processing

• 1 Linux Multi-command collaboration: Pipelines and redirects
  • Pipelines and redirects
• 2 Linux command line Text Processing tool
  • File browsing
  • Search-grep based on keyword
  • Column-based processing of text-cut
  • Text Statistics-WC
  • Text Sort-sort
  • Delete duplicate rows
  • Text comparison-diff
  • Check spelling-aspell
  • Working with text content-tr
  • Search for replacement-sed

An important idea of open source: Don't reinvent the wheel .
Many open source software is the re-grouping of existing software, code, and functionality.
"The Cathedral and the Bazaar"

1 Linux Multi-command collaboration: pipelines and redirect pipelines and redirects

Linux, most of the commands are simple, often only one or a few simple functions, we can combine the commands of different functions together to achieve the purpose of a complex function.
In Linux, almost all commands are returned in plain text, whereas plain text data is the input format for most commands, which makes collaboration possible.
The command behavior of Linux we provide pipelines and redirection mechanisms, and multi-command collaboration is done through pipelines and redirects.

• The command line shell data flow has the following definitions:

name	Description	numbering	default
Stdin	Standard input	0	Keyboard
STDOUT	Standard output	1	Terminal
STDERR	Standard error	2	Terminal

The command receives the parameter or data through the stdin, stdout the output or passes the STDERR output error.

• Pipeline and redirection to control the data flow of the CLI
• Pipelines are often used to combine different commands for more complex functions
• Redirection is typically used to save output or error messages for a command

category	Key Words	definition	Example
redirect	>	redirect stdout to file (overwrite)	echo "linuxcast.net" > outfile
redirect	>>	redirect stdout to file (append)	echo "linuxcast.net" >> outfile
redirect	2>	redirect stderr to file (overwrite)	echo "linuxcast.net" 2> outfile
redirect	2>&1	Combine stderr with stdout	echo "linuxcast.net" 2>&1 outfile
redirect	<	Stdin	grep linuxcast < /etc/passwd
Pipeline	|	StdOut a command as a stdin of another command	ls -l | grep linuxcast

A more complex example: Find the video file of the user Linuxcast, redirect the lookup error to null (discard)
find / -user linuxcast 2> /dev/null | grep Video

2 Linux command line Text Processing tool file browsing

• cat/more/less/head/tail

Search-grep based on keyword

This command uses regular expressions to search for a specified plain text file or plain text (pipe)

• -iIgnore case
• -nNumber of rows in the actual result (file)
• -vOutput lines without keywords (inverse selection)
• -AxThe output contains the specified number of rows after the row of the result
• -BxThe output contains the specified number of rows before the row of the result

Column-based processing of text-cut

• -dSpecify the split character, which is tab by default
• -fSpecify the column number of the output
• -cCut based on character, fixed length

Cases:
cut -d: -f1 /etc/passwdTo : display the first column as a separator
grep linuxcast /etc/passwd | cut -d: -f3UID of output user linuxcast (third column)
cut -c2-6 /etc/passwd第2-6个 characters

Text Statistics-WC

• -lCount Rows Only Line
• -wCount only word Word
• -cCount bytes Only
• -mCount characters only

Text Sort-sort

• -rMake a flashback sort
• -nSort based on a number
• -fIgnore case
• -uDelete duplicate rows
• -tcUse C as a delimiter to sort columns
• -kxSpecifies which column to sort on when sorting based on the specified character is divided into columns

Delete duplicate rows

• sort -uDelete duplicate rows when sorting
• uniqDelete a duplicate adjacent row

Text comparison-diff

• -iIgnore case
• -bIgnore the change in the number of spaces
• -uUnified display of comparison information (typically used to generate patch files)
  diff -u linuxcast linuxcast-new > final.patch

Check spelling-aspell

• aspell check linuxcast
• aspell list < linuxcast

Working with text content-tr

Translate

• tr -d ‘TMD‘ < linuxcastDelete keyword
• tr ‘a-z‘ ‘A-Z‘ < linuxcastConvert case

Search for replacement-sed

Working with regular expressions to work with text files

• sed ‘s/linux/unix/g‘ linuxcastReplace Linux for UNIX, /g replace all
• sed ‘1,50s/linux/unix/g‘ linuxcastReplace 1-50 lines
• sed -e ‘s/linux/unix/g‘ -e ‘s/nash/nash_su/g‘ linuxcastMultiple regular expressions
• sed -f sededit linuxcastTo save a match expression in an Sededit file

Getting Started with Linux-7 Linux pipelines, redirects, and text processing