[Linux] Stream, pipe (Pipeline), filter-note

Flow (Stream)

1. A stream refers to a sequence of data elements that can be used.

2. Flow, can be imagined as the belt on the waiting for processing of goods, can also be imagined as a factory assembly line items.

3. Stream, can be unlimited data.

4. There is a function that processes this one stream at the same time producing another stream. This function is filtered (filter). Use Pipelines (Pipelie) to connect these features.

Unix Pipeline (Pipeline)

1. The pipeline is connected to the processing element, and the output of one processing element is the input to the next processing element.

2. The pipeline can speed up data processing.

2. The pipelines below Unix are illustrated below.

For example, list all the files in the current directory (ls-l), keep only the file name with the string key (grep key), and view it in the scrolling screen (less)

ls grep  Less

Filter

1. Refers to the program used to process the flow. Although filter can be used alone, they are often used in combination with pipelines.

2. The default is to obtain data from standard input and write to standard output. You can use the input operator < change the data source, through the output operator > change the Write target, by appending the operator >> append data.

3. Through the pipe symbol | , link multiple Filter.

4. Example. Get the data from/etc/passwd, and each row will be divided according to ":" and get the first word field (cut-d:-F 1/etc/passwd), lift the line output containing foo string (grep foo)

Cut 1 /etc/passwdgrep foo

5. Common Filter procedures are:

grep: From the text dataset, search for the line of the regular expression matched to, in one line.

Cat fruitlist.txtappleapplespineappleapple-Apple-fruitfruit-grep  Apple Fruitlist.txt appleapplespineappleapple-Apple-fruitfruit-apple

Cat: Read multiple files and write to standard output in turn.

Cat [Options] [File_names] > Newfile.txt

Cut: Extracts a portion of each row as output. Less use, examples slightly

Head: Displays the contents of the text file, or the beginning of the pipeline data.

Head  filename

Tail: Displays the contents of the text file, or the end of the pipeline data.

Tail  filename

Sort: Sorts the contents of a text file, or the pipe data, and outputs it. You can specify the row sequence

$CatZipcodeadam12345Bob34567Joe56789Sam45678Wendy23456   $ Sort-k 2n Zipcodeadam12345Wendy23456Bob34567Sam45678Joe56789

Resources

Stream (computing), Wikipedia

Pipeline (Unix), Wikipedia

Bubble (computing), Wikipedia

Filter, Wikipedia

[Linux] Stream, pipe (Pipeline), filter-note