Instance details Linux go down except duplicate line command Uniq

Address: http://blog.51yip.com/shell/1022.html

One, what's uniq for?

The duplicate lines in the text are basically not what we want, so we're going to get rid of them. There are other commands in Linux that can remove duplicate lines, but I think Uniq is a convenient one. When using Uniq, pay attention to the following two points

1, when working with text, it is generally used in combination with the sort command, because Uniq does not check for duplicate rows unless they are adjacent rows. If you want to sort the inputs first, use Sort-u.

2, for text operations, if the field is preceded by a null character (usually including spaces and tabs), and then a non-null character, the null character before the character in the field is skipped

Two, uniq parameter description

1[Email protected] ~]$ Uniq-- Help2 usage: uniq [options] ... [File]3 filters adjacent matching rows from the input file or standard input and writes to the output file or standard output. 4       5 when no option is attached, the matching row is merged at the first occurrence. 6       7 The long option must use parameters that are also required for short options. 8-C,--count//precede each line with a prefix number that indicates the number of occurrences of the corresponding line item9-D,--repeated//only duplicate rows are outputTen-D,--all-repeated//only duplicate rows are output, but several rows are output One-F,--skip-fields=n//-F Ignore number of segments,-F 1 ignores first paragraph A-I.,--ignore- Case       // Case insensitive --S,--skip-chars=n//root-F is a bit like, but-S is ignored, followed by how many characters-s 5 ignores the next 5 characters --U,--unique//after removing the duplicates, all shows up, the root MySQL distinct function is a bit like the-Z,--zero-terminated end lines with0 byte, not newline --W,--check-chars=n//No comparison of the contents of the nth character of each line ---help//Display this help message and exit ---version//display version information and exit

Where-Z doesn't know what's the use

Three, Test text file Uniqtest

1[Email protected] mytest]$ Uniq-C Uniqtest2      3  This  isa test3      1I am tank4      2I love tank5      1  This  isA test//and the first line is repeated6      1Whom has aTry  7      1WhoM has aTry  8      1You have aTry  9      1I want to abroadTen      1Those is good men One      1We are good Men

Four, detailed examples

[Email protected] mytest]$ Uniq-C Uniqtest3  This  isa test1I am tank2I love tank1  This  isA test//and the first line is repeated     1Whom has aTry       1WhoM has aTry       1You have aTry       1I want to abroad1Those is good men1We are good Men

As we can see from the above example, a feature of uniq that checks for duplicate rows only checks adjacent rows. Duplicate data, there must be many are not adjacent together.

[Email protected] mytest]$ sort uniqtest |uniq-C1WhoM has aTry       1I am tank2I love tank1I want to abroad4  This  isa test1Those is good men1We are good men1Whom has aTry       1You have aTry  

This will solve the problem mentioned in the last example.

    [[email protected] mytest]$ uniq-d-c uniqtest       3 This is a test        2 I love tank  

Uniq-d only duplicate rows are displayed

    [Email protected] mytest]$ Uniq-D uniqtest      This is  a test        This  is a test        This  is a test      I love tank      I love tank  

uniq-d only displays duplicate rows, and displays the repeating lines. He can't use with-c

[Email protected] mytest]$ uniq-f1-C Uniqtest3  This  isa test1I am tank2I love tank1  This  isa test2Whom has aTry       1You have aTry       1I want to abroad2Those is good Men//only one row, showing two rows

Here those only one row, the display is repeated, this is because, F 1 ignores the first column, check the repetition from the second field start.

[Email protected] mytest]$ Uniq-i-C Uniqtest3  This  isa test1I am tank2I love tank1  This  isa test2Whom has aTry  //one uppercase, one lowercase     1You have aTry       1I want to abroad1Those is good men1We are good Men

When checking, case insensitive

[Email protected] mytest]$ uniq-s4-C Uniqtest3  This  isa test1I am tank2I love tank1  This  isa test3Whom has aTry   //What is the difference between an example on the root    1I want to abroad1Those is good men1We are good Men

When checking, do not consider the first 4 characters, so whom have a try and you have a try is the same.

    [Email protected] mytest]$ Uniq-u uniqtest      I am tank      This is  a test       Try Try youtry I want to      abroad those is                Good Men      We were good   men

To repeat the items, and then show them all.

[Email protected] mytest]$ uniq-w2-C Uniqtest3  This  isa test3I am tank1  This  isa test1Whom has aTry       1WhoM has aTry       1You have aTry       1I want to abroad1Those is good men1We are good Men

The 2nd character of each line after the content is not checked, so I am tank root I love tank is the same.

Instance details Linux go down except duplicate line command Uniq