Linux command (UNIQ)

Reprint Address: http://blog.51yip.com/shell/1022.html

Instance details Linux go down except duplicate line command Uniq

One, what's uniq for?

The duplicate lines in the text are basically not what we want, so we're going to get rid of them. There are other commands in Linux that can remove duplicate lines, but I think Uniq is a convenient one. When using Uniq, pay attention to the following two points

1, when working with text, it is generally used in combination with the sort command, because Uniq does not check for duplicate rows unless they are adjacent rows. If you want to sort the inputs first, use Sort-u.

2, for text operations, if the field is preceded by a null character (usually including spaces and tabs), and then a non-null character, the null character before the character in the field is skipped

Two, uniq parameter description

1[Email protected] ~]$ Uniq-- Help2 usage: uniq [options] ... [File]3 filters adjacent matching rows from the input file or standard input and writes to the output file or standard output. 4 5 when no option is attached, the matching row is merged at the first occurrence. 6 7 The long option must use parameters that are also required for short options. 8-C,--count//precede each line with a prefix number that indicates the number of occurrences of the corresponding line item9-D,--repeated//only duplicate rows are outputTen-D,--all-repeated//only duplicate rows are output, but several rows are output One-F,--skip-fields=n//-f ignored number of segments,-F 1Ignore first paragraph A-I.,--ignore-case// Case insensitive --S,--skip-chars=n//root-f a bit like, but-S is ignored, the number of characters behind-s 5 ignores the next 5 characters --U,--unique//after removing the duplicates, all shows up, the root MySQL distinct function is a bit like the-Z,--zero-terminated end lines with 0 byte, notNewLine --W,--check-chars=n//No comparison of the contents of the nth character of each line ---HELP//Display this help message and exit ---version//Display version information and exit

Three, Test text file Uniqtest

1This isa test2This isa test3This isa test4 I am tank5 I love tank6 I love tank7This isa test8Whom has aTry  9WhoM has aTry  TenYou have aTry   One I want to abroad A Those is good men -We are good Men

Four, detailed examples

As we can see from the example, a feature of uniq that checks for duplicate rows only checks adjacent rows. Duplicate data, there must be many are not adjacent together.

[[email protected] mytest]$ Uniq-c uniqtest       is  a test       1 I am tank       2  I love Tank       is a test           // and the       first line is a repeating try       try       1  try       1 I want to abroad       1 Those is good       men 1 We is good Men  

This will solve the problem mentioned in the last example.

    [[email protected] mytest]$ sort uniqtest |uniq-c       try       1 I am tank       2 I love Tan K       1 I want to abroad       was               A test 1 those is good Men 1< c16> We is good       mentry       1 try    

Uniq-d only duplicate rows are displayed

[Email protected] mytest]$ uniq-d-c uniqtest   is  a test   

uniq-d only displays duplicate rows, and displays the repeating lines. He can't use with-c

[Email protected] mytest]$ Uniq-D uniqtest  is  a test  is  a test  
     is a test  I love tank  

Here those only one row, the display is repeated, this is because, F 1 ignores the first column, check the repetition from the second field start.

[[email protected] mytest]$ uniq-f 1-c uniqtest   is  a test   1 I am tank   2< c7> I love tank   are  a test   try   1 You  try   1 I want to abroad    2 Those is good Men   //Only one row, showing two rows

When checking, case insensitive

[Email protected] mytest]$ uniq-i-c   uniqtestis  a test   1 I am tank   2
   
     I love tank   is
     
     a test   
    try  // 
    one uppercase, one lowercase    1 You  
    try   1
     I want to abroad   1
       Those is goodmen 1 we is good Men
   

When checking, do not consider the first 4 characters, so whom have a try and you have a try is the same.

[[email protected] mytest]$ uniq-s 4-C uniqtest  is  a test  1 I am tank  2< c7> I love tank  is  a test  try   // Root on an example what's the  difference 1 I want to abroad   1 Those is good   men

to repeat the items, and then show them all.

    [Email protected] mytest]$ Uniq-u uniqtest      I am tank      is  a      testTry Try youtry I want to      abroad those is good                Men      We is good   men

The 2nd character of each line after the content is not checked, so I am tank root I love tank is the same.

[[email protected] mytest]$ uniq-w 2-C uniqtest   is  a test   3 I am tank   
   
    is 
     a test   
    try   
    try   1 You  
    try   1
     i want To abroad   1
     Those is   good men
   

Linux command (UNIQ)