Reprint Address: http://blog.51yip.com/shell/1022.html
Instance details Linux go down except duplicate line command Uniq
One, what's uniq for?
The duplicate lines in the text are basically not what we want, so we're going to get rid of them. There are other commands in Linux that can remove duplicate lines, but I think Uniq is a convenient one. When using Uniq, pay attention to the following two points
1, when working with text, it is generally used in combination with the sort command, because Uniq does not check for duplicate rows unless they are adjacent rows. If you want to sort the inputs first, use Sort-u.
2, for text operations, if the field is preceded by a null character (usually including spaces and tabs), and then a non-null character, the null character before the character in the field is skipped
Two, uniq parameter description
1[Email protected] ~]$ Uniq-- Help2 usage: uniq [options] ... [File]3 filters adjacent matching rows from the input file or standard input and writes to the output file or standard output. 4 5 when no option is attached, the matching row is merged at the first occurrence. 6 7 The long option must use parameters that are also required for short options. 8-C,--count//precede each line with a prefix number that indicates the number of occurrences of the corresponding line item9-D,--repeated//only duplicate rows are outputTen-D,--all-repeated//only duplicate rows are output, but several rows are output One-F,--skip-fields=n//-f ignored number of segments,-F 1Ignore first paragraph A-I.,--ignore-case// Case insensitive --S,--skip-chars=n//root-f a bit like, but-S is ignored, the number of characters behind-s 5 ignores the next 5 characters --U,--unique//after removing the duplicates, all shows up, the root MySQL distinct function is a bit like the-Z,--zero-terminated end lines with 0 byte, notNewLine --W,--check-chars=n//No comparison of the contents of the nth character of each line ---HELP//Display this help message and exit ---version//Display version information and exit
Three, Test text file Uniqtest
1This isa test2This isa test3This isa test4 I am tank5 I love tank6 I love tank7This isa test8Whom has aTry 9WhoM has aTry TenYou have aTry One I want to abroad A Those is good men -We are good Men
Four, detailed examples
As we can see from the example, a feature of uniq that checks for duplicate rows only checks adjacent rows. Duplicate data, there must be many are not adjacent together.
[[email protected] mytest]$ Uniq-c uniqtest is a test 1 I am tank 2 I love Tank is a test // and the first line is a repeating try try 1 try 1 I want to abroad 1 Those is good men 1 We is good Men
This will solve the problem mentioned in the last example.
[[email protected] mytest]$ sort uniqtest |uniq-c try 1 I am tank 2 I love Tan K 1 I want to abroad was A test 1 those is good Men 1< c16> We is good mentry 1 try
Uniq-d only duplicate rows are displayed
[Email protected] mytest]$ uniq-d-c uniqtest is a test
uniq-d only displays duplicate rows, and displays the repeating lines. He can't use with-c
[Email protected] mytest]$ Uniq-D uniqtest is a test is a test
is a test I love tank
Here those only one row, the display is repeated, this is because, F 1 ignores the first column, check the repetition from the second field start.
[[email protected] mytest]$ uniq-f 1-c uniqtest is a test 1 I am tank 2< c7> I love tank are a test try 1 You try 1 I want to abroad 2 Those is good Men //Only one row, showing two rows
When checking, case insensitive
[Email protected] mytest]$ uniq-i-c uniqtestis a test 1 I am tank 2
I love tank is
a test
try //
one uppercase, one lowercase 1 You
try 1
I want to abroad 1
Those is goodmen 1 we is good Men
When checking, do not consider the first 4 characters, so whom have a try and you have a try is the same.
[[email protected] mytest]$ uniq-s 4-C uniqtest is a test 1 I am tank 2< c7> I love tank is a test try // Root on an example what's the difference 1 I want to abroad 1 Those is good men
to repeat the items, and then show them all.
[Email protected] mytest]$ Uniq-u uniqtest I am tank is a testTry Try youtry I want to abroad those is good Men We is good men
The 2nd character of each line after the content is not checked, so I am tank root I love tank is the same.
[[email protected] mytest]$ uniq-w 2-C uniqtest is a test 3 I am tank
is
a test
try
try 1 You
try 1
i want To abroad 1
Those is good men
Linux command (UNIQ)