Vim remove duplicate rows

Method 1:

Sort first and then duplicate

: Sort // direct sorting: g/^ \(. * \) $ \ n \ 1 $/d // remove duplicate rows: g/\ % (^ \ 1 $ \ n \) \@<= \(. * \) $/d // function same as above, also remove duplicate rows: g/\ % (^ \ 1 \>. * $ \ n \) \@<=\ (\ k \ + \). * $/d // the same as above, also removing duplicate rows

Method 2: Use awk

awk ‘!a[$0]++’ file

Resolution:

Borrow http://bbs.51cto.com/thread-964013-1.html

The awk process is processed row by row. By default, the process starts from the first line of the file to the last line of the file, we also need to know that the basic command format of awk is 'pattern {action} ', which matches various styles first, and then how to print the output in braces, the default value is {print $0} If pattern matches pattern. If pattern does not hit the false value (0), {action} will not be processed; when pattern is hit, {action} is processed when the judgment value is true (not 0 }.
The simplest example is that awk '1' file and awk '{print $0}' file are all printed from start to end.

'! A [$0] ++'
It can be divided into several parts for simple explanation.
This command does not have {action}, that is, the row is printed as long as the pattern part determines that the value is true (not 0). Otherwise, the row is skipped and not printed.
! In awk, the opposite meaning is taken, that is, the right turns into a false one. In this command, it will be explained by the magic horse;
A [$0] is very easy to understand. The variable used to create array a is each row in the text. $1 in awk is the first column and $2 is the second column, similarly, $ NF is the last column, and $0 represents all columns and delimiters, that is, a whole row. If pattern is true, a whole row is printed.
++ Indicates that after the variable is obtained for Array a, the value of array + 1
Find the simplest document to explain.

Cat file
Xxx
Yyy
Xxx
Zzz

This file contains four rows, the first and third of which are repeated. The process of using this command is as follows:
Obtain the first line a [xxx] because this is the first line. If array a has never seen the variable xxx, its value is false (0) that is to say, a [xxx] = 0. At this time! It has a major effect. He converts a [xxx] false (0) to a [xxx] True (! 0) at this time, the first line that should not have been printed will be printed, and the value of a [xxx] + 1 after logical inversion will be taken and the second line will be processed.
Line a [yyy] is the same as line a [xxx ].
The situation changes when the third row is reached, because the first row has already seen a [xxx] and has been ++, and its value is not 0 rather than the first two rows, it should have been printed! You do not need to print the logic.
Line 4 a [zzz] is the same as line 1 and Line 2.
This is the result after execution.
Awk '! A [$0] ++ 'file
Xxx
Yyy
Zzz

Make the file a little more complex.

awk '{print NR,$0}' file1 xxx2 yyy3 zzz4 xxx5 yyy6 zzz7 xxx8 yyy9 zzz

There are 9 lines of text in total, and 3 rows are repeated at a time. To make it clearer, the default {print $0} is changed to {print NR, $0 }. NR indicates the row number.
Now, let's give it a try.

awk '!a[$0]++{print NR,$0}' file1 xxx2 yyy3 zzz

awk 'a[$0]++{print NR,$0}' file4 xxx5 yyy6 zzz7 xxx8 yyy9 zzz

Obviously, yes! The command is to print only the first $0 that appears, that is, to remove duplicates! The command is just opposite to him, that is, only remove the first $0.