Remove duplicate data rows in linux (remove duplicate ip addresses)

This article describes how to remove duplicate data rows from a file in linux.

1. remove adjacent duplicate data rows

Copy codeThe code is as follows:
$ Cat data1.txt | uniq
Output:
Beijing
Wuhan
Beijing
Wuhan

2. remove all duplicate data rows

Copy codeThe code is as follows:
$ Cat data1.txt | sort | uniq

Note:
Only the uniq command removes the adjacent duplicate data lines.
If sort is used, all duplicate data rows are converted into adjacent data rows, and then uniq is used to remove all duplicate data rows.

Output:
Beijing
Wuhan

Appendix: data1.txt

Copy codeThe code is as follows:
[Root @ syy ~] # Cat data1.txt
Beijing
Beijing
Wuhan
Wuhan
Wuhan
Beijing
Beijing
Beijing
Wuhan
Wuhan

Note: it is useful for filtering IP addresses in logs.

Delete rows with duplicate fields in big data files in Linux

A recently written data collection program generates a file containing more than million rows of data. the data consists of four fields, and the second field must be deleted as required, no suitable tool was found in linux. stream processing tools such as sed/gawk can only process one row but cannot find rows with duplicate fields. It seems that I had to use a python program, and suddenly remembered to use mysql, so I made a big shift:

1. use mysqlimport -- local dbname data.txt to import data to the table. The table name must be consistent with the file name.
2. execute the following SQL statement (the unique field must be uniqfield)

Copy codeThe code is as follows:
Use dbname;
Alter table tablename add rowid int auto_increment not null;
Create table t select min (rowid) as rowid from tablename group by uniqfield;
Create table t2 select tablename. * from tablename, t where tablename. rowid = t. rowid;

Drop table tablename;
Rename table t2 to tablename;