Seven examples of the uniq command: Remove duplicate lines from text files

Seven examples of the uniq command: The uniq command in Linux can be used to process repeated lines in text files, this tutorial explains some of the most common usage of the uniq command, which may be helpful to you. The following file test will be used as a test file to explain how the uniq command works. $ Cat testaaaabbbbbbxx1. Syntax: $ uniq [-options] When the uniq command does not add any parameters, it will only remove duplicate rows and print the remaining rows: $ uniq testaabbxx2. use the-c parameter to display the number of occurrences of each row. This parameter displays the number of occurrences of each row in the file. $ Uniq-c test 2 aa 3 bb 1 xx3. use the-d parameter to print only duplicate rows. This parameter will print only duplicate rows in the file, as shown below, "xx" is not printed because it is not a duplicate row in the test file. $ Uniq-d testaabb the above example prints all duplicate rows, but only prints them once. However, the-D parameter prints all duplicate rows. For example, if "aa" appears twice in the test file, it will be printed twice in the following example. $ Uniq-D testaaaabbbbbb4. use the-u parameter to print only non-duplicate rows. Only "xx" rows in the test file are non-duplicated, so only the rows are printed. $ Uniq-u testxx 5. Use the-w parameter to compare only the first N characters of each line. This parameter will only compare the first N characters of each line to determine whether it is a repeated row. In this example, we will use test2 as the test file. $ Cat test2hi Linuxhi LinuxUhi LinuxUnixhi commands under Unix use the-w parameter to compare the first eight characters of each line, and then use the-c parameter to print the number of times each line appears. $ Uniq-c-w 8 testNew 3 hi Linux 1 hi Unix Command use the-w parameter to compare the first eight characters of each line, and then use the-D parameter to print all repeated rows. $ Uniq-D-w 8 testNewhi Linuxhi LinuxUhi LinuxUnix6. use the-s parameter to ignore the first N characters of each line. When you use this parameter for comparison, the first N characters of each line are ignored. In this example, we will use the following test3 as the test file. $ Cat test3aabbxxbbbbcbbd command use-s to ignore the first two characters of each line, and then use the-D parameter to print all repeated rows. In this example, the first two characters in each row, such as "aa" in the first line and "xx" in the second line, are ignored, the last two characters of the two rows, "bb", are the same. Therefore, the two rows are printed as duplicate rows. $ Uniq-D-s 2 test3aabbxxbb7. the-f parameter is used to ignore the first N fields of each line. This parameter is similar to the-s parameter, but not a character, but a 'region'. The 'region' is separated by spaces by default. $ Cat test2hi hello Linuxhi friend Linuxhi hello LinuxUnix the commands below will ignore the first two 'regions' of each line and print the duplicate lines with the-D parameter. In this example, the "hi hello" of the first line and the "hi friend" of the second line are the first two fields, so they will not be compared, the next domain "Linux" of these two rows is the same, so these two rows are considered to be duplicate rows. $ Uniq-D-f 2 test2hi hello Linuxhi friend Linux