Several methods of shell script de-weight

• Test file

cat >testjasonjasonjasonfffff

Press CTR + D to save

1, Sort-u

sort -u testfffffjason

2, Uniq

Uniq Testjasonfffffjason

This shows that Uniq needs to be sorted before it can be re-

3, sort, pipe and uniq combined use

• Remove duplicates and keep a duplicate of the data

Sort test| Uniq Fffffjason

• Remove all duplicate rows (do not keep duplicate rows), and calculate the number of rows

Sort test| uniq -UC      1 fffff

• Find all duplicate rows and calculate the number of rows

Sort test| uniq -DC      4 Jason

• According to a field to weight

Change the original data to

 more 1 2 12 3

Sort 1,1 -21

Man sort view the use of sort, as described by the-K parameter:

-K,--key=pos1[,pos2]
Start a key at POS1 (Origin 1), end it at POS2 (default end of line)

That is,-K which columns are sorted, POS1 is the start column, Pos2 is the end column

Several methods of shell script de-weight