Python's way to remove list duplicates or similar elements

Recently, bloggers want to use Python to remove repetitive or similar SQL when analyzing database slow query logs, so you don't have to look at a lot of similar SQL. Repeating the same data is simpler and can be done using the built-in set command.

For example:

L1 = [' A ', ' B ', ' C ', ' d ', ' e ', ' A ', ' B ', ' F ']
L2 = List (set (L1))
Print L2

This is easy to implement with Python.

How do I get rid of similar strings, the SQL statements I use here? Fortunately Python is convenient, there are built-in library difflib can be used. The method for calculating the similarity of Difflib is similar to the following:

>>> Import Difflib
>>> Difflib. Sequencematcher (None, ' abcde ', ' abcde '). Ratio ()
1.0
>>> Difflib. Sequencematcher (None, ' abcde ', ' zbcde '). Ratio ()
0.80000000000000004
>>> Difflib. Sequencematcher (None, ' abcde ', ' zyzzy '). Ratio ()
0.0

With this method, let the blogger I have the idea of the first.

Python-list-similar-remove

Start with the first element of the list, iterate through each element after the element and compare. If two strings are judged to be similar, delete the element. After the traversal, start with the second element, loop through, and finally get the list after removing the similar elements.

Temporary bloggers think that this is the way, if more efficient and better welcome to communicate. The code resembles the following:

def remove_similar (lists,similarity=0.9):
I=0
L=len (lists)
While I<l:
J=i+1
While J<l:
Seq=difflib. Sequencematcher (None,lists[i],lists[j])
Ratio=seq.ratio ()
If ratio>=similarity:
Del Lists[j]
L=l-1
Else
J+=1
I+=1
return lists

The default similarity of the above function is set to 0.9, which can be modified according to your request.

What's easier to remember is using the built-in set

L1 = [' B ', ' C ', ' d ', ' B ', ' C ', ' A ', ' a ']
L2 = List (set (L1))
Print L2

There's also a speed difference, which is said to be faster, without testing the two.

L1 = [' B ', ' C ', ' d ', ' B ', ' C ', ' A ', ' a ']
L2 = {}.fromkeys (L1). Keys ()
Print L2

Both have a disadvantage, the elimination of duplicate elements after the order changed:

1 [' A ', ' C ', ' B ', ' d ']

If you want to keep their original sort:

Using the Sort method of the list class

L1 = [' B ', ' C ', ' d ', ' B ', ' C ', ' A ', ' a ']
L2 = List (set (L1))
L2.sort (Key=l1.index)
Print L2

You can also write this

L1 = [' B ', ' C ', ' d ', ' B ', ' C ', ' A ', ' a ']
L2 = sorted (set (L1), Key=l1.index)
Print L2

You can also use traversal

L1 = [' B ', ' C ', ' d ', ' B ', ' C ', ' A ', ' a ']
L2 = []
For I in L1:
If not I in L2:
L2.append (i)
Print L2

The code above can also be written like this

L1 = [' B ', ' C ', ' d ', ' B ', ' C ', ' A ', ' a ']
L2 = []
[L2.append (i) for I-L1 if not I in L2]
Print L2

This guarantees that the sort will be unchanged:

? 1 [' B ', ' C ', ' d ', ' a ']