python list去重複值

Suppose you have a list in python that looks like this:

 [
'a'
,
'b'
,
'a'
]

 # or like this:

 [
1
,
2
,
2
,
2
,
3
,
4
,
5
,
6
,
6
,
6
,
6
]

and you want to remove all duplicates so you get this result:

 [
'a'
,
'b'
]

 # or

 [
1
,
2
,
3
,
4
,
5
,
6
]

How do you do that? ...the fastest way? I wrote a couple of
alternative implementations and did a quick benchmark loop on the
various implementations to find out which way was the fastest. (I
haven't looked at memory usage). The slowest function was 78 times
slower than the fastest function.

However, there's one very important difference between the various
functions. Some are order preserving and some are not. For example, in
an order preserving function, apart from the duplicates, the order is
guaranteed to be the same as it was inputted. Eg, uniqify([1,2,2,3])==[1,2,3]

Here are the functions:

def f1(seq):<br /> # not order preserving<br /> set = {}<br /> map(set.__setitem__, seq, [])<br /> return set.keys()<br /> def f2(seq):<br /> # order preserving<br /> checked = []<br /> for e in seq:<br /> if e not in checked:<br /> checked.append(e)<br /> return checked<br /> def f3(seq):<br /> # Not order preserving<br /> keys = {}<br /> for e in seq:<br /> keys[e] = 1<br /> return keys.keys()<br /> def f4(seq):<br /> # order preserving<br /> noDupes = []<br /> [noDupes.append(i) for i in seq if not noDupes.count(i)]<br /> return noDupes<br /> def f5(seq, idfun=None):<br /> # order preserving<br /> if idfun is None:<br /> def idfun(x): return x<br /> seen = {}<br /> result = []<br /> for item in seq:<br /> marker = idfun(item)<br /> # in old Python versions:<br /> # if seen.has_key(marker)<br /> # but in new ones:<br /> if marker in seen: continue<br /> seen[marker] = 1<br /> result.append(item)<br /> return result<br /> def f6(seq):<br /> # Not order preserving<br /> set = Set(seq)<br /> return list(set)

And what you've all been waiting for (if you're still reading). Here
are the results:

 *
 f2
 13.24

 *
 f4
 11.73

 *
 f5
 0.37

 f1
 0.18

 f3
 0.17

 f6
 0.19

 (*
 order
 preserving
)

Clearly f5
is the "best" solution. Not only is it really
really fast; it's also order preserving and supports an optional
transform function which makes it possible to do this:

 >>>
 a
=
list
(
'ABeeE'
)

 >>>
 f5
(
a
)

 [
'A'
,
'B'
,
'e'
,
'E'
]

 >>>
 f5
(
a
,
 lambda
 x
:
 x
.
lower
())

 [
'A'
,
'B'
,
'e'
]