Removing duplicates in lists

One of the best ways of removing duplicates is by using the set() function in python, and then converting it back into list. The code is as follows-

In [2]: some_list = ['a', 'a', 'v', 'v', 'v', 'c', 'c', 'd']

In [3]: list(set(some_list))

Out [3]: ['a', 'c', 'd', 'v']

Simplifying it,
mylist = list(set(mylist))
You may also use the iterator to remove the duplicates-

def uniqify(iterable):

seen = set()

for item in iterable:

if item not in seen:

seen.add(item)

yield item

This will return an iterator or generator so that you may use it wherever you can use an iterator.
For list:

unique_list = list(uniqify([1, 2, 3, 4, 3, 2, 4, 5, 6, 7, 6, 8, 8]))

print(unique_list)

3 years ago

The best approach to get a unique collection of items is to use a set. Sets are basically the unordered collections of distinct objects. If you want to create a set from any iterable, then simply pass it to the built-in set() function.
For example,

>>> t = [1, 2, 3, 1, 2, 5, 6, 7, 8]

>>> t

[1, 2, 3, 1, 2, 5, 6, 7, 8]

>>> list(set(t))

[1, 2, 3, 5, 6, 7, 8]

>>> s = [1, 2, 3]

>>> list(set(t) - set(s))

[8, 5, 6, 7]

3 years ago