1. 遍历整个列表

# removing duplicated from list 
# using naive methods 

# initializing list
test_list = [1, 3, 5, 6, 3, 5, 6, 1]
print ("The original list is : " +  str(test_list))

# using naive method to remove duplicated from list 
res = []
for i in test_list:
    if i not in res:
        res.append(i)

# printing list after removal 
print ("The list after removing duplicates : " + str(res))

# 输出结果：
# 原始列表是：[1, 3, 5, 6, 3, 5, 6, 1]
# 删除重复项后的列表：[1, 3, 5, 6]

2. 使用 set()

最大的缺点之一是set后列表中元素的顺序不再和原来一样

# removing duplicated from list 
# using set()

# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : " +  str(test_list))

# using set()to remove duplicated from list 
test_list = list(set(test_list))

# printing list after removal 
# distorted ordering
print ("The list after removing duplicates : " + str(test_list))

# 输出结果：
# 原始列表是：[1, 5, 3, 6, 3, 5, 6, 1]
# 删除重复项后的列表：[1, 3, 5, 6]

3. 使用列表理解 + enumerate()

此方法使用枚举根据列表理解删除重复元素。通过检查该元素是否已存在于列表中来跳过该元素。此方法保持列表中元素的顺序。

# removing duplicated from list 
# using list comprehension + enumerate()

# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : " +  str(test_list))

# using list comprehension + enumerate()
# to remove duplicated from list 
res = [i for n, i in enumerate(test_list) if i not in test_list[:n]]

# printing list after removal 
print ("The list after removing duplicates : " + str(res))

4. 使用 collections.OrderedDict.fromkeys()

这是完成特殊任务的最快方式,它首先删除列表中的重复项并返回一个字典，最后将其转换为列表。此方法也可用于字符串，之后列表中元素的顺序也发生了变化。

# removing duplicated from list 
# using collections.OrderedDict.fromkeys()
from collections import OrderedDict

# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : " +  str(test_list))

# using collections.OrderedDict.fromkeys()
# to remove duplicated from list 
res = list(OrderedDict.fromkeys(test_list))

# printing list after removal 
print ("The list after removing duplicates : " + str(res))

5. 处理嵌套列表中的重复元素

用于多维列表（列表嵌套）重复元素移除。这里假设列表（也是一个列表）中具有相同元素（但不一定是相同顺序）的元素被认为是重复的。

使用 set() + sorted() 方法：

# removing duplicate sublist 
# using set() + sorted()

# initializing list
test_list = [[1, 0, -1], [-1, 0, 1], [-1, 0, 1],
                           [1, 2, 3], [3, 4, 1]]

# printing original list
print("The original list : " + str(test_list))

# using set() + sorted()
# removing duplicate sublist
res = list(set(tuple(sorted(sub)) for sub in test_list))

# print result
print("The list after duplicate removal : " + str(res))

# 输出结果：
# 原始列表：[[1, 0, -1], [-1, 0, 1], [-1, 0, 1], [1, 2, 3], [3, 4, 1]]
# 去重后的列表：[(-1, 0, 1), (1, 3, 4), (1, 2, 3)]

使用set() + map() + sorted()方法：

# removing duplicate sublist 
# using set() + map() + sorted()

# initializing list
test_list = [[1, 0, -1], [-1, 0, 1], [-1, 0, 1],
                           [1, 2, 3], [3, 4, 1]]

# printing original list
print("The original list : " + str(test_list))

# using set() + map() + sorted()
# removing duplicate sublist
res = list(set(map(lambda i: tuple(sorted(i)), test_list)))

# print result
print("The list after duplicate removal : " + str(res))

# 输出结果：
# 原始列表：[[1, 0, -1], [-1, 0, 1], [-1, 0, 1], [1, 2, 3], [3, 4, 1]]
# 去重后的列表：[(-1, 0, 1), (1, 3, 4), (1, 2, 3)]

Reference：webqdkf.com