Python - Remove duplicate values across Dictionary Values

Last Updated : 27 Jan, 2025

Dictionaries often contain lists or sets as their values and sometimes we want to remove duplicate values across all dictionary keys. For example, consider the dictionary d = {'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6]}. The number 3 appears under both 'a' and 'b', and 5 appears under both 'b' and 'c'. To clean up such overlaps, we can use various methods to ensure that each value is unique across all dictionary keys. Let's explore different methods to achieve this.

Using set

This approach uses a set to keep track of values that have already been encountered, ensuring they are added only once.

Python

d = {'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6]}
s = set()

# Remove duplicates across dictionary values
for key in d:
    d[key] = [v for v in d[key] if v not in s and not s.add(v)]

print(d)

Output

{'a': [1, 2, 3], 'b': [4, 5], 'c': [6]}

Explanation:

A set named 's' is initialized to track values that have been processed.
For each key in the dictionary, the comprehension filters out values that are already in 's'.
The add() method is used to update the seen set during iteration.

Let's explore some more ways and see how we can remove duplicate values across dictionary items.

Using Nested Loops

This method manually iterates over each value in the dictionary and removes duplicates by checking against a set.

Python

d = {'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6]}
s = set()

# Remove duplicates across dictionary values
for key, values in d.items():
    a = []
    for v in values:
        if v not in s:
            a.append(v)
            s.add(v)
    d[key] = a

print(d)

Output

{'a': [1, 2, 3], 'b': [4, 5], 'c': [6]}

Explanation:

A new list 'a' is created for each dictionary key.
The method checks if a value is already in the set 's'. If not, it is added to both 'a' and 's'.
After processing, the dictionary is updated with the filtered list of values.

Using a Flat List and Dictionary Comprehension

This method flattens all dictionary values into a single list, processes it to remove duplicates, and then reconstructs the dictionary.

Python

d = {'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6]}

# Flatten values and remove duplicates
all_val = sum(d.values(), [])
unique_val = list(dict.fromkeys(all_val))

# Reconstruct the dictionary
d = {k: [v for v in unique_val if v in d[k]] for k in d}

print(d)

Output

{'a': [1, 2, 3], 'b': [3, 4, 5], 'c': [5, 6]}

Explanation:

All dictionary values are flattened into a single list.
Duplicate values are removed by using dict.fromkeys(), which preserves the order of first appearance.
The dictionary is reconstructed with filtered values.