Python Program To Remove all duplicates words from a given sentence

In this tutorial, we will learn to write a Python program for removing all duplicate words from a given sentence.

For a given string we will have to check if there are any duplicate words or not. If found then we have to remove the duplicate word from the string and return the string after removing all the duplicates.

We will be using a dictionary data structure in our program.

Look at the input-output format to understand better.

Input: Trees are are important

Output: Trees are important

Input: Welcome to study tonight to coders!

Output: Welcome to studytonight coders!

To solve this problem quickly we can use the following approaches:

1. using dictionary and Counter() method
2. using fromkeys() method
3. using count() method

Approach 1: dictionary and Counter()

In this, we will use the Counter() method and create a dictionary where the words in the string will be the keys and the frequency of each word will be the value. Since a key is unique we will join all the keys to get the string.

Algorithm

Follow the algorithm to understand the approach better.

Step 1- Define a function that will remove duplicates from the string

Step 2- Split the string to get words

Step 3- Create a dictionary using Counter()

Step 4- Declare a string and add the keys to it

Step 5- Print this string as the result

Step 6- Declare a string that will be checked for duplicates

Step 7- Call the function and pass the string in it

Python Program 1

Look at the program to understand the implementation of the above-mentioned approach. We have imported Counter from the collections module in our program. To get the words from the string we have used the split() method and specified space as the separator which will return a list of words in the string.

To get the keys of a dictionary we will use the key() method. It returns all the keys which are in a dictionary.

from collections import Counter 
def remove_duplicate(s):

    s = s.split(" ")
    word_dic = Counter(s)
    result = " ".join(word_dic.keys())
    print (result)
 
st = 'There are two children children playing in the park'
remove_duplicate(st)

There are two children playing in the park

Approach 2: Using fromkeys() Method

In this approach, we will use the fromkeys() method. This method returns a new dictionary with the elements which are passed in it. We will pass the words in the given string in this method to generate keys. Then we will print all the keys as a string.

Algorithm

Follow the algorithm to understand the approach better.

Step 1- Define a string

Step 2- Split the string to get words

Step 3- Pass the value returned by split() in fromkeys()

Step 4- Join the value returned by fromkeys() and print as a string

Python Program 2

Look at the program to understand the implementation of the above-mentioned approach. First, we have to split the string using split() method to get the words, then using the fromkeys() we have created a dictionary and printed the keys in that dictionary as a string using the join() method.

# remove duplicate words from a sentence
# using fromkeys()

string = "Python is good Python is for beginners"
 
print(' '.join(dict.fromkeys(string.split())))

Python is good for beginners

Approach 3: using count() Method

In this approach, we will use the count() method to find the frequency of each word. We will add only those words in a string that have a frequency of 1 or greater than 1 and are not already present in the string.

Algorithm

Follow the algorithm to understand the approach better.

Step 1- Define a string that will be checked for duplicates function that will remove duplicates from the string

Step 2- To get the words we will split the string

Step 3- Declare an empty list to store non-duplicate words

Step 4- Run a loop to traverse the words

Step 5- Check if the count of the word is greater than or equal to 1 and is not already present in the list

Step 6- Add words that satisfy the condition in the list

Step 7- Print the result list

Python Program 3

Look at the program to understand the implementation of the above-mentioned approach. For traversing the list of words we have used a for loop. To add elements to the list we have used the append() method.

# remove duplicate words from a sentence
# using count()

string = "Python is good and Java is also good"
words_str = string.split()
result = []
for word in words_str:
    # store unique words in list
    if (string.count(word)>=1 and (word not in result)):
        result.append(word)
print(' '.join(result))

Python is good and Java also

Conclusion

In this tutorial, we have seen three different ways of removing duplicate words from a string. In the first approach, we have used a dictionary and the Counter() method. In the second approach, we have used the fromkeys() method. And in the third approach, we have used the count() method.