Pandas Find Duplicate Rows

Most simple way to find duplicate rows in DataFrame is by using the duplicated() method. This method returns a boolean Series indicating whether each row is a duplicate of a previous row.

import pandas as pd

data = {'Name': ['John', 'Alice', 'Bob', 'Eve', 'John', 'Charlie'], 
        'Age': [25, 30, 22, 35, 25, 28], 
        'Gender': ['Male', 'Female', 'Male', 'Female', 'Male', 'Male'], 
        'Salary': [50000, 55000, 40000, 70000, 50000, 48000]}

df = pd.DataFrame(data)
# Find duplicate rows
duplicates = df[df.duplicated()]
print(duplicates)

   Name  Age Gender  Salary
4  John   25   Male   50000

In addition to the this method, there are several other approaches to find out the Duplicate rows in Pandas.

1. Identifying Duplicates Based on Specific Columns

Sometimes, you may only want to check for duplicates in specific columns, rather than the entire row. we can use duplicated() with the subset parameter to check for duplicates in specific columns.

The subset parameter allows you to specify which columns to consider when looking for duplicates. In this case, we're checking for duplicates based on the Name and Age columns.

# Find duplicates based on 'Name' and 'Age' columns
duplicates_by_columns = df[df.duplicated(subset=['Name', 'Age'])]
print(duplicates_by_columns)

   Name  Age Gender  Salary
4  John   25   Male   50000

2. Removing Duplicate Rows Using drop duplicates

Once duplicates are identified, you can remove them using the drop_duplicates() method. This method returns a new DataFrame with the duplicate rows removed.

The drop_duplicates() method removes all rows that are identical to a previous row. By default, it keeps the first occurrence of each row and drops subsequent duplicates.

# Remove duplicate rows
df_no_duplicates = df.drop_duplicates()
print(df_no_duplicates)

      Name  Age  Gender  Salary
0     John   25    Male   50000
1    Alice   30  Female   55000
2      Bob   22    Male   40000
3      Eve   35  Female   70000
5  Charlie   28    Male   48000

Here are some key takeaways:

• Identify duplicates using duplicated()
• Find duplicates based on specific columns
• Remove duplicate rows with drop_duplicates()