How to delete duplicates from Pandas DataFrame in Python

Pandas is a popular data analysis library in Python, and it provides an efficient way to store and manipulate data in the form of a DataFrame. However, data can often contain duplicates, and it’s important to remove them to get accurate results. In this article, we’ll go over how to delete duplicates from a Pandas DataFrame in Python.

The first step is to create a DataFrame. You can do this by passing a dictionary or a list of lists to the pandas.DataFrame() function. For example, if you have a list of names and addresses, you can create a DataFrame by passing a list of dictionaries to the function:

import pandas as pd
data = [{"Name": "John", "Address": "123 Main St"},
{"Name": "Jane", "Address": "456 Elm St"},
{"Name": "John", "Address": "123 Main St"}]
df = pd.DataFrame(data)

In this example, the DataFrame contains a duplicate of the first record. To remove these duplicates, we use the drop_duplicates() method of the DataFrame:

df = df.drop_duplicates()

By default, drop_duplicates() removes all duplicates from the DataFrame, but you can also specify which columns to consider when checking for duplicates. For example, if you only want to consider the “Name” column:

df = df.drop_duplicates(subset="Name")

You can also specify multiple columns to consider by passing a list of column names to the subset argument:

df = df.drop_duplicates(subset=["Name", "Address"])

In addition to removing duplicates, you can also specify how to handle duplicates using the keep argument. By default, keep='first', which means that the first occurrence of a duplicate record is kept and the rest are removed. You can also specify keep='last' to keep the last occurrence, or keep=False to remove all duplicates.

In conclusion, removing duplicates from a Pandas DataFrame is a straightforward process. You can use the drop_duplicates() method to remove all duplicates or specify which columns to consider when checking for duplicates. With these tools, you’ll be able to handle duplicates in your data and ensure that you’re getting accurate results.

Here is another example:

Let’s say you have a DataFrame that contains information about different fruits, including the type of fruit and the number of pieces available:

import pandas as pd
data = [{"Fruit": "Apple", "Pieces": 10},
{"Fruit": "Banana", "Pieces": 20},
{"Fruit": "Apple", "Pieces": 10},
{"Fruit": "Grape", "Pieces": 5}]
df = pd.DataFrame(data)

In this case, we have two records with the same information (type of fruit is “Apple” and pieces available is 10). To remove these duplicates, we can use the drop_duplicates() method:

df = df.drop_duplicates()

If we want to consider only the “Fruit” column when checking for duplicates, we can specify the subset argument:

df = df.drop_duplicates(subset="Fruit")

And if we want to keep only the last occurrence of a duplicate record, we can specify the keep argument:

df = df.drop_duplicates(subset="Fruit", keep='last')

This will result in a DataFrame with only unique records:

  Fruit Pieces
1 Banana 20
3 Grape 5
2 Apple 10

As you can see, removing duplicates from a Pandas DataFrame is simple and flexible, allowing you to consider specific columns and keep the desired occurrences of duplicates.

Disclaimer: The information and code presented within this recipe/tutorial is only for educational and coaching purposes for beginners and developers. Anyone can practice and apply the recipe/tutorial presented here, but the reader is taking full responsibility for his/her actions. The author (content curator) of this recipe (code / program) has made every effort to ensure the accuracy of the information was correct at time of publication. The author (content curator) does not assume and hereby disclaims any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from accident, negligence, or any other cause. The information presented here could also be found in public knowledge domains.