How to deal with missing values in Pandas DataFrame in Python

Hits: 86

How to deal with missing values in Pandas DataFrame in Python

Dealing with missing values is a common task when working with datasets in Pandas. Missing values can occur for many reasons, such as data entry errors, missing data in the source, or outliers. It’s important to handle missing values effectively to ensure the accuracy and reliability of your data analysis.

Pandas provides several options for handling missing values in a DataFrame. One common strategy is to simply drop any rows or columns that contain missing values. This can be done using the dropna() function.

df = df.dropna()


This will drop any rows that contain at least one missing value.

Another strategy is to fill in missing values with a specific value, such as the mean or median of the column. This can be done using the fillna() function.

df = df.fillna(value)


This will fill in all missing values in the DataFrame with the specified value.

In the case you know the missing values are just for some specific columns, you can use the fillna() function with the specific columns name as well.

df[["column_name1","column_name2"]] = df[["column_name1","column_name2"]].fillna(value)


Another strategy is to fill the missing values with a value calculated based on some other values within the dataset. This can be done using the interpolate() function. This function will interpolate the missing values based on the method specified. Some common methods are ‘linear’, ‘time’, ‘index’, and ‘values’.

df = df.interpolate(method='linear')


It will fill the missing values linearly based on the other values present.

Additionally, you can drop missing value based on a specific condition, using the dropna() function with the thresold parameter, which is a numeric value that represent the minimum number of non-NA values to be retained.

df = df.dropna(thresold=n)   where n is the number of non-NA values.


Finally, it’s also possible to create a Boolean mask indicating the presence of missing values using isna() or notna() function which can be then used to select specific subset of data based on the missing values.

Handling missing values is an important step in data preparation, and the choice of strategy will depend on the specific dataset and the desired outcome of the analysis. Understanding the different options provided by Pandas and when to use each one will help ensure accurate and reliable results in your data analysis.

In this Learn through Codes example, you will learn: How to deal with missing values in Pandas DataFrame in Python.


Personal Career & Learning Guide for Data Analyst, Data Engineer and Data Scientist

Applied Machine Learning & Data Science Projects and Coding Recipes for Beginners

A list of FREE programming examples together with eTutorials & eBooks @ SETScholars

95% Discount on “Projects & Recipes, tutorials, ebooks”

Projects and Coding Recipes, eTutorials and eBooks: The best All-in-One resources for Data Analyst, Data Scientist, Machine Learning Engineer and Software Developer

Topics included: Classification, Clustering, Regression, Forecasting, Algorithms, Data Structures, Data Analytics & Data Science, Deep Learning, Machine Learning, Programming Languages and Software Tools & Packages.
(Discount is valid for limited time only)

Disclaimer: The information and code presented within this recipe/tutorial is only for educational and coaching purposes for beginners and developers. Anyone can practice and apply the recipe/tutorial presented here, but the reader is taking full responsibility for his/her actions. The author (content curator) of this recipe (code / program) has made every effort to ensure the accuracy of the information was correct at time of publication. The author (content curator) does not assume and hereby disclaims any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from accident, negligence, or any other cause. The information presented here could also be found in public knowledge domains.

Learn by Coding: v-Tutorials on Applied Machine Learning and Data Science for Beginners