Data Cleaning
Data cleaning is the process of identifying and correcting errors and inconsistencies in data to improve its quality. Clean data is essential for accurate analysis and reliable results. This section will cover various techniques for handling missing values and outliers, which are common issues in real-world datasets.
Handling Missing Values
- Understanding Missing Values: Missing values occur when no data is recorded for a variable in an observation. They can arise due to various reasons, such as data entry errors, equipment malfunctions, or respondents skipping questions in surveys.
Types of missing values:
-
- MCAR (Missing Completely At Random): The missingness is entirely random and unrelated to any observed or unobserved data.
- MAR (Missing At Random): The missingness is related to observed data but not to the missing data itself.
- MNAR (Missing Not At Random): The missingness is related to the missing data itself, leading to potential bias.
Techniques for Identifying Missing Values:
- Summary Statistics: Use functions like
isnull()in Python’s Pandas library to count missing values in each column. - Visualization: Use heatmaps or bar charts to visualize the presence of missing values. Libraries like Seaborn can help create these visualizations.
Example:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
# Load dataset
df = pd.read_csv('data.csv')
# Check for missing values
print(df.isnull().sum())
# Visualize missing values
sns.heatmap(df.isnull(), cbar=False, cmap='viridis')
plt.title('Missing Values Heatmap')
plt.show()
Strategies for Handling Missing Values:
- Removing Missing Values: Drop rows or columns with missing values using
dropna(). This is effective when the missing data is minimal.
df_cleaned = df.dropna() # Drops any rows with missing values - Imputation: Fill in missing values using statistical methods:
- Mean/Median/Mode Imputation: Replace missing values with the mean, median, or mode of the column.
df['column_name'].fillna(df['column_name'].mean(), inplace=True) # Mean imputation - K-Nearest Neighbors (KNN) Imputation: Use KNN to predict and fill missing values based on the values of similar data points.
- Predictive Modeling: Use regression or classification models to predict missing values based on other features.
Handling Outliers
- Understanding Outliers:
Outliers are data points that differ significantly from other observations in the dataset. They can skew results and affect statistical analyses.
Outliers can be caused by measurement errors, data entry errors, or they may indicate variability in the data.
- Techniques for Identifying Outliers:
Statistical Methods:
- Z-Score: A Z-score indicates how many standard deviations an element is from the mean. A Z-score greater than 3 or less than -3 is often considered an outlier.
from scipy import stats
z_scores = stats.zscore(df['column_name'])
outliers = df[(z_scores > 3) | (z_scores < -3)] - Interquartile Range (IQR): Calculate the IQR and identify outliers as values below Q1 – 1.5 * IQR or above Q3 + 1.5 * IQR.
Q1 = df['column_name'].quantile(0.25)
Q3 = df['column_name'].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df['column_name'] < (Q1 - 1.5 * IQR)) | (df['column_name'] > (Q3 + 1.5 * IQR))]
Strategies for Handling Outliers:
- Removing Outliers: If outliers are due to errors, they can be removed from the dataset.
df_cleaned = df[~df['column_name'].isin(outliers)]
- Capping Values: Limit the values of outliers to a certain threshold (e.g., capping at the 1st and 99th percentiles).
df['column_name'] = np.where(df['column_name'] > upper_limit, upper_limit, df['column_name'])
- Transformation: Apply transformations (e.g., log transformation) to reduce the impact of outliers.