Data Preprocessing
Transform raw data into a clean and analysis-ready format
The Importance of Data Preprocessing
Data preprocessing is a crucial step in the data analysis pipeline that involves transforming raw data into a clean and analysis-ready format. It includes handling missing values, removing duplicates, dealing with outliers, and feature engineering.
Preprocessing Pipeline
Data Collection
Gather raw data
Cleaning
Handle missing values
Transformation
Scale and encode
Feature Engineering
Create new features
Common Preprocessing Tasks
Data Cleaning
- • Handle missing values
- • Remove duplicates
- • Fix inconsistencies
- • Deal with outliers
Feature Engineering
- • Create new features
- • Transform variables
- • Encode categories
- • Scale numerical data
Implementation Example
import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.impute import SimpleImputer
# Load data
df = pd.read_csv('raw_data.csv')
# Handle missing values
imputer = SimpleImputer(strategy='mean')
df['numeric_column'] = imputer.fit_transform(df[['numeric_column']])
# Remove duplicates
df = df.drop_duplicates()
# Handle outliers
Q1 = df['numeric_column'].quantile(0.25)
Q3 = df['numeric_column'].quantile(0.75)
IQR = Q3 - Q1
df = df[~((df['numeric_column'] < (Q1 - 1.5 * IQR)) |
(df['numeric_column'] > (Q3 + 1.5 * IQR)))]
# Encode categorical variables
le = LabelEncoder()
df['category'] = le.fit_transform(df['category'])
# Scale numerical features
scaler = StandardScaler()
df['numeric_column'] = scaler.fit_transform(df[['numeric_column']])
# Feature engineering
df['new_feature'] = df['feature1'] / df['feature2']
df['date'] = pd.to_datetime(df['date'])
df['day_of_week'] = df['date'].dt.dayofweek
print("Preprocessing complete!")Best Practices
Data Quality
- • Document data sources
- • Validate data types
- • Check for anomalies
- • Maintain data integrity
Process Efficiency
- • Automate repetitive tasks
- • Create reusable functions
- • Monitor memory usage
- • Version control datasets