Data Normalization with Pandas

Data normalization is the process of scaling numeric features to a standard range, preventing large values from dominating the learning process in machine learning models. It is a important step in machine learning and data analysis ensure that numerical features are on a similar scale for optimal model performance. Normalization helps to improve algorithm performance particularly for distance-based models like K-Nearest Neighbors (KNN) and Support Vector Machines (SVM). It is important because:

• Avoids numerical instability in models
• Speeds up convergence in gradient-based algorithms
• Ensures all features contribute equally to the analysis

Steps for Data Normalization in Pandas

Here we will apply some techniques to normalize the data and discuss these with the help of examples. For this let's understand the steps needed for data normalization with Pandas.

1. Import the required libraries
2. Load or create a dataset
3. Apply different normalization techniques
4. Visualize the results

Let's create a sample dataset using Pandas and visualize it.

import pandas as pd  
import matplotlib.pyplot as plt  

df = pd.DataFrame([
    [180000, 110, 18.9, 1400],  
    [360000, 905, 23.4, 1800],  
    [230000, 230, 14.0, 1300],  
    [60000, 450, 13.5, 1500]
], columns=['Col A', 'Col B', 'Col C', 'Col D'])

print(df)

df.plot(kind='bar')
plt.show()

Output:

Normalization Techniques in Pandas

1. Maximum Absolute Scaling

This technique rescales each feature between -1 and 1 by dividing all values by the maximum absolute value in that column. This technique is especially useful when your data doesn’t contain negative numbers and you want to preserve the data’s sparsity. We can apply the maximum absolute scaling in Pandas using the .max() and .abs() methods as shown below. Let's apply normalization techniques one by one.

max_scaled = df.copy()

for column in df_max_scaled.columns:
    max_scaled[column] = max_scaled[column] / max_scaled[column].abs().max()

print(max_scaled)

max_scaled.plot(kind='bar')
plt.show()

Output :

As we can see in above output all values now lie between -1 and 1. Each value is shown in relation to the largest value in that column.

2. The min-max feature scaling

The min-max approach also called normalization rescales the feature to a hard and fast range of [0,1] by subtracting the minimum value of the feature then dividing by the range. . It works well for models like K-Nearest Neighbors (KNN) which compare distance between data points. We can apply the min-max scaling in Pandas using the .min() and .max() methods.

scaled = df.copy()

for column in df_min_max_scaled.columns:
    scaled[column] = (scaled[column] - scaled[column].min()) / (scaled[column].max() - scaled[column].min())

print(scaled)
scaled.plot(kind='bar')
plt.show()

Output :

After scaling the smallest value becomes 0 and the largest becomes 1. All other values lie between these two. This makes it easier for the machine learning model to handle features fairly.

3. The z-score method

The z-score method often called standardization changes the values in each column so that they have a mean of 0 and a standard deviation of 1. This technique is best when your data follow a normal distribution or when you want to treat values in terms of how far they are from the average.

z_scaled = df.copy()

for column in z_scaled.columns:
    z_scaled[column] = (z_scaled[column] - z_scaled[column].mean()) / z_scaled[column].std()

print(z_scaled)

z_scaled.plot(kind='bar')
plt.show()

Output : 

After applying this method each feature is centered around zero and its spread is standardized. This helps in models like logistic regression, SVM and neural networks to perform better.