The Ultimate Guide to Handling Class Imbalance with 11 Techniques: CRISP-DM Data Preparation

---

The Ultimate Guide to Handling Class Imbalance with 11 Techniques: CRISP-DM Data Preparation

Data Mastery Series — Episode 9: Imbalance Handling

If you are interested in articles related to my experience, please feel free to contact me: linkedin.com/in/nattapong-thanngam

---

CRISP-DM framework (Image by Author)

Imbalanced data refers to a situation where the number of observations in one class is significantly lower than in the other class(es) in a binary or multi-class classification problem. This is a common problem in various domains, such as medical diagnosis, fraud detection, and credit scoring, where the minority class contains important information and the goal is to detect it accurately. In these cases, a model that simply predicts the majority class can achieve high accuracy but fail to identify the minority class, which is often the more important class.

Imbalance handling is the process of addressing this issue by applying various techniques to balance the distribution of classes in the dataset. The goal is to increase the representation of the minority class or reduce the representation of the majority class, depending on the specific technique applied.

By balancing the dataset, we can improve the performance of the classification model, reducing the risk of under-representing the minority class and providing more accurate predictions. Imbalance handling techniques can also help to reduce the overall misclassification rate, improve the precision and recall, and make the model more robust and generalizable to new data.

Data Set:

• This code generates a synthetic binary classification dataset comprising of 10,000 data points, classified into two classes: minority and majority. The minority class contains 500 data points, whereas the majority class contains 9,500 data points. The dataset comprises of two input features, namely ‘X1’ and ‘X2’, and one target variable labeled as ‘Y’.

Dataset (Image by Author)

Techniques for handling imbalances can be broadly divided into three types.

1. Over-sampling techniques: Oversampling techniques produce artificial minority class points. Some methods of oversampling include Random Oversampling, SMOTE, SMOTE-NC, Borderline SMOTE, SVM SMOTE, KMeans SMOTE, etc.
2. Under-sampling techniques: Undersampling techniques include removing the majority of the class points. Some methods of oversampling include Random Undersampling, Tomek Links, Cluster Centroids, etc.
3. Combination of Oversampling and Undersampling techniques: Some methods of combination include SMOTE-ENN, SMOTE-Tomek, etc.

11 Popular Handling Class Imbalance Techniques

1. Random Oversampling Random oversampling is a technique that involves duplicating random instances of the minority class in the dataset to balance the class distribution. This technique can be effective in simple cases, where the dataset is not too large and the imbalance is not too severe. However, it can lead to overfitting, as the model may be trained on identical instances of the minority class.

Random Oversampling — Scatter plot (Image by Author)

Random Oversampling — Box plot (Image by Author)

Random Oversampling — Histogram (Image by Author)

2. SMOTE SMOTE (Synthetic Minority Over-sampling Technique) is a widely used technique for balancing class distributions. SMOTE works by generating synthetic samples of the minority class by interpolating between existing samples. The algorithm identifies the k nearest neighbors of a minority class sample and creates new synthetic samples along the line connecting the original sample and its neighbors. The number of new synthetic samples created is determined by the ratio of the minority and majority class sizes.

SMOTE — Scatter plot (Image by Author)

SMOTE — Box plot (Image by Author)

SMOTE — Histogram (Image by Author)

3. SMOTE-NC SMOTE-NC (SMOTE for Nominal and Continuous features) is an extension of SMOTE that can handle datasets with both continuous and categorical features. SMOTE-NC first separates the data into continuous and categorical subsets and then applies SMOTE to each subset separately. The algorithm generates synthetic samples for continuous features by interpolating between the nearest neighbors in the same way as SMOTE. For categorical features, the algorithm randomly selects one of the neighboring samples and then randomly selects a feature value from that sample.

SMOTE-NC — Scatter plot (Image by Author)

SMOTE-NC — Box plot (Image by Author)

SMOTE-NC — Histogram (Image by Author)

4. Borderline SMOTE Borderline SMOTE is a variation of SMOTE that generates synthetic samples only for samples that are near the decision boundary between the minority and majority classes. The idea is to focus the synthetic samples on the areas that are most difficult to learn, rather than just blindly oversampling the entire minority class.

Borderline SMOTE — Scatter plot (Image by Author)

Borderline SMOTE — Box plot (Image by Author)

Borderline SMOTE — Histogram (Image by Author)

5. SVM SMOTE SVM SMOTE is a combination of SMOTE and support vector machines (SVMs). The algorithm first applies SMOTE to the minority class to create synthetic samples. It then trains an SVM on the original and synthetic samples and uses the decision function of the SVM to identify the samples that are most difficult to classify. These samples are then further oversampled using SMOTE.

SVM SMOTE — Scatter plot (Image by Author)

SVM SMOTE — Box plot (Image by Author)

SVM SMOTE —Histogram (Image by Author)

6. KMeans SMOTE
KMeans SMOTE is a hybrid of KMeans clustering and SMOTE. The algorithm first clusters the minority class using KMeans and then applies SMOTE to each cluster separately. This technique is useful when the minority class is not well defined and is spread out across the feature space.

KMeans SMOTE— Scatter plot (Image by Author)

KMeans SMOTE — Box plot (Image by Author)

KMeans SMOTE — Histogram (Image by Author)

7. Random Undersampling Random undersampling is a technique that involves removing random instances of the majority class to balance the class distribution. This technique can be effective in simple cases, where the dataset is not too large and the imbalance is not too severe. However, it can lead to a loss of information and may remove important instances of the majority class.

Random Undersampling — Scatter plot (Image by Author)

Random Undersampling — Box plot (Image by Author)

Random Undersampling — Histogram (Image by Author)

8. Tomek Links Tomek Links are pairs of instances from different classes that are close to each other but are not of the same class. These pairs of instances can be removed from the dataset, as they are likely to be misclassified by any classifier. Tomek Links can be used to undersample the majority class or to improve the boundary between classes.

Tomek Links — Scatter plot (Image by Author)

Tomek Links — Box plot (Image by Author)

Tomek Links — Histogram (Image by Author)

9. Cluster Centroids Cluster Centroids is an undersampling technique that works by creating clusters of the majority class and then replacing each cluster with its centroid. The centroids are calculated by averaging the feature values of the instances in each cluster. This technique can be effective when the majority class is well-defined and can be clustered into groups.

Cluster Centroids— Scatter plot (Image by Author)

Cluster Centroids — Box plot (Image by Author)

Cluster Centroids — Histogram (Image by Author)

10. SMOTE-ENN
SMOTE-ENN is a combination of SMOTE and Edited Nearest Neighbors (ENN). The algorithm first applies SMOTE to the minority class to create synthetic samples. It then removes any synthetic samples that are misclassified by a k-nearest neighbor classifier. Next, it applies ENN to remove any remaining samples that are misclassified by the same classifier. The resulting dataset is a combination of the original and synthetic samples, with the misclassified samples removed.

SMOTE-ENN— Scatter plot (Image by Author)

SMOTE-ENN — Box plot (Image by Author)

SMOTE-ENN — Histogram (Image by Author)

11. SMOTE-Tomek
SMOTE-Tomek is a combination of SMOTE and Tomek Links. The algorithm first applies SMOTE to the minority class to create synthetic samples. It then identifies any Tomek Links between the minority and majority classes and removes the instances from the majority class. The resulting dataset is a combination of the original and synthetic samples, with the Tomek Links removed.

SMOTE-Tomek — Scatter plot (Image by Author)

SMOTE-Tomek — Box plot (Image by Author)

SMOTE-Tomek —Histogram (Image by Author)

In conclusion, handling imbalanced data is a crucial step in building an accurate and robust classification model. Imbalance handling techniques can help to balance the class distribution, reduce the risk of under-representing the minority class, and improve the overall performance of the model. These techniques can be broadly classified into oversampling, undersampling, and combination of both. Each technique has its advantages and disadvantages, and the choice of the technique should depend on the specific problem and dataset. By using the appropriate imbalance handling technique, we can build models that accurately detect the minority class and make reliable predictions on new data.

---

Thank you for taking the time to read this article! If you found it enjoyable, we recommend checking out these other articles for your reading pleasure.

Data Science _Edit description_medium.com

Dashboard _Edit description_medium.com

Course_Review _Thai version_medium.com

Please feel free to contact me, I am willing to share and exchange on topics related to Data Science and Supply Chain. Facebook: facebook.com/nattapong.thanngam
Linkedin: linkedin.com/in/nattapong-thanngam