Evaluating Multiclass Imbalanced Data Classification: Computing Precision, Recall, Accuracy and F1-Score with scikit-learn

Fundamental Principles of Class Weights and Evaluation Metrics

In machine learning classification tasks, class weights (class_weight) and evaluation metrics represent two distinct yet closely related concepts. The class weight parameter primarily adjusts model attention to different classes during training, while evaluation metrics measure model performance after training completion.

In scikit-learn, the class_weight parameter influences model training by modifying class weights within the loss function. When set to 'auto', the algorithm automatically computes weights based on class distribution in training data, assigning higher weights to minority classes. This mechanism effectively mitigates negative impacts of class imbalance on model training.

Averaging Strategies for Multiclass Evaluation Metrics

In multiclass classification scenarios, metrics such as precision, recall, and F1-score require specific averaging strategies to generate single comprehensive scores. scikit-learn provides three primary averaging approaches:

Macro averaging: Computes metric values for each class then takes arithmetic mean, assigning equal importance to all classes. This approach suits scenarios requiring balanced performance across classes, but may be dominated by rare class performance under extreme class imbalance.

Micro averaging: Calculates global metrics by aggregating true positives, false positives, and false negatives across all classes. This method focuses on sample-level performance and tends to reflect majority class performance under class imbalance.

Weighted averaging: Applies support-based weighting to macro average, considering importance differences among classes while avoiding micro average's excessive bias toward majority classes.

Code Implementation and Best Practices

Proper evaluation workflow requires testing models on unseen data. The following code demonstrates complete process using StratifiedShuffleSplit for data partitioning and model evaluation:

from sklearn.datasets import make_classification
from sklearn.model_selection import StratifiedShuffleSplit
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score

# Generate sample data
X, y = make_classification(n_samples=1000, n_informative=10, n_classes=3)

# Create stratified shuffle splitter
sss = StratifiedShuffleSplit(n_splits=1, test_size=0.3, random_state=42)

for train_idx, test_idx in sss.split(X, y):
    X_train, X_test = X[train_idx], X[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]
    
    # Train SVM model with class weights
    svc = SVC(kernel='linear', C=1, class_weight='balanced')
    svc.fit(X_train, y_train)
    
    # Predict and compute metrics
    y_pred = svc.predict(X_test)
    
    print("Macro F1-score:", f1_score(y_test, y_pred, average="macro"))
    print("Weighted precision:", precision_score(y_test, y_pred, average="weighted"))
    print("Micro recall:", recall_score(y_test, y_pred, average="micro"))

Warning Handling and Version Compatibility

In scikit-learn version 0.18 and above, multiclass evaluation metrics must explicitly specify the average parameter. For migrating legacy code, update all relevant metric calls to:

# Correct usage (avoiding DeprecationWarning)
f1 = f1_score(y_true, y_pred, average='weighted')
precision = precision_score(y_true, y_pred, average='macro')
recall = recall_score(y_true, y_pred, average='micro')

In cross-validation, corresponding scoring parameters should be updated to forms like scoring="f1_weighted" to ensure compatibility with new API versions.

Strategies for Handling Imbalanced Data

Beyond using class weights, several strategies can address class imbalance:

Resampling techniques: Balance dataset distribution through oversampling minority classes or undersampling majority classes. Common methods include SMOTE oversampling and random undersampling.

Threshold adjustment: For probabilistic classifiers, optimize specific class performance by adjusting decision thresholds, particularly important in precision-recall trade-offs.

Ensemble methods: Employ algorithms specifically designed for imbalanced data, such as EasyEnsemble or BalanceCascade, to effectively enhance model recognition capability on rare classes.

Guidelines for Evaluation Metric Selection

In practical applications, evaluation metric selection should be based on specific business requirements:

When all classes are equally important, macro averaging is recommended as it fairly reflects each class's performance.

If data distribution matches real scenarios and overall sample classification effectiveness is desired, micro averaging becomes more appropriate.

Weighted averaging provides optimal compromise when balancing class importance and data distribution.

Accuracy often proves unreliable under severe class imbalance since it can be dominated by majority class performance, masking model deficiencies in minority classes.