Keywords: Data Splitting | Training Set | Validation Set | Test Set | NumPy | Pandas | Machine Learning
Abstract: This article provides a comprehensive guide on splitting datasets into training, validation, and test sets for machine learning projects. Using NumPy's split function and Pandas data manipulation capabilities, we demonstrate the implementation of standard 60%-20%-20% splitting ratios. The content delves into splitting principles, the importance of randomization, and offers complete code implementations with practical examples to help readers master core data splitting techniques.
Importance of Data Splitting
In machine learning projects, properly splitting datasets into training, validation, and test sets is a fundamental step in model development. The training set is used for model training, the validation set for hyperparameter tuning, and the test set for final model evaluation. This splitting approach effectively prevents overfitting and ensures model generalization capability.
Core Implementation Using NumPy Splitting
Using NumPy's np.split() function combined with Pandas data randomization enables efficient data splitting. First, the entire dataset needs to be randomly shuffled to ensure each subset represents the distribution characteristics of the original data.
import numpy as np
import pandas as pd
# Create sample dataframe
df = pd.DataFrame(np.random.rand(10, 5), columns=list('ABCDE'))
# Randomly shuffle data and split
train, validate, test = np.split(
df.sample(frac=1, random_state=42),
[int(0.6 * len(df)), int(0.8 * len(df))]
)In-depth Analysis of Splitting Principles
The indices_or_sections parameter of the np.split() function defines the split point positions. For a dataset of length N, [int(0.6*N), int(0.8*N)] indicates splitting at the 60% and 80% positions, resulting in three subsets: the first 60% as training set, the middle 20% as validation set, and the final 20% as test set.
Critical Role of Randomization Processing
Using df.sample(frac=1, random_state=42) to randomly shuffle data is crucial. frac=1 indicates sampling all data, while random_state=42 ensures reproducible results. This processing avoids potential impacts of data order on model training.
Practical Application Example
Considering a dataset containing 20 samples, we can demonstrate how to split according to an 80%-10%-10% ratio:
a = np.arange(1, 21)
result = np.split(a, [int(0.8 * len(a)), int(0.9 * len(a))])
print(result)The output will display three arrays: the first 16 elements, followed by 2 elements, and finally 2 elements, perfectly matching the expected splitting ratio.
Comparison with Alternative Methods
Although using train_test_split twice can achieve similar functionality, the NumPy method is more concise and efficient. Directly using np.split() avoids creating intermediate variables, making the code clearer and more understandable.
Best Practice Recommendations
In actual projects, it's recommended to always set the random_state parameter to ensure reproducible results. Meanwhile, adjust splitting ratios according to specific problems, and consider alternative solutions like cross-validation for small datasets.