Methods and Performance Analysis for Row-by-Row Data Addition in Pandas DataFrame

Basic Methods for Row-by-Row Data Addition

In data science and machine learning projects, there is often a need to dynamically add new data rows to DataFrame. Although Pandas DataFrame is designed as an immutable data structure, row-by-row data addition becomes necessary in certain scenarios.

Adding Single Rows Using loc Indexing

The most straightforward approach is using DataFrame's loc attribute. This method allows us to add new rows by specifying index positions:

import pandas as pd
import numpy as np

# Create empty DataFrame
df = pd.DataFrame(columns=['lib', 'qty1', 'qty2'])

# Add data row by row
for i in range(5):
    df.loc[i] = ['name' + str(i)] + list(np.random.randint(10, size=2))

print(df)

The above code creates a DataFrame containing 5 rows, where each row's 'lib' column contains the string 'name' plus index number, and 'qty1' and 'qty2' columns contain random integers. This method is simple and intuitive, particularly suitable for adding small amounts of data.

Performance-Optimized Data Collection Methods

When dealing with large amounts of data, performance becomes a critical consideration. Collecting data into lists and then creating DataFrame in one go can significantly improve efficiency:

# Collect data using lists
rows_list = []

for i in range(1000):
    row_dict = {
        'lib': 'name' + str(i),
        'qty1': np.random.randint(10),
        'qty2': np.random.randint(10)
    }
    rows_list.append(row_dict)

# Create DataFrame in one operation
df = pd.DataFrame(rows_list)
print(f"DataFrame shape: {df.shape}")

Performance Comparison Analysis

We conducted performance tests on four common methods, revealing significant differences:

From the test results, it's evident that the method of collecting data in dictionary lists and then creating DataFrame in one operation has overwhelming performance advantages, being dozens of times faster than row-by-row addition methods.

Advanced Methods Using concat Function

Pandas' concat function provides more flexible DataFrame merging capabilities:

def append_single_row(df, row_data):
    """
    Add single row using concat
    """
    new_row_df = pd.DataFrame([row_data], columns=df.columns)
    return pd.concat([df, new_row_df], ignore_index=True)

# Example usage
df = pd.DataFrame(columns=['lib', 'qty1', 'qty2'])
new_row = {'lib': 'new_lib', 'qty1': 15, 'qty2': 25}
df = append_single_row(df, new_row)

Avoiding Deprecated append Method

It's important to note that DataFrame's append method has been completely removed in Pandas 2.0. In earlier versions, although it could be used:

# Deprecated method - do not use in Pandas 2.0+
df = df.append(new_row, ignore_index=True)

This method should be avoided in new projects to prevent compatibility issues in future versions.

Best Practice Recommendations

Based on performance testing and practical application experience, we recommend the following best practices:

1. Small data amounts: Use loc method for row-by-row addition, providing simple and readable code
2. Large data amounts: Use dictionary lists to collect data, then create DataFrame in one operation
3. Streaming data processing: Set appropriate batch sizes and periodically convert data to DataFrame
4. Memory optimization: For extremely large datasets, consider using distributed computing frameworks like Dask or PySpark

Practical Application Scenario Examples

In machine learning model evaluation, we often need to record performance metrics for each model:

def evaluate_models(models, X_test, y_test):
    """
    Evaluate multiple models and collect results
    """
    results = []
    
    for model_name, model in models.items():
        y_pred = model.predict(X_test)
        accuracy = accuracy_score(y_test, y_pred)
        precision = precision_score(y_test, y_pred, average='weighted')
        
        results.append({
            'model': model_name,
            'accuracy': accuracy,
            'precision': precision,
            'timestamp': pd.Timestamp.now()
        })
    
    return pd.DataFrame(results)

# Usage example
models = {
    'RandomForest': RandomForestClassifier(),
    'LogisticRegression': LogisticRegression(),
    'SVM': SVC()
}

results_df = evaluate_models(models, X_test, y_test)

Conclusion

Adding data row by row to Pandas DataFrame is a common requirement in data science. Although multiple implementation methods exist, performance differences are significant. For small data amounts, the loc method provides good readability; for large data amounts, the dictionary list collection method shows clear performance advantages. In practical projects, appropriate methods should be chosen based on data volume and performance requirements, following best practices to ensure code efficiency and maintainability.