Comprehensive Methods for Adding Multiple Columns to Pandas DataFrame in One Assignment

Introduction

In data analysis and processing, it is often necessary to add multiple new columns to an existing Pandas DataFrame. Many developers expect to use syntax like df[['col1', 'col2', 'col3']] = [val1, val2, val3] to achieve one-time assignment, but this approach encounters issues in practice.

Problem Analysis

When using column list syntax to create new columns, Pandas requires the right-hand side to be a DataFrame object. This is because Pandas needs to ensure data alignment and index matching. Direct list assignment causes errors since lists cannot provide the necessary index information.

import pandas as pd
import numpy as np

data = {'col_1': [0, 1, 2, 3],
        'col_2': [4, 5, 6, 7]}
df = pd.DataFrame(data)

# Incorrect approach
# df[['column_new_1', 'column_new_2', 'column_new_3']] = [np.nan, 'dogs', 3]

Solutions

Method 1: List Unpacking Assignment

This is the most intuitive method, using Python's list unpacking feature to assign values to multiple columns simultaneously:

df['column_new_1'], df['column_new_2'], df['column_new_3'] = [np.nan, 'dogs', 3]

This method is straightforward and suitable for cases with a small number of columns, but requires manual specification of each column name.

Method 2: DataFrame Expansion Assignment

Leveraging DataFrame's index matching特性，create a single-row DataFrame and automatically expand it to match the original DataFrame's index:

df[['column_new_1', 'column_new_2', 'column_new_3']] = pd.DataFrame([[np.nan, 'dogs', 3]], index=df.index)

This method maintains the semantics of multi-column assignment but requires explicit index specification.

Method 3: Concat Merging

Use the pd.concat function to merge the original DataFrame with a newly created DataFrame along the column axis:

df = pd.concat(
    [
        df,
        pd.DataFrame(
            [[np.nan, 'dogs', 3]], 
            index=df.index, 
            columns=['column_new_1', 'column_new_2', 'column_new_3']
        )
    ], axis=1
)

This method offers strong flexibility and can handle more complex data merging scenarios.

Method 4: Join Connection

Use the join method to connect the new DataFrame to the original DataFrame:

df = df.join(pd.DataFrame(
    [[np.nan, 'dogs', 3]], 
    index=df.index, 
    columns=['column_new_1', 'column_new_2', 'column_new_3']
))

This method has concise syntax but may be less efficient when handling large datasets.

Method 5: Dictionary DataFrame Creation

Create a new DataFrame using a dictionary, then merge via join:

df = df.join(pd.DataFrame(
    {
        'column_new_1': np.nan,
        'column_new_2': 'dogs',
        'column_new_3': 3
    }, index=df.index
))

This method aligns better with Python's dictionary syntax but note that column order may be sorted alphabetically.

Method 6: Assign Method

Use DataFrame's assign method to add multiple columns in one operation:

df = df.assign(column_new_1=np.nan, column_new_2='dogs', column_new_3=3)

This is one of the most Pythonic methods, with clear syntax and support for method chaining.

Method 7: Reindex Technique

First expand the column space, then assign values to existing columns:

new_cols = ['column_new_1', 'column_new_2', 'column_new_3']
new_vals = [np.nan, 'dogs', 3]
df = df.reindex(columns=df.columns.tolist() + new_cols)   # Add empty columns
df[new_cols] = new_vals  # Multi-column assignment works for existing columns

This method leverages the fact that multi-column assignment works for existing columns.

Method 8: Separate Assignments

The most direct approach, assigning values to each new column separately:

df['column_new_1'] = np.nan
df['column_new_2'] = 'dogs'
df['column_new_3'] = 3

Although this requires more lines of code, it offers the best readability and is easy to understand and maintain.

Method Comparison and Selection Recommendations

When choosing a specific method, consider the following factors:

• Code Conciseness: assign method and list unpacking are most concise
• Performance Considerations: For large datasets, separate assignments typically perform best
• Readability: Separate assignments and assign method are easiest to understand
• Flexibility: concat and join methods support more complex data merging

Supplement: Application of Insert Method

In addition to the above methods, the insert method can be used to insert new columns at specific positions:

# Insert single column at specified position
df.insert(2, "Marks", [90, 70, 45, 33, 88, 77], True)
df.insert(3, "ID", [101, 201, 401, 303, 202, 111], True)

This method is suitable for scenarios requiring precise control over column positions.

Conclusion

Pandas provides multiple methods for adding multiple columns to a DataFrame, each with its own applicable scenarios. For simple column additions, the assign method or separate assignments are recommended; for scenarios requiring complex data merging, consider concat or join methods. Understanding the principles and characteristics of these methods helps in selecting the most appropriate solution in practical work.