Complete Guide to File Iteration and Path Manipulation in Bash Scripting

Fundamentals of File Iteration and Path Manipulation

File traversal and path manipulation are common requirements in automated script development. Implementing these functionalities through Bash scripting can significantly enhance work efficiency and code reusability. This article delves into efficient file path processing and dynamic output filename generation within the Bash environment, based on practical use cases.

Core Script Implementation

The following script demonstrates how to iterate through text files in a specified directory and generate multiple numbered output paths for each file:

#!/bin/bash
# Iterate through all .txt files in the Data directory
for filepath in Data/*.txt; do
    # Check file existence to handle unmatched glob patterns
    if [ ! -e "$filepath" ]; then
        echo "Warning: No matching files found, skipping processing"
        continue
    fi
    
    # Inner loop to generate multiple output files
    for ((counter=0; counter<=3; counter++)); do
        # Extract base filename (without path and extension)
        base_name=$(basename "$filepath" .txt)
        
        # Construct output file path
        output_path="Logs/${base_name}_Log${counter}.txt"
        
        # Execute target program
        ./MyProgram.exe "$filepath" "$output_path"
    done
done

Key Technical Points Analysis

The script implementation involves several core concepts of Bash programming:

File Globbing and Path Expansion

The Data/*.txt expression uses wildcard patterns to match all text files in the specified directory. This pattern matching automatically expands into a complete list of file paths, providing the foundation for subsequent processing.

Path Processing and Filename Extraction

The basename command is a crucial tool for path manipulation:

# Extract filename from full path
filename=$(basename "/path/to/file.txt")
# Result: file.txt

# Remove specified extension simultaneously
base_name=$(basename "/path/to/file.txt" .txt)
# Result: file

Parameter Expansion Techniques

Bash provides powerful parameter expansion capabilities for string operations:

filepath="/Data/example.txt"

# Remove path prefix
relative_path="${filepath#/}"
# Result: Data/example.txt

# Extract directory path
dir_path="${filepath%/*}"
# Result: /Data

# Extract file extension
extension="${filepath##*.}"
# Result: txt

Boundary Conditions and Error Handling

Robust scripts must consider various edge cases:

File Existence Validation

Checking file existence before processing prevents unexpected errors:

for filepath in Data/*.txt; do
    # Check if file exists and is a regular file
    if [ ! -f "$filepath" ]; then
        echo "File does not exist or is not a regular file: $filepath"
        continue
    fi
    
    # Check file readability
    if [ ! -r "$filepath" ]; then
        echo "File is not readable: $filepath"
        continue
    fi
    
    # Normal processing logic
    # ...
done

Output Directory Verification

Ensuring output directory existence prevents runtime errors:

# Check if output directory exists, create if not present
if [ ! -d "Logs" ]; then
    mkdir -p "Logs" || {
        echo "Error: Cannot create Logs directory"
        exit 1
    }
fi

Advanced Application Scenarios

Dynamic Prefix Mapping

Drawing inspiration from dictionary mapping concepts in reference materials, similar functionality can be implemented in Bash:

#!/bin/bash

# Define prefix mapping relationships
declare -A prefix_mapping=(
    ["data"]="processed"
    ["temp"]="temporary"
    ["log"]="archived"
)

for filepath in Data/*.txt; do
    base_name=$(basename "$filepath" .txt)
    
    # Extract file prefix
    file_prefix="${base_name%%_*}"
    
    # Check mapping relationship
    if [[ -n "${prefix_mapping[$file_prefix]}" ]]; then
        new_prefix="${prefix_mapping[$file_prefix]}"
        
        for ((i=0; i<=3; i++)); do
            output_path="Logs/${new_prefix}_${base_name#*_}_Log${i}.txt"
            ./MyProgram.exe "$filepath" "$output_path"
        done
    else
        echo "Skipping file with unmapped prefix: $base_name"
    fi
done

Parallel Processing Optimization

For large-scale file processing, consider parallel execution to improve efficiency:

#!/bin/bash

# Maximum parallel processes
MAX_JOBS=4
current_jobs=0

process_file() {
    local filepath="$1"
    local base_name=$(basename "$filepath" .txt)
    
    for ((i=0; i<=3; i++)); do
        output_path="Logs/${base_name}_Log${i}.txt"
        ./MyProgram.exe "$filepath" "$output_path" &
    done
}

for filepath in Data/*.txt; do
    # Wait for available process slots
    while [ $current_jobs -ge $MAX_JOBS ]; do
        wait -n
        ((current_jobs--))
    done
    
    process_file "$filepath" &
    ((current_jobs++))
done

# Wait for all remaining processes to complete
wait

Best Practices Summary

When implementing file iteration and path manipulation scripts, follow these best practices:

1. Always Validate Input: Check file existence, type, and permissions before processing to avoid runtime errors.

2. Use Quotes to Protect Variables: Enclose all file path variables in double quotes to prevent issues caused by spaces and other special characters.

3. Clear Error Handling: Provide meaningful error messages to help users understand the nature of problems.

4. Modular Design: Break complex logic into independent functions to improve code readability and maintainability.

5. Performance Considerations: For large-scale file processing, consider using parallel execution or batch processing techniques.

By mastering these core techniques and best practices, developers can build robust and efficient automated file processing scripts to meet various complex business requirements.