Variable Expansion Control and Best Practices for Here Documents in Shell Scripting

Basic Mechanism of Here Documents and Variable Expansion Issues

In Unix/Linux shell programming, Here Document is a common input redirection method that allows embedding multi-line text directly into scripts as command input. Its basic syntax format is:

command << delimiter
text content
delimiter

However, when the delimiter is unquoted, all lines within the Here Document undergo full shell expansion, including parameter expansion, command substitution, and arithmetic expansion. This can lead to unexpected issues in practical applications.

Analysis of Practical Problem Cases

Consider the following typical scenario: a user attempts to append a Bash script to a file using cat <<EOF >> brightup.sh:

cat <<EOF >> brightup.sh
#!/bin/bash
curr=`cat /sys/class/backlight/intel_backlight/actual_brightness`
if [ $curr -lt 4477 ]; then
   curr=$((curr+406));
   echo $curr  > /sys/class/backlight/intel_backlight/brightness;
fi
EOF

Since the delimiter EOF is unquoted, the shell immediately expands variables and command substitutions within the Here Document during execution. The backtick command `cat /sys/class/backlight/intel_backlight/actual_brightness` is executed before writing to the file, and the variable $curr is replaced with its current value. The final file content becomes:

#!/bin/bash
curr=1634
if [  -lt 4477 ]; then
   curr=406;
   echo   > /sys/class/backlight/intel_backlight/brightness;
fi

This completely destroys the original code's logical structure, rendering the script non-functional.

Solutions: Two Methods to Disable Expansion

To prevent expansion in Here Documents, the key is to properly quote the delimiter. According to Bash documentation, when the delimiter is quoted, no lines undergo any expansion process.

Method One: Single-Quoting the Delimiter

The simplest solution is to add single quotes around the delimiter:

cat <<'EOF' >> brightup.sh
#!/bin/bash
curr=`cat /sys/class/backlight/intel_backlight/actual_brightness`
if [ $curr -lt 4477 ]; then
   curr=$((curr+406));
   echo $curr  > /sys/class/backlight/intel_backlight/brightness;
fi
EOF

This ensures all content is written to the file exactly as is, preserving code integrity.

Method Two: Backslash-Escaping the Delimiter

An equivalent approach is to escape the delimiter with a backslash:

cat <<\EOF >>brightup.sh
#!/bin/bash
curr=`cat /sys/class/backlight/intel_backlight/actual_brightness`
if [ $curr -lt 4477 ]; then
   curr=$((curr+406));
   echo $curr  > /sys/class/backlight/intel_backlight/brightness;
fi
EOF

Both methods produce identical results, and developers can choose based on personal preference.

Strategies for Partial Expansion Control

In some scenarios, mixing expansion and literal text may be necessary. This can be achieved through selective escaping:

cat <<EOF >>brightup.sh
#!/bin/sh
# Created on $(date)
echo "\$HOME will not be evaluated because it is backslash-escaped"
EOF

In this example:

• $(date) is immediately executed and replaced with the current date
• \$HOME remains as the literal value $HOME due to the escaped dollar sign

The output result is:

#!/bin/sh
# Created on Fri Feb 16 11:00:18 UTC 2018
echo "$HOME will not be evaluated because it is backslash-escaped"

Handling Escape Sequences and Color Codes

In scenarios involving terminal color output, Here Document's handling of escape sequences also requires attention. Referencing the forum discussion case:

NORMAL="\033[0m"
WHITE="\033[1;37m"
cat << EOF
Syntax:
${WHITE}${0##*/} -h|--help${NORMAL}
${WHITE}${0##*/} [-k|--kernel] [--no-sync] [-v|--verbose] FILE${NORMAL}
EOF

Since the cat program itself does not process backslash escape sequences, the output directly displays literal values like \033[1;37m instead of actual color codes.

Solution One: Predefining Escape Sequences

Use printf or echo -e to preprocess escape sequences:

e=$(printf "\e")
NORMAL="${e}[0m"
WHITE="${e}[1;37m"
cat << EOF
${WHITE}Colored Text${NORMAL}
EOF

Solution Two: Using tput Command

For better terminal compatibility, using the tput command is recommended:

BOLD=$(tput bold)
NORM=$(tput sgr0)
cat << EOF
${BOLD}HAI THERE${NORM} this is a test.
EOF

This method leverages the terminfo database, automatically adapting to different terminal characteristics.

Bash Official Documentation Analysis

According to relevant sections from the Bash manual:

If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. If word is quoted, no expansion is performed on the lines.

This mechanism explains why quoting the delimiter completely disables expansion, while selective escaping enables fine-grained control.

Best Practices Summary

Based on the above analysis, the following best practices should be followed when using Here Documents:

• Disable Expansion by Default: Always use quoted delimiters unless immediate expansion is explicitly required
• Choose Appropriate Quoting Method: Either single quotes or backslash escaping works, maintain consistency
• Handle Escape Sequences: For special sequences like color codes, use predefined variables or tput
• Test and Verify: Carefully check output content before actual deployment to ensure it meets expectations

By properly understanding and utilizing Here Document's expansion control mechanisms, developers can avoid many common script writing errors, improving code reliability and maintainability.