String Length Calculation in Bash: From Basics to UTF-8 Character Handling

Fundamentals of String Length Calculation in Bash

In Bash scripting, calculating string length is one of the most fundamental operations. The most straightforward approach uses the ${#string} syntax, which returns the number of characters through parameter expansion. For example:

myvar="some string"
size=${#myvar}
echo "$size"  # Output: 11

This method is simple and efficient, suitable for most ASCII string scenarios. However, when dealing with UTF-8 strings containing multi-byte characters, ${#string} returns character count rather than byte count, which may cause discrepancies in certain application scenarios.

UTF-8 String Length Processing

UTF-8 encoding uses variable-length bytes to represent characters, with ASCII characters occupying 1 byte while other characters may occupy 2-4 bytes. This creates a distinction between character length and byte length.

Using wc Command for Length Calculation

The wc command provides options for counting both bytes and characters:

echo -n "Généralité" | wc -c    # Count bytes: 13
echo -n "Généralité" | wc -m    # Count characters: 10

By combining these two options, both character count and byte count can be obtained simultaneously:

for string in "Généralités" "Language" "Théorème" "Février" "Left: ←" "Yin Yang ☯"; do
    strlens=$(echo -n "$string" | wc -mc)
    chrs=$((${strlens% *}))
    byts=$((${strlens#*$chrs }))
    printf " - %-*s is %2d chars length, but uses %2d bytes\n" \
        $((14 + $byts - $chrs)) "$string" $chrs $byts
done

The output clearly demonstrates the difference between character length and byte length:

 - Généralités    is 11 chars length, but uses 14 bytes
 - Language       is  8 chars length, but uses  8 bytes
 - Théorème       is  8 chars length, but uses 10 bytes
 - Février        is  7 chars length, but uses  8 bytes
 - Left: ←        is  7 chars length, but uses  9 bytes
 - Yin Yang ☯     is 10 chars length, but uses 12 bytes

Pure Bash Solutions

Although the wc command is powerful, each invocation requires creating a subprocess, resulting in poor performance when used frequently in loops. Here are several pure Bash solutions:

Through Environment Variable Settings

By temporarily modifying LANG and LC_ALL environment variables, Bash can be forced to calculate string length in bytes:

myvar='Généralités'
chrlen=${#myvar}
oLang=$LANG oLcAll=$LC_ALL
LANG=C LC_ALL=C
bytlen=${#myvar}
LANG=$oLang LC_ALL=$oLcAll
printf "%s is %d char len, but %d bytes len.\n" "${myvar}" $chrlen $bytlen

While effective, this method requires saving and restoring environment variables, making the code somewhat cumbersome.

Using printf %n Format

A more elegant solution utilizes the printf command's %n format, which stores the number of bytes processed into a specified variable:

myvar='Généralités'
chrlen=${#myvar}
printf -v _ %s%n "$myvar" bytlen
printf "%s is %d char len, but %d bytes len.\n" "${myvar}" $chrlen $bytlen

Here, printf -v _ redirects output to temporary variable _, while the %n format stores byte count into bytlen variable. This approach doesn't require modifying environment variables and results in cleaner code.

Practical Function Encapsulation

For ease of reuse, string length calculation functionality can be encapsulated into functions:

Character and Byte Length Difference Calculation

strU8DiffLen() {
    local -i bytlen
    printf -v _ %s%n "$1" bytlen
    return $((bytlen - ${#1}))
}

This function returns the difference between byte length and character length, facilitating dynamic adjustments in formatted output.

Complete String Analysis Function

showStrLen() {
    local -i chrlen=${#1} bytlen
    printf -v _ %s%n "$1" bytlen
    printf "String '%s' is %d bytes, but %d chars len: %q.\n" "$1" $bytlen $chrlen "$1"
}

Usage example:

showStrLen "théorème"
# Output: String 'théorème' is 10 bytes, but 8 chars len: $'th\303\251or\303\250me'

Performance Comparison Analysis

To evaluate performance differences between methods, we conducted benchmark tests with 1000 string length calculations:

wc Command Method

string="Généralité"
time for i in {1..1000}; do
    strlens=$(echo -n "$string" | wc -mc)
done
echo $strlens

Test result: Execution time approximately 2.637 seconds

printf %n Method

string="Généralité"
time for i in {1..1000}; do
    printf -v _ %s%n "$string" bytlen
    chrlen=${#string}
done
echo $chrlen $bytlen

Test result: Execution time approximately 0.005 seconds

Performance comparison shows that the printf %n method is approximately 500 times faster than the wc command method, primarily due to avoiding frequent subprocess creation.

Special Character Handling

When processing strings containing special characters, attention must be paid to quote usage:

special_string='Hello, $name!'
echo ${#special_string}  # Output: 14

Due to single quotes, $name is treated as a literal rather than a variable reference.

Alternative Calculation Methods

Besides the aforementioned methods, Bash supports other string length calculation approaches:

Using expr Command

str="Test String@#$"
n=$(expr "$str" : '.*')
echo "Length of the string is : $n"

Using awk Command

str="this is a string"
n=$(echo $str | awk '{print length}')
echo "Length of the string is : $n"

Using while Loop

str="this is a string"
n=0
while read -n1 character; do
    n=$((n+1))
done < <(echo -n "$str")
echo "Length of the string is : $n"

Practical Application Recommendations

When selecting string length calculation methods, consider the following factors:

Performance Requirements: For scenarios requiring frequent string length calculations, prioritize the printf %n method.

Encoding Environment: In pure ASCII environments, ${#string} syntax is sufficient; in UTF-8 multilingual environments, consider the distinction between characters and bytes.

Code Readability: For simple length calculations, ${#string} syntax is most intuitive; for complex character processing, encapsulation into functions improves code maintainability.

Compatibility: ${#string} syntax works in most Bash versions, while printf %n format requires newer Bash version support.

Conclusion

Bash provides multiple methods for calculating string length, each with its applicable scenarios. The ${#string} syntax is simple and efficient, suitable for most ASCII string scenarios. When handling UTF-8 multi-byte characters, distinguishing between character length and byte length becomes necessary, where the printf %n format provides the optimal solution. By appropriately selecting calculation methods and encapsulating functions, developers can write both efficient and maintainable Bash scripts.