Replacing Whitespace with Line Breaks Using sed to Create Word Lists

When working with text data, it is often necessary to separate words from continuous text into individual lines, such as creating vocabulary lists for language learning. sed (stream editor) is a powerful command-line tool that can achieve this through regex matching and substitution operations. This article uses Greek text as an example to explain in detail how to use sed to replace whitespace characters with line breaks.

Basic sed Command Structure

The basic format of a sed command is sed 's/pattern/replacement/flags', where s denotes substitution, pattern is the regex to match, replacement is the content to replace with, and flags are modifiers. For the task of replacing whitespace with line breaks, the key is constructing the correct pattern to match whitespace characters.

Using Character Classes to Match Whitespace

The POSIX standard defines character classes to match specific types of characters. The [[:blank:]] character class matches spaces or tabs, while [[:space:]] matches all whitespace characters including newlines. In most cases, [[:blank:]] is preferable as it avoids accidentally matching existing newlines in the text.

$ echo 'τέχνη βιβλίο γη κήπος' | sed -E -e 's/[[:blank:]]+/\n/g'

In the above command, [[:blank:]]+ matches one or more consecutive spaces or tabs, \n represents a newline, and the /g modifier ensures global replacement. The output is as follows:

τέχνη
βιβλίο
γη
κήπος

Quantifiers and Extended Regular Expressions

Quantifiers control how many times a pattern matches. The + quantifier means match one or more times, enabled by the -E option for Extended Regular Expression (ERE) syntax. Without -E, \+ must be used instead, which is part of Basic Regular Expression (BRE) syntax.

# Using extended regex
sed -E -e 's/[[:blank:]]+/\n/g'
# Using basic regex
sed -e 's/[[:blank:]]\+/\n/g'

Handling File Input and Output

sed can process files directly and redirect the output to a new file. For example, to replace whitespace in files lesson1 and lesson2 and save the result to all-vocab:

sed -E -e 's/[[:blank:]]+/\n/g' lesson1 lesson2 > all-vocab

Compatibility Considerations

Different versions of sed vary in regex support. Older versions may not recognize \n for newlines or the + quantifier. In such cases, a more compatible but verbose syntax can be used:

sed -e 's/[ \t][ \t]*/\
/g'

Here, [ \t] matches a single space or tab, and [ \t]* matches zero or more, simulating the + quantifier. The newline is inserted literally, with a backslash escaping the newline in the command line.

Perl-Compatible Regular Expressions

For sed implementations that support Perl-Compatible Regular Expressions (PCRE), \s can be used to match any whitespace character, offering a more concise alternative to POSIX character classes.

sed -E -e 's/\s+/\n/g' old > new

However, note that \s availability depends on the specific sed implementation and may not be cross-platform compatible.

Importance of Quote Usage

In the shell, single and double quotes handle backslashes differently. Single quotes preserve the literal value of all characters, making them ideal for protecting backslashes in regex patterns; double quotes allow variable expansion and certain escape sequences. Therefore, single quotes are recommended for sed commands.

# Correct: using single quotes
sed -e 's/[[:blank:]]+/\n/g'
# Potentially problematic: double quotes may cause unintended escaping
sed -e "s/[[:blank:]]+/\n/g"

Practical Application Example

Assume a text file greek.txt contains Greek vocabulary: τέχνη βιβλίο γη κήπος. Use the following command to generate a word list:

sed -E -e 's/[[:blank:]]+/\n/g' greek.txt > vocabulary.txt

After execution, vocabulary.txt will contain one word per line, facilitating further study or analysis.

Through this article, readers can master the basics of using sed for text substitution, understand the application of character classes and quantifiers in regex, and be mindful of compatibility issues across environments. Although sed syntax can be complex, its powerful text-processing capabilities make it an indispensable tool in the command-line arsenal.