Python subprocess Module: A Comprehensive Guide to Redirecting Command Output to Variables

Problem Background and Common Misconceptions

Executing external commands and capturing their output in Python is a common task, but beginners often face issues where output is printed to the terminal instead of stored in a variable. For example, the following code:

def storels():
   a = subprocess.Popen("ls", shell=True)
storels()

directly displays the output of the ls command in the terminal, rather than storing it in variable a. Similarly, using redirection to a file:

def storels():
   subprocess.Popen("ls > tmp", shell=True)
   a = open("./tmp")
   # Rest of code
storels()

also fails to prevent terminal output, as subprocess.Popen does not automatically capture stdout or stderr by default.

Core Solution: Using stdout=subprocess.PIPE

To capture command output, explicitly specify the stdout=subprocess.PIPE parameter. For instance, capturing the output of ls:

proc = subprocess.Popen('ls', stdout=subprocess.PIPE)
output = proc.stdout.read()
print(output)

This code redirects output to a pipe, allowing it to be read via proc.stdout.read() and stored in variable output. Sample output might be:

bar
baz
foo

Handling Standard Error (stderr)

Some commands, such as cdrecord --help, send output to stderr instead of stdout. If only stdout is captured, errors like OSError: [Errno 2] No such file or directory may occur due to command not found or path issues. The correct approach is to capture stderr:

proc = subprocess.Popen(['cdrecord', '--help'], stderr=subprocess.PIPE)
output = proc.stderr.read()
print(output)

Sample output:

Usage: wodim [options] track1...trackn
Options:
    -version    print version information and exit
    dev=target  SCSI target to use as CD/DVD-Recorder
    gracetime=# set the grace time before starting to write to #.
...

Merging stdout and stderr

For commands that output to both stdout and stderr, use stderr=subprocess.STDOUT to redirect stderr to stdout and capture them together:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = proc.stdout.read()

Using communicate() Instead of read()

While proc.stdout.read() works, proc.communicate() is preferred as it handles buffering correctly and avoids deadlocks. For example:

proc = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
print('stdout:', out)
print('stderr:', err)

In the output, stdout may be empty, while stderr contains the help text.

Security and Command Passing

Using shell=True can introduce security risks, especially if the command string comes from user input. It is advisable to break the command into a list of tokens:

proc = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)

This prevents shell injection attacks and improves portability.

Simplified Methods in Python 2.7 and Later

For Python 2.7+, subprocess.check_output() offers a more concise interface:

output = subprocess.check_output('ls')

To redirect stderr as well:

output = subprocess.check_output('ls', stderr=subprocess.STDOUT)

For commands with arguments, use a list or shell string:

output = subprocess.check_output(['ls', '-a'])
output = subprocess.check_output('ls -a', shell=True)

Summary and Best Practices

Key points for capturing external command output include: using stdout=subprocess.PIPE or stderr=subprocess.PIPE to redirect streams, preferring communicate() for reading output, avoiding shell=True unless necessary, and leveraging check_output() for simplicity. These methods ensure output is correctly stored in variables without interfering with terminal display.