Updated July 28, 2023

Introduction to Python Subprocess

The process is defined as a running program; each process has its own different states like its memory, list of open files, etc. As we know that every process executes one statement at a time. In Python versions 2 and 3, there are several different ways to run the process externally to interact with the operating system. Earlier versions had os modules to do this, but now in the latest versions, these os modules are replaced by subprocess modules to call external commands from Python code. Subprocess modules are independent because they run completely independent entities in their private system state and execute the main thread concurrently with the original process.

Working of Python Subprocess with Syntax and Example

In the latest versions of Python as it supports the subprocess module instead of the old os modules. The process that creates the subprocess work with other things where the subprocess carries out its work even though the process is working with something else. The subprocess module is used to run new programs through Python code by creating a new process that helps gain input or output or error pipes and gain exit codes of various commands.

The Subprocess provides a high-level interface than the older os modules like os.system(), os.spawn(), os.popen(), etc. these all are replaced with a new spawn process, connect to their input or output or error pipes and also to gain their return codes.

Syntax:

subprocess.call (args, *, stdin=None, stdout=None, stderr=None, shell=False)

args: this is used when you want the commands to be executed.
stdin: it carries the value of standard input to be passed.
stdout: it carries the value of output, which is obtained by standard output stream.
stderr: this carries the error obtained from the standard error stream.
shell: It returns a Boolean value. True when commands get executed.

So you can run the command using the args parameter, wait until the command gets executed, and then return the returncode attribute; if it’s zero, it returns the function; else, it raises CalledProcessError.

Example #1

We want to execute the echo command, a UNIX command in the Python script, but this will give us an error saying it’s a syntax error as the echo is not a built-in command Python. Echo is a UNIX command to print the value or statement, whereas, in Python, we have printed as a command to print the value or statement.

So to run these UNIX commands in Python, we have to create a new process that can be done using subprocess modules. This was earlier done using the os.system(), but this was only for the older versions; instead, we can use subprocess.call(), which executes a program as a child process in which we cannot complete our needs. So we need to use open in advanced cases as popen is a class and not just a method.

It is as shown below:

import subprocess
subprocess.Popen('ls –la', shell = True)

The above program can also use a subprocess.call(), but this executes the program as a child process which is executed by the operating system as a separate process, so as per the Python documentation, it is recommended to use popen class. It can be written as

import subprocess
subprocess.call('ls –la', shell =True)

In the above, both the programs shell argument will take ‘false’ as default, but we want the commands to be executed, then we have to assign the shell argument as ‘True’.

If the subprocess is not imported, then as discussed earlier, no commands can be executed externally in Python codes, so to do this, we need to use subprocess modules. To do this, we need to first import the subprocess; else, you will get the error saying no such file is available, which means the ls command cannot be executed in Python as its UNIX command. It is shown in the below screenshot.

In subprocess documentation, you get a list of popen class methods few of them are popen.wait(), which will wait for the child process to get terminated, popen, kill () to kill the child process, popen.terminate() stops the child process, popen.communicate() this is the most commonly used method which helps to interact with the process. Let us consider about communicate() method, which allows reading and sending data from and to the standard input and standard output, respectively.

Example #2

Let us take an example to execute windows commands like dir and sort. So there are two commands; to execute these commands, we have to create two subprocesses as we want the directories to be sorted in reverse order in which we use the ‘\R’ option in sort call. In the below program, we use stdout of p1 as input to p2, so we declare PIPE in stdout for p1. So we have to close p1 as we have used it as input to p2. And as usual, the communication is done using the communicate() method.

import subprocess
p1 = subprocess.Popen('dir', shell =True, stdin = None, stdout = subprocess.PIPE, stderr = subprocess.PIPE)
p2 = subprocess.Popen('sort /R', shell=True, stdin=p1.stdout)
p1.stdout.close()
out, err = p2.communicate()

Output:

The above example has two subprocess dir and sort commands to be executed through Python script as given above, and the dir command process is an input to the sort command process to sort the obtained directories in reverse order.

Conclusion

In Python, versions 2 and 3 have separated ways of calling commands externally through the Python code. In Python 2, we had operating system (os) methods as a better way to call the child process of other languages through the Python script. However, in the present version, it uses the subprocess module, which has several methods. Popen is one of the most used classes in both versions, but in the latest versions, it again has many other methods in it to make it very efficient. Subprocess also has a call(), check_stdout(), check_stdin() are also some of the methods of a subprocess which are used instead of Popen class’s method communicate().