Python subprocess module

Last Updated : 21 Aug, 2024

The subprocess module present in Python(both 2.x and 3.x) is used to run new applications or programs through Python code by creating new processes. It also helps to obtain the input/output/error pipes as well as the exit codes of various commands. In this tutorial, we’ll delve into how to effectively use subprocessing modules to execute external functions from your Python code, pass data to them, and get results Whether you’re a beginner or an experienced Python user developer, this guide will prove useful with your skills and abilities.

An Introduction to Subprocess in Python

The subprocess module in Python is a built-in module that allows us to create new child processes. We can get exit codes, output, and error messages from these child processes. It is a valuable tool for executing external functions or commands in your Python code. Think of it as a way to programmatically interact with your computer using Python instead of manually entering commands in the terminal. This module simplifies the implementation process and seamlessly integrates external programming into your Python workflow.

For example, we can use a subprocessing module to initiate shell commands such as “ls” or “ping” and retrieve their output directly from your Python script. It also makes it easy to execute other Python scripts or executable files, such as .exe files on Windows. Also, the subprocess module gives you control over the input and output of the executed process. This means that you can accurately monitor the data sent to the application and capture the data it returns.

Uses of Subprocess Module

Run system commands or external programs from your Python script.
They can handle input, output, and errors caused by child programs in your Python code
Capture the output (stdout) and error messages (stderr) generated by these commands.
Communicate with the running process by sending input (stdin).
Wait for the command to complete and obtain its return code to check for success or failure.

What is subprocess Popen in Python?

`subprocess.Popen` is a low-level interface for running subprocesses, while subprocess.run is a high-level wrapper around Popen intended for ease of use. Popen allows you to start a new process and interact with its standard input, output, and error streams. It returns a handle to the running procedure that can be used to wait for the procedure to complete, check its return code, or abort it. run is a more flexible function that allows you to run a command and capture its results in a single call, allowing you to specify options for a command without having to create a Popen object and manage the streams yourself, such as whether to raise or exception if the command fails.

In general, you should use run if you just want to run the command and capture its output and Popen if you need more control over the process, such as interacting with its input and output streams. The Popen class takes the same arguments as run(), including args specifying which commands to run and other optional arguments such as stdin, stdout, stderr, shell, cwd, and env

Python Subprocess check_output

In the subprocess module, check_output is a function that closely resembles run(). It has the specific behavior of returning only the standard output of the command. However, it differs in how it handles command execution outcomes: if the command’s return code is non-zero, it triggers a CalledProcessError exception. The check_output function shares the same set of arguments as the run() function. This includes the ‘args’ parameter, which defines the command to be executed, and it also accepts additional optional arguments like ‘stdin’, ‘stderr’, ‘shell’, ‘cwd’, and ‘env’.

Python

import subprocess
try:
    ans = subprocess.check_output(["python", "--version"], text=True)
    print(ans)

except subprocess.CalledProcessError as e:
    print(f"Command failed with return code {e.returncode}")

Output

Python 3.7.17

Python Subprocess Pipe

The Python subprocess module empowers the creation and interaction with child processes, which enables the execution of external programs or commands. A standout feature of this module is the capacity to establish pipes, facilitating communication between the parent and child processes. To create such pipes, you can employ the subprocess module in combination with the Popen class. By specifying the ‘stdout’ or ‘stdin’ argument as ‘subprocess.PIPE’, you establish this interprocess communication channel. For instance, the code snippet below illustrates a pipe that connects the output of the ‘ls’ command to the input of the ‘grep’ command, which filters the output to display only the lines containing the word “file”.

Python

import subprocess

lsProcess = subprocess.Popen(["ls"], stdout=subprocess.PIPE, text=True)
grepProcess = subprocess.Popen(
    ["grep", "sample"], stdin=ls_process.stdout,
    stdout=subprocess.PIPE, text=True)
output, error = grepProcess.communicate()

print(output)
print(error)

sample_file

Python Subprocess Call

subprocess.call() is a Python function within the subprocess module. It is employed to execute a command in a separate process and wait for its completion. The function returns a return code, which is typically zero for success and non-zero for failure. This function shares its arguments with subprocess.run(), such as ‘args’ for specifying the command to be executed, along with optional parameters like ‘stdin’, ‘stdout’, ‘stderr’, ‘shell’, ‘cwd’, and ‘env’.

By default, the command’s standard output and error are directed to the same output streams as the parent process unless you explicitly redirect them using the ‘stdout’ and ‘stderr’ arguments.

Python

import subprocess
ans = subprocess.call(["python", "--version"])
if ans == 0:
    print("Command executed.")
else:
    print("Command failed.", return_code)

Output

Python 3.7.17
Command executed.

Compile and Run Different Languages in Python using Subprocess

To execute different programs using Python two functions of the subprocess module are used:

1.subprocess.check_call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
Parameters:
args=The command to be executed.Several commands can be passed as a string by separated by “;”.
stdin=Value of standard input stream to be passed as (os.pipe()).
stdout=Value of output obtained from standard output stream.
stderr=Value of error obtained(if any) from standard error stream.
shell=Boolean parameter.If True the commands get executed through a new shell environment.
Return Value:
The function returns the return code of the command.If the return code is zero, the function simply returns(command executed successfully) otherwise CalledProcessError is being raised.

2.subprocess.check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False)
Parameters:
args=The command to be executed. Several commands can be passed as a string by separated by “;”.
stdin=Value of standard input stream to be passed as pipe(os.pipe()).
stdout=Value of output obtained from standard output stream.
stderr=Value of error obtained(if any) from standard error stream.
shell=boolean parameter.If True the commands get executed through a new shell environment.
universal_newlines=Boolean parameter.If true files containing stdout and stderr are opened in universal newline mode.
Return Value:
The function returns the return code of the command.If the return code is zero, the function simply returns the output as a byte string(command executed successfully) otherwise CalledProcessError is being raised.

Example: Let us consider the following code example in different languages.

C++

#include <iostream> 
using namespace std; 
int main() 
{ 
    int a, b; 
    cin >> a >> b; 
    cout << "Hello World from C++.Values are:" << a << " " << b; 
    return 0; 
}

#include<stdio.h> 
int main() 
{ 
    printf("Hello World from C"); 
    return 0; 
}

Java

class HelloWorld { 
    public static void main(String args[]) 
    { 
        System.out.print("Hello World from Java."); 
    } 
}

Output

Hello World from C++.Values are:0 4196176

Python Code to Execute the Above Programs

In this code there are three functions created separate each for executing C,C++ and JAVA program respectively. ‘executeC()’ is used to execute the C program file.‘executeCpp()’ is used to execute the C++ program file.‘executeJava()’ is used to execute the JAVA program file.

In each method It employs the ‘subprocess.check_call’ method to execute the necessary compilation and execution commands. After execution, it prints the return code, which can indicate whether the program ran successfully.

Python

# Python 3 program to demonstrate subprocess
import subprocess
import os

def excuteC():
    # store the return code of the c program(return 0)
    # and display the output
    s = subprocess.check_call("gcc HelloWorld.c -o out1;./out1", shell=True)
    print(", return code", s)

def executeCpp():
    # create a pipe to a child process
    data, temp = os.pipe()
    # write to STDIN as a byte object(convert string
    # to bytes with encoding utf8)
    os.write(temp, bytes("5 10\n", "utf-8"))
    os.close(temp)

    # store output of the program as a byte string in s
    s = subprocess.check_output(
        "g++ HelloWorld.cpp -o out2;./out2", stdin=data, shell=True)

    # decode s to a normal string
    print(s.decode("utf-8"))

def executeJava():
    # store the output of
    # the java program
    s = subprocess.check_output(
        "javac HelloWorld.java;java HelloWorld", shell=True)
    print(s.decode("utf-8"))


# Driver function
if __name__ == "__main__":
    excuteC()
    executeCpp()
    executeJava()

Output

Hello World from C, return code 0
Hello World from C++. Values are:5 10
Hello World from Java.

Python subprocess module – FAQs

What is the Python Subprocess Module?

The Python subprocess module is a standard utility module that allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. This module is intended to replace older modules and functions like os.system, os.spawn, os.popen, popen2, and commands. It provides more powerful facilities for spawning new processes and retrieving their results.

Is Subprocess Part of Python Standard Library?

Yes, the subprocess module is part of the Python standard library. It is included with Python and does not require any additional installation.

What is the Difference Between Import and Subprocess in Python?

Import: The import statement in Python is used to bring modules into the current namespace so that you can access functions, classes, and variables defined within those modules. It’s a fundamental feature of Python that allows code modularization and reusability.
Subprocess: subprocess is a specific module within the Python standard library used for creating and managing additional processes. It provides the capability to spawn new processes, connect to their input/output/error pipes, and retrieve their results. It is not related to the import statement except that you need to import the subprocess module to use its functionality.

What is the Difference Between os and Subprocess in Python?

os: The os module provides a way of using operating system dependent functionality like reading or writing to the filesystem, managing paths, and accessing environment variables. It includes some functions for process management as well, such as os.system which allows you to run shell commands, but it is more limited.
subprocess: The subprocess module is specifically designed for spawning new processes, connecting to their input/output/error pipes, and obtaining their return codes. It offers more control and flexibility than os.system by allowing interaction with the process’s input and output.

Is Subprocess Run Synchronous?

Yes, by default, the subprocess.run() function is synchronous, meaning it waits for the command to complete and returns a CompletedProcess instance. It blocks the calling thread until the process whose creation it has requested has completed, and then it proceeds with the rest of the code. If you need asynchronous behavior, you can use subprocess.Popen which provides more control over how processes are run, including non-blocking behavior.

These explanations clarify the functionality and utility of the subprocess module in Python, distinguishing it from other similar modules and concepts within the language