Best Way to Run All Python Files in a Directory

To run a Python file in a directory, we generally use the Python or python3 command. However, it only runs a single file at a time. But executing one file each on a shell script seems hectic. Therefore, we must come up with a way to execute all files present in a directory simultaneously.

In this article, let's go through the different methods to run all the Python files in a directory at a time.

• Using subprocess Module
• Using exec() function
• Using shell scripts
• Using multiprocessing

Using subprocess.run() 

The subprocess module allows us to run Python scripts as separate processes, which ensures isolation between them. You can execute all the Python files in a directory using the <b>subprocess.run()</b> method.

Example

Following is the example in which all the Python files will be executed in the defined path -

import subprocess
from pathlib import Path

directory = Path(r"D:\Tutorialspoint")

# Loop through all Python files and execute them
for file in directory.glob("*.py"):
   print(f"Running: {file}")
   subprocess.run(["python", str(file)], check=True)

Following is the output of the executed Python files in the given directory -

Running: D:\Tutorialspoint\example.py
Welcome to Tutorialspoint.
Running: D:\Tutorialspoint\sample.py
Happy Learning With Tutorialspoint

Using exec() function

The exec() function can execute Python code within the same process as the subprocess module. This method is less isolated but can be useful if the scripts are tightly coupled.

Example

Here is the example usage of the exec() function to run all Python files -

from pathlib import Path

directory = Path(r"D:\Tutorialspoint")

# Loop through and execute scripts within the same process
for file in directory.glob("*.py"):
   print(f"Executing: {file}")
   with open(file) as f:
      exec(f.read())

Here is the output of executing all the Python files in the defined path -

Executing: D:\Tutorialspoint\example.py
Welcome to Tutorialspoint.
Executing: D:\Tutorialspoint\sample.py
Happy Learning With Tutorialspoint

Using Shell Script

A Shell Script is used to run all Python files in a directory from outside Python. It's a simple and efficient way to execute multiple scripts, especially in Unix-based environments such as Linux and macOS.

Example

Below is an example in which we use a shell script to run all Python files in a directory -

#!/bin/bash
# Directory containing Python scripts
directory="D:\Tutorialspoint"

# Loop through all Python files and execute them
for f in "$directory"/*.py; do
   echo "Running: $f"
   python "$f"
done

Below is the output of executing all the Python files in the defined path -

Running: /path/to/your/scripts/example.py
Welcome to Tutorialspoint.
Running: /path/to/your/scripts/sample.py
Happy Learning With Tutorialspoint

Using Multiprocessing in Python

The multiprocessing module in Python allows us to run multiple Python files in parallel in which can significantly speed up execution when dealing with many scripts. Each script is executed in its own separate process to ensure isolation and better performance.

Example

Below is an example of using multiprocessing to run all Python files in a directory -

import subprocess
import multiprocessing
from pathlib import Path

# Directory containing Python scripts
directory = Path("D:/Tutorialspoint")

# Function to run each script
def run_script(file):
   print(f"Running: {file}")
   subprocess.run(["python", str(file)], check=True)

# Use multiprocessing to run scripts in parallel
if __name__ == "__main__":
   python_files = list(directory.glob("*.py"))
   with multiprocessing.Pool() as pool:
      pool.map(run_script, python_files)

Below is the output of executing all the Python files in parallel from the defined path -

Running: D:\Tutorialspoint\example.py
Running: D:\Tutorialspoint\sample.py
Welcome to Tutorialspoint.
Happy Learning With Tutorialspoint

We can choose any of the above best ways to run all Python files in a directory, depending on our specific use cases.