Summary

The website provides guidance on running Python scripts in the background using nohup and the ampersand, managing background jobs, and checking logs.

Abstract

The article is a technical guide aimed at Python developers, detailing methods to run Python scripts in the background on Unix-like systems. It covers the use of the ampersand to run scripts without occupying the terminal and introduces the nohup command to ensure scripts continue running even if the terminal is closed. The article also explains how to redirect output to log files, check background processes, and terminate them if necessary. It emphasizes the importance of using nohup for scripts running on remote servers to prevent accidental termination. The guide includes code examples, command outputs, and a recommendation to use nohup for all background jobs. Additionally, it references related posts for further reading on Linux commands and executing shell commands in Python.

Opinions

The author suggests that developers should run scripts with nohup to avoid accidental job termination, especially on remote servers.
It is implied that capturing logs in a file rather than displaying them in the terminal is the preferred method for background processes.
The article conveys that using the -u option with Python can be beneficial for live logging when running scripts in the background.
The author provides a clear recommendation to use nohup in conjunction with output redirection for running scripts in the background to ensure their resilience against terminal disconnection.
The use of jobs, kill, lsof, ps, and grep commands is presented as standard practice for managing and monitoring background jobs in Unix-like environments.

How to Run Python Scripts in the Background

Learn to run Python scripts with nohup and check the process and logs

As a Python developer, it’s a common job to write Python scripts and run them in the background. In this post, we will introduce how to run Python scripts in the background with the ampersand and the nohup command. We will also introduce how to check the processes and logs, especially when the terminal running the script is closed.

Using ampersand (&)

First, let’s write a simple Python script sleep.py which takes 60 seconds to run and will print something every second:

import time

for i in range(60):
    time.sleep(1)
    print(i)

The easiest way to run a script in the background is by appending an ampersand (&) to the end of the command.

python sleep.py &

However, since our script constantly prints logs in the console, the logs will just keep popping up, and we can’t do anything else in the same terminal:

When a script is run in the background, we normally don’t want to see the logs popping up all the time. Instead, we want the logs to be saved in a file that can be checked later when the job is completed. This can be done with STDOUT and STDERR redirection as shown in this post:

 python sleep.py > sleep.log 2>&1 &

> sleep.log 2>&1 means redirecting both STDOUT and STDERR to the same file sleep.log.

However, if you check the log file before the script is completed, you will see nothing logged there. This is because the logs are buffered by default to improve IO efficiency.

For demonstration, we can use the -u option of Python to force the STDOUT and STDERR streams to be unbuffered so we can see the logs live:

python -u sleep.py > sleep.log 2>&1 &

An issue with the ampersand is that if you exit the terminal, the running job will be stopped. This may not be an issue if you run the script on your local computer. However, it is a critical issue if the script is run on a remote computer as we can get disconnected for various reasons.

To solve this problem, we can use the nohup command which will keep the job running even after the terminal is closed.

Using nohup (no hang up)

To prevent the process from being stopped when the terminal is closed, we can append nohup to the command above:

nohup python -u sleep.py > sleep.log 2>&1 &

In this way, the script will continue running even when the terminal is closed.

Use jobs to check background jobs

The jobs command in Linux can be used to manage background and suspended jobs in the current shell. Note that it can only be used in the current shell where the background jobs were started:

$ jobs
[1]  + running    nohup python -u sleep.py > sleep.log 2>&1

We can kill the job by job id, which is 1 in this case:

$ kill %1
[1]  + 26549 terminated  nohup python -u sleep.py > sleep.log 2>&1

We can use kill -9 for a more forceful kill:

$ kill -9 %1
[1]  + 26639 killed     nohup python -u sleep.py > sleep.log 2>&1

Find the process running the script when the terminal is closed

It’s a bit more challenging to find the process running the script with nohup when the terminal is closed. There are various ways to do it, we will introduce two that we use frequently.

Using lsof to check the log file

$ lsof sleep.log          
COMMAND      PID  USER  FD  TYPE DEVICE SIZE/OFF     NODE NAME
python3.12 27852  lynn  1w   REG   1,18       62 48848553 sleep.log

lsof lists all processes that have sleep.log open, showing you which commands are writing to it.

We can then kill the process by process ID (PID):

kill 27852

Using ps and grep to identify the process ID

We can also find the PID by script name with the ps and grep commands:

$ ps -ef | grep sleep.py
501 27852   896   0 11:54PM ttys001    0:00.04 python -u sleep.py

In this post, we introduced how to run Python scripts in the background with the ampersand and the nohup command. It’s recommended to always use the nohup command to run scripts in the background in case the terminal is closed for any reason. Otherwise, your job may be stopped accidentally and you will have to restart it, if it’s restartable, otherwise you will need to run it from scratch.

In Plain English 🚀

Thank you for being a part of the In Plain English community! Before you go:

Be sure to clap and follow the writer ️👏️️
Follow us: X | LinkedIn | YouTube | Discord | Newsletter
Visit our other platforms: CoFeed | Differ
More content at PlainEnglish.io