Home / ubuntu / Questions / 1540157
Accepted
gatorback
Asked: 2025-02-01 02:56:53 +0800 CST

python script fails to start in service

  • 772

CONTEXT

  • Ubuntu 22.04 Desktop: MQTT server

  • Goal: run python (paho with logger) script as a service

  • .py logging routine tests as expected (not in an venv):

python3 mqtt2log.py

line 6 of the script is: import paho.mqtt.client as mqtt

A .service file is coped to /etc/system/system:

# run python script as service
# https://unix.stackexchange.com/a/634422/182280e
# copy this .service description to /etc/systemd/system/myscript.service
# /home/user/mqtt2log.py

[Unit]
Description=mqtt2log.py

[Service]
ExecStart=/usr/bin/python3 /home/user/mqtt/mqtt2log.py

[Install]
WantedBy=multi-user.target

OBSERVATIONS

ls -l /etc/systemd/system/mqtt2log.service returns:

-rwxrwxrwx 1 root root 314 Jan 11 17:41 /etc/systemd/system/mqtt2log.service

sudo systemctl status mqtt2log returns:

Jan 31 13:01:30 mqtt systemd[1]: Started mqtt2log.py.
Jan 31 13:01:30 mqtt python3[158388]: Traceback (most recent call last):
Jan 31 13:01:30 mqtt python3[158388]:   File "/home/user/mqtt/mqtt2log.py", line 6, in <module>
Jan 31 13:01:30 mqtt python3[158388]:     import paho.mqtt.client as mqtt
Jan 31 13:01:30 mqtt python3[158388]: ModuleNotFoundError: No module named 'paho'
Jan 31 13:01:30 mqtt systemd[1]: mqtt2log.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 13:01:30 mqtt systemd[1]: mqtt2log.service: Failed with result 'exit-code'.

The error messages seem to indicate the paho module can not be found in line 6: import paho.mqtt.client as mqtt

QUESTIONS

Error returned: ModuleNotFoundError: No module named 'paho'

  1. Why is the paho module not found?

  2. What diagnostic steps can be taken?

I am puzzled by the fact that the python script runs from the command line. Maybe this is somehow a permissions issue?

MQTT / Paho INSTALL

sudo apt update
sudo apt-get install mosquitto
sudo vi /etc/mosquitto/mosquitto.conf 
    allow_anonymous true
    listener 1883
sudo systemctl restart mosquitto
sudo apt install python3-pip
pip3 install paho-mqtt
sudo apt install -y mosquitto-clients
22.04

2 Answers

  1. Best Answer

    The problem is pip. Your dependency - paho-mqtt - when installed via pip will only install in your userspace or into a venv because it can't write to the system.

    However, due to PEP 668 (and only enabled on Python 3.12 in Ubuntu 24.04 and later, it seems, currently), pip will NOT install into an externally managed environment (aka system Python on Ubuntu systems). You may run into this, in which case you should create a dedicated venv and then call the Python script with the full path to the venv's Python executable so it knows which PYTHON_ROOT to use (the venv sets this at runtime)

    My suggestion would be to make a dedicated venv for your application, and then your system call the Python executable and file directly with the full path to the executable in the venv.

    For example, if your working directory for your service is /opt/myservice/ you would do:

    python3 -m venv /opt/myservice/venv
source /opt/myservice/venv/bin/activate
pip3 install paho-mqtt

    ... and then in your service execution line (assuming SystemD), you'd have:

    ...

[Service]
...
WorkingDirectory=/opt/myservice
ExecStart=/opt/myservice/venv/bin/python /opt/myservice/script.py
...

    and thus directly call the service's scripts, etc. from the venv's Python.

    Make sure that the ownership of all the files, the venv, etc. is the user and group your service runs as though!

    If you absolutely want to ignore PEP 668, and introduce the potential system/module conflicts that the protections are intentionally trying to shield you from, recognize that you run the risk of damaging your system packages, etc. by following the next instructions. You accept all risk on yourself by doing this!

    In versions of Python earlier than 3.12 in Ubuntu, you can run this to install your module of choice at the system level:

    sudo pip3 install paho-mqtt

    In Python 3.12 and later, you have to do this:

    sudo pip3 install --break-system-packages

    Note that you accept all the risk of breaking your system Python or introducing conflicts with your system's installed packages by doing this.

    • 1

  2. Jobs run through cron (or startup scripts, or systemd scripts), aren't run in the same runtime environment that you have on your desktop. Startup scripts are run as root. None of your PATH changes, or other environment variable settings from ~/.bashrc are automatically propagated to your cron job. For example, there's no $DISPLAY, so GUI programs need special treatment (read man xhost).

    One can set environment variables for all one's cron jobs in the crontab file Read man 5 crontab.

    Look at the results of echo "=== id ===";id;echo "=== set ===";set;echo "=== env ===";env | sort;echo "=== alias ===";alias in each of your environments.
    An easy way is to store the commands in a bash script, and execute it from your terminal session, saving the output, then, execute the script from the "other" environment, saving the output. Compare the saved outputs using diff.

    #!/bin/bash
echo "=== id ===";id
echo "=== set ===";set
echo "=== env ===";env | sort
echo "=== alias ===";alias`

    Since the command part of the crontab line is, by default, interpreted by /bin/sh, which has a simpler syntax than /bin/bash, I recommend having command be a call to a bash script (executable, mounted, starts with #!/bin/bash) which sets up the environment, then calls the desired program.

    • -1