User402841's questions

I have set up gunicorn to server two Django sites, as per this setup. This works well. If I run:

# systemctl start gunicorn@my_website.service

Then all is well and my website is being served as expected. So I move to enable this on boot:

# systemctl enable gunicorn@my_website.service

Let's check it's enabled:

# systemctl is-enabled gunicorn@my_website.service
enabled

Looks good. Now, let's reboot...

Turns out my site is not up. nginx is working, it just seems that gunicorn isn't doing it's thing. Let's investigate:

# systemctl status gunicorn@my_website.service
 ● gunicorn@my_website.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/[email protected]; indirect; vendor preset: enabled)
   Active: inactive (dead)

Hmmm... that looks weird. Checking out the log since boot:

# journalctl -u gunicorn@my_website.service -b
-- Logs begin at Mon 2019-04-08 06:04:03 UTC, end at Thu 2020-10-15 13:23:30 UTC. --
-- No entries --

Very puzzling... not even a log entry! When I start the service manually we're back in operations:

# systemctl start gunicorn@my_website.service
# systemctl status gunicorn@my_website.service
● gunicorn@my_website.service - gunicorn daemon
   Loaded: loaded (/etc/systemd/system/[email protected]; indirect; vendor preset: enabled)
   Active: active (running) since Thu 2020-10-15 13:25:29 UTC; 30s ago
 Main PID: 1272 (gunicorn)
    Tasks: 4 (limit: 1151)
   CGroup: /system.slice/system-gunicorn.slice/gunicorn@my_website.service
           ├─1272 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application
           ├─1294 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application
           ├─1297 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application
           └─1298 /usr/bin/python3 /usr/local/bin/gunicorn --access-logfile - --workers 3 --bind unix:/home/my_website/gunicorn.sock my_website.wsgi:application

Oct 15 13:25:29 web systemd[1]: Started gunicorn daemon.
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1272] [INFO] Starting gunicorn 19.9.0
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1272] [INFO] Listening at: unix:/home/my_website/gunicorn.sock (1272)
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1272] [INFO] Using worker: sync
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1294] [INFO] Booting worker with pid: 1294
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1297] [INFO] Booting worker with pid: 1297
Oct 15 13:25:29 web gunicorn[1272]: [2020-10-15 13:25:29 +0000] [1298] [INFO] Booting worker with pid: 1298

And indeed, my website works now! But why is this seemingly not even run on boot when it's enabled? How to debug this?

Running Ubuntu 18.04.

As requested, the content of the [email protected] file below:

# cat /etc/systemd/system/[email protected]
[Unit]
Description=gunicorn daemon
After=network.target
PartOf=gunicorn.target
# Since systemd 235 reloading target can pass through
ReloadPropagatedFrom=gunicorn.target

[Service]
User=ubuntu
Group=www-data
WorkingDirectory=/home/%i/
ExecStart=/usr/local/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/home/%i/gunicorn.sock \
          %i.wsgi:application

[Install]
WantedBy=gunicorn.target

I'd like to invoke sending an e-mail (or running a particular script, if that would be possible) when a certain iptables DROP rule is being run. (For instance, when I'm blocking an IP after 5 failed SSH logins, I'd like to run mail -s "SSH Blocked" [email protected] or something similar. Is there a way to directly have iptables execute this?

If not, then I guess I'd need to scan the logs with an external tool and then send out emails. Any recommended tool for this? Please note that I'm using systemd so I am using journalctl and not old fashioned log files.

I've got three Fit-PCs in use. They are being used as light-weight Linux servers. Unfortunately, on Jun 30, the first of them failed to start due to the leap-second bug. I tried rebooting it a few times, but the screen remained blank after the third bootup-attempt. This appeared to be hardware-related and we took it to a repair-man. He told us something had overheated and that the motherboard was broken. He was able to recover the data, but the fit-pc was written off.

The second Fit-PC was unable to reboot a few days later (first time we actually tried to reboot). With apparently sheer luck, it rebooted on the third attempt, and it is now working fine.

The third Fit-PC had not given any problems. When I found out the other ones failed due to the Leap-Second, I actually thought we were lucky with this third one. Fact is, the recent slowness of the server was most likely due to this same bug, and now that I rebooted this machine (first time after Jun 30), it's giving me the exact same symptoms as the other ones. These symptoms are:

• Initial reboot attempt fails; OS does not load.
• I connect a screen to see what is going on. Remains black.
• I reboot again. I now see the regular loading screen ("Intel Atom..."), but this freezes
• I try to reboot again.
• Screen now simply does not activate at all. It does now show any sign of life. The monitor simply acts as if nothing is sending any signal, so I have no way to interact with the CPU whatsoever.

I've trying to reboot about 4 times now, but am very much fearing the same problem as before. Where I live the Fit-PCs are uncommon and I am not sure if there are qualified techs who actually know how to repair this (and I am not even sure if the diagnosis of the other tech was correct). So I am asking: do you also think my motherboard was overheated and was yet another Fit-PC bricked, or is there something else I can do?

EDIT: Using Ubuntu 12.04 on all of the Fit-PCs.

EDIT:

I also considered a power-failure. But there are a few inconsistencies:

• the servers are on three different sites,
• no power surge was reported and no other hardware was affected - weather was sunny and calm,
• the only similarity between the three machines was that they started acting odd every since Jun 30 (the third one was having high loads but I failed to recognize this until the first reboot since Jun 30, which I did today).

I could also not find other Fit-PCs affected by the leap-second, but am simply not sure what else could cause this...