I have a (formerly working) logrotate setup for OpenResty that I took from a previous Ubuntu 18.04 installation. Yet the logrotate.service fails with this error now ...
error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst
to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
... and I have difficulties understanding why. The new machines are running Ubuntu 20.04, but I don't see why that should make a difference in this case.
First, here is the config:
$ cat /etc/logrotate.d/custom-openresty
/usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
{
daily
rotate 60
maxsize 1G
missingok
notifempty
compress
compresscmd /usr/bin/zstd
uncompresscmd /usr/bin/unzstd
compressoptions -9 --long -T1
compressext .zst
delaycompress
sharedscripts
postrotate
test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
endscript
}
/etc/logrotate.conf
is unchanged and looks like this:
# see "man logrotate" for details
# rotate log files weekly
weekly
# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.
Here is the state of my files (I would expect to get access.log.1
after logrotate ran):
$ ls -alhg /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x 2 root 4.0K Sep 16 20:04 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r-- 1 root 10G Sep 16 19:56 access.log
-rw-r--r-- 1 root 5.5K Sep 16 19:56 error.log
However, the logrotate.service fails with this error:
~$ systemctl status logrotate.service
● logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago
TriggeredBy: ● logrotate.timer
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 27403 (code=exited, status=1/FAILURE)
$ sudo journalctl --unit logrotate.service
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files...
Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result 'exit-code'.
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files.
When I run it (without systemd) in debug mode as root, I get the following output:
# logrotate -v -d /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
...
Creating new state
Handling 1 logs
rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
Now: 2020-09-16 20:24
Last rotated at 2020-09-16 20:11
log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
Now: 2020-09-16 20:24
Last rotated at 2020-09-16 19:00
log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60),
renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59),
...
/logs/access.log.3.zst (rotatecount 60, logstart 1, i 2),
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1),
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0),
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
running script with arg /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
: "
test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
"
To me, everything looks normal. If I run it, it also works:
# logrotate -v /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
...
Creating new state
Handling 1 logs
rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
Now: 2020-09-16 20:26
Last rotated at 2020-09-16 20:11
log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
Now: 2020-09-16 20:26
Last rotated at 2020-09-16 19:00
log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60),
old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist
...
old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1),
old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0),
old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
No error and in the end access.log.1
is created as expected:
# ls -algh /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x 2 root 4.0K Sep 16 20:26 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r-- 1 root 10G Sep 16 19:56 access.log.1
-rw-r--r-- 1 root 5.5K Sep 16 19:56 error.log
Note that it said ...
log /usr/local/openresty/nginx/logs/access.log.61.zst
doesn't exist -- won't try to dispose of it
... instead of error renaming ... Read-only file system
(as logrotate.service
does)
Why does it work when running manually as root, but fails when it is executed by the logrotate.service?
I did not make any changes to logrotate.service
. For completeness, here is the unit file:
$ systemctl cat logrotate.service
# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true
[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7
# hardening options
# details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
# no ProtectHome for userdir logs
# no PrivateNetwork for mail deliviery
# no ProtectKernelTunables for working SELinux with systemd older than 235
# no MemoryDenyWriteExecute for gzip on i686
PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=full
RestrictRealtime=true
Now I'm running out of options. Any help in troubleshooting the problem is much appreciated.
I think, I found it. The logrotate file is not the problem.
Instead, it is caused by the hardening features in the systemd unit file. After I disabled the
ProtectSystem=full
hardening options, it worked. The reason is that logrotate has to operate on the/usr
directory in my case, which is readonly if the option is enabled.From the documentation on
ProtectSystem=
To fix it properly, I added now the following line to
/lib/systemd/system/logrotate.service
:Then
ProtectSystem=full
will mount everything as readonly except the directory where the logs are. Now, the readonly error is gone:I assume, I never hit the problem on Ubuntu 18.04 because the hardening options were not yet the defaults. Ubuntu 20.04 has it now, but I don't know in what Ubuntu release they were first introduced.
Nice find to the same problem i was seeing. Editing nginx.conf to log to different directory (eg. the common /var/log/nginx) is an alternative that also works.