Home / server / Questions / 1036286
Accepted
liori
Asked: 2020-10-04 12:55:06 +0800 CST

Best way to archive journald logs in a space-efficient way?

  • 772

It has been established that journald logs are huge. On this specific system, the logs grow about 3GB per week. For audit purposes I would like to preserve system logs for a longer time than what can I comfortably store on that system; this would only be "just-in-case" storage that doesn't have to be quick to retrieve from. What would be the best way to archive these logs in a space-efficient way?

I am thinking of periodically doing some kind of journalctl | xz >>/var/log/old-logs.txt.xz, but that would be quite wasteful in terms of CPU usage, re-archiving the same parts of data again and again and might miss logs if they get deleted by journalctl, so maybe there's a better way?

disk-space-utilization archive journald

2 Answers

  1. Best Answer

    journald rotates its own files, so it's safe to copy the old files (which always have an @ in the filename) elsewhere, compress them, or do whatever you want with them. If it's necessary to look into them, journalctl can be pointed at specific journal files with the --file command line option.

    You should also see journald.conf for options to limit journald's disk usage.

    • 3

  2. Well it's true that xz compression might give you the best compression level.

    Yet xz compressed logs can not be queried with journalctl.

    The other approach is to use filesystem compression. I use btrfs for this with zstd compression and it gives me about 10x space usage reduction and logs are still available for interaction via journalctl. I do not recommend using btrfs volume for /var/log/journal as journalctl keeps marking its directories with No_COW attribute (see lsattr) which prevents compression. It's possible to overcome this like I've done in my setup, but it is more fragile and complicated. Another way is to move old log files to separate compressed dir. Something like this:

    mkfs.btrfs "$BLOCK_DEVICE_PATH"
mkdir /var/log/journal_archive

# With my data zstd compression level gives best compression/speed ratio.
# Your mileage may vary.
#Default level 3 is quite good so you might just stick with the default
mount -o compress=zstd:8 "$BLOCK_DEVICE_PATH" /var/log/journal

# To archive old logs (older than a week):
find /var/log/journal/$MACHINEID/ -name '*@*.journal' -mtime +7 -exec mv -v {} /var/log/journal_archive \;

# To query old logs
journalctl --directory=/var/log/journal_archive

# Cleanup archive to keep 20GB of uncompressed files
journalctl --vacuum-size=20G --directory /var/log/journal_archive
    • 1