On our production server there is a small drive for the root mount point /
,
/var/log
is taking too much space and I have to manually delete some files.
How can I move /var/log/
to let's say /home/log
WITHOUT REBOOTING?
Here is the thing I thought:
$ mkdir /home/log
$ rsync -a /var/log /home/log
$ mount --bind /home/log /var/log
$ /etc/init.d/rsyslof restart
But I know that some services use file descriptors, so they'll continue to use /var/log
or inodes.
Proper design
I assume you are unable to simply extend the filesystem in question (using
lvextend && ext2online
), because you do not use LVM or use wrong filesystem type.Your approach
What you've proposed might work if you signal the daemons with SIGHUP (kill -1 pid). Obviously you would need to later on "mount -o bind / /somewhere" and clean up what has been left underneath mounted /var/log. But it has a bad smell for me, especially for production.
Avoid downtime, have a clean result (but complicated to do)
Forget about "mount -o bind" idea, create a new LV/partition, but don't mount it yet.
For each daemon that has any open file (I would expect at least syslog, inetd, sshd):
kill -1
or/etc/init.d/script reload
)lsof | grep /var/log
that daemon has closed its filesMount over /var/log. Restore old configurations, SIGHUP/reload daemons again.
Easy way (downtime)
Create a new LV/partition and mount it properly over either /var or /var/log. The easy way is to take down the server to maintenance mode (single-user mode), and use the actual console (not ssh) for the operation.
Everyone else's answers are excellent and correct, and you should definitely read them first.
I just thought I'd share this because it makes for easy copy-and-paste, if your case turns out to be quite a simple one like mine was:
Stop the syslog and copy current logs out:
then, mount your new location over
/var/log
. Say it's a new device called/dev/sdb
now you can copy files back and restart the syslog:
Assuming this all happens quite early on in the life of your machine,
rsyslog
is likely to be the only daemon running. YMMV!PS - you'll be wanting to add it to your
fstab
as well probably. Here's one way of doing that, again assuming a very straightforward mount:(cf https://serverfault.com/a/267610/80606 about catting mtab to fstab)
Another thing that you could do is:
/var/log
/var/log
(usinglsof
as kubanskamac suggested)/var/log
to another partition with enough free space (following your example, that would be/home/log
)ln -s /home/log /var/log
)Please note that this is far from what I'd consider as a good practice. It's just a workaround so that you don't have to shutdown the server. The right solution would be to create a new
/var
or/var/log
partition with enough space (or expand the current one),To move the log directory to a different location:
Another solution based on @hwjp , if you can't use another drive volume to move them, you can create a virtual drive volume into another volume who has more space available (that is my case) :
Create virtual volume:
A) do :
sudo dd if=/dev/zero of=VHD-log.img bs=1M count=1200
B) do :
sudo mkfs -t ext4 /thevolumeofyourchoice/VHD-log.img
Format the EXT4 file system type in the VHD-log image file with the mkfs utility.C) do :
sudo mkdir /thevolumeofyourchoice/vlog
Mount VHD-log to a directory (mount point)D) do :
sudo mount -t auto -o loop /thevolumeofyourchoice/VHD-log.img /thevolumeofyourchoice/vlog
D1) To mount VHD-log at system boot to the final directory, add this entry in the /etc/fstab file.
Move old log files:
E) do :
service rsyslog stop
lsof | grep /var/log
to lists open files in /var/log and switch off needed daemon (apach2,freshclam in my case)cp -rp /var/log/* /thevolumeofyourchoice/vlog
(cp -p same as --preserve=mode,ownership,timestamps)F) do :
sudo umount /thevolumeofyourchoice/vlog
sudo mv /var/log /var/log-old
sudo mkdir /var/log
sudo chgrp syslog /var/log
sudo mount -t auto -o loop /thevolumeofyourchoice/VHD-log.img /var/log
G) do :
service rsyslog start
and restart others services you stoppedFinally double check :
You ca do a -
lsof | grep /var/log
to lists open files in /var/log and verify they point to /var/log and not /var/log-oldYou can mv, backup or delete /var/log-old after all is ok.
I would just: