We have (more than) a linux server on AWS that use a network file system, in our case AWS EFS, mounted on /logs directory.
It happens sometimes that:
- the machine is rebooted
- for any reason the network file system is not mounted, so all process starts to log in /logs but on the main (system) partition
- disk space quickly starts to be low and, in worst case, the disk become completely full.
While debugging the issue a reboot happened and network file system comes back to normal so that my server can mount the disk and starts logging in the correct partition.
BUT the logs generated while network file system are still on the main partition, they are NOT accessible since a disk is mounted in the same /logs directory and they use disk space that cannot be reclaimed.
Is there any way apart unmount the network file system to access that logs so that I can trash or move them to the correct location avoiding to use precious space in system partition?