I have a MariaDB master-slave setup on 2 CentOS 7.0 servers. Today, I extended a LVM volume on the master server (because it was getting full). After extending the partition, I rebooted the server and reconnected the slave to it. All went well and the server appeared to sync any queries run on the master again.
But, in my monitoring (Cacti) I noticed a very "suspicious" freefall in the disk space usage on the slave host.
I am not sure why. Did the slave drop/rotate binlogs when reconnecting to the master maybe? Then again, almost 600GB in binlogs seems insane.
I ran the pt-table-checksum
tool from the Percona Toolkit to verify integrity between the hosts, running this command returns no diffs. So everything seems well, I just can't explain the massive drop in disk space usage, does anybody have an idea?
After some further research, I concluded that it actually was due to rotating binlogs. When I checked my MariaDB data directory with:
I noticed that the "oldest" file creation time was around the same time as the disk space usage drop in my monitoring:
Further inspection of the same folder learns that over the last weekend alone, about 25GB of new binlogs were created. So I guess I should take another look at my binlog rotation as it takes up a lot of space currently.
I'd also advice the same to anyone else facing the same issue. Check your binlog rotation to make sure it doesn't take up too much space and optimize it where possible. Also, a drop in disk space after rebooting the master seems to be a regular procedure, as the binlogs are indeed removed/reset on the slave host.