Home / server / Questions / 819362
In Process
a coder
Asked: 2016-12-08 11:18:19 +0800 CST

MySQL won't start, ran out of disk space on /var/log

  • 772

Long story short, I discovered expire_log_days = 3 a little too late. My /var/log directory is completely filled with old mysql-bin files, and per google/SE search, it is strongly recommended not to manually delete them.

The trouble is, I can't get into a MySQL shell to PURGE them, nor am I able to start the service with the new config setting expire_log_days.

I removed some older non-MySQL logs (about 500mb worth), but am still unable to get mysql to start.

Is there a less painful way out of this? 

[root@box mariadb]# systemctl restart mariadb.service
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
[root@box mariadb]# journalctl -xe
-- Subject: Unit session-203.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit session-203.scope has begun starting up.
Dec 07 14:01:35 my.sql.box systemd[1]: Starting MariaDB database server...
-- Subject: Unit mariadb.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has begun starting up.
Dec 07 14:01:36 my.sql.box mysqld_safe[23884]: 161207 14:01:36 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Dec 07 14:01:36 my.sql.box mysqld_safe[23884]: 161207 14:01:36 mysqld_safe Starting mysqld daemon with databases from /data/db/mysql
Dec 07 14:01:36 my.sql.box mysqld_safe[23884]: 161207 14:01:36 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Dec 07 14:01:37 my.sql.box systemd[1]: mariadb.service: control process exited, code=exited status=1
Dec 07 14:01:37 my.sql.box systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has failed.
-- 
-- The result is failed.
Dec 07 14:01:37 my.sql.box systemd[1]: Unit mariadb.service entered failed state.
Dec 07 14:01:37 my.sql.box systemd[1]: mariadb.service failed.
Dec 07 14:01:37 my.sql.box audispd[4973]: node=my.sql.box type=SERVICE_START msg=audit(1481137297.301:1657): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg=
Dec 07 14:01:37 my.sql.box polkitd[4981]: Unregistered Authentication Agent for unix-process:23850:300226 (system bus name :1.460, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_U

[root@box log]# systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-12-07 14:09:17 EST; 7s ago
  Process: 24840 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 24838 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 24810 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 24838 (code=exited, status=0/SUCCESS)

Dec 07 14:09:12 my.sql.box systemd[1]: Starting MariaDB database server...
Dec 07 14:09:13 my.sql.box mysqld_safe[24838]: 161207 14:09:13 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Dec 07 14:09:13 my.sql.box mysqld_safe[24838]: 161207 14:09:13 mysqld_safe Starting mysqld daemon with databases from /data/db/mysql
Dec 07 14:09:16 my.sql.box mysqld_safe[24838]: 161207 14:09:16 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
Dec 07 14:09:17 my.sql.box systemd[1]: mariadb.service: control process exited, code=exited status=1
Dec 07 14:09:17 my.sql.box systemd[1]: Failed to start MariaDB database server.
Dec 07 14:09:17 my.sql.box systemd[1]: Unit mariadb.service entered failed state.
Dec 07 14:09:17 my.sql.box systemd[1]: mariadb.service failed.
[root@box log]#
mysql disk-space-utilization mysql-replication mariadb

2 Answers

  1. Assuming it is your binary logs that are occupying most of your disk space. All you need to do is move at least one binary log to a separate partition that has extra disk space. Only move do not delete. Then execute PURGE BINARY LOGS TO 'binlog.file'; in mysql console and choose a binary log older than the log file you moved in the previous step. That should free up more disk space. Then move back the binary log file to it's proper location and if needed purge more binary logs.

    • 2

  2. You should disable temporarily those logs by commenting all the expire_log_days or log_bin at the my.cnf file in order to turn off server logging and then manually move/delete the /var/log binaries.

    Then start the server, enable again the commented flags and restart MariaDB.

    • 0