Every hour one MariaDB version 10.4 database stops responding and I see this output in the log file:
Feb 9 10:00:02 maria55 mysqld: 2023-02-09 10:00:02 0x7f72547a5700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.13/storage/innobase/btr/btr0cur.cc line 7940
Feb 9 10:00:02 maria55 mysqld: InnoDB: Failing assertion: space_id == page_get_space_id(page)
Feb 9 10:00:02 maria55 mysqld: InnoDB: We intentionally generate a memory trap.
Feb 9 10:00:02 maria55 mysqld: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Feb 9 10:00:02 maria55 mysqld: InnoDB: If you get repeated assertion failures or crashes, even
Feb 9 10:00:02 maria55 mysqld: InnoDB: immediately after the mysqld startup, there may be
Feb 9 10:00:02 maria55 mysqld: InnoDB: corruption in the InnoDB tablespace. Please refer to
Feb 9 10:00:02 maria55 mysqld: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
Feb 9 10:00:02 maria55 mysqld: InnoDB: about forcing recovery.
The server is running Kopano and although I can't find any prominent CRONs that cause this problem exactly on the hour, I fully suspect Kopano has some maintenance routines every hour triggering the crash.
The advice about corruption in InnoDB sounds legitimate, but I don't know how to do that the links are not clear.
For now we did a mysqlcheck kopano and database checks works 100%
Googling further doesn't really show anything.
I'm at a loss what to try next.
My one idea is to upgrade the database to see if any 'bugs' might be squashed, but when we did a dump to the same disk, this happened:
mysqldump --single-transaction --routines kopano -r kopanodb.sql
mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table 'lob' at row: 166961
Now I have a secondary problem in that I can't make a backup. The total size of the database is 80GB!
My idea was to upgrade MariaDB but now I am scared things go catastrophically wrong and I won't have a good backup.
My other idea was to maybe stop the MariaDB service and then copy /var/lib/mysql to a temporary location, all 80 GB, and then at least maybe this will be an okay backup.
Any other ideas or tips?
The solution was to do a MariaDB upgrade. The clues were version 10.4 of MariaDB is very old, and this line:
To get around the problem of database backup, I took the database offline by stopping the service, and then just did a directory/file copy. It's not an ideal backup but I think I would have been able to restore had something gone wrong.