I have a linux machine dedicated to MySQL server with a pretty high load. Today I woke up and was terrified to see that database server is down. I could connect to it via SSH, but it was responding with bus error
to each and every command:
[root@r1304 home]# ls
Bus error
[root@r1304 home]# tail /var/log/messages
Bus error
[root@r1304 home]# reboot
Bus error
[root@r1304 home]# free -m
Bus error
[root@r1304 home]# chkdisk
Bus error
I went to Data Center and did a hard reset, which seemed to help, but after a half an hour situation reapeated and now I can't even connet via SSH anymore.
Any ideas what this could be? how to diagnose such a problem and what are possible fixes?
Server has 32 GB RAM, 2xSSD drives with software RAID
UPDATE According to Zabbix, when MySQL died, number of processes stated to increase drammaticaly, until I did a hard reset. What could those be?
Number of processes
Your storage system seems to have failed. You can't read from disk at this point.
Can you see the messages in the kernel ring buffer? Look at the output of
dmesg
if the command even works at this point.Do you have any backups or replication enabled?