I have a lot of data (svn repository and postgresql database) in EC2 instance, yesterday the server doesn't respond to http requests, and I `sudo reboot', after that the server couldn't boot up again, what should I do?
The system log in the AWS console shows nothing output from the instance, and the most recent sys log is:
Ubuntu 9.10 domU-12-31-39-00-A0-06 tty1
domU-12-31-39-00-A0-06 login: [1796075.232506] INFO: task rsyslogd:22423 blocked for more than 120 seconds.
[1796075.232522] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232509] INFO: task pdflush:17 blocked for more than 120 seconds.
[1796195.232525] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232639] INFO: task kjournald:296 blocked for more than 120 seconds.
[1796195.232646] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232728] INFO: task rsyslogd:22423 blocked for more than 120 seconds.
[1796195.232735] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232858] INFO: task amuled:874 blocked for more than 120 seconds.
[1796195.232865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.232962] INFO: task master:1716 blocked for more than 120 seconds.
[1796195.232968] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233065] INFO: task qmgr:1724 blocked for more than 120 seconds.
[1796195.233071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233190] INFO: task java:4554 blocked for more than 120 seconds.
[1796195.233196] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233296] INFO: task java:4745 blocked for more than 120 seconds.
[1796195.233302] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1796195.233353] INFO: task java:11614 blocked for more than 120 seconds.
[1796195.233360] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
It seems I can't do anything now, I can't SSH into it, and can't operate in single user mode, too. Any idea??
EC2 instances cannot be booted into safe mode.
What is the output of ec2-get-console-output for that instance? It might contain clues as to what's preventing the instance from booting properly.
If this is an EBS-backed instance you'll be able to "stop" the instance, detach the root EBS volume from it, attach the root EBS volume to another instance and make changs to the contents, then reattach it to the original instance and "start" it running again.
If this is an instance-store instance then you're out of luck - there's no way to go in and fix one of those puppies.
Actually... an EC2 Linux instance can not be booted into safe mode, I've got one stuck there now. Its just not something you'd ever want to do. Apparently 14.04 doesn't play nice with the 'reboot now' command. This tells the server to restart in single user mode which requires console action. The AWS web GUI is hardwired to use this command. Because you don't have access to the console you're effectively screwed.
I'm adding this note so someone doesn't take your statement as a challenge and get stuck in the same hole I fell down.