Monday, January 16, 2012

Rebooting When Filesystem Fails

SkyHi @ Monday, January 16, 2012
Recently I found myself stuck with a system that allowed me to ssh into it but the root filesystem was read-only and all the binaries in /sbin, /bin, or /usr/bin were simply not accessible.
I attempted to run shutdown -r now and reboot without any success. I was really stuck as I did not have physical or KVM access to the server which was located in our office and no one were going to be there until 9 AM. Needless to say, I was also trying to avoid going to the office in the middle of the night as I had a early morning drive to the airport to drop off my business partner.
So I decided to explore how I could reboot the system while the disk subsystem was not responding well. I wondered if I could somehow reboot the system by setting some values in the kernel memory via /proc filesystem. My research led me to the following commands:

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

The system rebooted immediately and after automatically running a fsck check on the filesystems, it came back as if nothing had happened, which was a very very lucky thing. It reminded myself that every now and then a little bit of good luck can be great blessing.