On Friday I came home to find that the server was reporting that the storage was full and that it was unable to write to the drives, with over 6tb of storage remaining available on the drives.
I first attempted a reboot of the server and the server came back up and reported that it was attempting to access space outside of the drive causing an immediate kernel panic. I attempted dozens of fixes and not one of them would work. I finally ran out of things to attempt, gave up on saving the data and deleted the partitions on the drives, reformatted, and reinstalled the operating system from scratch.
Upon rebooting I again received the message that the system was still attempting to access space outside of the drive and the kernel panic occurred again. More google searches and I found a tool that was able to fix the problem, just wish I had found it before I did the full reinstall but didn't find it until after that had already been done.
My running theory is that something occurred that damaged the master boot record on the drive, cosmic rays anyone?
I began the process of re-configuring the server to our needs and started the process of recovering the files from our remote backup provider, which was a multi-day nightmare on it's own. Something I will talk about in my next message.
When I finally was able to get the backups downloaded I began the process of getting everything ready to put the site back up, finishing just a bit ago.
I worked 60+ hours over the last 4 days, sleeping only when I was not able to continue any longer. You start making stupid mistakes when you get too fatigued.
As near as I can tell we lost a few hours of data but the majority of it was able to be recovered! If you find any problems on the site please let me know asap!
