On Friday I came home to find that the server was reporting that the storage was full and that it was unable to write to the drives, with over 6tb of storage remaining available on the drives.
I first attempted a reboot of the server and the server came back up and reported that it was attempting to access space outside of the drive causing an immediate kernel panic. I attempted dozens of fixes and not one of them would work. I finally ran out of things to attempt, gave up on saving the data and deleted the partitions on the drives, reformatted, and reinstalled the operating system from scratch.
Upon rebooting I again received the message that the system was still attempting to access space outside of the drive and the kernel panic occurred again. More google searches and I found a tool that was able to fix the problem, just wish I had found it before I did the full reinstall but didn't find it until after that had already been done.
My running theory is that something occurred that damaged the master boot record on the drive, cosmic rays anyone?
I began the process of re-configuring the server to our needs and started the process of recovering the files from our remote backup provider, which was a multi-day nightmare on it's own. Something I will talk about in my next message.
When I finally was able to get the backups downloaded I began the process of getting everything ready to put the site back up, finishing just a bit ago.
I worked 60+ hours over the last 4 days, sleeping only when I was not able to continue any longer. You start making stupid mistakes when you get too fatigued.
As near as I can tell we lost a few hours of data but the majority of it was able to be recovered! If you find any problems on the site please let me know asap!
Susan,
Thanks for you efforts and energies.
We appreciate all that you did. :)
Chrissy
Oh my next post will be after I have had something to eat and some rest...
And it i wonderful to have a home again!
Thank you!
Susan, thank you so much for your hard work! Not just the superhuman marathon this weekend, but all the earlier work to make sure that there were backups to restore. We missed Susan's Place, and we are glad to have it back.
Yes, thank you!
Sent from my SM-G935V using Tapatalk
As the former owner of a web/email host and internet service provider, I can fully sympathize with you, Susan! I had some of those marathons, and don't miss them a bit! All my thanks and best wishes for your own recovery after the site's recovery. Thank you, thank you, thank you!
Remote backups are great for safely preserving the data offsite, but bandwidth is never enough to quickly get back terabytes of data. We tried offering that service ourselves, but it usually comes down to Fedex'ing a hard drive containing the data is quicker. It's a great safety tool, but there probably needs to be a local backup that can be used in all but a true disaster.
Thank You Susan for all. Your tireless/tiring efforts!!!
Onward we go!!!!!!!!!!!!!
Ashley 🙋♀️💕🌻
Susan
Thanks for all your hard work and dedication!!! I am glad it all turned out well. I can only imagine the things that were running through your mind during the crash....LOL
That was as huge amount of effort and I take my bonnet off for all the dedication to the site and all it's members and viewers
Thank you
Thank you
Thank you
Hugs and love Jessica
I was going through withdrawal! ;)
One thing I've notice though, is after reading messages and going back to the topic list, the page reloads, which means the topic I just read is no longer listed. Previously, I had to manually reload the page. This means that if, after reading a topic, I no longer want to see it, I can't use mark that topic to be ignored, until the next time it appears.
It should be fixed. I still had the page set to not cache.
Quote from: Susan on January 16, 2019, 02:50:20 PM
It should be fixed. I still had the page set to not cache.
tnx
Thnx Susan for getting us back online. I thought for a sec the site was gone.
Lisa