hey y'all. I really having trouble assimilating comments and typing out replies. Sometimes I am blank right from the start, other times a whole reply flashes through me head and by the time I put fingers to keyboard it's all blank again. I've also been very clumsy lately, like coordination is off + eyesight depth perception is wrong. Knocking things over and such. Last night I set something down on the counter .. the counter wasn't there, it went straight to the floor.
PP and blood work .. sorted? NOT!. They reviewed the blood work and called me back. "Oh, we're so sorry. they only did the lipid tests, which look great by the way, they did not do the full Comprehensive Metabolic Panel. We need you to come in again for another blood draw."
So, on my perfectly nice Friday with 8.5hrs of overtime, I had to leave early to get to them before they close. They're 2nd 'free' redo blood draw cost me $100 off my paycheck. GRRRRRRR
speaking of overtime, For you techies out there, I'll describe why I had OT.
As some of you may recall, we ordered and installed a new SAN (network and DA storage), server, assorted switch. It was installed and configured in the nice brand new shiny tech area. oo pretty.
On Wed, it was a fairly good day for me, things went well, I felt good, low humidity. I got home and felt well enough to mow our yard. 2 hours later (on a small rider, large yard) and my required shower, I notice the blinky light on my phone.
I check messages, well that new server had a glitch, the techs on-hand restarted it all improperly. I was going to have to check the whole system for issues. but WAIT, I had some email come in .. equipment unreachable via network. After the 4th such showed up in the space of a few minutes I remembered that the monitoring app that I use on my phone was still off for vacation.
I restart the app and buzz .. buzz .. 60+ alerts about failed equipment. No recourse, I'm beat and tired but I have to go back in to work.
I get there, open the tech door and I am greeted by a missing ceiling tile, exposed drain pipe, a waterfall running down the last rack of equipment onto the floor where the missing tile was mush in a huge puddle. Yep, the last rack was the main switch stack (3-4 months old) and the new san, operational for maybe a week.
My counterpart shows up just as I am recovering and she goes into shock as well. Standing there gaping.
While gathering a tarp and such to deflect the water and clean up a bit, mind is racing. How to get things back online for the morning shift. 1 switch dead, server storage head dead, everything full of water and could die at any time.
Click, click, brain goes into recovery mode. Despite my recent muddled thinking I get it sorted. My brain is auto-mode when it comes to computers and network. I can be totally miserable and do computer work.
1st step - pull the wet/dead stuff. At least half gallon of water out of two pieces of equipment .. I didn't know there was enough room in there for that much water.
2nd step - while my counterpart does the clean-up I go off to prep an older switch to swap out with a (completely separate) new switch in another part of the building - which just happens to be new. I swap them out and reset the new switch to insert into the stack replacing the failed switch. OO pretty lights as it all comes online. A few dead ports on the 2nd switch, nothing I can do about that.
3rd step - replace the new server. We don't have one. WAIT, our old SAN!! I had built out a new'ish server (I was going to use it for archival). It's already set up for connecting to a SAN. We just need to use the 10G card from the dead one .. luckily for us, the card is still good. Do all the reconfiguration set up shares and up we go.
4th step - Plug in the redundant power (which has a tripped breaker). I get to the last power cord, plug it in and POP, there goes another breaker. One working switch, 2nd power supply. I pull the power supply out and a cup of water pours out onto the floor. Why was that thing working at all? Glad it was though, I didn't have another one to swap out.
By 4 am everything is clean, dry .. except inside some of it .. (tarp still hanging, jic) and fully operational. Time to go home and go to bed, which I did and back to work by 8 am to follow-up on anything that might have been missed or failed.
Yes, everyone knows that you don't install equipment under any plumbing, our 'bosses' didn't care, "it'll be OK" .. yep, it was fine :/ They haven't learned a thing, they pan on installing our new VOIP phone system in another room .. under the upstairs bathrooms that have already been known to leak .. AND .. has a warranty clause tht invalidates the warranty if installed under plumbing.
What happened? combination. A new cleaning crew used a 'never used' utility closet to empty a mop bucket. Well, that mop bucket missed. The drain was fine but the floor outside the drain had crack, the water went under the basin, down the pipe and into the new server room.
equipment damage? eh, no big deal. We have the whole rack approved and ordered for replacement ... only about $70,000.
ps. one 'boss' though that we could clean up the wet equipment with a spray cleaner O.o
That's enough spewing for today, I guess I had to get that out somewhere. lucky you, you got the read it .. or you got seriously bored, skimmed, and moved on as fast as your mouse could click 'next topic'
