PC/Server/OS cataclysms

[Riek]

So, what horror stories do you all have where your PC or Server or even your OS has died a horrid death, either temporary or permenant?

[Riek]

To kick this off, it was late one night, or actually early one morning, and I was at work plugging away at adding a new network interface to the banks firewalls and rerouting a bunch of stuff due to IP range changes.

Anyways, so I was reading some man pages on this ace command called "routeflush". Rather than remove twenty odd routes manually (out of approximately 200) and run the routing script, I could remove them all at once and run the script!

This seemed like a brilliant idea once I hit the enter key, up until a blank screen appeared and my logging GUI which I had open just stopped dead. It took me all of three seconds to realize that the route to my remote connection had also been severed, and it was now impossible to run the routing script to readd anything. Management generally frowns upon the bank losing its extranet capability, so a quick sprint into the cells and a logon host connection fixed that little problem!

I still thank my lucky stars that I had decided to update the server at my local site rather than to one sitting an hours drive away.

[Aristotle]

Originally posted by Haigen

I still thank my lucky stars that I had decided to update the server at my local site rather than to one sitting an hours drive away.

Oh man, that would have sucked.

[kestra]

I was deleting some useless files on my hard drive to get more space.
Unbeknowst to me at the time, there were two spyware programs that I saw in my "add/remove" window of my computer. When I removed them, they took all my start up programs with them.
This sucked total ass, as I couldn't retreave any of my recipes from my computer (over 200 that I use for teaching).
I remember thinking, holy fuck, if I can't get these back, I'm out of a job.
Lesson learned. Thankfully my catering manager had copies of my recipes as well.

[Pyrosama]

Once my company had a space issue and we didn't know what was causing our production as400 to experience huge memory leaks, so in the meantime while they were trying desparately to figure out what was going on, I was asked by my boss to write a monitoring utility.

The problem was that at around 95% of used system resources, the OS is set to IPL which in as400 language is to restart. My utility was to run in the background and when the system reached about 90% the program would shut down the domino servers (4 that were huge hogs) and restart them, thus freeing up some memory to keep the OS from auto IPLing if it reached 95%.

My big mistake is that the production as400 at that time normally ran between 75% and 80% and my math was off a bit because I was testing on the test machine at a low percent to force the shutdowns to make sure that the program was doing what it was supposed to do. This was about 3 or 4 years ago so I don't recall the details, just the horror of it all.

I forgot to change the condition on the percentage or something freakish like that before moving it to production. When I moved the program to production and started it, the domino servers kept shutting down and starting up all the time, not just when they were supposed to. Not really a crash because they were doing exactly what the program told them to do, but it caused me and my boss some discomfort trying to come up with a plausible explanation for the downtime.

[Talsek]

I was monitoring some SQL Server activity on a pair of clustered servers a while back, and was just finishing up. So I went to log off...

Start... shut down... *flick of the mouse wheel* ... OK.

My -intended- action was to log off, but my laggy terminal server connection changed from log off to restart just as I clicked OK. WHOOPS! I instantly severed connections from all of our users to the production system as the cluster went into failover mode. This happened during a peak usage period.

Talsek the Dumbass

[Gesslar]

That is definately an issue, though. Terminal Server should NOT offer a "Restart Computer" option or let it be so easily done.

[Talsek]

It's turned off for all but Administrators, because, you know, we're trustworthy :P. I am generally a keyboard whore anyway, and accuracy is a part of my reasoning! Silly mouses/mice...