Server Downtime
 

The Ford Truck Enthusiasts main server experienced several hours of intermittent down time between 10:15 pm and 4:00 am, Eastern Standard Time.

Someone at the server facility was moving a server in the same rack as ours and apparently the power plug to our main server was accidently loosened. We thought it was a server hang and called the server facility asking for a reboot. The system rebooted, 2 minutes later it died again. Turns out the power plug was in just enough to work for a few minutes yet loose enough to cause a glitch (vibrations, someone walking by, who knows) and causing it to go down again. Unfortunately, a UPS doesn't help when the plug is pulled on the server!

Not knowing it was a power cable we suspected the worst and headed down to the facility. We figured it out pretty quick but had to keep the server unplugged from the Internet. One of the hard drives underwent a full system check and we had to restore some data from backups (nothing lost on that drive, we're very careful about backups!).

Powered the server back up and after about 20 minutes we noticed a lot of database console error messages. Took the server back down, did a check on the database and found several of the index files were corrupt (apparently they were being written to when the system died).

Nothing seems to be lost (we had to rebuild the Ranger forum title and counts but everything else looks good). If you notice anything missing let us know, we can restore data from our replicated tables.

Its now 5:15 am, I've had a long night and I'm turning in. Needless to say, I'm a "wee bit" upset that someone else's carelessness cost several hours of my time. Our power plug is now zip-tied to the box!

On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then!

Re: Server Downtime
 

Are we slow now? :D Seriously I have never had the feeling that the site was slow...

Great site! thanks to you, and the rest of the admin and mods, for making this the best site on the net.

Did you mean "Our power plug is now zip-tied to the box!"
instead of "not zip-tied"
If so... that's pretty funny :)
Zip ties, super glue, and duct tape will fix anything

BUMMER....
was wondering what was wrong...
How can I refer other people to the better site if it's down ?!?!

And actually, although maddening, it's a best it was just the plug :D

It's always good to back up from a problem and slow down... usually we fly in and think it's a nuclear clean up an it willbe just spilled milk in the end :-D

FYI, I mentioned "we" in the above post. I need to give credit where credit is due and mention that Frank Fenn, a close friend and FTE officer, was there with me.

Yes, just fixed it. Lack of sleep will do that to you!

Thanks for the great work Ken... I know less dedicated people would have said "oh well it will wait til monday, or tomorrow" etc

Thanks for the dedication ken. :)