The Ford Truck Enthusiasts main server experienced several hours of intermittent down time between 10:15 pm and 4:00 am, Eastern Standard Time.
Someone at the server facility was moving a server in the same rack as ours and apparently the power plug to our main server was accidently loosened. We thought it was a server hang and called the server facility asking for a reboot. The system rebooted, 2 minutes later it died again. Turns out the power plug was in just enough to work for a few minutes yet loose enough to cause a glitch (vibrations, someone walking by, who knows) and causing it to go down again. Unfortunately, a UPS doesn't help when the plug is pulled on the server!
Not knowing it was a power cable we suspected the worst and headed down to the facility. We figured it out pretty quick but had to keep the server unplugged from the Internet. One of the hard drives underwent a full system check and we had to restore some data from backups (nothing lost on that drive, we're very careful about backups!).
Powered the server back up and after about 20 minutes we noticed a lot of database console error messages. Took the server back down, did a check on the database and found several of the index files were corrupt (apparently they were being written to when the system died).
Nothing seems to be lost (we had to rebuild the Ranger forum title and counts but everything else looks good). If you notice anything missing let us know, we can restore data from our replicated tables.
Its now 5:15 am, I've had a long night and I'm turning in. Needless to say, I'm a "wee bit" upset that someone else's carelessness cost several hours of my time. Our power plug is now zip-tied to the box!
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then!
This forum is owned and operated by Internet Brands, Inc., a Delaware corporation. It is not authorized or endorsed by the Ford Motor Company and is not affiliated with the Ford Motor Company or its related companies in any way. Ford® is a registered trademark of the Ford Motor Company.