The Ford Truck Enthusiasts main server experienced several hours of intermittent down time between 10:15 pm and 4:00 am, Eastern Standard Time.
Someone at the server facility was moving a server in the same rack as ours and apparently the power plug to our main server was accidently loosened. We thought it was a server hang and called the server facility asking for a reboot. The system rebooted, 2 minutes later it died again. Turns out the power plug was in just enough to work for a few minutes yet loose enough to cause a glitch (vibrations, someone walking by, who knows) and causing it to go down again. Unfortunately, a UPS doesn't help when the plug is pulled on the server!
Not knowing it was a power cable we suspected the worst and headed down to the facility. We figured it out pretty quick but had to keep the server unplugged from the Internet. One of the hard drives underwent a full system check and we had to restore some data from backups (nothing lost on that drive, we're very careful about backups!).
Powered the server back up and after about 20 minutes we noticed a lot of database console error messages. Took the server back down, did a check on the database and found several of the index files were corrupt (apparently they were being written to when the system died).
Nothing seems to be lost (we had to rebuild the Ranger forum title and counts but everything else looks good). If you notice anything missing let us know, we can restore data from our replicated tables.
Its now 5:15 am, I've had a long night and I'm turning in. Needless to say, I'm a "wee bit" upset that someone else's carelessness cost several hours of my time. Our power plug is now zip-tied to the box!
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then!
FTE Founder. Enjoying life, hitting the throttle and hearing my Ford squeal 'em....
was wondering what was wrong...
How can I refer other people to the better site if it's down ?!?!
And actually, although maddening, it's a best it was just the plug
It's always good to back up from a problem and slow down... usually we fly in and think it's a nuclear clean up an it willbe just spilled milk in the end
People are much more opposed to fur than leather because it's much easier to harass rich women than motorcycle gangs.
13 F350 Platinum 6.7 CCSB being chased by 14 Redwood 38GK 5er,
13 Explorer Sport 3.5 Twin Turbo 365 hp Ecoboost.
all gone:12 MVP Destiny 355RE 5er,12 F250 6.7 CCSB Grey Metallic Lariat w/ Ultimate Pkg,05 Ex 4x4 6L,03 Ex 4x4 6L,02 Ex 4x2 7.3L
This forum is owned and operated by Internet Brands, Inc., a Delaware corporation. It is not authorized or endorsed by the Ford Motor Company and is not affiliated with the Ford Motor Company or its related companies in any way. Ford® is a registered trademark of the Ford Motor Company.