The Ford Truck Enthusiasts main server experienced several hours of intermittent down time between 10:15 pm and 4:00 am, Eastern Standard Time.
Someone at the server facility was moving a server in the same rack as ours and apparently the power plug to our main server was accidently loosened. We thought it was a server hang and called the server facility asking for a reboot. The system rebooted, 2 minutes later it died again. Turns out the power plug was in just enough to work for a few minutes yet loose enough to cause a glitch (vibrations, someone walking by, who knows) and causing it to go down again. Unfortunately, a UPS doesn't help when the plug is pulled on the server!
Not knowing it was a power cable we suspected the worst and headed down to the facility. We figured it out pretty quick but had to keep the server unplugged from the Internet. One of the hard drives underwent a full system check and we had to restore some data from backups (nothing lost on that drive, we're very careful about backups!).
Powered the server back up and after about 20 minutes we noticed a lot of database console error messages. Took the server back down, did a check on the database and found several of the index files were corrupt (apparently they were being written to when the system died).
Nothing seems to be lost (we had to rebuild the Ranger forum title and counts but everything else looks good). If you notice anything missing let us know, we can restore data from our replicated tables.
Its now 5:15 am, I've had a long night and I'm turning in. Needless to say, I'm a "wee bit" upset that someone else's carelessness cost several hours of my time. Our power plug is now zip-tied to the box!
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then!
__________________
-Ken
- If you think education is expensive--try ignorance.
- Knowing is not enough; we must apply. Willing is not enough; we must do.
- Only religious books are pronounced infallible on the strength of the contradictions they contain.
See "IB Tim" for support.
2006 Roush F150 Project: Too many mods to list.
04 F250 King Ranch Crewcab 6L PSD project
00 Ranger Supercab XLT 3.0 V6
RIP: 67 & 75 F100
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then! [/b]
Are we slow now? Seriously I have never had the feeling that the site was slow...
Great site! thanks to you, and the rest of the admin and mods, for making this the best site on the net.
Did you mean "Our power plug is now zip-tied to the box!"
instead of "not zip-tied"
If so... that's pretty funny
Zip ties, super glue, and duct tape will fix anything
BUMMER....
was wondering what was wrong...
How can I refer other people to the better site if it's down ?!?!
And actually, although maddening, it's a best it was just the plug
It's always good to back up from a problem and slow down... usually we fly in and think it's a nuclear clean up an it willbe just spilled milk in the end
__________________
05 Ex 4x4 6L -Walker BTM, 9005's fogs, GOSRACING headlight wiring harness, Silverstar 9008's, SCT is all - 8.9 @ 75 1/8th!
Gone but not forgotten -2003 Ebony Ex, 4x4 6L Westin Bull Bar, PIAA 520's, PIAA Headlamp UpgradeTurbo-back Exhaust...
even goner-2002 4x2 7.3L Ex Limited-buy back
FYI, I mentioned "we" in the above post. I need to give credit where credit is due and mention that Frank Fenn, a close friend and FTE officer, was there with me.
__________________
-Ken
- If you think education is expensive--try ignorance.
- Knowing is not enough; we must apply. Willing is not enough; we must do.
- Only religious books are pronounced infallible on the strength of the contradictions they contain.
See "IB Tim" for support.
2006 Roush F150 Project: Too many mods to list.
04 F250 King Ranch Crewcab 6L PSD project
00 Ranger Supercab XLT 3.0 V6
RIP: 67 & 75 F100
Originally posted by Andysutt Did you mean "Our power plug is now zip-tied to the box!"
instead of "not zip-tied"
If so... that's pretty funny
Zip ties, super glue, and duct tape will fix anything
Yes, just fixed it. Lack of sleep will do that to you!
__________________
-Ken
- If you think education is expensive--try ignorance.
- Knowing is not enough; we must apply. Willing is not enough; we must do.
- Only religious books are pronounced infallible on the strength of the contradictions they contain.
See "IB Tim" for support.
2006 Roush F150 Project: Too many mods to list.
04 F250 King Ranch Crewcab 6L PSD project
00 Ranger Supercab XLT 3.0 V6
RIP: 67 & 75 F100
Seriously I have never had the feeling that the site was slow...
