Server Downtime
#1
Join Date: Jan 1997
Location: Enjoying the real world.
Posts: 23,165
Likes: 0
Received 7 Likes
on
6 Posts
Server Downtime
The Ford Truck Enthusiasts main server experienced several hours of intermittent down time between 10:15 pm and 4:00 am, Eastern Standard Time.
Someone at the server facility was moving a server in the same rack as ours and apparently the power plug to our main server was accidently loosened. We thought it was a server hang and called the server facility asking for a reboot. The system rebooted, 2 minutes later it died again. Turns out the power plug was in just enough to work for a few minutes yet loose enough to cause a glitch (vibrations, someone walking by, who knows) and causing it to go down again. Unfortunately, a UPS doesn't help when the plug is pulled on the server!
Not knowing it was a power cable we suspected the worst and headed down to the facility. We figured it out pretty quick but had to keep the server unplugged from the Internet. One of the hard drives underwent a full system check and we had to restore some data from backups (nothing lost on that drive, we're very careful about backups!).
Powered the server back up and after about 20 minutes we noticed a lot of database console error messages. Took the server back down, did a check on the database and found several of the index files were corrupt (apparently they were being written to when the system died).
Nothing seems to be lost (we had to rebuild the Ranger forum title and counts but everything else looks good). If you notice anything missing let us know, we can restore data from our replicated tables.
Its now 5:15 am, I've had a long night and I'm turning in. Needless to say, I'm a "wee bit" upset that someone else's carelessness cost several hours of my time. Our power plug is now zip-tied to the box!
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then!
Someone at the server facility was moving a server in the same rack as ours and apparently the power plug to our main server was accidently loosened. We thought it was a server hang and called the server facility asking for a reboot. The system rebooted, 2 minutes later it died again. Turns out the power plug was in just enough to work for a few minutes yet loose enough to cause a glitch (vibrations, someone walking by, who knows) and causing it to go down again. Unfortunately, a UPS doesn't help when the plug is pulled on the server!
Not knowing it was a power cable we suspected the worst and headed down to the facility. We figured it out pretty quick but had to keep the server unplugged from the Internet. One of the hard drives underwent a full system check and we had to restore some data from backups (nothing lost on that drive, we're very careful about backups!).
Powered the server back up and after about 20 minutes we noticed a lot of database console error messages. Took the server back down, did a check on the database and found several of the index files were corrupt (apparently they were being written to when the system died).
Nothing seems to be lost (we had to rebuild the Ranger forum title and counts but everything else looks good). If you notice anything missing let us know, we can restore data from our replicated tables.
Its now 5:15 am, I've had a long night and I'm turning in. Needless to say, I'm a "wee bit" upset that someone else's carelessness cost several hours of my time. Our power plug is now zip-tied to the box!
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then!
#2
Re: Server Downtime
[i]
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then! [/B]
On a brighter note, the additional server we've been working on has passed all tests and it should go live Monday or Tuesday evening. Expect blazing speeds then! [/B]
Great site! thanks to you, and the rest of the admin and mods, for making this the best site on the net.
#3
#4
BUMMER....
was wondering what was wrong...
How can I refer other people to the better site if it's down ?!?!
And actually, although maddening, it's a best it was just the plug
It's always good to back up from a problem and slow down... usually we fly in and think it's a nuclear clean up an it willbe just spilled milk in the end
was wondering what was wrong...
How can I refer other people to the better site if it's down ?!?!
And actually, although maddening, it's a best it was just the plug
It's always good to back up from a problem and slow down... usually we fly in and think it's a nuclear clean up an it willbe just spilled milk in the end
#7
Trending Topics
Thread
Thread Starter
Forum
Replies
Last Post