Webmaster Forum Rules | Posting Guide | Contact Us | Testimonials | Contributing Geek Program | Advertise on Geek/Talk
Welcome to the GeekTalk Webmaster Discussion Forums from GeekVillage.com

Click Here To Register. It's Free!

Go Back   geek/talk: Signature-free discourse for serious web publishers > YOUR FRIENDS: The geek/talk Community > Technical Questions about geek/talk
User Name
Password
Register FAQ Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Display Modes
Old 06-25-2006, 04:47 PM   #1
Steve_S
I am a Contributing Geek. Are You?
 
Steve_S's Avatar
 
Join Date: Aug 1998
Location: Las Vegas, Nevada, USA
Posts: 5,224
Arrow The GV Dedicated Server Had a Bad Hair Day...

A few hours ago I noticed that GV, the Forums, and the DNS Domain were all off line and would not resolve from various locations on planet earth.

Czar/Josh is the owner and our emails crossed on this issue. Tribal Chief Czar indicated that the primary hard Disk had failed and of course that includes everything plus /home.

This is a catastrophic event and only those of us who run dedicated servers can appreciate the pain and agstant, not to mention sleep deprevation. Fortunetly for all of us, Czar AND the talented Data393 techs appear to have recovered everything from the SECOND hard drive which this box has always had. If your dedicated server and or virtual account server doesn't have at least 2 hard disks (seperate physical devices), respectfully, your nutso Plus regular and atutomated complete backups to the second hard disk, and a few cron tabs to safely store the data. If your freeky like I am, run your own crontab backups and automatically download same to your own machine and then copy it over to your network drive.

Next time YOU go looking for an elcheapo dedicated server run by a 13 year old kid on his lunch break, remember this story and exspect to be "down" for anywhere from a day or two to a week. Iv'e seen years of data lost forever and business closed with this kind of failure. Quality hosts are worth their weight in gold, as this issue illustrates. This kind of event and the fix is exactly what you pay for, because when a Hard Disk FAILS, they don't send you an email warning you that they will be toast in 6 hours. Not to mention that it's Sunday in their data center.

Congrats to Josh and Data393. Welcome to the club.

Last edited by Steve_S; 06-25-2006 at 05:18 PM.
Steve_S is offline   Reply With Quote
Old 06-25-2006, 05:29 PM   #2
Czar

Webmaster
 
Join Date: Aug 1999
Location: Gold Coast, Queensland, Australia
Posts: 9,506
Default

...And this all had to start at 11pm in Australian time, too.

I'll have the full update shortly. For now, a few systems are still offline and a few scripts are having trouble adjusting to a new operating system, since Data393 no longer support RedHat Linux. When everything is under control, I'll be back with a fleshier post.
__________________
Czar

Follow Geek/Talk's Twitter Feed and Facebook Page to stay up to date with new discussion threads and online ad industry highlights.

Important GeekVillage Links: Home | Rules | Posting Guide | Report Trouble | Feedback | Advertise on GV
Czar is offline   Reply With Quote
Old 06-25-2006, 07:25 PM   #3
Jan

Administrator
 
Join Date: May 2001
Location: Beautiful Darwin
Posts: 4,753
Default

Glad you see we got back up relatively unscathed It wasn't looking too bright at your 4am update.
Jan is offline   Reply With Quote
Old 06-25-2006, 08:24 PM   #4
Alex|Canep Media
I am a Contributing Geek. Are You?
 
Join Date: Apr 2006
Location: USA
Posts: 361
Default

I run a hosting company and know how terrible hard drive failures are. You should invest in a RAID Drive Controller to make sure theres no downtime if it happens again.

If your having some trouble with scripts my system admin can take a look for you.
Alex|Canep Media is offline   Reply With Quote
Old 06-25-2006, 09:07 PM   #5
emmzee
I am a Contributing Geek. Are You?
 
emmzee's Avatar
 
Join Date: Jul 1999
Location: Mississauga, Ontario
Posts: 1,372
Default

Glad to hear things are okay! I got concerned when I tried to access the site and got 404's for every page. Good to hear things are back up. Also makes me glad my host now provides automated remote backups with all their accounts
__________________
Darren Hewer
NEW: Simpsons Trivia Quiz
My sites: DOS Games - Free Web Games - Video Game Stores - Windows Games - Why Faith
emmzee is offline   Reply With Quote
Old 06-25-2006, 09:33 PM   #6
Steve_S
I am a Contributing Geek. Are You?
 
Steve_S's Avatar
 
Join Date: Aug 1998
Location: Las Vegas, Nevada, USA
Posts: 5,224
Arrow

Yipppppie.............AdvertPro is breathing again. Nice!

Thats a wounderfull offer proxy.

On a very few occassions during almost 8 years, the village has had some very interesting server issue.

I actually found my thread on the last HD failure, circa 3/02. Even though it's over 4 years ago, I recall the pain At the time, GV was on a dedicated box with Dialtone and it had dual SCSI drives. As I recall the primary SCSI started to fail gradually with errors in a file I can't remeber at the moment. Tons of read and write request errors were not only causing errors but driving the load and memory consumption to dangerous levels that I had never seen. As I'm sure some of you know, my stomach was in nots and I can't remember if I every slept.

The vB MySQL database was running fine with no corruption and so was the script. I closed her down because it only takes a single sector to nuke the DB. Still freeking, they swapped a new SCSI disk into the box and we were back up in a few hours.

The gory details at:

http://www.geekvillage.com/forums/sh...ight=corrupted
Steve_S is offline   Reply With Quote
Old 06-25-2006, 11:19 PM   #7
Czar

Webmaster
 
Join Date: Aug 1999
Location: Gold Coast, Queensland, Australia
Posts: 9,506
Default

Phew!

My heart has sure received a workout during the past 15 hours.

Okay, I've finished checking all essential services and content areas (and spent a couple of hours asleep after having crashed at 9am this morning ), and all appears to be well.

THE LOWDOWN
This was a story of error after error; possibly the result of running on a box that has been reliably plugging along for several years without incident - and without a major hardware overhaul.

I'll save you some of the gory details, but the trigger of today's downtime was indeed the failure of a harddrive, or at least part of a harddrive. The /home partition of our web hosting drive became unreadable by fsck after a simple reboot request about 15 hours ago. The machine simply wouldn't start until a Data393 techie commented out the reference to /home from the boot sequence. Once the machine was back online, WHM and cPanel worked, but all accounts listed on the server were empty because their web and email data was on the /home partition. Some data was eventually retrieved from that partition, thankfully, including the latest vB database, while other data was either installed from scratch (such as the new OS and cPanel) or retrieved from backups.

To their credit, Data393 responded extremely proactively. They answered my first support ticket within 5 minutes and although I lost track of them for a couple of hours here and there as they were installing and configuring a different flavour of Linux and fresh copies of cPanel, plus retrieving backups and mounting the failed drive, two techies alternated throughout the day to produce a brilliant outcome.

I'll absolutely migrate the site to a more robust server setup during the next couple of months, and will definitely invest in a RAID Drive Controller as suggested by proxy, but it's reassuring to know that when a catastophic failure such as partial of full loss of a harddrive occurs, there are a certain breed of hosts who can respond this quickly to patch things up.

Kudos to Data393 and to Jan, Steve and everyone else who assisted to get things back up and running as seamlessly as possible. No-one likes downtime, but being supported by a network like this makes it so much easier to bear.
__________________
Czar

Follow Geek/Talk's Twitter Feed and Facebook Page to stay up to date with new discussion threads and online ad industry highlights.

Important GeekVillage Links: Home | Rules | Posting Guide | Report Trouble | Feedback | Advertise on GV
Czar is offline   Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Interland sells dedicated server division Larwee Web Hosting Discussion 0 09-06-2005 03:53 PM
To get a dedicated host or not... DJ_B Web Hosting Discussion 4 01-29-2001 12:25 AM
CGI Usage Necessitating Dedicated Server? wsz Web Hosting Discussion 1 01-26-2001 05:48 PM
Cheap Dedicated Server For You! Steve_S Web Design and Webmaster Issues 10 07-26-2000 08:10 PM
How do you manage your dedicated server? Bryan Web Design and Webmaster Issues 6 05-03-2000 01:33 PM

Please support our advertisers. They ensure our survival.

All times are GMT -5. The time now is 02:32 PM.


GeekVillage.com is copyright © 1998-2015 Curiosity Cave - Science gifts for clever kids. All rights reserved.
Powered by vBulletin® Version 3.7.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.