Catastrophic Hardware Failure - Going to Chicago

[FliesLikeABrick]

The Soldat homepage, forums, and everything else hosted by U13 has been down since this morning (6:45 AM EST).  I don't know the exact time of the failure, but I woke up around 11am and found the server completely unresponsive.

Right now the Soldat homepage and soldatforums are running on another server of mine temporarily.  Due to DNS changes being required to redirect this traffic, Soldat.pl and forums.soldat.pl may not be working for many people for a few hours.

I will edit this post to reflect updates in what I find out.

-- Initial findings --
The server kernel panicked with an error making it clear that it was not a software problem.  All hard drives, raid arrays, and data are intact.

Memtest results indicate that the entire second bank of RAM is not usable.  Consistency across the many errors and across the two modules of RAM in that bank (1GB each) indicates that it is most likely a failing memory controller on the CPU tied to that memory bank, or that there is a problem with the motherboard or physical connectors to the RAM. 

I have placed a support call to my host and asked them to reseat all RAM, hoping that maybe it is just a connector acting wonky.  I will personally be in Chicago in 2 weeks and would be able to take more permanent measures then.

With any luck, this will be narrowed down to one physical component soon.  I am checking on the warranty status of all components right now to see what costs may be incurred.  I would estimate that at most this could be $1000 to repair if the failing component(s) is/are not under warranty.

-- Update 2:52 PM --
I have contacted the datacenter to have them swap the RAM around as well as reseat it firmly.  I should hear back from them within the next 20-30 minutes.


-- UPdate 8:00 PM --
I've gone through 2 or 3 troubleshooting steps over the phone, but it is starting to get expensive to pay them to do it all for me.  Instead, I'm going to go out there for the next few days and fix it myself.  Hopefully that works out. 

What does this mean?  All U13 servers and everything else I host will likely remain in a state similar to what it is now until Friday night or Saturday afternoon.  I will continue to try and bring things online before then, but I make no promises.

The most you can do to help is be patient.  This kind of thing doesn't happen often.  The machine that died has been up since mid-december with no problems of any sort whatsoever, and once this is fixed I expect we'll have a lot more time problem-free.

I'm going out to chicago tomorrow night to fix this.  1500 miles, 16 hours on the train.

You all better love me... feel free to donate to help pay for my train tickets.  $170 total.  Donations can be sent to flieslikeabrick@gmail.com via paypal or mailed to:

Ryan Rawdon
43 Eagle Street
Troy, NY 12180

Donations received:
The Geologist - $150.00
ChrisGBK - $100.00
Elephant Hunter - $100.00
Jerry Dixon - $15.00
MWTBDLTR -$6

-- 2:14 PM, the next day --
Stop donating! I have enough to pay for my costs (I think).
much love to the people who donated!


-- 05/13/2007 --
The long and complete version of this tale can be found at http://tns.u13.net/?p=13

[Iron Man]

yeah, i just got home from school and saw that it was down.  thanks for hosting on your own FLAB!

[-Skykanden-]

yes that was all the day, i was worried, thanks for the infromation

[Sytrus]

Yah.

I wasn't really worried, things like this can just happen....

[Iq Unlimited]

wow, what the hell did that. At least your getting it all fixed. Nice work as usual, I was wondering why the servers were acting strange.

[FliesLikeABrick]

I'm going out to chicago tomorrow night to fix this.  800 miles, 16 hours on the train.

You all better love me... feel free to donate to help pay for my train tickets.  $170 total.  Donations can be sent to flieslikeabrick@gmail.com via paypal or mailed to:

Ryan Rawdon
43 Eagle Street
Troy, NY 12180

[Silverflame]

hope everythings ok flab

[rfreak]

Hope someone will send you money for your good job

P.S. you can let money to me to if yes just PM me lol

hope you a good run to chicago

[iDante]

Oh dear, I was wondering why u13 servers were down. Well have fun in Chicago, it may be bigger but it isn't as fun as Seattle.

[RabidTreeFrog]

so that's why the forums were down.

Thanks for going through all the trouble to fix this Flies.

[bja888]

Onece you learn what is causing it, fix it, then learn how to prevent it. Remember, you always get what you pay for.

[FliesLikeABrick]

Onece you learn what is causing it, fix it, then learn how to prevent it. Remember, you always get what you pay for.

yeah, and I paid $1200 for that motherboard+chasis.  You may get what you pay for, but that doesn't mean you won't get the bad egg once in a while.  I learned a long time ago not to buy cheap shit, especially for server hardware.  Everything in this server that is having the problem (outcry.u13.net) is top-of-the-line.

There's nothing I can do to prevent something like this from happening again

[bja888]

Onece you learn what is causing it, fix it, then learn how to prevent it. Remember, you always get what you pay for.

yeah, and I paid $1200 for that motherboard+chasis.  You may get what you pay for, but that doesn't mean you won't get the bad egg once in a while.  I learned a long time ago not to buy cheap ****, especially for server hardware.  Everything in this server that is having the problem (outcry.u13.net) is top-of-the-line.

There's nothing I can do to prevent something like this from happening again

Its all so hopeless!!!!!!!!!!

<snip>Image macro</snip>

[NinjaGimp369]

I'm going out to chicago tomorrow night to fix this.  800 miles, 16 hours on the train.

You all better love me... feel free to donate to help pay for my train tickets.  $170 total.  Donations can be sent to flieslikeabrick@gmail.com via paypal or mailed to:

Ryan Rawdon
43 Eagle Street
Troy, NY 12180

Wow, the trouble you go through to keep us all happy. Thanks Flab <3

[TBDM]

thanks for the info and all FLAB, your the best!

hope you have a safe trip!!

[Zamorak]

Sorry to hear this Flies, I hope you have a nice trip.

[FannyBoy]

Wow amazing trip to Chicago! Thanks FLAB, you are second god (after MM)

[Morik]

Good luck with your server repairing expedition! Eat some Chicago style pizza while your up there for me.

[FliesLikeABrick]

Donation list added to the first post.  Thanks a ton to ChrisGBK

[Michal Marcinkowski]

Good luck with your server repairing expedition! Eat some Chicago style pizza while your up there for me.

Hah, yeah give 5$ donations for the pizza.