Slow forums today: server has a broken RAID controller FIXED

[Gwyneth Llewelyn]

[FIXED]
Hi all,

Early this morning, I got a message from the tech support team of DreamHost, where these forums are hosted, informing me that the RAID controller (the hardware that "connects" the computer to its disks) of the server has broken down. While they fix the hardware — possibly moving everything to a brand new computer — they're running all websites on this server from the backups, which are being mounted over the network. So in theory, no data has been lost.

In practice, of course, network-mounted disks are perhaps 1000x slower than local RAID-based disks (or possibly even slower, depending on lots of factors). This means a general slowdown and loss of performance for all sites. In fact, among the 100+ sites I host on this server, the CDS forums are one of the very few that actually still respond to requests Aye, they're far slower today, but at least they are still up and running... well, up and "walking"...

I have no idea how long it takes until DreamHost fixes the hardware problem. From discussions on their forums, it can take as little as "a few hours" (assuming they have spare RAID controllers somewhere...), "up to 36 hours", or "several days". A few users, last week, waited 10 days until they managed to get all their sites up again in new hardware. So all we can do right now is to be very, very patient

[Rosie Gray]

Hi Gwyn,

I always find these forums excruciatingly slow. Maybe if there is a new server it will improve things?

[Gwyneth Llewelyn]

That would indeed be good news, Rosie In the mean time, even do sporadically the forums work, the situation is NOT resolved, so if anyone with access to user notices/in-world group is able to read this, it would be nice to tell everybody that we're aware that the forums are slow/unresponding/down and hoping that the techs fix it soon.

Although the news are not very encouraging: I've heard that restoring everything on a new server can take a whole week

[Gwyneth Llewelyn]

Just to let you know: after some painful ten days of patiently waiting, the forums were finally moved to a brand new server.

Everything is supposed to be working just as before, just faster (well, as fast as possible, considering the hosting provider's own limitations...)

If you notice something wrong — like, say, a missing picture or being unable to login — let me know. Login problems can usually be fixed by cleaning the browser's cookie for this forum — it just might have an old IP address in it or something weird like that.

[Gwyneth Llewelyn]

Bad news: if you're reading this, you're lucky. The 'new' server is showing some exceptionally high CPU load. I'm hoping that it's just hundreds of users checking in and making sure their backups are all where they should. Nevertheless, what this means is a general slowness overall, as bad — or worse! — than on the 'old' server, so I've filed another ticket to support.

Whatever is happening, it didn't happen this morning, when pretty much everything was peachy and running without faults. It's something unexpected and new...

Oh well. It just means that we need to be even more patient for a while, and hope that this time the issue is not so serious!

[Rosie Gray]

It seems to be running nicely now.

R.

[Gwyneth Llewelyn]

Aye... as far as I can tell, the new server, while possibly slightly less powerful (just 4 CPUs, but running at three times the speed than the other 'old' server, and with far more CPU cache per core), actually seems to run much faster. What I can see is that it has perhaps just half the load as before. How exactly the hosting provider managed that I have no idea, but they seemed to have done a great job so far. Let's hope it remains like that