ithought.org status and service notices

Q2 2009 has been pretty smooth! There haven't been any unplanned outages, and the only planned outage was performed successfully on a Sunday evening with minimal interruptions to a subset of ithought services.

• 6 new shared hosting customers
• 295GB of web traffic served on the primary webserver
• Average of 1.87 website requests per second

Perpahs the biggest news of this quarter, ithought is now hosting a Tor relay with 1.5Mbps of bandwidth allocated for it. You can see some details about it here, but the short version is that tor helps people use the internet anonymously which is becoming more and more important, especially in places like Iran. ithought's tor server is one of less than 2000 public servers in the world!

This server is a 1U Dual-CPU Opteron machine thats only other job is running Folding@home which it has been doing for over a year. Stats for this are available here. By using ithought to provide your hosting, you're helping find cures for diseases and helping subvert oppressive regiemes in other countries!

maintenance complete. All production servers are back up, but the firmware update ate the main backup storage system so a full backup will be run tonight to repopulate it

will post here when things are back

This Sunday, April 26th, sometime around 10:00pm eastern time, the primary webserver will be down for firmware updates on it's storage system to prevent a data unavailability problem identifed by the vendor. It is expected that this downtime will only be 15 minutes, but there is a possibility it may take longer. All data will be backed up before these updates, so there is 0 chance of any data loss if something goes wrong. An update will be posted here once this maintenance is complete

It's been a bit of a rough start at ithought's new hosting provider in Q1 2009, but they've always been responsive to trouble tickets and these 2 outages were completely unrelated and the worst 2 outages in company history. Once our amount of equipment doubles from a half rack to a full rack, we'll be able to work with tier-one providers to be able to keep things up and running, and with current growth rates, it should only be a year or two! Here's some stats from this quarter:

• 10 new shared hosting customers
• 190GB of web traffic served on the primary webserver
• Average of 1.36 website requests per second
• Average of 101 database queries per second
• 5 minutes of downtime of ithought equipment due to issues with ithought equipment (this was a software upgrade issue)
• Significant downtime due to issues with my provider as explained above

There is still plenty of RAM, CPU, and disk space availabe to keep up this growth rate for a while without negatively affecting anyone, and I'll buy even more hardware as soon as there is enough in the bank to do so.

Thanks for your continued business!

After 4+ hours of downtime, all service was restored but my providers websites have been very slow coming back online. Turns out that the root cause here is that someone hacked into their power systems and was trying to use social engineering to get money wired to some overseas bank account or something, and they are working as fast as they can to get things back online while preserving forensics. I will update here if I find out any more information, and I apologize again for the downtime

Wow. Everydns was down last monday for an hour. They explained that this hour was their longest outage since 1998 or something like that and that a full writeup would be posted shortly, but they have since removed that notice and there is no explanation. Now, my colo provider is down again, starting at 9:05am this morning and they are having an 'internal issue' according to the administrative assistant that answered the tech support line because so many people are calling in. They're frantically scrambling to get things back up as their website is down as well, and things will be up as soon as they can get them back up. Once I have twice as much equipment as I have now, it will be possibile for me to rent square footage in a building, and pay for power and internet for it directly instead of having to go through one of these smaller providers. Once that happens, there will not be any more issues like this. Sorry for the trouble, once this resolves, let me know if you would like a credit applied to your account.

Looks like everydns.net, my DNS provider, is having some latency issues currently. In the past, these have always been a result of a DDOS attack against them, and are usually resolved fairly quickly. If you experience slow access to your site or timeouts, this is probably the cause and I will update this if I find out any more information

I finally have outage details from my provider, and the gist of it is that they had 3h30m of downtime due to a partial switch failure that dropped network traffic and prevented failing over to backup hardware, and the VLAN that my equipment is on did not come back up when the other VLANs came back up. They were not monitoring any of my IPs due to the size of things that I have, so were unaware that anything was still broken. After notifying them of the issue, they took several hours to track down the problem and resolve it. My outage was the largest unplanned outage in company history (since 2002 with 7 current datacenter locations) so this should not happen again. They are now monitoring my equipment (on top of the monitoring that I do), I now have a direct phone number to a manager there, and given their explnations, I believe that there shouldn't be any more issues

everything is back up. I will post all the details tomorrow. again, my apologies for the downtime, let me know if you would like a credit to your account.