ithought.org status and service notices

Q4 2011: A move across the street, new power hardware, new pricing

• ithought's equipment was moved across the street one weekend to a shiny new facility that should remain it's home for the forseable future.
• Power system issues led to some power management hardware being replaced.
• Continual growth in traffic served: 878GB of web traffic on web servers averaging 4 requests/second
• Continued usage of 1Mbps in a tor relay, and 2 Opeterons hard at work 100% of the time on Folding@Home
• Both Minecraft servers updated to 1.0!

The new power distribution unit has been installed and is working properly. All systems were offline for less than 10 minutes

Tomorrow, Wednesday Nov 30th around 10pm all ithought systems will be shut down for replacement of the power distribution unit. This should take no more than 30 minutes to complete.

The outage this morning was caused by a power distribution unit failing. A temporary replacement has been installed and all services are back up and running, and a permanent replacement is being orderd. Servers will all go offline later this week during non-business hours for it to be installed, and a notice will be posted here 24 hours in advance of the scheduled downtime

All ithought services are currently experiencing an outage of unknown origin. I am waiting on my hosting provider to respond with an explanation

Overnight on Friday Oct 14 / Saturday Oct 15, My rack of equipment was moved by my hosting provider from one building to another without any advance notice to me. While they were able to get power up and running quickly, they made a configuration mistake with their network equipment (Didn't add my VLAN) so it took much longer than expected to troubleshoot this and get my equipment back online. This ended up being an outage of about 9 hours. Servers were shut down cleanly so no filesystems or database systems were badly damaged.

Today, my hosting provider experienced a power failure in this new facility. Power was cut to all servers and networking hardware, and while this was somewhat quickly resolved, my equipment didn't appear back on the network until I opened a support ticket with them which their network team ended up resolving. Once in, filesystems and mysql files on several systems had to be repaired by hand. The primary web/db server was not online like the rest of my equipment, so a remote console was connected and it seems that the storage system attached to the primary web/db server was not powered on before that server, so it was unable to boot because it didn't have any of it's disks present. A reboot fixed this, and after running filesystem checks on several multi-TB volumes (which takes some time if you've ever tried!), this server was able to boot back up.

There may be issues with individual corrupted MySQL databases on pongo.ithought.org, please e-mail me if you are having any trouble with your website. I'll try and get through all of them as quickly as I can, but the list is pretty long and most of them look ok so far.

As always, all e-mail was spooled offsite so no e-mails were lost.

Please accept my deepest apologies for the inconveinence this has been, I know that this must be at least as frustrating to you as it has been to me. I'm speaking with a senior manager at my hosting provider tomorrow to see what can be done to prevent this from happening, and investigating other hosting options. (If you have a favorite colocation provider in Atlanta with a good track record that can quickly get a full rack provisioned, plese let me know!) Don't hesitate to e-mail me if you have any questions.

There is currently an ongoing network outage for all equipment in ithought's Atlanta facility due to an upstream network provider equipment failure. There is no estimated time of repair, but they are actively working to resolve the problem

Q3 2011: Lots of change this quarter, and the vast majority had no impact on any customers.

• All DNS moved from EveryDNS to Dynect. This should improve availability of all sites whenever a DDOS happens, and makes it a lot easier for me to manage. Additionally, I've shut down the ithought PowerDNS system and Dynect is now acting as the primary DNS resolver for all ithought properties instead of acting as a secondary. Unfortunately, the move to Dynect means that for any new domains, there will be a $15/year charge for ithought managing your DNS, but all existing customers have been grandfathered in.
• The remtote mail spool, status blog, and monitoring systems have been moved from Slicehost over to the Rackspace Cloud which should have better network availablity.
• ithought SSL certificates have all been updated, and all now expire in 2012.
• Lots of traffic served: 762GB

Q2 2011: DDOSes, power outages, hardware replacements, and more! There was more downtime this quarter than usual, but every step has been taken to address the problems. Unfortunately there's not much that can be done about provider power/network issues but new hardware was installed to handle high traffic sites which improved performance on the low-traffic site system that most customers are on, and DNS is getting switched from EveryDNS over to Dynect which has a much better ability to handle DDOS attacks on their infrastructure than EveryDNS. Numbers:

• 578G of web traffic on web servers
• 1 new shared hosting customer
• 4 Requests/second on primary web server

As always, let me know if there is anything you need!

It has been over an hour, but most services are back up and running. The new web/db server automatically recovered after about 20 minutes from power restoration, the mail server automatically recovered after about 10 minutes, most dedicated servers came up immediately, and the primary web/db server took almost 1.5 hours as it needed some help with filesystem repairs on its massive storage system. Currently 1 dedicated server is down and pending some assistance from remote hands, and the tor/foldingathome/minecraft server is down pending remote hands as well