ithought.org status and service notices

Q2 2010 was the first new hardware in a while, due to a few servers dying during the power issues on April 4th. Thanks to the ram added last quarter in the main web/db server, there is still plenty of headroom for handling more traffic. The stats:

• 496GB of web traffic on the pimrary web server (15% increase from Q1 2010)
• 4 new shared hosting customers
• Average of 2.2 Requests/second on the primary web server (10% increase from Q1 2010)

As always, let me know if there is anything you need!

Tomorrow morning, my hosting provider will be conducting some emergency maintenance that involves rebooting some of their network equipment. There should be a single period of approximately 3-5 minutes where ithought hosted services will be unavailable, sometime between 3am and 4am eastern time, tomorrow June 16.

It turns out that the ithought outage described in the previous update today was due to a power brownout in the facility. This affected all users in this 160,000 square foot facility and the root cause has yet to be found. The two servers that did not boot back up were damaged and replacements are being ordered.

Today at around 3:20pm eastern, 5 of my servers were powered off and back on for reasons unknown to me. I have an open ticket with my hosting provider about this and will update when I have more information. The implications so far:

• Most websites were down for ~15 minutes while the primary web/db server rebooted. No major damage here.
• The primary mail server was down for ~1 hour. It required manual rebooting, and fixing several problems via a remote console, before it would boot. No mail was lost, and all inbound mail was queued on the on-site bakup mail server which was not rebooted.
• The backup database system that replicates from the primary web/db server was corrupted and not repairable and had to be rebuilt. This finished at around 4:45pm eastern, and its outage did not cause any customer facing issues.
• One auxiliary server did not boot back up, but it hosts no customer services.
• Onc customers dedicated server did not boot back up, a ticket is open with my hosting provider to resolve this.

My apologies for the outage, I will ll update here as soon as I have any more information about the cause of this.

Q1 2010 was a quarter of small changes and upgrades including:

• DNS move for system infrastructure from EveryDNS to DynInc for better stability
• Doubling RAM in primary web server for better performance

The stats update is:

• 13 new shared hosting customers
• 432GB of web traffic on primary web server (35% increase from Q4 2009, biggest quarter to date)
• Average of 2 web site requests per second (8% increase from Q4 2009)

Thanks for your continued business and support! E-mail me if there is any info that you would like to see included future quarterly updates.

Maintenance today was successful, RAM in the primary web/db server has been doubled to 16G and all services are back up. There was around 15 minutes of downtime

On Sunday around 1PM eastern time, the primary web/db server will be shut down to install additional memory and upgrade its kernel. Best case scenario is about 15 minutes of downtime, but things may be down for up to 2 hours. Mail service will not be affected. status.ithought.org will remain up through the maintenance and will be updated when maintenance is complete, as well as during maintenance if there are any significant changes from the expected 15 minute outage.

This morning there was a brief outage of the primary webserver due to what looks to be some malicious traffic that used up all resources allocated to the web server process which prevented it from responding to legitimate requests. Steps have been taken to prevent this from happening in the future including increasing the resources allocate to the webserver process. Sorry for the inconveinence! Brief outages seem to only happen when ATTs email-textmessage gateway is performing slowly, so I was not able to respond to this as quickly as I would have liked

The source of the white "Server Error" page that some of you have been seeing over the last few days has been identified as something that can be fixed by adding more memory to the server that you are on. An additional 8GB of RAM is in the mail and a maintenance window will be scheduled for installation once it arrives. Temporary fixes have been made that should prevent this problem from occuring until the new memory can be installed. Additionally, sometime in the close future, a new server will be purchased and customers will be moved over to is as needed to prevent any future issues.

ithought.org DNS entries are moving tonight from EveryDNS infrastructure to DynDNS infrastructure. This should be seemless but if anything seems broken, feel free to send me an e-mail.