ithought.org status and service notices

The previousy announced maintenance for May 8th has been moved to 5am on May 15th

Sometime between 5am and 6am eastern time, iThought servers will be moving to new physical space in the datacenter. The best-case scenario for this outage is 10-20 minutes, but the expected duration is several hours. I will post an update here once maintenance is complete, or if it looks like things will go past 9am eastern.

The scheduled maintenance did not go is planned, but everything is resolved and in better shape than before maintenance. You can get all the details at http://ckdake.com/content/2013/upgrading-udev-and-kernels-on-a-stack-of-gentoo-servers.html"

Tomorrow evening starting at around 9pm eastern, every ithought.org server will be rebooted for kernel upgrades. Each server should be down for no more than several minutes. If it looks like any outage will take longer than 1 hour, notice will be posted here.

All is well. No issues this month, consistent traffic, onward to 2013.

No news is good news, things ran pretty well this quarter. iThought is no longer directly providing e-mail hosting and instead outsourcing through Rackspace which has been working great, and the old mailserver hardware has been repurposed to improve our ability to recover from any MySQL problems tha tmay arise in the future.

• 1.2TB of traffic on primary web servers averaging 4 requests/second

Power Outages. That sums up this quarter, there have been several interruptions in power at iThought's hosting facility. If this happens again, we'll likely be purchasing a (expensive) rack-level UPS to prevent further issues from these extremly brief issues that cause all servers to reboot. No data has been lost, but the 5minute to several hour wait for servers to reboot and repair their storage and database systems is extermly frustrating.

• Shutdown of ithought.org e-mail hosting systems almost complete. Only one forwarding-only domain left, and no more hosted e-mail. Rackspace has much better spam filtering and it's a marked improvement for everyone.
• 1+TB of traffic on primary web servers averaging 4 requests/second
• New automated invoicing system via Less Accounting has been working well, better invoices that are easier to pay is a win for everyone

Yesterday around 3pm, iThoughts hosting provider experienced a failure of a major circuit breaker. This was routed around, but caused a ~5 minute power loss for all iThought hardware. Within 10 minutes, most servers were back on line, but the main web/db server took about an hour to boot due to repairs needed on it's filesystem. Some non-customer facing services were down for up to 3 hours, but as of 6:30pm yesterday, everything was back online

Not much to report!

• One outage due to MySQL server crashing on a Sunday morning
• Slight slowdown during backups of web servers, we're looking into how to make backups better
• Shutdown of ithought.org e-mail hosting systems almost complete. Only a few domains to go!
• 707 GB of traffic on primary web servers averaging 4 requests/second
• New automated invoicing system via Less Accounting

Q4 2011: A move across the street, new power hardware, new pricing

• ithought's equipment was moved across the street one weekend to a shiny new facility that should remain it's home for the forseable future.
• Power system issues led to some power management hardware being replaced.
• Continual growth in traffic served: 878GB of web traffic on web servers averaging 4 requests/second
• Continued usage of 1Mbps in a tor relay, and 2 Opeterons hard at work 100% of the time on Folding@Home
• Both Minecraft servers updated to 1.0!