ithought.org status updates

Maintenance rescheduled to 5am on 5/22

Tue, 21 May 2013 23:14:06 +0000

My provider has again rescheduled the maintenance. This will now happen tomorrow morning around 5am

maintenance moved to 5am on May 15th

Tue, 07 May 2013 13:28:26 +0000

The previousy announced maintenance for May 8th has been moved to 5am on May 15th

Scheduled Maintenance: May 8th early morning Eastern Time

Tue, 30 Apr 2013 22:45:07 +0000

Sometime between 5am and 6am eastern time, iThought servers will be moving to new physical space in the datacenter. The best-case scenario for this outage is 10-20 minutes, but the expected duration is several hours. I will post an update here once maintenance is complete, or if it looks like things will go past 9am eastern.

Outage report for last 24 hours

Fri, 08 Feb 2013 02:09:24 +0000

The scheduled maintenance did not go is planned, but everything is resolved and in better shape than before maintenance. You can get all the details at http://ckdake.com/content/2013/upgrading-udev-and-kernels-on-a-stack-of-gentoo-servers.html"

Restarts tomorrow evening for kernel upgrades

Wed, 06 Feb 2013 01:05:55 +0000

Tomorrow evening starting at around 9pm eastern, every ithought.org server will be rebooted for kernel upgrades. Each server should be down for no more than several minutes. If it looks like any outage will take longer than 1 hour, notice will be posted here.

ithought quarterly update: Q4 2012

Tue, 01 Jan 2013 16:35:31 +0000

All is well. No issues this month, consistent traffic, onward to 2013.

ithought quarterly update: Q3 2012

Tue, 02 Oct 2012 13:53:47 +0000

No news is good news, things ran pretty well this quarter. iThought is no longer directly providing e-mail hosting and instead outsourcing through Rackspace which has been working great, and the old mailserver hardware has been repurposed to improve our ability to recover from any MySQL problems tha tmay arise in the future.

1.2TB of traffic on primary web servers averaging 4 requests/second

ithought quarterly update: Q2 2012

Mon, 02 Jul 2012 18:02:41 +0000

Power Outages. That sums up this quarter, there have been several interruptions in power at iThought's hosting facility. If this happens again, we'll likely be purchasing a (expensive) rack-level UPS to prevent further issues from these extremly brief issues that cause all servers to reboot. No data has been lost, but the 5minute to several hour wait for servers to reboot and repair their storage and database systems is extermly frustrating.

Shutdown of ithought.org e-mail hosting systems almost complete. Only one forwarding-only domain left, and no more hosted e-mail. Rackspace has much better spam filtering and it's a marked improvement for everyone.
1+TB of traffic on primary web servers averaging 4 requests/second
New automated invoicing system via Less Accounting has been working well, better invoices that are easier to pay is a win for everyone

Brief power-related outage yesterday

Thu, 31 May 2012 13:22:30 +0000

Yesterday around 3pm, iThoughts hosting provider experienced a failure of a major circuit breaker. This was routed around, but caused a ~5 minute power loss for all iThought hardware. Within 10 minutes, most servers were back on line, but the main web/db server took about an hour to boot due to repairs needed on it's filesystem. Some non-customer facing services were down for up to 3 hours, but as of 6:30pm yesterday, everything was back online

ithought quarterly update: Q1 2012

Mon, 02 Apr 2012 15:35:07 +0000

Not much to report!

One outage due to MySQL server crashing on a Sunday morning
Slight slowdown during backups of web servers, we're looking into how to make backups better
Shutdown of ithought.org e-mail hosting systems almost complete. Only a few domains to go!
707 GB of traffic on primary web servers averaging 4 requests/second
New automated invoicing system via Less Accounting