Status

This page is updated manually with status of current and recent (30ish days) events.

(Times are US/Arizona UTC-7)

Current status is: Amber: One of our ISPs are doing maintenance sometime between midnight and 6AM on July 8th (Arizona time). They expect downtime of between 5 and 15 minutes

20200707 @20:00 – One of our ISPs will be doing maintenance tonight – sometime between midnight and 6AM. Expected downtime of the link is between 5 and 15 minutes.
As most sites are configured to use multiple network links, those sites will automatically shift over to use the other network interface.
(Sites using Cloudflare will likely be down during this window as they can’t use both interfaces.)

20200513@18:08 Services are returning. Still investigating.

20200513@17:10PM – Due to a networking issue, at least one server has gone down. Currently investigating, and a tech has been expedited to the data center to assess and fix.

20200511@8:35PM – Something in one of the database queries (or just bad luck) triggered a bug in MySQL which caused the database process (not data) to crash in a not-automatically-recoverable state.
Bug info here: https://jira.mariadb.org/browse/MDEV-21826
We used their workaround to restart the database and verify data integrity, then restarted the database again to restore our performance settings.

Green: I am completely operational, and all my circuits are functioning perfectly normally.

AMBER: External network issues.

RED: Zombie Apocalypse

Magenta – a service is down, but not an emergency.