Status

This page is updated manually with status of current and recent (30ish days) events.

(Times are US/Arizona UTC-7)

Current status is: Magenta – Server now up, and services are resuming

20200513@18:08 Services are returning. Still investigating.

20200513@17:10PM – Due to a networking issue, at least one server has went down. Currently investigating, and a tech has been expedited to the data center to assess and fix.

20200511@8:35PM – Something in one of the database queries (or just bad luck) triggered a bug in MySQL which caused the database process (not data) to crash in a not-automatically-recoverable state.
Bug info here: https://jira.mariadb.org/browse/MDEV-21826
We used their workaround to restart the database and verify data integrity, then restarted the database again to restore our performance settings.

20200511 @8:30 PM – Currently investigating an issue with the database on one server. 

20200404 @7PM – We changed the program we use for FTP so we can roll out SFTP (FTP over SSH) in the coming weeks. Simultaneously, we’ll likely also change FTP to require encryption (i.e. FTPS – FTP with TLS encryption) for greater security.

20200318 @5:00AM – Our ISP’s maintenance went off without any issues. We didn’t observe any downtime on their link.

20200318 @2:00AM – One of our network providers will be doing maintenance between 2 and 4AM local time. This will require them to take our link offline. All traffic should flow to our other Phoenix link. They estimate the link will be down for 40 minutes.

Green: I am completely operational, and all my circuits are functioning perfectly normally.

AMBER: External network issues.

RED: Zombie Apocalypse

Magenta – a service is down, but not an emergency.