Status

This page is updated manually with status of current and recent (30ish days) events.

(Times are US/Arizona UTC-7)

Current status is: RED: Two Phoenix web hosting servers hung unexpectedly – investigating.

20181210 @12:32AM – Two web hosting servers hung in Phoenix. Power cycled them and they came back up. Investigating why they hung

20181113 @2:59AM – They’ve resolved the problem.

Our engineers have taken the steps necessary to restore full power to our DC. At this time we are seeing service up and stable and do not anticipate any further disruptions to power to devices. We will continue to monitor this issue and update with any new status.

20181113 @2:11AM – The data center is suffering additional power flickers on their A power channel. Now that our second switch is back up, we’re much more resilient during these additional flickers. Here’s the current status:

Our engineers have isolated the root cause of the power issue occurring at our PHX DC. We are working to resolve this and bring all systems back online. In the process of this final work there is a possibility of additional disruptions occuring. Once all this work has been completed we will update again and let you know that all work is done.

I’m going to make a couple small changes to the settings on the backup core router to ensure proper failover if the circuit the primary router is on drops again.

20181113 @12:35AM – Secondary switch tested fine under load and was updated to latest version of vendor software. Things now seem stable, but we’re remaining on site to monitor the situation.

20181113 @12:09 – The networking issues caused two webhosting servers to need a swift kick in the restart button. Getting them back up now.

20181112@11:59AM – Our second score network switch in Phoenix crashed hard due to a power flicker (or possible software bug). We were limping along, but Jay left for the data center to reset it. When he was 10 minutes from the location that switch seems to have confused the primary.

The switch has been reset and we’re doing some testing on it to see if it needs to be replaced.

 

20180913 @ 11:32PM – Server backup second attempt completed without incident.We’ll be watching this and looking to see if the backup storage is not accepting data fast enough to prevent a backlog on the web server.

 


 

Green: I am completely operational, and all my circuits are functioning perfectly normally.

AMBER: External network issues.

RED: Zombie Apocalypse

Magenta – a service is down, but not really an emergency.