Status

This page is updated manually with status of current and recent (30ish days) events.

(Times are US/Arizona UTC-7)

Current status is: RED: One server is down

20210615@0700AM Server has an error with mysql, looking at it now and working to get it back up.

20210511@3:30AM – A few SSL certificates broke, so Apache on one server failed to restart when clearing logs. 20 minutes of weird until we could revoke and replace the bad certs. 

20210506 @11:30AM – Testing going well with only one false positive (VaultPress), so rolling to a handful of servers.

20210506 @8:00AM – We’re testing adding the mod_security firewall to block the worst of the worst traffic. Currently testing it on the sandbox servers, but will be rolling it out slowly so we can watch for, and minimize false positives.

202010426 @10:50AM – Secondary ISP has a fix in to the packet loss issue. Also, looks like there was a middle of the night few minute blip on their link at around 3:10AM. That’s likely related. 

20210426 @9:45AM – Our remote monitoring tools noticed some packet loss on our secondary ISP. They’re aware of the problem and are investigating. 

20210409 @7:00AM – Waiting on the outage report from Cogent. We’ll post the summary here in a future update

20210408 @1:30PM – Routers are back up. Can’t wait to hear their explanation as to what happened. On our end, I’m going to write some tools to quickly (20ish minutes) flip all sites off from whatever ISP is having problems so the next time this happens, it won’t be as impacting. -J

20210408 @12:37PM – Update from Cogent: “Customers in Phoenix Arizona may be experiencing a loss of network connectivity. This is being caused by a large power outage. Our field engineering team is working to resolve it. The ticket number for this issue is HD11580887.”

20210408 @12:05PM – Our other Network Provider (Cogent) is having an issue with their core routers in Phoenix. (I’m wondering if this is a Cisco bug as our other provider had a problem last week)

20210405 @9:50PM – Network Provider had an issue with their router. It’s back up.

20210405 @9:40PM – One of our network providers connections dropped. Looks like their router – we’ll update this status when we have an update.

Status key with examples:

Green: I am completely operational, and all my circuits are functioning perfectly normally.

AMBER: External network issues or planned maintenance

RED: Server down or critical bug/security flaw needing immediate fix

Magenta – a non-core web service is down, but not a critical emergency.