This page is updated manually with status of current and recent events.
(Times are US/Arizona UTC-7)
Current status is: Green: I am completely operational, and all my circuits are functioning perfectly normally.
20170104 @ 2:03PM – Jay Goofed & when testing the hard drives on a server, ran a more intensive test than he should have. It slowed one of the servers in Quebec to a crawl for about 15 minutes. *blush* Sorry! On the plus side, the drives are all fine.
20161213 @ 12:47PM – Everything has been perfect since. *knocks wood*
20161213 @ 9:47AM – Our network provider installed a patch to one of their router pairs to fix the bug. The first router came back up fine, and is handling all the traffic. They’ll watch it for a bit & then install the patch on the second (currently idle) router.
20161213 @ 9:30AM – Traffic on one of our providers is suffering packet loss. Our second provider is fine, but about a third of packets are being lost on the first which isn’t bad enough for browsers to ignore it completely, and use the other.
20161209 1:30AM – One of our Phoenix network providers is doing planned maintenance. Their network will have a few outages between now and 4AM AZ time. Web traffic is failing over to the second network, so this is minimally impacting.
20161205 @ 9:58AM – Everything was back to normal with the last update from the provider. We have a ticket in with them on why they did the maintenance four days early.
20161205 @ 3:58AM – Their monthly maintenance window ends in an hour, but they claim to be done.
20161205 @ 2:30AM – One of our network providers in Phoenix had some network maintenance planned for Dec 9th. For whatever reason, they did it tonight – probably the bug they were hoping to avoid bit them. It shouldn’t have much impact on us, as all Phoenix servers have redundant network connections & browsers will try the second if the first fails.
20161204 @ 9:00PM – Figured out the network issue – one server’s 10G network connection (10 gigabit) was not moving the traffic it should have been. It was maxing out individual connections at 0.01% of its capacity. Sites and virtual servers which were on that server have been moved elsewhere and are functioning properly now. This issue was only noticeable with file over about 2MB in size.
Green: I am completely operational, and all my circuits are functioning perfectly normally.
AMBER: External network issues .