Status

This page is updated manually with status of current and recent (30ish days) events.

(Times are US/Arizona UTC-7)

Current status is: Green: I am completely operational, and all my circuits are functioning perfectly normally.

20190324 @ 5PM – Social Warfare has an updated version of their plugin. Seems better, but use at your own risk. Be sure to update the plugin BEFORE reactivating it. They have made the latest version available to all clients – free and paid.

20190321 @12:25 PM – We’ve deactivated the Social Warfare plugin on all sites pending a fix to the plugin being hacked.

20190320 @5PM – We updated all sites to the newest version of woo-mailerlite as it fixes a bug causing massive site slowness.

20190319 @ 6:33AM: Our ISP has fixed the issue. Watching to see if it is stable.

20190319 @ 5:59AM: One of our ‘ISPs’ in Phoenix seems to be suffering an outage – looks like at their peering point with Zayo. All browser traffic will automatically use our other link – only Cloudflare traffic (which we had defaulted to the now-down link) is suffering issues.

20190316 @ 10:18PM – Network latency returned to normal. Am awaiting an update on what caused the trouble.

20190316 @ 10:04PM – Found higher network latency than normal. Not sure why – could be anything from a flaky network switch to an attack on the server or one of its peers on said switch. Opened ticket with facility for them to troubleshoot it.

20190316 @ 9:57PM – Server is back up from reboot – all seems better. Watching for oddness.

20190316 @ 9:48PM – Server came back on its own – destination backup storage was probably a bottleneck which made the server non-responsive until the backup finished. (The reboot command was canceled last moment.)

20190316 @ 9:43PM – One server in Quebec hung during backups. Remotely rebooting it now. Readying an on-call tech at the facility to check it if the remote reboot fails.

20190215 12:06AM – Switch upgrade and power cycle was successful.

Green: I am completely operational, and all my circuits are functioning perfectly normally.

AMBER: External network issues.

RED: Zombie Apocalypse

Magenta – a service is down, but not really an emergency.