Status

This page is updated manually with status of current and recent (30ish days) events.

(Times are US/Arizona UTC-7)

Current status is: Green: I am completely operational, and all my circuits are functioning perfectly normally.

20170518 @1:35PM – One of our network links dropped for 7 minutes. Traffic (almost completely) moved to the other links as it should have.

20170517 @ 1:30 PM – Migrating all the site databases to a different server, rebuilding the database service, and moving the sites databases backed was successful. (And yes, we took lots of backups along the way.)

20170517 @10:45 AM – Looks like the fix isn’t complete. The database server process on whphx7 is working, but behaving oddly. We’re going to migrate all sites on that server to the database process on our spare server. After & rebuild whphx7’s database

20170517 @9:25AM – Fixed. The database process wouldn’t restart until all bad tables were found and fixed.

20170517 @9:19AM – The WordFence plugin used on many sites, (and maybe a second plugin used on one site) corruputed their respective database tables on a PHX webhost server. The tables have all been repaired.  (It’s so secure, it keeps the baddies and the goodies from getting to a site. Sigh…). This is the second or third time the plugin has broken sites.

20140510 @ 11:50PM – Server is back up and happy. Thanks for your patience.

20140510 @ 10:30PM – We have a motherboard in Quebec that is failing (network ports are flaking out) – will be doing an emergency hardware change on it tonight- around 2AM Eastern. About 7 sites will be affected. Downtime will be approximately 1 hour.

20170426 @9:45PM – Some of the Quebec servers were unable to resolve the name of their respective database server when an internal network link failed. To prevent this from happening again, resolution for all critical infrastructure is being hard coded. (15 min)

20170422 @3PM – TS web server is updated. In a classic case of taking care of client needs before our own, we finally upgraded the server with our sites on it (techsurgeons.com, webmail, support, and portal) to SSD drives – more than a year after upgrading the last server hosting client sites.

20170422 @8AM – We’re doing updates on the TS webmail & main server including a major update to the web server software. Things might be odd for a bit

20170422 @8AM – We’re still affected by the occasional crashing bug under high I/O load (backups). RC7 of the latest Linux kernel is out & might be safe enough to use.

 

 


 

Green: I am completely operational, and all my circuits are functioning perfectly normally.

AMBER: External network issues.

RED: Zombie Apocalypse

Magenta – a service is down, but not really an emergency.