Folks
Apologies for the outages over the last couple of days.
The co-location company that we use, Titan Internet, were bought out a while ago, and frankly things have not been the same since.
The recent outage was caused by a re-build of the NAS (provided by Titan and shared with loads of other users) that we use for backups. It's attached to the server via an NFS mount. The work on the NAS was not announced in advance, so I was unable to take any precautionary measures.
With the backup NFS mount down, all overnight backup jobs failed. For some reason I don 't understand, the MySQL backups froze rather than crashed. Eventually, the MySQL server became locked up as all available connections were in use serving frozen backups!
TMC uses a MySQL database to store all dynamic content and to store authentication details for users. Additionally, MySQL is used to store authentication details for mail users, and is used by the SMTP mail server (Exim), as well as the POP/IMAP servers (Courier IMAP).
So TMC was down, as was the email service! As a result of the latter problem, no-one could email me to say that there was a problem! I eventually found out when my wife complained that she couldn't get her personal email.
I maintain a Gmail account as a backup to the email accounts managed by the TMC server. I've passed the details of this email account to Cakeboy. He has my phone number as well, which he knows he's welcome to use. Hopefully, this sort of problem won't happen again in the future - I have complained to Titan about their failure to notify me - but if it does, Cakeboy and the team should be able to let me know in good time.
All the best to everyone.
Phil