November 21: We got reports of corrupted tables. Wastrel could not access the hosting site and filled out a ticket. After some back and forth we got this reply:
I do have some bad news. It does appear that with the latest outage, the Virtual Machine is corrupt and is no longer available. What we can do is to restore from a local backup that you may have. I did a check on the backups that we have had recently and they are also corrupt.
Wastrel and I both scrambled around for backups and I sent the following to the hosting company:
How corrupt is corrupt? Can we try any sort of restoration on the backups? We had a remote RAID backup system, but the HDD failed about a month ago.
November 23 reply:
It is not possible to restore from any backups and at this point, it looks like even all backup data is not available. If you have any sort of backup, we can try to restore it for you as well.
After the last outage I had created a rsync job to run a periodic backup to my own RAID array. In late October, my NAS (where I was storing this backup) crashed and became unrecoverable. This resulted in NO backup being available on my hardware, but thankfully (I thought) I have Crashplan running on my NAS......
My Crashplan subscription backs up my laptop, my desktop, and my NAS, but, stupid me, neglected to add the NAS SGU directory to the Crashplan backup job, so the only backup I had of my NAS did not include the SGU directory. (meaning I didn't have a backup of my backup)
After I restored the NAS to operation, I did NOT restart the rsync job, realized that this was the case and scheduled myself to do it on November 23rd. (two days after the server crashed)
In the mean time, Wastrel had set up a seperate on-server backup. This WAS supposed to be a cloud server where everything was backed up, but since the entire virtual machine was corrupted, this backup was destroyed when the virtual machine crashed.
So, with the hosting company being no help, Wastrel and I began digging around in our various storage locations for any other backup we might have. Unfortunately, the ONLY one we had was the one from the 2012 forum move (the same one we used earlier this year.)
So now we are back to square one on a new server, but this time, I have put in additional measures to make certain we don't ever end up in this place again. Those procedures are:
- The new hosting company allowed an install of Crashplan directly on the server.
- Crashplan is running a new backup every 12 hours
- This backup is being stored on the Crashplan Servers
- This backup is being stored on my NAS
- This backup is being stored on my Desktop computer
- Crashplan keeps every 12 hour backup for 90 days
- Beyond 90 days, Crashplan keeps the last backup from every week for one year
- Beyond a year, Crashplan keeps a the last backup from each month, FOREVER
Based on the above, we should NEVER be in this position again.