Like a neighbourhood burglar alarm that goes off with the slightest provocation; backup error emails are really starting to annoy me.
Actual backups are working correctly. The problem is that the process of verifying that the backup is an exact copy of your database sometimes returns an error when in fact the databases are perfect replicas. This sends emails overnight incorrectly stating that there are backup errors.
The cause is largely the recent addition of fiche diagrams into c9. These increase the size of the database significantly. This increases the time it takes for online backup to perform a verification cycle. Because of the increase in verification time there is increased chance that while a verification is in progress, a change to the database is made that invalidates the verification. Result : a false alarm.
I will rethink the design of online backups and come up with a method that eliminates the inherent 'race condition' (as it is known in IT lingo), that triggers the false alarm during verification.
Inclusion of fiche diagrams has been a great addition to c9; but it has not been without its technical challenges; which we are slowly mastering and resolving.
Update: Worked out a better system. Deployed bulk of changes to our webserver. Will take a few days to monitor and ensure it is working as designed before complete migration to new verification methods; but if they are working correctly should provide instant relief. Next version of c9 will also include some minor changes to remove some 'race conditions' that exist in c9 itself during the process of creating verification statistics; but this is a much lesser issue than the server issues now corrected. False alarms are generated mainly because of how our backup web server was doing its job of cross checking fidelity of the backup. Another advantage of new system is that our webserver will generally be much more responsive. Verification workload on server has now been significantly reduced. Previously, running verification would load the webserver so much you could notice a difference in performance when loading our web pages for example; but new design this no longer happens. Also, I expect that online backup will be alot quicker, especially when installing a new system, as server code has now been tuned for increased performance.