Automatically Test Your Database Backups
The solution is to frequently and automatically test backups by:
- Regularly downloading the latest backup from S3 (or wherever) and performing a full restore onto a clean server.
- Testing its validity in a way that a human is sure to notice if it stops working properly.
The first part sounds hard, but isn’t. For Overcast, I run an inexpensive Linode server devoted to automatically fetching, installing, and testing the latest backup every day and emailing me a report.