We recently developed a new internal tool for verifying database backup integrity. As you may know, our managed database-as-a-service plans include daily, weekly, and monthly backups, archived for up to 6 years.
As the frequency of daily backups has increased, it made us realize we needed a better mechanism for globally tracking the status of individual backups, without relying on brittle solutions like email / Slack notifications. Our new solution leverages S3 bucket notifications that trigger a custom AWS Lambda function which is responsible for updating a centralized database backups registry. We've implemented a reporting interface on top of this registry, giving us visibility into issues such as missing backups, corrupt files, and more. Best of all - our customers don't need to worry about these scenarios, as our DevOps team monitors them daily.
The backups registry will also be available to customers later this year as part of a new suite of internal APIs and tools we'll be launching.