The Importance of Backups: Lessons from the UniSuper and Google Cloud Incident
May 26, 2024
The Importance of Backups: Lessons from the UniSuper and Google Cloud Incident
Data is the lifeblood of businesses, governments, and individuals. With so much reliance on cloud services, ensuring data security and availability has never been more critical. A recent incident involving UniSuper and Google Cloud has brought this issue into sharp focus, underscoring the importance of having reliable backups.
What Happened?
In May, 2024, UniSuper and Google Cloud issued a joint statement revealing a major disruption that impacted UniSuper's private cloud services. UniSuper, an Australian superannuation fund that provides services to employees of Australia's higher education and research sector, manages over 615,000 members and A$124 billion in funds. The incident was caused by an inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud by Google Cloud. This led to the unintended deletion of UniSuper’s Private Cloud subscription across multiple geographies, affecting numerous virtual machines, databases, and applications. None of UniSuper’s customers were able to access their accounts and were effectively locked out of their funds.
Scope of the Impact
According to Google Cloud's detailed post-mortem, the incident impacted only one customer (UniSuper) and one specific service (Google Cloud VMware Engine - GCVE) in one cloud region. No other Google Cloud services or customers were affected. The misconfiguration, which left an essential input parameter blank during deployment, caused the system to default to a fixed-term subscription that automatically expired after one year, leading to the unexpected deletion.
Why the Outage Lasted So Long
UniSuper had duplication in two geographies to safeguard against outages. However, the deletion event caused a simultaneous loss in both locations. Restoring the services required extensive efforts and collaboration between UniSuper and Google Cloud. Fortunately, UniSuper had backups with an additional service provider, which minimized data loss and significantly improved the recovery process.
Recovery and Remediation
The recovery process was intensive, involving 24/7 efforts over several days to restore UniSuper's GCVE Private Cloud, network configurations, applications, and data. Key factors in the recovery included:
Robust Backup Strategy: UniSuper’s backups stored in Google Cloud Storage were crucial for data restoration.
Third-Party Backup Software: This provided an additional layer of resilience.
Close Collaboration: The joint efforts of UniSuper's and Google Cloud's technical teams facilitated a rapid and comprehensive recovery.
Lessons Learned
This incident highlights several critical points about data management and the importance of backups:
Even Major Cloud Providers Can Fail: Despite rigorous protocols, errors and unforeseen incidents can occur, impacting critical services.
Multiple Layers of Backup: Relying solely on a single provider, even one as reputable as Google Cloud, can be risky. Having independent backups with another service provider was instrumental in UniSuper’s recovery.
Continuous Monitoring and Updates: Regularly reviewing and updating deployment and backup procedures can prevent similar incidents.
Why You Should Always Have Your Own Backup
The UniSuper and Google Cloud incident serves as a stark reminder that having control over your own backups is indispensable. Here are key takeaways:
Independence and Control: Maintaining your own backups ensures you are not solely dependent on a single provider's infrastructure and policies.
Rapid Recovery: In the event of a failure, having immediate access to backups can significantly reduce downtime and data loss.
Peace of Mind: Knowing that your data is safely backed up in multiple locations allows you to focus on your core activities without constant worry.
Conclusion
While cloud services offer unparalleled convenience and scalability, they are not infallible. The UniSuper incident demonstrates that even the most reliable services can encounter unforeseen issues. By maintaining your own backups, you safeguard your data against such incidents, ensuring continuity and resilience in the face of unexpected disruptions.
Being proactive about backups is not just advisable—it’s essential.
Additional Recommendations
Regularly review and update your backup procedures.
Implement a 3-2-1 backup strategy: three copies of your data, on two different storage devices, with one copy offsite.
Consider using services that offer backups where you are the recipient of your files.
Develop a disaster recovery plan to ensure rapid recovery in the event of a failure.
