Google Cloud Accidentally Deletes Customer’s Account
Richard Speed (via Hacker News):
Google’s Cloud CEO Thomas Kurian has weighed in on the UniSuper fiasco and confirmed that UniSuper’s Private Cloud subscription was accidentally deleted.
In a joint statement with UniSuper CEO Peter Chun, Kurian admitted that an “inadvertent misconfiguration” during the provisioning of UniSuper’s Private Cloud services resulted in the deletion of the subscription.
In a cascade of catastrophe familiar to anyone using duplication, the deletion of the account resulted in deletion across other regions.
Two weeks later, they are finally fully restored:
Restoring UniSuper’s Private Cloud instance has called for an incredible amount of focus, effort, and partnership between our teams to enable an extensive recovery of all the core systems. The dedication and collaboration between UniSuper and Google Cloud has led to an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.
UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.
The customer isn’t exactly small either - “UniSuper is an Australian superannuation fund that provides superannuation services to employees of Australia’s higher education and research sector. The fund has over 620,000 members and $120 billion in assets (funds under management and total member accounts at 7 July 2021).”
Previously:
- Google Cloud Services Outages
- Google Account Deleted Due to CSAM False Positive
- Relying on Google Cloud
Update (2024-05-30): Ron Amadeo (via Slashdot):
In the immediate aftermath, the explanation we got was that “the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.” Two weeks later, Google Cloud’s internal review of the problem is finished, and the company has a blog post up detailing what happened.
Google has a “TL;DR” at the top of the post, and it sounds like a Google employee got an input wrong.
[…]
The most shocking thing about Google’s blunder was the sudden and irreversible deletion of a customer account. Shouldn’t there be protections, notifications, and confirmations in place to never accidentally delete something? Google says there are, but those warnings are for a “customer-initiated deletion” and didn’t work when using the admin tool.
[…]
In its post-mortem, Google now says, “Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and, along with third-party backup software, were instrumental in aiding the rapid restoration.” It’s hard to square these two statements, especially with the two-week recovery period. The goal of a backup is to be quickly restored; so either UniSuper’s backups didn’t get deleted and weren’t effective, leading to two weeks of downtime, or they would have been effective had they not been partially or completely wiped out.
