{"id":39253,"date":"2023-05-02T13:59:55","date_gmt":"2023-05-02T17:59:55","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=39253"},"modified":"2023-09-04T14:54:35","modified_gmt":"2023-09-04T18:54:35","slug":"google-cloud-services-outages","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2023\/05\/02\/google-cloud-services-outages\/","title":{"rendered":"Google Cloud Services Outages"},"content":{"rendered":"

Thomas Claburn<\/a> (Hacker<\/a> News<\/a>):<\/p>\n

Google Cloud stopped operating in Paris early on Wednesday morning local time due to “water intrusion,” said the off-prem biz, which a day earlier reported profitability for the first time.<\/p>

[…]<\/p>

“Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone,” the company’s status page<\/a> explains. “There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted.”<\/p>

A short while later, the incident description changed to “a multi-cluster failure and has led to an emergency shutdown of multiple zones.”<\/p>

[…]<\/p>

Though more brief, the load balancing problems were far broader, affecting not just the europe-west9 zone but multiple zones in Asia, Australia, Europe, North America, and South America.<\/p><\/blockquote>\n\n

Gergely Orosz<\/a> (via Drew Thaler<\/a>):<\/p>\n

\n

I have questions. How does water intrusion into one<\/em> data center take a whole zone (which should be multiple, physically separate and redundant DCs) offline?<\/p>\n

The point of availability zones is to avoid issues in one DC taking down the whole zone.<\/p>\n

Oh, I just see: an issue in one DC took down a whole region! So all AZs within that region are down.<\/p>\n

Wow, this is very bad: the point of AZs is exactly for this to not happen.<\/p>\n<\/blockquote>\n\n

Joshua Burgin<\/a>:<\/p>\n

Both Google and Microsoft don’t guarantee that all zones are physically separate buildings or separated by at least <x> km\/miles. Many of their “zones” in smaller regions are just separate buildings by the same DC facility<\/p><\/blockquote>\n\n

Dylan Tack<\/a>:<\/p>\n

“[AWS] AZs are physically separated<\/a> by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.”<\/p><\/blockquote>\n\n

Update (2023-09-04): Ry Crozier<\/a> (via Hacker News<\/a>):<\/p>\n

\n

Microsoft had “insufficient” staff levels at its data centre campus last week when a power sag knocked its chiller plant for two data halls offline, cooking portions of its storage hardware.<\/p>\n

[…]<\/p>\n

“We have temporarily increased the team size from three to seven, until the underlying issues are better understood and appropriate mitigations can be put in place.”<\/p>\n<\/blockquote>","protected":false},"excerpt":{"rendered":"

Thomas Claburn (Hacker News): Google Cloud stopped operating in Paris early on Wednesday morning local time due to “water intrusion,” said the off-prem biz, which a day earlier reported profitability for the first time.[…]“Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone,” the company’s status page explains. “There is […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2023-05-02T17:59:58Z","apple_news_api_id":"bdbdef2a-3131-40a6-bb15-35b1ab39567c","apple_news_api_modified_at":"2023-09-04T18:54:38Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAA==","apple_news_api_share_url":"https:\/\/apple.news\/Avb3vKjExQKa7FTWxqzlWfA","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[672,1354,856,2190,96,50],"class_list":["post-39253","post","type-post","status-publish","format-standard","hentry","category-technology","tag-amazon-web-services","tag-google-cloud-platform","tag-microsoft-azure","tag-outage","tag-web","tag-webapi"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/39253","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=39253"}],"version-history":[{"count":2,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/39253\/revisions"}],"predecessor-version":[{"id":40531,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/39253\/revisions\/40531"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=39253"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=39253"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=39253"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}