{"id":33816,"date":"2021-10-05T14:45:47","date_gmt":"2021-10-05T18:45:47","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=33816"},"modified":"2022-04-14T14:43:01","modified_gmt":"2022-04-14T18:43:01","slug":"facebook-bgp-outage","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2021\/10\/05\/facebook-bgp-outage\/","title":{"rendered":"Facebook BGP Outage"},"content":{"rendered":"

Celso Martinho and Tom Strickx<\/a> (Hacker News<\/a>):<\/p>\n

Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had “pulled the cables” from their data centers all at once and disconnected them from the Internet.<\/p>

This wasn’t a DNS issue itself, but failing DNS was the first symptom we’d seen of a larger Facebook outage.<\/p>

[…]<\/p>

BGP<\/a> stands for Border Gateway Protocol. It’s a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn’t know what to do, and the Internet wouldn’t work.<\/p>

The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.<\/p><\/blockquote>\n\n

Santosh Janardhan<\/a>:<\/p>\n

\n

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many of the internal tools and systems we use in our day-to-day operations, complicating our attempts to quickly diagnose and resolve the problem.<\/p>\n<\/blockquote>\n\n

See also: Brian Krebs<\/a> (Hacker News<\/a>), Bruce Schneier<\/a>, Hacker<\/a> News<\/a>.<\/p>\n\n

Update (2021-10-20): Santosh Janardhan<\/a>:<\/p>\n

Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.<\/p><\/blockquote>\n\n

Via Nick Heer<\/a>:<\/p>\n

\n

For comparison, as I write this, Apple’s System Status page<\/a> shows a resolved outage in Apple Pay and Wallet. For over seven hours yesterday, “users were not able to add, suspend, or remove existing cards to Apple Pay”, and this issue has simply been marked as “Resolved” but there are no more details. This explanation-free status update has been the standard for every iCloud-related outage, including serious incidents<\/a>. It does not build confidence.<\/p>\n<\/blockquote>\n\n

Reuters<\/a> (via Hacker News<\/a>):<\/p>\n

Messaging app Telegram gained over 70 million new users during Monday’s Facebook outage, its founder Pavel Durov said on Tuesday, as people worldwide were left without key messaging services for nearly six hours.<\/p><\/blockquote>\n\n

Mark Zuckerberg<\/a> (via Hacker News<\/a>):<\/p>\n

\n

First, the SEV that took down all our services yesterday was the worst outage we’ve had in years. We’ve spent the past 24 hours debriefing how we can strengthen our systems against this kind of failure. This was also a reminder of how much our work matters to people. The deeper concern with an outage like this isn’t how many people switch to competitive services or how much money we lose, but what it means for the people who rely on our services to communicate with loved ones, run their businesses, or support their communities.<\/p>\n<\/blockquote>","protected":false},"excerpt":{"rendered":"

Celso Martinho and Tom Strickx (Hacker News): Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had “pulled the cables” from their […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"2021-10-05T18:45:51Z","apple_news_api_id":"7b11ea75-633f-4de6-8a4e-dd94cd6e47ed","apple_news_api_modified_at":"2022-04-14T18:43:04Z","apple_news_api_revision":"AAAAAAAAAAAAAAAAAAAAAQ==","apple_news_api_share_url":"https:\/\/apple.news\/AexHqdWM_TeaKTt2UzW5H7Q","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[728,25,1286,2190,96,1363],"class_list":["post-33816","post","type-post","status-publish","format-standard","hentry","category-technology","tag-domain-name-system-dns","tag-facebook","tag-instagram","tag-outage","tag-web","tag-whatsapp"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/33816","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=33816"}],"version-history":[{"count":2,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/33816\/revisions"}],"predecessor-version":[{"id":33971,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/33816\/revisions\/33971"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=33816"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=33816"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=33816"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}