{"id":17310,"date":"2017-02-24T16:40:36","date_gmt":"2017-02-24T21:40:36","guid":{"rendered":"http:\/\/mjtsai.com\/blog\/?p=17310"},"modified":"2017-03-16T15:06:35","modified_gmt":"2017-03-16T19:06:35","slug":"cloudbleed-cloudflares-https-traffic-leak","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2017\/02\/24\/cloudbleed-cloudflares-https-traffic-leak\/","title":{"rendered":"Cloudbleed: Cloudflare&rsquo;s HTTPS Traffic Leak"},"content":{"rendered":"<p><a href=\"https:\/\/bugs.chromium.org\/p\/project-zero\/issues\/detail?id=1139\">Tavis Ormandy<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=13718752\">Hacker News<\/a>):<\/p>\n<blockquote cite=\"https:\/\/bugs.chromium.org\/p\/project-zero\/issues\/detail?id=1139\"><p>On February 17th 2017, I was working on a corpus distillation project, when I encountered some data that didn&rsquo;t match what I had been expecting. It&rsquo;s not unusual to find garbage, corrupt data, mislabeled data or just crazy non-conforming data...but the format of the data this time was confusing enough that I spent some time trying to debug what had gone wrong, wondering if it was a bug in my code. In fact, the data was bizarre enough that some colleagues around the Project Zero office even got intrigued.<\/p>\n<p>It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time. That solved the mystery, but some of the nearby memory had strings and objects that really seemed like they could be from a reverse proxy operated by cloudflare - a major cdn service.<\/p><\/blockquote>\n\n<p><a href=\"https:\/\/blog.cloudflare.com\/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug\/\">Cloudflare<\/a>:<\/p>\n<blockquote cite=\"https:\/\/blog.cloudflare.com\/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug\/\">\n<p>It turned out that in some unusual circumstances, which I&rsquo;ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.<\/p>\n<p>[&#8230;]<\/p>\n<p>It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.<\/p>\n<p>Once we knew that the bug was being caused by the activation of cf-html (but before we knew why) we disabled the three features that caused it to be used. Every feature Cloudflare ships has a corresponding <a href=\"https:\/\/en.wikipedia.org\/wiki\/Feature_toggle\">feature flag<\/a>, which we call a &lsquo;global kill&rsquo;. We activated the Email Obfuscation global kill 47 minutes after receiving details of the problem and the Automatic HTTPS Rewrites global kill 3h05m later. The Email Obfuscation feature had been changed on February 13 and was the primary cause of the leaked memory, thus disabling it quickly stopped almost all memory leaks.<\/p><\/blockquote>\n\n<p><a href=\"http:\/\/gizmodo.com\/everything-you-need-to-know-about-cloudbleed-the-lates-1792710616\">Adam Clark Estes<\/a>:<\/p>\n<blockquote cite=\"http:\/\/gizmodo.com\/everything-you-need-to-know-about-cloudbleed-the-lates-1792710616\"><p>You might not be familiar with Cloudflare itself, but the company&rsquo;s technology is running on a lot of your favorite websites. Cloudflare describes itself as a &ldquo;web performance and security company.&rdquo; Originally an app for tracking down the source of spam, the company now offers a whole menu of products to websites, including performance-based services like content delivery services; reliability-focused offerings like domain name server (DNS) services; and security services like protection against direct denial of service (DDoS) attacks.<\/p><\/blockquote>\n\n<p><a href=\"https:\/\/twitter.com\/lapcatsoftware\/status\/835128791907328000\">Jeff<\/a> <a href=\"https:\/\/twitter.com\/lapcatsoftware\/status\/835130492630491137\">Johnson<\/a>:<\/p>\n<blockquote cite=\"https:\/\/twitter.com\/lapcatsoftware\/status\/835128791907328000\"><p>The scandal is not that Cloudflare exposed private info. The scandal is that Cloudflare has access to private info.<\/p><\/blockquote>\n<blockquote cite=\"https:\/\/twitter.com\/lapcatsoftware\/status\/835130492630491137\"><p>Nobody should ever use a third-party HTTPS proxy. You might as well not even use HTTPS. That&rsquo;s not end-to-end encryption.<\/p><\/blockquote>\n\n<p>1Password&rsquo;s hosted service <a href=\"http:\/\/mjtsai.com\/blog\/2016\/08\/15\/1password-cloud-services-incompatible-with-vpns\/\">uses Cloudflare<\/a>, but it <a href=\"https:\/\/blog.agilebits.com\/2017\/02\/23\/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails\/\">does<\/a> use end-to-end encryption:<\/p>\n<blockquote cite=\"https:\/\/blog.agilebits.com\/2017\/02\/23\/three-layers-of-encryption-keeps-you-safe-when-ssltls-fails\/\"><p>No secrets are transmitted between 1Password clients and <a href=\"https:\/\/1password.com\">1Password.com<\/a> when you sign in and use the service. Our sign-in uses <a href=\"https:\/\/en.wikipedia.org\/wiki\/Secure_Remote_Password_protocol\">SRP<\/a>, which means that server and client prove their identity to each other without transmitting any secrets. This means that users of 1Password <strong>do not<\/strong> need to change their Master Passwords.<\/p>\n<p>Your actual data is encrypted with three layers (including SSL\/TLS), and the other two layers remain secure even if the secrecy of an SSL\/TLS channel is compromised.<\/p>\n<\/blockquote>\n\n<p><a href=\"https:\/\/www.dropbox.com\/en\/help\/27\">Dropbox<\/a> and <a href=\"http:\/\/mjtsai.com\/blog\/2016\/12\/23\/investigating-chronosync-4-7-for-cloud-backup\/\">ChronoSync<\/a> rely on HTTPS, only encrypting the user data after it gets to the server.<\/p>\n\n<p>See also: this <a href=\"https:\/\/github.com\/pirate\/sites-using-cloudflare\/blob\/master\/README.md\">list of affected sites<\/a> (via <a href=\"https:\/\/news.ycombinator.com\/item?id=13721452\">Hacker News<\/a>).<\/p>","protected":false},"excerpt":{"rendered":"<p>Tavis Ormandy (via Hacker News): On February 17th 2017, I was working on a corpus distillation project, when I encountered some data that didn&rsquo;t match what I had been expecting. It&rsquo;s not unusual to find garbage, corrupt data, mislabeled data or just crazy non-conforming data...but the format of the data this time was confusing enough [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"apple_news_api_created_at":"","apple_news_api_id":"","apple_news_api_modified_at":"","apple_news_api_revision":"","apple_news_api_share_url":"","apple_news_coverimage":0,"apple_news_coverimage_caption":"","apple_news_is_hidden":false,"apple_news_is_paid":false,"apple_news_is_preview":false,"apple_news_is_sponsored":false,"apple_news_maturity_rating":"","apple_news_metadata":"\"\"","apple_news_pullquote":"","apple_news_pullquote_position":"","apple_news_slug":"","apple_news_sections":"\"\"","apple_news_suppress_video_url":false,"apple_news_use_image_component":false,"footnotes":""},"categories":[2],"tags":[284,131,1485,339,270,355,48,96],"class_list":["post-17310","post","type-post","status-publish","format-standard","hentry","category-technology","tag-1password","tag-bug","tag-cloudflare","tag-html","tag-parser","tag-privacy","tag-security","tag-web"],"apple_news_notices":[],"_links":{"self":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/17310","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/comments?post=17310"}],"version-history":[{"count":3,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/17310\/revisions"}],"predecessor-version":[{"id":17435,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/posts\/17310\/revisions\/17435"}],"wp:attachment":[{"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/media?parent=17310"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/categories?post=17310"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mjtsai.com\/blog\/wp-json\/wp\/v2\/tags?post=17310"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}