Cloudbleed: Cloudflare’s HTTPS Traffic Leak
Tavis Ormandy (via Hacker News):
On February 17th 2017, I was working on a corpus distillation project, when I encountered some data that didn’t match what I had been expecting. It’s not unusual to find garbage, corrupt data, mislabeled data or just crazy non-conforming data...but the format of the data this time was confusing enough that I spent some time trying to debug what had gone wrong, wondering if it was a bug in my code. In fact, the data was bizarre enough that some colleagues around the Project Zero office even got intrigued.
It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time. That solved the mystery, but some of the nearby memory had strings and objects that really seemed like they could be from a reverse proxy operated by cloudflare - a major cdn service.
It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
[…]
It turned out that the underlying bug that caused the memory leak had been present in our Ragel-based parser for many years but no memory was leaked because of the way the internal NGINX buffers were used. Introducing cf-html subtly changed the buffering which enabled the leakage even though there were no problems in cf-html itself.
Once we knew that the bug was being caused by the activation of cf-html (but before we knew why) we disabled the three features that caused it to be used. Every feature Cloudflare ships has a corresponding feature flag, which we call a ‘global kill’. We activated the Email Obfuscation global kill 47 minutes after receiving details of the problem and the Automatic HTTPS Rewrites global kill 3h05m later. The Email Obfuscation feature had been changed on February 13 and was the primary cause of the leaked memory, thus disabling it quickly stopped almost all memory leaks.
You might not be familiar with Cloudflare itself, but the company’s technology is running on a lot of your favorite websites. Cloudflare describes itself as a “web performance and security company.” Originally an app for tracking down the source of spam, the company now offers a whole menu of products to websites, including performance-based services like content delivery services; reliability-focused offerings like domain name server (DNS) services; and security services like protection against direct denial of service (DDoS) attacks.
The scandal is not that Cloudflare exposed private info. The scandal is that Cloudflare has access to private info.
Nobody should ever use a third-party HTTPS proxy. You might as well not even use HTTPS. That’s not end-to-end encryption.
1Password’s hosted service uses Cloudflare, but it does use end-to-end encryption:
No secrets are transmitted between 1Password clients and 1Password.com when you sign in and use the service. Our sign-in uses SRP, which means that server and client prove their identity to each other without transmitting any secrets. This means that users of 1Password do not need to change their Master Passwords.
Your actual data is encrypted with three layers (including SSL/TLS), and the other two layers remain secure even if the secrecy of an SSL/TLS channel is compromised.
Dropbox and ChronoSync rely on HTTPS, only encrypting the user data after it gets to the server.
See also: this list of affected sites (via Hacker News).
11 Comments RSS · Twitter
What do we know about Amazon CloudFront or Google Firebase?
For that matter, when is mjtsai.com going SSL?
I have written an article about this intending to foster discussion about possible regulatory responses to this incident.
It also contains a list of technical recommendations that companies like cloudflare should be following to prevent issues like this.
@Sean I haven’t read about those. I’m not sure about this site. I suspect that for this type of content, SSL would slow it down, and I don’t currently have a way to make the internal links keep you on HTTPS if you start there. Unless I redirect HTTP to HTTPS, but I wonder if that would cause problems for people.
Michael, FYI, in HTML, you can leave the protocol off of the URL (e.g. "//mjtsai.com/blog/2017/02/24/cloudbleed-cloudflares-https-traffic-leak") and it will use the current protocol. No need for server-side shenanigans.
@Clark Right, but do I want to do some sort of automated replacement to convert all the existing links and future ones? I’m not sure. Also, I can tell from server error logs that a lot of software doesn’t understand the “//”.
Frankly, while there are certainly reasons on the margin for wanting mjtsai.com to go SSL, it seems a pretty damn low priority to me, given what the site is.
OTOH, shouldn't c-command.com go SSL with alacrity? Seems as if making damn sure no middleman are present on software downloads is important, even if it's a somewhat unlikely vector for various reasons.
@Chucky c-command.com has had SSL for a while now. The downloads and forum are always SSL. The whole site is available as SSL, but if you start out with HTTP it doesn’t redirect (yet). I didn’t want to inadvertently break anything. I’ll probably try it soon.
@Michael re SSL - the all too frequent problem of content manipulation/injection on hotel/airport/etc... networks. The speed issue seems minor these days in comparison to the value of basic public hygiene.
"Google wants to kill the unencrypted internet, and will soon flag two thirds of the web as "unsafe.""
"@Chucky c-command.com has had SSL for a while now. The downloads and forum are always SSL."
Hmm... I genuinely don't know if this is meaningful or not, but when I click the EagleFiler link from this page, the download link shows "http://c-command.com/downloads/EagleFiler-1.7.4.dmg". And when I download, the link the Finder shows the file was downloaded from is the same.
Like I say, I don't know if this actually means the file was downloaded w/out SSL or with, but I thought I should share.
@Chucky The link that you clicked was a relative one. If you had been viewing the secure version of the page, you would have gotten a secure download. Since you were on the insecure version of the page, it doesn’t make a whole lot of difference what the URL was because an attacker could have replaced it with a totally different link, anyway. For the first download, the ultimate line of defense is that the .dmg file and code are signed.
My comment above was referring to using SSL within the apps. The URL that they check for new versions and the downloads of the new versions themselves have been using SSL since Apple started talking about this. So if you trust the initial download, you don’t have to worry about an update downloading bad code.
In any case, I’m proceeding with converting the entire C-Command site to use SSL. As of now, all internal links except those inside of blog and forum posts should use HTTPS. The links in the apps and online help will also switch over in their next versions. And if you follow a HTTP link to the site it should redirect you to HTTPS.
If this goes smoothly, I plan to do the same thing with this site.
"For the first download, the ultimate line of defense is that the .dmg file and code are signed."
Ah. Makes perfect sense. I figured I was perhaps missing some important piece of the puzzle.
