Yesterday, web-infrastructure provider Cloudflare disclosed the discovery and subsequent mitigation of a bug — now known as “Cloudbleed” — in its system. This was a big-deal bug, and it’s been patched (though, in some cases, the more paranoid might want to change their passwords), and you’re welcome to take my word for it and stop reading now. But if you’d like to know what happened, why, and what you should do, read on.
Before understanding the bug, it’s important to understand what Cloudflare is. Cloudflare provides a number of services to millions of websites, mostly focused on maintaining those sites’ stability and security. They’ll mirror sites and set up redundancies if the site suddenly becomes swarmed with traffic, or handle a site’s implementation of SSL, the system that provides secure web traffic. They’re used by many large tech companies — some of which you almost certainly use yourself — but generally speaking, Cloudflare works unseen in the background; and if you’re just a casual web browser, there’s no reason you should have heard of it.
Still, it’s an extremely important company for the infrastructure of the internet. Long story short, a portion of the traffic between you and the websites you use flows through Cloudflare. And, in fact, many different websites use the same Cloudflare hardware at the same time.
So: Cloudbleed. The big problem with the bug was that Cloudflare would return sensitive data stored on uninitialized memory when an HTTP request was made under very specific circumstances and technological configurations. The Google team that found the bug was finding “private messages from major dating sites, full messages from a well-known chat service, online password-manager data, frames from adult-video sites, [and] hotel bookings.”
According to Cloudflare CEO Matthew Prince, who walked me through the whole saga over the phone this morning, the bug actually stems from a piece of code that was written about five years ago, but it was unleashed when Cloudflare made a change to its system last September. If its HTML parser was fed a bad piece of code written by a website that uses Cloudflare, the bug could happen under “an extremely rare set of circumstances that had to happen in a particular order.”
At that point last fall, these memory leaks started happening, according to Prince, at a very low frequency — ten or so times a day is how it was characterized. That all changed on February 13, when another system change increased the circumstances under which Cloudbleed could be executed. Still, a site would have to have a piece of poorly written HTML and a specific combination of Cloudflare features enabled in order to trigger the bug. According to Prince, the vast majority of these bug instances came from search engines, which automatically request web pages in order to index them. That’s how Google got involved.
On the 17th, Tavis Ormandy, who works on Google’s Project Zero, a team devoted to finding security vulnerabilities and patching them, “was working on a corpus distillation project, when I encountered some data that didn’t match what I had been expecting.” Figuring out how to reproduce the issue, the team “observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users.” That’s otherwise known as data from one of Cloudflare’s six million clients that had previously flowed through the server’s network. These are not things that should be publicly accessible, even through complicated technical maneuvering. Like the Heartbleed glitch a few years ago, it involves extracting what is essentially “leftover” data from computer memory not in use.
Among what Google observed was what Prince referred to as Cloudflare’s “NSA key.” When the company’s servers communicate with each other, that data is encrypted using said key. “We always internally called it the ‘NSA key’ because if the NSA was sitting on a piece of fiber connecting two of our data centers,” Prince said, “this was the key that kept that data from being listened in on.”
Ormandy quickly contacted Cloudflare, and according to the company’s timeline, it mitigated most of the problem in a matter of hours. “Within 44 minutes, 99 percent of the problem was patched across our network,” and the final one percent came about seven hours later. (It also didn’t publicly disclose the bug until yesterday, which clearly frustrated the Project Zero team.)
But there was another issue: Search engines had already scraped and retained portions of the memory leaks, so Cloudflare then had to contact search engines like Google and Bing and get all of the leaks taken down.
“There were about 150 Cloudflare customers where we were able to identify that some chunk of users’ data flowed through the system and had ended up in a [search engine],” Prince recalled. Cloudflare notified those customers, who, if they’re competent, notified their users and mandated a password reset or similar security maneuver.
The Cloudbleed glitch is not the same as the attacks that leaked millions of LinkedIn and Yahoo login credentials, and it appears to have been fixed before it could be widely exploited. Still, Prince readily admitted that “it could have been extremely bad. I think that we largely dodged a bullet.” It’s generally a good practice to update passwords on a regular basis, but the catastrophic implications of Cloudbleed appear, as of now, hypothetical. The bug was uncovered by a large tech organization that pokes at web infrastructure in ways that most hackers can’t, and the likelihood that it was exploited before it was fixed is very low.
Another part of understanding Cloudbleed is that precisely whose information could be exposed is dependent on what proportion of Cloudflare traffic its customers use. As Prince put it, on a highway, you’re far more likely to see a Toyota instead of a Maserati. Similarly, large tech companies that constantly pass through Cloudflare at a higher rate, and thus are more likely to be exposed, have their own security protocols.
Cloudflare is now going through the process of reviewing the rest of its systems, and Prince said that they were inserting data into their code that, if it appeared publicly accessible online, would act as a canary for leak issues.
Do you have to, as Gizmodo put it, “Change Your Passwords. Now”? Not necessarily. Much of that hand-wringing comes from an enormous list of sites that use Cloudflare, whose author admits, “just because a domain is on the list does not mean the site is compromised, and sites may be compromised that do not appear on this list.” Kinda broad, no? That said, if resetting all of your passwords gives you peace of mind, it would be foolish for me to stop you.
