Yesterday, a blockbuster Bloomberg Businessweek story was published claiming that China’s People’s Liberation Army had managed to crack the manufacturing supply chain and hack chips in servers manufactured by a company called Supermicro — chips used by Amazon, Apple, and around 30 other American companies. It means that the foundation of the last 25 years of electronics manufacturing — the use of cheap and serpentine supply chains, many based out of China, to manufacture electronics designed and assembled across the globe — has been compromised.
It’s a world-shaking story … if it holds up. There’s a part of the whole thing that I cannot get past: Both Apple and Amazon have issued vehement, detailed denials, with both companies casting aspersions on Bloomberg Businessweek’s reporting and flat-out denying key parts of its story. True, it’s not uncommon for big companies to carefully deny damaging stories whose main thrust is true. But it’s unthinkable for a large and publicly traded company to categorically and comprehensively deny the claims of an article like this unless they’re really not true.
On the other hand, Bloomberg Businessweek is known as both one of the best, and one of the most cautious, publications in the world of business and national-security reporting. It’s equally unthinkable that it would have published a story this shocking without having reported it deeply (the reporters write that they have 17 sources) and having thoroughly checked it out legally.
Which means that one of two unthinkable things must be happening, and it’s driving me crazy. I’ve read through Bloomberg’s story and Apple’s and Amazon’s statements over and over, trying to see the angle where somehow both Businessweek’s story and Apple’s and Amazon’s statements are true, and I just can’t.
Let’s take a look at what each company is saying, and what Bloomberg claims, starting with Amazon, which gets much more focus in the Bloomberg piece.
Amazon
Amazon issued two statements. One was a statement denying much of the story and was provided to Bloomberg before Bloomberg published its piece. After Bloomberg published its story, Amazon issued a stronger denial, posted to its own site and written by Steve Schmidt, chief information security officer at Amazon Web Services. I’m looking mainly at the second statement because it’s much more wide-ranging and a more vigorous denial of the Bloomberg story.
Bloomberg: “Nested on [Supermicro] servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design.”
Amazon: “At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems.”
Bloomberg: “Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community.”
Amazon: “Nor have we engaged in an investigation with the government.”
Bloomberg: “The first pass uncovered troubling issues, prompting AWS to take a closer look at Elemental’s main product: the expensive servers that customers installed in their networks to handle the video compression. […] In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says.”
Amazon: “[W]hen Amazon was considering acquiring Elemental, we did a lot of due diligence with our own security team, and also commissioned a single external security company to do a security assessment for us as well. That report did not identify any issues with modified chips or hardware.”
Bloomberg: “Mindful of the Elemental findings, Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered.”
Amazon: “The article also claims that after learning of hardware modifications and malicious chips in Elemental servers, we conducted a network-wide audit of SuperMicro motherboards and discovered the malicious chips in a Beijing data center. […] We never found modified hardware or malicious chips in servers in any of our data centers.”
Apple
Apple’s denial is in many ways much stronger than Amazon’s. This may be because, as Bloomberg alleges, the company refused to cooperate with authorities except to alert the FBI when it discovered a Supermicro server with hardware modification. Still, Apple’s statement and Bloomberg’s stand in stark contrast to each other.
Bloomberg: “Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline.”
Apple: “On this we can be very clear: Apple has never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server.”
Bloomberg: “Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally.”
Apple: “Apple never had any contact with the FBI or any other agency about such an incident.”
Bloomberg: “As for Apple, one of the three senior insiders says that in the summer of 2015, a few weeks after it identified the malicious chips, the company started removing all Supermicro servers from its data centers, a process Apple referred to internally as ‘going to zero.’ Every Supermicro server, all 7,000 or so, was replaced in a matter of weeks, the senior insider says.”
Apple: “We did not uncover any unusual vulnerabilities in the servers we purchased from Super Micro when we updated the firmware and software according to our standard procedures.”
So what happened? Perhaps Bloomberg Businessweek, a very cautious publication with extremely stringent editing standards, and two veteran reporters with decades of experience between them, got badly played by over a dozen sources spread across Apple, Amazon, and the national intelligence community. Some of the sourcing in Bloomberg’s story feels thin. All of the sources are anonymous, and much of Bloomberg’s assertions are secondhand information. For instance, Bloomberg’s descriptions of the modified motherboards are based not on what Bloomberg reporters saw, but what someone talking to Bloomberg reporters saw.
But 17 sources spoke about modified Supermicro hardware to Bloomberg, and there are multiple sources who assert that the modified hardware found its way to both Apple and Amazon. It’s hard to imagine so many sources all steering the reporters so wrong in the same direction, barring some massive conspiracy. Keep in mind, too, that Bloomberg’s editors (and legal department) saw these very strong denials from Apple and Amazon before running the story, and felt confident enough in their reporting to move forward.
And yet, if the Bloomberg story is true, Apple’s and Amazon’s denials also don’t make sense! Both companies are press shy, often more comfortable staying silent than making a statement, even in the face of controversy. Bloomberg’s story is explosive, so it’s understandable that both felt the need to issue statements, but both statements were far more expansive than they needed to be. (And remember that these statements, vetted by both companies’ legal teams, could now be subject to SEC scrutiny as materially false statements that would be misleading to investors.)
It’s beyond puzzling as to why both companies are being strident in refuting the story — including both denying ever contacting U.S. authorities or ever discovering motherboards with hardware modifications — especially when they had the option to either stay quiet or offer a much more muted denial to Bloomberg.
If Apple’s and Amazon’s denials weren’t so vigorous, I could concoct a far-fetched scenario where Amazon’s and Apple’s statements were made out of ignorance. Bloomberg’s story says the intelligence community was aware that the Chinese government planned a hardware attack via Supermicro servers as early as 2014, but the intelligence community was unsure what to do with the information — alerting companies about its suspicions would likely destroy a San Jose–based manufacturer and alert the Chinese intelligence community that U.S. spies were on to them. So it’s possible that Amazon and Apple found these odd Supermicro servers with hardware modifications, alerted authorities, and removed them from use, without being aware that they were actually being targeted by the Chinese military intelligence. But Apple and Amazon both deny that they ever found any malicious hardware, period, and deny that they ever worked with authorities.
Perhaps you could even go with the even more implausible theory that the denials are part of some government master plan, with Apple and Amazon working in concert with, or at the orders of, authorities, but it just doesn’t hold up; Apple flatly states: “We are not under any kind of gag order or other confidentiality obligations.”
What makes all of this feel like a toothache that I can’t stop touching with my tongue is, I can’t see who actually benefits in any of the possible scenarios laid out here. Bloomberg wouldn’t blow up its reputation on a flimsy story. I can see no reason why 17 different sources would all have the desire to work with Bloomberg reporters to advance a narrative of China’s hardware-hacking attempts becoming a reality. But I also can’t understand why Apple and Amazon seem to be overly broad in their denials unless they are very, very sure of what the facts are — they’re exposing themselves to too much risk otherwise.
If I blur my eyes and give all sides the benefit of the doubt to an absurd degree, I could imagine a world where all of them believe that they are telling the truth. What I cannot see, no matter how many times I turn Bloomberg’s story and Amazon’s and Apple’s statements over in my mind, is a world where all parties are actually right.
