How China Used a Tiny Chip to Infiltrate U.S. Companies
Jordan Robertson and Michael Riley (Hacker News):
Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community. Elemental’s servers could be found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers.
[…]
Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline. Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally.
[…]
Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.
On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident.
Julie Bort (in 2016, via iStumbler):
Still, Apple is motivated to design build its own hardware, the same as Google and Amazon does, and run it on its own for one pretty scary reason: security. It suspects that the servers it has been ordering from others are being captured during shipping, and backdoors added to them that will make them susceptible to being hacked.
At one point, the company even had people taking photographs of the motherboards in the computer servers it was using, then mark down exactly what each chip was, to make sure everything was fully understood.
Update (2018-10-05): Amir Efrati (in 2017):
In early 2016, Apple discovered what it believed was a potential security vulnerability in at least one data center server it purchased from a U.S.-based manufacturer, Super Micro Computer, according to a Super Micro executive and two people who were briefed about the incident at Apple. The server was part of Apple’s technical infrastructure, which powers its web-based services and holds customer data.
Apple ended up terminating its yearslong business relationship with Super Micro, according to Tau Leng, a senior vice president of technology for Super Micro, and a person who was told about the incident by a senior infrastructure engineering executive at Apple. The tech giant even returned some of Super Micro’s servers to the company, according to one of the people briefed about the incident.
Today, Bloomberg BusinessWeek published a story claiming that AWS was aware of modified hardware or malicious chips in SuperMicro motherboards in Elemental Media’s hardware at the time Amazon acquired Elemental in 2015, and that Amazon was aware of modified hardware or chips in AWS’s China Region.
As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this is untrue. At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
I have to say, this is all really bizarre. The Bloomberg story is very detailed, citing documents and inside sources. But the company denials are also detailed and emphatic. You don’t often see the latter when a company is trying to hide something or be coy.
John Gruber (tweet):
I see no way around it: either Bloomberg’s report is significantly wrong, at least as pertains to Amazon and Apple, or Apple and Amazon have issued blatantly false denials.
Apple (Hacker News, MacRumors):
We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.
[…]
Finally, in response to questions we have received from other news organizations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.
What sense does it make that Apple discovered a profound security problem in Super Micro motherboards in May 2015, so serious that the company reported it to the FBI, but then didn’t sever ties with Supermicro until at least eight months later? That timeline makes no sense.
After reading this Bloomberg story I have two questions:
1) Why not name the “third party company” that found this hack? What security firm wouldn’t want credit for this?
2) FBI and DNI/CIA/NSA declined comment on this story primarily sourced from “US officials.” What’s left?
In fairness to Bloomberg, chief among Apple’s complaints is a claim that Bloomberg’s reporters were vague in their questioning. Given the magnitude of the story, you don’t want to reveal all of your cards — but still want to seek answers and clarifications without having the subject tip off another news agency — a trick sometimes employed by the government in the hope of lighter coverage.
Yet, to Apple — and Amazon and other companies implicated by the report — they too might also be in the dark. Assuming there was an active espionage investigation into the alleged actions of a foreign government, you can bet that only a handful of people at these companies will be even cursorily aware of the situation. U.S. surveillance and counter-espionage laws restrict who can be told about classified information or investigations. Only those who need to be in the know are kept in a very tight loop — typically a company’s chief counsel. Often their bosses, the chief executive or president, are not told to avoid making false or misleading statements to shareholders.
This story has been rattling around my head all day today. My early thought was that perhaps the Bloomberg reporters did a Judith Miller. Maybe their government sources had a specific angle they wished to present to create a political case against China or in favour of further sanctions — or actions far more serious — and needed a credible third-party, like a news organization, to create a story like this. But Robertson and Riley’s seventeen sources include several individuals at Amazon and Apple with intimate knowledge of the apparent discovery of unauthorized hardware modifications, something they later confirmed in a statement to Alex Cranz of Gizmodo. This doesn’t seem likely.
[…]
Indeed, Kieren McCarthy of the Register did a fine job parsing each company’s statements, albeit with his usual unique flair. But, though there is absolutely some wiggle-room in each denial, there are remarks made by each company that, were they found to be wrong, would be simple lies.
[…]
Either manufacturing of these components becomes increasingly diversified or, more likely, far greater control and oversight is required by companies and end-client governments alike.
As to the reports – from both Amazon and Apple – that Bloomberg says its sources have seen. It is worth noting that Bloomberg does not claim to have seen those reports itself. How closely were its sources able to scrutinize those reports? Could they have been mistaken?
From that point, it is very possible that the other sources that Bloomberg felt were confirming its story were confirming something else: that China is trying to get into the hardware supply chain. Which is no doubt true, as US intelligence agencies have repeatedly warned in the past year, particularly with respect to mobile phones.
So it is possible that the reporters did an excellent job but ended up in the wrong place, with half a story but going down the wrong path. It is equally possible that they have got 90 per cent of the way there and Apple and Amazon are carefully using the last 10 per cent to issue careful denials.
Update (2018-10-10): Joe Rossignol:
Apple’s recently retired general counsel Bruce Sewell told Reuters he called the FBI’s then-general counsel James Baker last year after being told by Bloomberg of an open investigation into Supermicro, and was told that nobody at the federal law enforcement agency knew what the story was about.
John Paczkowski and Charlie Warzel (Hacker News):
Multiple senior Apple executives, speaking with BuzzFeed News on the condition of anonymity so that they could speak freely, all denied and expressed confusion with a report earlier this week that the company’s servers had been compromised by a Chinese intelligence operation.
What of The Information’s article Feb ’17? I don’t think this would be a conspiracy between the two news orgs. Something’s up.
Worth noting same Bloomberg reporters put out a story a few years citing multiple sources that the US knew about Heartbleed. That story was flat out wrong. Bloomberg didn’t follow it up or comment.
The U.S. Department of Homeland Security today said it has “no reason to doubt” the companies who denied a bombshell Bloomberg Businessweek report this week about Chinese spies using a tiny chip to infiltrate U.S. companies.
Reuters also reports that a division of GCHQ, Britain’s signals intelligence agency, does not presently doubt Apple and Amazon’s denials.
[…]
That’s a lot of reputable organisations — and the American government — who have staked their credibility on widely varying accounts of the veracity of this story.
Bloomberg’s Big Hack story should eventually be fully-corroborated, if true. According to their report, there are thousands of compromised servers out there. If there are, security experts will eventually identify these rogue chips and document them.
The Bloomberg article has no actionable information for industry or consumers. All claimed involved parties have denied the events described ever happened.
It’s unclear what the purpose of this is.
Apple (Hacker News):
In light of your important leadership roles in Congress, we want to assure you that a recent report in Bloomberg Businessweek alleging the compromise of our servers is not true. You should know that Bloomberg provided us with no evidence to substantiate their claims and our internal investigations concluded their claims were simply wrong. We are eager to share the facts in this matter because, were this story true, it would rightly raise grave concerns.
Hardware security researcher Joe Fitzpatrick was one of the very few named sources in Bloomberg’s blockbuster “The Big Hack” story. He provided only background information on the potential of hardware exploits in general — he claimed no knowledge of this specific case. On Patrick Gray’s Risky Business (great name) podcast, he expresses serious unease with the story Bloomberg published.
Jason Koebler, Joseph Cox, and Lorenzo Franceschi-Bicchierai:
Even sources used in the original story are confused about what’s going on. The cybersecurity podcast Risky Business interviewed one of the few named sources in the original Businessweek article, hardware security expert Joe Fitzpatrick, who expressed doubts about the article, and said he had never been contacted by any Bloomberg fact-checker. Fitzpatrick was used as an expert source to comment on the technical details of what Bloomberg described and does not have any firsthand knowledge of the actual alleged hack.
what kind of source elicits so much confidence that you don’t provide evidence for review to the companies involved, single source some key details, and stand by your story when two tech bigs are shooting you in the face with both barrels while multiple telecoms say “not us”?
For what it’s worth, I don’t want Robertson and Riley to have egg on their faces. I hope the story is not entirely as described because, if it is, it is truly one of the biggest security breaches in modern history — Supermicro has supplied a lot of servers to industry giants. But I don’t want the reporters to be wrong; Bloomberg has a great reputation for publishing rigorously-researched and fact-checked longform stories; I don’t want to have lingering doubts about their future reporting. And I’m not defending the biggest corporations in the world out of loyalty or denial — they have PR teams for that, and should absolutely be criticized when relevant. And I think the central point of the article — that the supply chain of a vast majority of the world’s goods is monopolized by an authoritarian and privacy-averse government is a staggering risk — is absolutely worth taking seriously.
Rob Joyce, Senior Advisor for Cybersecurity Strategy at the NSA, is the latest official to question the accuracy of Bloomberg Businessweek’s bombshell “The Big Hack” report about Chinese spies compromising the U.S. tech supply chain.
“I have pretty good understanding about what we’re worried about and what we’re working on from my position. I don’t see it,” said Joyce, speaking at a U.S. Chamber of Commerce cyber summit in Washington, D.C. today, according to a subscriber-only Politico report viewed by MacRumors.
See also: Upgrade.
Update (2018-10-19): BuzzFeed (Hacker News):
Apple CEO Tim Cook, in an interview with BuzzFeed News, went on the record for the first time to deny allegations that his company was the victim of a hardware-based attack carried out by the Chinese government. And, in an unprecedented move for the company, he called for a retraction of the story that made this claim.
Update (2018-10-25): John Gruber:
The longer they drag this out before a full retraction, the more damage they’re taking to their long-term credibility. Read their statement closely — they’re not saying their story is true or that Apple and Tim Cook are wrong. All they say is they spent a year on the story and spoke to 17 sources multiple times.
This is one of the most baffling sagas I can remember. Either the supply chain is hosed and companies like Apple and Amazon really have no idea, they do know and their executives are covering it up in flagrant violation of the law, or an esteemed news organization fucked up to an immense degree.
@tim_cook is right. Bloomberg story is wrong about Amazon, too. They offered no proof, story kept changing, and showed no interest in our answers unless we could validate their theories. Reporters got played or took liberties. Bloomberg should retract.
Supermicro sold tens of thousands of server motherboards to the US companies mentioned in the story. Were they all infected with the offending spyware chip? Probably not, but there must have been thousands of motherboards released into the wild with the purported mission of penetrating US infrastructures. Yet, despite “more than a year of reporting” and “more than 100 interviews…including government officials and insiders at the companies” (from Bloomberg’s reply to Tim Cook), Bloomberg and its (anonymous) sources were unable to come up with a single infected motherboard.
A missing weapon doesn’t mean the crime didn’t happen. But not finding any weapons after thousands of crimes should have troubled the authors — or, more important, their hierarchy of editors.
Erik Wemple (via John Gruber):
Sources tell the Erik Wemple Blog that the New York Times, the Wall Street Journal and The Post have each sunk resources into confirming the story, only to come up empty-handed.
[…]
Bloomberg, on the other hand, gives readers virtually no road map for reproducing its scoop, which helps to explain why competitors have whiffed in their efforts to corroborate it.
Michael Riley, one of the reporters on the story, quickly asserted after the story’s publication that the physical evidence assured that corroborating stories would soon be published. Not only has that not happened, it’s the inverse that has: source after source raising doubts about the accuracy of the story’s core arguments.
Today, Supermicro Charles Liang joined Cook in calling for a retraction. In a statement shared by CNBC, Liang said that Supermicro has not found malicious hardware components in its products, nor has Bloomberg produced an affected Supermicro motherboard. Bloomberg, he says, should "act responsibly" and retract its "unsupported allegations."
We have discussed two patently false technical details in the Bloomberg article. Anyone involved in the server industry will know this as they are common foundational elements regarding how servers work. Beyond the false points in the Bloomberg article, there are a number of other elements that are at best implausible.
[…]
In this article, we have shown why the technical details of the Bloomberg alleged hack are inaccurate and/or implausible. These technical details were offered to Bloomberg through anonymous sources, so we have no way of doing further fact-checking. We showed why, even if a chip can be produced and placed it would not work as Bloomberg reports.
Update (2018-12-12): Erik Wemple:
Not only did industry and government officials denounce the conclusions on the record, but the story itself was short on hard evidence of a supply-chain compromise. It relied on “17 people” who “confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information,” noted the story. What it lacked were documents, photos, reports — any of the artifacts that would logically go along with such a scary intrusion into the U.S. economy.
Despite such shortcomings, Bloomberg continues to stand by the same stand-by statement it issued weeks ago: “We stand by our story and are confident in our reporting and sources.”
I don’t think it’s real. Yes, it’s plausible. But first of all, if someone actually surreptitiously put malicious chips onto motherboards en masse, we would have seen a photo of the alleged chip already. And second, there are easier, more effective, and less obvious ways of adding backdoors to networking equipment.
Joseph Menn (via Rene Ritchie, Hacker News, MacRumors):
Computer hardware maker Super Micro Computer Inc told customers on Tuesday that an outside investigations firm had found no evidence of any malicious hardware in its current or older-model motherboards.
I’ve been told by reporters that they don’t have any journalistic requirement to protect sources that intentionally deceive them. Are we there yet? Because at this point, it seems like either:
1. REALLY bad reporting
2. Coordinated “leaks”
I’m guessing the latter.
4 Comments RSS · Twitter
I've read some arguments against using Firewalls because they can be used to create DoSes that would not happen if the Firewalls were not there.
The Register has a detailed dissection of how they suspect Bloomberg's reporters got this wrong.
https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/
[…] How China Used a Tiny Chip to Infiltrate U.S. Companies […]
What I don't get, is that even the most advanced chip can easily be detected and block by a properly configured firewall.
So why in hell would companies paranoid about security and their data don't have such device on there network ?