Thursday, October 4, 2018 [Tweets] [Favorites]

How China Used a Tiny Chip to Infiltrate U.S. Companies

Jordan Robertson and Michael Riley (Hacker News):

Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community. Elemental’s servers could be found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers.

[…]

Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline. Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally.

[…]

Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

Apple’s response:

Over the course of the past year, Bloomberg has contacted us multiple times with claims, sometimes vague and sometimes elaborate, of an alleged security incident at Apple. Each time, we have conducted rigorous internal investigations based on their inquiries and each time we have found absolutely no evidence to support any of them. We have repeatedly and consistently offered factual responses, on the record, refuting virtually every aspect of Bloomberg’s story relating to Apple.

On this we can be very clear: Apple has never found malicious chips, “hardware manipulations” or vulnerabilities purposely planted in any server. Apple never had any contact with the FBI or any other agency about such an incident.

Julie Bort (in 2016, via iStumbler):

Still, Apple is motivated to design build its own hardware, the same as Google and Amazon does, and run it on its own for one pretty scary reason: security. It suspects that the servers it has been ordering from others are being captured during shipping, and backdoors added to them that will make them susceptible to being hacked.

At one point, the company even had people taking photographs of the motherboards in the computer servers it was using, then mark down exactly what each chip was, to make sure everything was fully understood.

Update (2018-10-05): Amir Efrati (in 2017):

In early 2016, Apple discovered what it believed was a potential security vulnerability in at least one data center server it purchased from a U.S.-based manufacturer, Super Micro Computer, according to a Super Micro executive and two people who were briefed about the incident at Apple. The server was part of Apple’s technical infrastructure, which powers its web-based services and holds customer data.

Apple ended up terminating its yearslong business relationship with Super Micro, according to Tau Leng, a senior vice president of technology for Super Micro, and a person who was told about the incident by a senior infrastructure engineering executive at Apple. The tech giant even returned some of Super Micro’s servers to the company, according to one of the people briefed about the incident.

Stephen Schmidt:

Today, Bloomberg BusinessWeek published a story claiming that AWS was aware of modified hardware or malicious chips in SuperMicro motherboards in Elemental Media’s hardware at the time Amazon acquired Elemental in 2015, and that Amazon was aware of modified hardware or chips in AWS’s China Region.

As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this is untrue. At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.

Kim Zetter:

I have to say, this is all really bizarre. The Bloomberg story is very detailed, citing documents and inside sources. But the company denials are also detailed and emphatic. You don’t often see the latter when a company is trying to hide something or be coy.

John Gruber (tweet):

I see no way around it: either Bloomberg’s report is significantly wrong, at least as pertains to Amazon and Apple, or Apple and Amazon have issued blatantly false denials.

Apple (Hacker News, MacRumors):

We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed. Our best guess is that they are confusing their story with a previously-reported 2016 incident in which we discovered an infected driver on a single Super Micro server in one of our labs. That one-time event was determined to be accidental and not a targeted attack against Apple.

[…]

Finally, in response to questions we have received from other news organizations since Businessweek published its story, we are not under any kind of gag order or other confidentiality obligations.

John Gruber:

What sense does it make that Apple discovered a profound security problem in Super Micro motherboards in May 2015, so serious that the company reported it to the FBI, but then didn’t sever ties with Supermicro until at least eight months later? That timeline makes no sense.

Matt Drance:

After reading this Bloomberg story I have two questions:

1) Why not name the “third party company” that found this hack? What security firm wouldn’t want credit for this?

2) FBI and DNI/CIA/NSA declined comment on this story primarily sourced from “US officials.” What’s left?

Zack Whittaker:

In fairness to Bloomberg, chief among Apple’s complaints is a claim that Bloomberg’s reporters were vague in their questioning. Given the magnitude of the story, you don’t want to reveal all of your cards — but still want to seek answers and clarifications without having the subject tip off another news agency — a trick sometimes employed by the government in the hope of lighter coverage.

Yet, to Apple — and Amazon and other companies implicated by the report — they too might also be in the dark. Assuming there was an active espionage investigation into the alleged actions of a foreign government, you can bet that only a handful of people at these companies will be even cursorily aware of the situation. U.S. surveillance and counter-espionage laws restrict who can be told about classified information or investigations. Only those who need to be in the know are kept in a very tight loop — typically a company’s chief counsel. Often their bosses, the chief executive or president, are not told to avoid making false or misleading statements to shareholders.

Nick Heer:

This story has been rattling around my head all day today. My early thought was that perhaps the Bloomberg reporters did a Judith Miller. Maybe their government sources had a specific angle they wished to present to create a political case against China or in favour of further sanctions — or actions far more serious — and needed a credible third-party, like a news organization, to create a story like this. But Robertson and Riley’s seventeen sources include several individuals at Amazon and Apple with intimate knowledge of the apparent discovery of unauthorized hardware modifications, something they later confirmed in a statement to Alex Cranz of Gizmodo. This doesn’t seem likely.

[…]

Indeed, Kieren McCarthy of the Register did a fine job parsing each company’s statements, albeit with his usual unique flair. But, though there is absolutely some wiggle-room in each denial, there are remarks made by each company that, were they found to be wrong, would be simple lies.

[…]

Either manufacturing of these components becomes increasingly diversified or, more likely, far greater control and oversight is required by companies and end-client governments alike.

Kieren McCarthy:

As to the reports – from both Amazon and Apple – that Bloomberg says its sources have seen. It is worth noting that Bloomberg does not claim to have seen those reports itself. How closely were its sources able to scrutinize those reports? Could they have been mistaken?

From that point, it is very possible that the other sources that Bloomberg felt were confirming its story were confirming something else: that China is trying to get into the hardware supply chain. Which is no doubt true, as US intelligence agencies have repeatedly warned in the past year, particularly with respect to mobile phones.

So it is possible that the reporters did an excellent job but ended up in the wrong place, with half a story but going down the wrong path. It is equally possible that they have got 90 per cent of the way there and Apple and Amazon are carefully using the last 10 per cent to issue careful denials.

Update (2018-10-10): Joe Rossignol:

Apple’s recently retired general counsel Bruce Sewell told Reuters he called the FBI’s then-general counsel James Baker last year after being told by Bloomberg of an open investigation into Supermicro, and was told that nobody at the federal law enforcement agency knew what the story was about.

John Paczkowski and Charlie Warzel (Hacker News):

Multiple senior Apple executives, speaking with BuzzFeed News on the condition of anonymity so that they could speak freely, all denied and expressed confusion with a report earlier this week that the company’s servers had been compromised by a Chinese intelligence operation.

Bob Burrough:

What of The Information’s article Feb ’17? I don’t think this would be a conspiracy between the two news orgs. Something’s up.

Kevin Beaumont:

Worth noting same Bloomberg reporters put out a story a few years citing multiple sources that the US knew about Heartbleed. That story was flat out wrong. Bloomberg didn’t follow it up or comment.

Joe Rossignol:

The U.S. Department of Homeland Security today said it has “no reason to doubt” the companies who denied a bombshell Bloomberg Businessweek report this week about Chinese spies using a tiny chip to infiltrate U.S. companies.

Nick Heer:

Reuters also reports that a division of GCHQ, Britain’s signals intelligence agency, does not presently doubt Apple and Amazon’s denials.

[…]

That’s a lot of reputable organisations — and the American government — who have staked their credibility on widely varying accounts of the veracity of this story.

John Gruber:

Bloomberg’s Big Hack story should eventually be fully-corroborated, if true. According to their report, there are thousands of compromised servers out there. If there are, security experts will eventually identify these rogue chips and document them.

See also: Hacker News.

SwiftOnSecurity:

The Bloomberg article has no actionable information for industry or consumers. All claimed involved parties have denied the events described ever happened.

It’s unclear what the purpose of this is.

Apple (Hacker News):

In light of your important leadership roles in Congress, we want to assure you that a recent report in Bloomberg Businessweek alleging the compromise of our servers is not true. You should know that Bloomberg provided us with no evidence to substantiate their claims and our internal investigations concluded their claims were simply wrong. We are eager to share the facts in this matter because, were this story true, it would rightly raise grave concerns.

John Gruber:

Hardware security researcher Joe Fitzpatrick was one of the very few named sources in Bloomberg’s blockbuster “The Big Hack” story. He provided only background information on the potential of hardware exploits in general — he claimed no knowledge of this specific case. On Patrick Gray’s Risky Business (great name) podcast, he expresses serious unease with the story Bloomberg published.

Jason Koebler, Joseph Cox, and Lorenzo Franceschi-Bicchierai:

Even sources used in the original story are confused about what’s going on. The cybersecurity podcast Risky Business interviewed one of the few named sources in the original Businessweek article, hardware security expert Joe Fitzpatrick, who expressed doubts about the article, and said he had never been contacted by any Bloomberg fact-checker. Fitzpatrick was used as an expert source to comment on the technical details of what Bloomberg described and does not have any firsthand knowledge of the actual alleged hack.

John Paczkowski:

what kind of source elicits so much confidence that you don’t provide evidence for review to the companies involved, single source some key details, and stand by your story when two tech bigs are shooting you in the face with both barrels while multiple telecoms say “not us”?

Nick Heer:

For what it’s worth, I don’t want Robertson and Riley to have egg on their faces. I hope the story is not entirely as described because, if it is, it is truly one of the biggest security breaches in modern history — Supermicro has supplied a lot of servers to industry giants. But I don’t want the reporters to be wrong; Bloomberg has a great reputation for publishing rigorously-researched and fact-checked longform stories; I don’t want to have lingering doubts about their future reporting. And I’m not defending the biggest corporations in the world out of loyalty or denial — they have PR teams for that, and should absolutely be criticized when relevant. And I think the central point of the article — that the supply chain of a vast majority of the world’s goods is monopolized by an authoritarian and privacy-averse government is a staggering risk — is absolutely worth taking seriously.

Joe Rossignol:

Rob Joyce, Senior Advisor for Cybersecurity Strategy at the NSA, is the latest official to question the accuracy of Bloomberg Businessweek’s bombshell “The Big Hack” report about Chinese spies compromising the U.S. tech supply chain.

“I have pretty good understanding about what we’re worried about and what we’re working on from my position. I don’t see it,” said Joyce, speaking at a U.S. Chamber of Commerce cyber summit in Washington, D.C. today, according to a subscriber-only Politico report viewed by MacRumors.

See also: Upgrade.

Update (2018-10-19): BuzzFeed (Hacker News):

Apple CEO Tim Cook, in an interview with BuzzFeed News, went on the record for the first time to deny allegations that his company was the victim of a hardware-based attack carried out by the Chinese government. And, in an unprecedented move for the company, he called for a retraction of the story that made this claim.

3 Comments

What I don't get, is that even the most advanced chip can easily be detected and block by a properly configured firewall.
So why in hell would companies paranoid about security and their data don't have such device on there network ?

Will Notbepublished

I've read some arguments against using Firewalls because they can be used to create DoSes that would not happen if the Firewalls were not there.

The Register has a detailed dissection of how they suspect Bloomberg's reporters got this wrong.

https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/

Stay up-to-date by subscribing to the Comments RSS Feed for this post.

Leave a Comment