Scanning iCloud Photos for Child Sexual Abuse
The Child Sexual Abuse Material (CSAM) Scanning Tool allows website owners to proactively identify and take action on CSAM located on their website. By enabling this tool, Cloudflare will compare content served for your website through the Cloudflare cache to known lists of CSAM. These lists are provided to Cloudflare by leading child safety advocacy groups such as the National Center for Missing and Exploited Children (NCMEC).
Financial Times (via Hacker News, reprint):
Apple plans to scan US iPhones for child abuse imagery
Matthew Green (via Hacker News):
I’ve had independent confirmation from multiple people that Apple is releasing a client-side tool for CSAM scanning tomorrow. This is a really bad idea.
These tools will allow Apple to scan your iPhone photos for photos that match a specific perceptual hash, and report them to Apple servers if too many appear.
[…]
This sort of tool can be a boon for finding child pornography in people’s phones. But imagine what it could do in the hands of an authoritarian government?
[…]
The way Apple is doing this launch, they’re going to start with non-E2E photos that people have already shared with the cloud. So it doesn’t “hurt” anyone’s privacy.
It’s implied but not specifically stated that they are not scanning the contents of iCloud Backup (which is not E2E), only iCloud Photo Library.
But you have to ask why anyone would develop a system like this if scanning E2E photos wasn’t the goal.
[…]
Hashes using a new and proprietary neural hashing algorithm Apple has developed, and gotten NCMEC to agree to use.
We don’t know much about this algorithm. What if someone can make collisions?
Or what if the AI simply makes mistakes?
Chance Miller (Apple, Hacker News, MacRumors):
Apple is today announcing a trio of new efforts it’s undertaking to bring new protection for children to iPhone, iPad, and Mac. This includes new communications safety features in Messages, enhanced detection of Child Sexual Abuse Material (CSAM) content in iCloud, and updated knowledge information for Siri and Search.
[…]
If there is an on-device match, the device then creates a cryptographic safety voucher that encodes the match result. A technology called threshold secret sharing is then employed. This ensures the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content.
[…]
Apple isn’t disclosing the specific threshold it will use — that is, the number of CSAM matches required before it is able to interpret the contents of the safety vouchers. Once that threshold is reached, however, Apple will manually review the report to confirm the match, then disable the user’s account, and sent a report to the National Center for Missing and Exploited Children.
There’s a technical summary here.
Many other cloud storage services are already doing that, in a much less privacy-preserving way. In a way, it’s their responsibility given that they’re storing the data and it is illegal to possess such content in many parts of the world.
I have always been concerned that this system could be weaponized as a way gain access to someone’s account. For example:
- Add the hash of a non-pornographic image to the database
- Using a burner email address, email the non-pornographic image to the target’s Gmail address. The target wouldn’t think anything of it.
- The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
I wonder how easy it is to add a photo to someone’s iCloud Photo Library.
What they say: “This algorithm will scan your images for potential child abuse”
What it will actually do: Looks at your nudes without your consent and sends them to a team who will of course have people who save them and share them when they see its not child abuse.
That would never happen, of course. Apple would probably argue that you don’t really have to trust their team because threshold secret sharing will prevent them from needing to review the images, anyway. But who knows what threshold they’re using or how reliable the perceptual hashing actually is.
One takeaway is that, CSAM detection aside, Apple already has access to these photos. You shouldn’t upload anything to the cloud that you want to keep private. But Apple isn’t giving users much choice. It doesn’t let you choose a truly private cloud backup or photo syncing provider. If you don’t use iCloud Photo Library, you have to use Image Capture, which is buggy. And you can’t use iCloud to sync some photos but not others. Would you rather give Apple all your photos or risk losing them?
And, now that the capability is built into Apple’s products, it’s hard to believe that they won’t eventually choose to or be compelled to use it for other purposes. They no longer have the excuse that they would have to “make a new version of the iPhone operating system.” It probably doesn’t even require Apple’s cooperation to add photo hashes to the database.
Previously:
- Apple Dropped Plans for End-to-End Encrypted iCloud Backups After FBI Objected
- The Time Tim Cook Stood His Ground Against the FBI
- Facebook Solicits Nude Photos to Stop Revenge Porn
- Yahoo’s FISA E-mail Scan
- FBI Asks Apple for Secure Golden Key
- Apple Patches “Find My iPhone” Exploit
Update (2021-08-06): Nick Heer, regarding my question about adding a photo to someone else’s iCloud Photo Library:
AirDropped images are automatically added to the photo library, aren’t they?
Because Apple is scanning iCloud Photos for the CSAM flags, it makes sense that the feature does not work with iCloud Photos disabled. Apple has also confirmed that it cannot detect known CSAM images in iCloud Backups if iCloud Photos is disabled on a user’s device.
I think a fair counterargument is that Apple’s more proactive approach to child safety takes away one of law enforcement’s favourite complaints about commonplace encryption.
But it represents a similar trade-off to the aforementioned iCloud backups example. Outside of the privacy absolutist’s fictional world, all of privacy is a series of compromises. Today’s announcements raise questions about whether these are the right compromises to be making. What Apple has built here is a local surveillance system that all users are supposed to trust. We must believe that it will not interfere with our use of our devices, that it will flag the accounts of abusers and criminals, and that none of us innocent users will find ourselves falsely implicated. And we must trust it because it is something Apple will be shipping in a future iOS update, and it will not have an “off” switch.
Perhaps this is the only way to make a meaningful dent in this atrocious abuse, especially since the New York Times and the NCMEC shamed Apple for its underwhelming reporting of CSAM on its platforms. But are we prepared for the likely expansion of its capabilities as Apple and other tech companies are increasingly pressured to shoulder more responsibility for the use of their products? I do not think so. This is a laudable effort, but enough academics and experts in this field have raised red flags for me to have some early concerns and many questions.
Andrew Orr (in 2019, MacRumors):
Occasionally I like to check up on Apple’s security pages and privacy policies. I noticed something new in the privacy policy, which was last updated May 9, 2019. Under the “How we use your personal information” header, one of the paragraphs now reads (emphasis added):
We may also use your personal information for account and network security purposes, including in order to protect our services for the benefit of all our users, and pre-screening or scanning uploaded content for potentially illegal content, including child sexual exploitation material.
Apple may have even been doing this for years, but this is the first time this has appeared in its privacy policy. And I checked earlier versions using the Wayback Machine.
[…]
Speaking at CES 2020, Apple’s chief privacy officer Jane Horvath mentioned photos backed up to iCloud in terms of scanning.
[…]
A search warrant revealed that Apple scans emails for this content.
Apple’s scanning does not detect photos of child abuse. It detects a list of known banned images added to a database, which are initially child abuse imagery found circulating elsewhere. What images are added over time is arbitrary. It doesn’t know what a child is.
Apple thinks photo scanning is non-negotiable — that for legal and PR reasons, you can’t be a major consumer tech company and not scan users’ photos — so the only way to encrypt photos on-device was to develop & implement client-side scanning.
My read is that the FBI keeps harping about CSAM and “going dark”. It’s the hardest thing to defend, so now they can say “no one can use iCloud to store CSAM and I won’t build a backdoor into iCloud encryption”
They are if they are moving server-side scanning to “client-side hashing then matching on the server-side”. If this is a pre-req for encrypted iCloud data, then this is potentially a win. But, this is all negated by absence of auditability of the hash DB.
If it came out that Apple was adding anything other than CSAM fingerprints to the database, it’d be ruinous to the company’s reputation. As bad as if they were pilfering from Apple Cash accounts.
It sounds like Apple is not adding anything to the database, so it’s not in a position to make any guarantees. It’s just using an opaque list of hashes supplied by a third party.
The hash databases used by CSAM scanning methods have little oversight.
[…]
In any case, all of this requires us to place trust in automated systems using unproven machine learning magic, run by technology companies, and given little third-party oversight. I am not surprised to see people worried by even this limited scope, never mind the possibilities of its expansion.
Government: <adds images known to be from target to database>
Apple: <matches, uploads contents of target’s phone to government server for further inspection>
Government: thanku appl
Whoever controls this list can search for whatever content they want on your phone, and you don’t really have any way to know what’s on that list because it’s invisible to you (and just a bunch of opaque numbers, even if you hack into your phone to get the list.)
The theory is that you will trust Apple to only include really bad images. Say, images curated by the National Center for Missing and Exploited Children (NCMEC). You’d better trust them, because trust is all you have.
[…]
This means that, depending on how they work, it might be possible for someone to make problematic images that “match” entirely harmless images. Like political images shared by persecuted groups. These harmless images would be reported to the provider. […] And the problem is that none of this technology was designed to stop this sort of malicious behavior. In the past it was always used to scan unencrypted content. If deployed in encrypted systems (and that is the goal) then it provides an entirely new class of attacks.
[…]
Regardless of what Apple’s long term plans are, they’ve sent a very clear signal. In their (very influential) opinion, it is safe to build systems that scan users’ phones for prohibited content.
That’s the message they’re sending to governments, competing services, China, you.
EFF (tweet, Hacker News, MacRumors):
All it would take to widen the narrow backdoor that Apple is building is an expansion of the machine learning parameters to look for additional types of content, or a tweak of the configuration flags to scan, not just children’s, but anyone’s accounts. That’s not a slippery slope; that’s a fully built system just waiting for external pressure to make the slightest change.
[…]
Apple and its proponents may argue that scanning before or after a message is encrypted or decrypted keeps the “end-to-end” promise intact, but that would be semantic maneuvering to cover up a tectonic shift in the company’s stance toward strong encryption.
But knowing this uses a neural net raises all kinds of concerns about adversarial ML, concerns that will need to be evaluated.
Apple should commit to publishing its algorithms so that researchers can try to develop “adversarial” images that trigger the matching function, and see how resilient the tech is.
I am vehemently opposed to scanning of personal information, be it in the cloud (under end-to-end encryption), or on our local devices. The long term risk for misuse of such technology far outweighs any short term benefit.
[…]
There are world governments of all kinds, and they all have questionable policies of varying degrees. As soon they tell a corporation implement their dubious dragnet or suffer the consequences, the corporation will promptly give them access to your photos, emails, any other data.
The reason Apple’s approach is going far too far comes down to one thing: the difference between law enforcement, where an agency needs good reason to access private data, and surveillance. Apple’s approach is surveillance. (And from the company that made the 1984 ad.)
A narrowly defined backdoor is still a backdoor. “Partial” digital privacy isn’t a thing -- you either have it or you don’t.
If you think you can design a system that violates privacy only for some people, you can’t. I don’t care who you are.
Apple has won enormous amounts of goodwill by declaring that privacy is a human right, and is about to destroy all of it at once by building a technology to have your phone scan your pictures and turn you over to law enforcement if they’re the wrong sort of pictures.
It doesn’t matter what sort of pictures motivated this feature; eventually governments will force its use for all sorts of things, and many governments do not respect human rights. I’m completely aghast that this is being contemplated.
Here’s the thing about “slippery slope” arguments: a slope is rarely slippery, but it still goes downhill.
It took 12 years to go from “your Mac app needs to be code signed for the keychain and firewall” to “you need to upload every build of your Mac app to Apple for approval”.
It is difficult for me to reconcile the Apple that makes ostensibly clever machine learning stuff that can match child abuse imagery, even after it has been manipulated, with the Apple that makes software that will fail to sync my iPhone for twenty minutes before I give up.
Same with the iMessage scanning feature and iMessage itself.
Now that Apple has willingly built spyware into iOS and macOS, within 10 years this tech will:
(1) be mandated by government in all end-to-end encrypted apps; and
(2) expand to scan for terrorism, disinformation, "misinformation", then eventually political images and memes.
This is not a drill.
Police are already misusing location data gathered for COVID contact tracing even though everyone SWORE it wouldn’t be used for anything by health purposes.
Clearly a rubicon moment for privacy and end-to-end encryption.
I worry if Apple faces anything other than existential annihilation for proposing continual surveillance of private messages then it won’t be long before other providers feel the pressure to do the same.
[…]
If Apple are successful in introducing this, how long do you think it will be before the same is expected of other providers? Before walled-garden prohibit apps that don’t do it? Before it is enshrined in law?
Really seems like Apple tried to protect customer data in the cloud by scanning for illegal material locally on the phone, thereby creating a new kind of risk for customer data on the phone.
To address these concerns, Apple provided additional commentary about its plans today.
Apple’s known CSAM detection system will be limited to the United States at launch, and to address the potential for some governments to try to abuse the system, Apple confirmed to MacRumors that the company will consider any potential global expansion of the system on a country-by-country basis after conducting a legal evaluation.
[…]
Even if the threshold is exceeded, Apple said its manual review process would serve as an additional barrier and confirm the absence of known CSAM imagery. Apple said it would ultimately not report the flagged user to NCMEC or law enforcement agencies and that the system would still be working exactly as designed.
I wonder how much manual review Apple is planning to do, given that it says there’s only a 1 in 1 trillion probability of incorrectly flagging an account.
In an internal memo distributed to the teams that worked on this project and obtained by 9to5Mac, Apple acknowledges the “misunderstandings” around the new features, but doubles down on its belief that these features are part of an “important mission” for keeping children safe.
It’s hard not to feel that a bait and switch is being presented. Apple announced that disabling iCloud Photos bypasses CSAM detection. This practically ensures failure, as anyone involved in child exploitation will of course disable iCloud Phots. So then what? Set up to fail...
So we already have the on-device detection, and limiting it to iCloud Photos will fail. This means that further measures will be required, i.e., scanning regardless of whether iCloud Photos is enabled.
Seems like Apple’s idea of doing iCloud abuse detection with this partially-on-device check only makes sense in two scenarios: 1) Apple is going to expand it to non-iCloud data stored on your devices or 2) Apple is going to finally E2E encrypt iCloud?
But if it is to enable end-to-end iCloud encryption and it is not applied to purely local files, that seems like an overall privacy benefit.
If we follow that line of speculation further, it makes me wonder why Apple would create so much confusion in its communication of this change. Why drop this news at the beginning of August, disconnected from any other product or service launch? Why not announce it and end-to-end iCloud encryption at the same time, perhaps later this year?
Update (2021-08-09): John Gruber:
The database will be part of iOS 15, and is a database of fingerprints, not images. Apple does not have the images in NCMEC’s library of known CSAM, and in fact cannot — NCMEC is the only organization in the U.S. that is legally permitted to possess these photos.
[…]
All of these features are fairly grouped together under a “child safety” umbrella, but I can’t help but wonder if it was a mistake to announce them together. Many people are clearly conflating them, including those reporting on the initiative for the news media.
[…]
In short, if these features work as described and only as described, there’s almost no cause for concern. […] But the “if” in “if these features work as described and only as described” is the rub. That “if” is the whole ballgame. If you discard alarmism from critics of this initiative who clearly do not understand how the features work, you’re still left with completely legitimate concerns from trustworthy experts about how the features could be abused or misused in the future.
Glenn Fleishman and Rich Mogull:
The problem is that exploitation of children is a highly asymmetric problem in two different ways. First, a relatively small number of people in the world engage in a fairly massive amount of CSAM trading and direct online predation.
[…]
The other form of asymmetry is adult recognition of the problem. Most adults are aware that exploitation happens—both through distribution of images and direct contact—but few have personal experience or exposure themselves or through their children or family. That leads some to view the situation somewhat abstractly and academically. On the other end, those who are closer to the problem—personally or professionally—may see it as a horror that must be stamped out, no matter the means. Where any person comes down on how far tech companies can and should go to prevent exploitation of children likely depends on where they are on that spectrum.
[…]
(Spare some sympathy for the poor sods who perform the “manual” job of looking over potential CSAM. It’s horrible work, and many companies outsource the work to contractors, who have few protections and may develop PTSD, among other problems. We hope Apple will do better. Setting a high threshold, as Apple says it’s doing, should dramatically reduce the need for human review of false positives.)
[…]
Apple’s head of privacy, Erik Neuenschwander, told the New York Times, “If you’re storing a collection of C.S.A.M. material, yes, this is bad for you. But for the rest of you, this is no different.”
Given that only a very small number of people engage in downloading or sending CSAM (and only the really stupid ones would use a cloud-based service; most use peer-to-peer networks), this is a specious remark, akin to saying, “If you’re not guilty of possessing stolen goods, you should welcome an Apple camera in your home that lets us prove you own everything.” Weighing privacy and civil rights against protecting children from further exploitation is a balancing act. All-or-nothing statements like Neuenschwander’s are designed to overcome objections instead of acknowledging their legitimacy.
What happens when China announces its version of the NCMEC, which not only includes the horrific imagery Apple’s system is meant to capture, but also images and memes the government deems illegal?
The fundamental issue — and the first reason why I think Apple made a mistake here — is that there is a meaningful difference between capability and policy. One of the most powerful arguments in Apple’s favor in the 2016 San Bernardino case is that the company didn’t even have the means to break into the iPhone in question, and that to build the capability would open the company up to a multitude of requests that were far less pressing in nature, and weaken the company’s ability to stand up to foreign governments. In this case, though, Apple is building the capability, and the only thing holding the company back is policy.
[…]
Apple is compromising the phone that you and I own-and-operate, without any of us having a say in the matter. Yes, you can turn off iCloud Photos to disable Apple’s scanning, but that is a policy decision; the capability to reach into a user’s phone now exists, and there is nothing an iPhone user can do to get rid of it.
@Apple now circulating a propaganda letter describing the internet-wide opposition to their decision to start checking the private files on every iPhone against a secret government blacklist as “the screeching voices of the minority.”
The NCMEC database […] contains countless non-CSAM pictures that are entirely legal not only in the U.S. but globally. […] Increasing the scope of scanning is barely a slippery slope, they’re already beyond the stated scope of the database.
This is where the human reviewers come in. In theory, it doesn’t matter if the database contains non-CSAM pictures—either because they were collected along with CSAM ones or because a government deliberately added them to the database—because the reviewers will see that the user did not actually have CSAM and so will decline to make a report. However, this assumes (1) a quality of review that Apple has not previously demonstrated, and (2) that Apple will not be pressured or tricked into hiring reviewers that are working towards another purpose.
What would you say if Apple announced that Siri will always listen and report private conversations (not just those triggered by “Hey Siri”) but only if a really good neural network recognizes them as criminal, and there’s PSI to protect you?
RE: Apple’s plan to scan every photo in iMessage with machine learning and alert parents to nudity. […] Let me share so you can imagine how it will be misused.
Steve Troughton-Smith (also Paul Haddad):
I feel like Apple could easily have built these new features to outright prevent explicit/illegal material from being viewed or saved on its platforms, while sidestepping the slippery slope outcry entirely. […] I mean why are they letting this stuff onto iCloud Photos in the first place?
Perhaps the thinking is that the matching needs to remain hidden so that people can’t learn how to evade it.
We’re past the point where giving Apple the benefit of the doubt can be interpreted as anything other than willful ignorance from a place of Western privilege. These aren’t hypotheticals, we already have examples of Apple’s policies failing people in other countries.
So end-to-end encryption means nothing?
Device maker can log/view/save your content right before it gets sent (encrypted) or right after it’s received (unencrypted), but your content was still E2E encrypted!
In my opinion, there are no easy answers here. I find myself constantly torn between wanting everybody to have access to cryptographic privacy and the reality of the scale and depth of harm that has been enabled by modern comms technologies.
[…]
I have friends at both the EFF and NCMEC, and I am disappointed with both NGOs at the moment. Their public/leaked statements leave very little room for conversation, and Apple’s public move has pushed them to advocate for their equities to the extreme.
[…]
Likewise, the leaked message from NCMEC to Apple’s employees calling legitimate questions about the privacy impacts of this move “the screeching voices of the minority” was both harmful and unfair.
[…]
One of the basic problems with Apple’s approach is that they seem desperate to avoid building a real trust and safety function for their communications products. There is no mechanism to report spam, death threats, hate speech, NCII, or any other kinds of abuse on iMessage.
As a result, their options for preventing abuse are limited.
Say you’re a big Apple fan who is really upset with the photo scanning announcement. In order to send a market signal by switching phones, you would also have to buy a new watch, give up AirDrop / iMessage with your friends, not watch Ted Lasso on your new phone, etc etc etc
At some point ecosystem lock-in creates to many different switching costs that the market can no longer send meaningful signals about what’s important, leaving only public opinion and government regulation to shape a company’s behavior. That feels real icky to me!
Apple’s dark patterns that turn iCloud uploads on by default, and flip it back on when moving to a new phone or switching accounts, exacerbate the problem.
More specifically, the concern involves where this type of technology could lead if Apple is compelled by authorities to expand detection to other data that a government may find objectionable. And I’m not talking about data that is morally wrong and reprehensible. What if Apple were ordered by a government to start scanning for the hashes of protest memes stored on a user’s phone? Here in the U.S., that’s unlikely to happen. But what if Apple had no choice but to comply with some dystopian law in China or Russia? Even in Western democracies, many governments are increasingly exploring legal means to weaken privacy and privacy-preserving features such as end-to-end encryption, including the possibility of passing legislation to create backdoor access into messaging and other apps that officials can use to bypass end-to-end encryption.
So these worries people are expressing today on Twitter and in tech forums around the web are understandable. They are valid. The goal may be noble and the ends just—for now—but that slope can also get slippery really fast.
While child exploitation is a serious problem, and while efforts to combat it are almost unquestionably well-intentioned, Apple’s proposal introduces a backdoor that threatens to undermine fundamental privacy protections for all users of Apple products.
[…]
Apple’s current path threatens to undermine decades of work by technologists, academics and policy advocates towards strong privacy-preserving measures being the norm across a majority of consumer electronic devices and use cases. We ask that Apple reconsider its technology rollout, lest it undo that important work.
Most of the heat RE: neuralMatch is rooted in ignorance of what it does. I’m not here to educate.
But there’s a valid worry that hostile governments could use it to rat out their citizens for non-CSAM offenses.
Some concrete actions Apple could take to fix that[…]
[…]
Guarantee the database is global, not a localized resource.
[…]
Publish neuralMatch as an all-purpose image matching API, so third parties can audit it on a technical level.
[…]
Allow third parties to test the neuralMatch API specifically against the CSAM hashes, so they can audit it for the kinds of politically-motivated matches people are worried about.
Looks like the NeuralHash is included in the current beta in the Vision framework.
Oliver Kuederle (via Hacker News):
At my company, we use “perceptual hashes” to find copies of an image where each copy has been slightly altered. This is in the context of stock photography, where each stock agency (e.g. Getty Images, Adobe Stock, Shutterstock) adds their own watermark, the image file ID, or sharpens the image or alters the the colours slightly, for example by adding contrast.
[…]
It shouldn’t come as a surprise that these algorithms will fail sometimes. But in the context of 100 million photos, they do fail quite often. And they don’t fail in acceptable ways[…]
The laws related to CSAM are very explicit. 18 U.S. Code § 2252 states that knowingly transferring CSAM material is a felony. (The only exception, in 2258A, is when it is reported to NCMEC.) In this case, Apple has a very strong reason to believe they are transferring CSAM material, and they are sending it to Apple -- not NCMEC.
It does not matter that Apple will then check it and forward it to NCMEC. 18 U.S.C. § 2258A is specific: the data can only be sent to NCMEC. (With 2258A, it is illegal for a service provider to turn over CP photos to the police or the FBI; you can only send it to NCMEC. Then NCMEC will contact the police or FBI.) What Apple has detailed is the intentional distribution (to Apple), collection (at Apple), and access (viewing at Apple) of material that they strongly have reason to believe is CSAM. As it was explained to me by my attorney, that is a felony.
The problem with any take on the Apple/CSAM stuff is that there are so many horrible people in the world that do horrible things to people, and so many governments that do horrible things to people, and any pretty much any tech that thwarts one of them enables the other one.
There’s an argument, with support from Game Theory, that says that Apple can set a high threshold for the number of matches, and only detect and report a few cases of CSAM. Indeed, even that may be unnecessary to drive anyone currently sharing CSAM to abandon the use of iCloud Photos altogether.
That would be a win for Apple but not really help solve the problem as a whole.
The worst case scenario for the initial implementation isn’t necessarily false positives, though those would certainly be awful.
Worst case scenario is child abusers don’t use iCloud Photos, and Apple’s NCMEC report #s don’t increase much.
CyberTipline is the nation’s centralized reporting system for the online exploitation of children, including child sexual abuse material, child sex trafficking and online enticement. In 2020, the CyberTipline received more than 21.7 million reports.
Only 265 were from Apple. I’m not sure how to square this with Apple’s chief privacy officer stating in January 2020 that it was already scanning photos server-side. Are the criminals already avoiding iCloud, or is Apple’s matching not very effective?
Stefano Quintarelli (via Hacker News):
The point I try to make is that it will do little to protect children (while weakening users’ privacy and pushing criminals to hide better) but it will be used as an excuse to justify a tight control of the devices in order to perpetuate their apparent monopolistic power through the app store in a time when such behavior is under the fire of competition authorities.
The whole point of end-to-end encryption is to prevent the provider of the service to itself be coerced into giving off information about its users. Apple is building exactly the opposite of that.
Will you even know when the system is abused? The US government already forced companies into coercion while preventing them from telling their users that this is happening.
This is about an infrastructure which can be put to use for any and all of your data. It doesn’t matter what Apple claims it is limited to doing now. What matters is that this is a general purpose capability.
[…]
And what is incredibly stupid about this approach is that only technology-ignorant child-abusers will fail to turn off iCloud photo syncing, which at the moment is what the Apple system counts on. Everyone else gets spied on.
Aral Balkan (via Hacker News):
If Apple goes ahead with its plans to have your devices violate your trust and work against your interests, I will not write another line of code for their platforms ever again.
[…]
When I wrote The Universal Declaration of Cyborg Rights, I wanted to get people thinking about the kind of constitutional protections we would need to protect personhood in the digital network age.
This document serves to address these questions and provide more clarity and transparency in the process.
Apple’s FAQ is really disingenuous.
Why is Apple doing this now?
One of the significant challenges in this space is protecting children while also preserving the privacy of users. With this new technology, Apple will learn about known CSAM photos being stored in iCloud Photos where the account is storing a collection of known CSAM. Apple will not learn anything about other data stored solely on device.
Existing techniques as implemented by other companies scan all user photos stored in the cloud. This creates privacy risk for all users. CSAM detection in iCloud Photos provides significant privacy benefits over those techniques by preventing Apple from learning about photos unless they both match to known CSAM images and are included in an iCloud Photos account that includes a collection of known CSAM.
This answer makes no sense in light of the facts that Apple was already doing server-side scanning and that the photos to now be scanned on device are ones that Apple would have access to via the cloud, anyway. [Update (2021-08-10): See the update below.]
Can the CSAM detection system in iCloud Photos be used to detect things other than CSAM?
Our process is designed to prevent that from happening.
The answer is clearly “yes,” because it relies on hashes, which Apple has not vetted; and depends on human review, which may not work as intended.
Could governments force Apple to add non-CSAM images to the hash list?
Apple will refuse any such demands.
This is not the right question. We don’t really care whether Apple is the one adding the hashes, but simply whether they can be added. And the answer to that is clearly “yes.” There are already non-CSAM hashes in the NCMEC database. Apple has no ability to “refuse” because it never even sees the images. It trusts the hashes that it’s been given by the government.
Let us be clear, this technology is limited to detecting CSAM stored in iCloud and we will not accede to any government’s request to expand it.
Apple has already compromised user privacy in response to Chinese law. If, say, US law compelled them to scan non-iCloud photos, what choice would they have but to accede? Would they stop selling iPhones? Have every single engineer resign? I don’t see how this is a promise any company could keep, even if it wanted to.
Yes, I fully believe that Apple will refuse when asked, and I don’t question their motives for why this feature should exist. The problem is that I don’t believe it’s remotely enough. Some states do not have a record of taking no for an answer, and when recent history shows impactful decisions, going against those same values and morals, that are the result of either successful pressure or regulatory capture, the situation recalls the words of a quite different Marx: “Those are my principles, and if you don’t like them… well, I have others.”
Apple isn’t “throwing a bone” to law enforcement. Apple is giving them an appetizer. When the biggest computer vendor in the US says it’s ok to put spyware on their own devices, this gives the green light to all legislators and agencies to start demanding everything they want.
Apple said that while it does not have anything to share today in terms of an announcement, expanding the child safety features to third parties so that users are even more broadly protected would be a desirable goal. Apple did not provide any specific examples, but one possibility could be the Communication Safety feature being made available to apps like Snapchat, Instagram, or WhatsApp so that sexually explicit photos received by a child are blurred.
Another possibility is that Apple’s known CSAM detection system could be expanded to third-party apps that upload photos elsewhere than iCloud Photos.
Update (2021-08-10): John Gruber and Rene Ritchie say that, actually, Apple’s servers have never scanned iCloud photo libraries for CSAM, only photos attached to certain messages stored on iCloud’s mail servers. Many sources reported Apple’s chief privacy officer saying at CES 2020 that photos uploaded to iCloud were scanned. However, some of these seem to be based on an article that has since been updated:
This story originally said Apple screens photos when they are uploaded to iCloud, Apple’s cloud storage service. Ms Horvath and Apple’s disclaimer did not mention iCloud, and the company has not specified how it screens material, saying this information could help criminals.
I have not found any official Apple statements saying what was scanned before.
In any case, this changes how I interpret Apple’s FAQ, as well as speculation for the future. If photo library scanning is new, Apple is not reimplementing a previously working system in a way that is potentially less private (since it could be easily tweaked to scan non-cloud photos). It also seems less likely to imply a switch to making iCloud Photos E2EE. It could simply be that Apple wanted to implement the fingerprinting in a way that took advantage of distributed CPU power. Or that it wanted to avoid having a server scanner that it could be compelled to use. This also explains why Apple only made 265 reports in 2020.
Apple’s Chief Privacy Officer seemed to say CSAM scanning of iCloud servers was already happening back in January 2020 and Apple’s Privacy Policy has allowed it since May 2019. However, it is now unclear whether iCloud server CSAM scanning has actually been happening.
Apple now seems to be telling media that server-based CSAM scanning will start when on-device scanning starts.
Or maybe it’s all done on-device when the old photos sync down from the cloud?
John Gruber (tweet):
I do wonder though, how prepared Apple is for manually reviewing a potentially staggering number of accounts being correctly flagged. Because Apple doesn’t examine the contents of iCloud Photo Library (or local on-device libraries), I don’t think anyone knows how prevalent CSAM is on iCloud Photos.
[…]
If the number is large, it seems like one innocent needle in a veritable haystack of actual CSAM collections might be harder for Apple’s human reviewers to notice.
Notice Apple changing the definition of “end-to-end encryption.” No longer is the message a private communication between sender and receiver.
Perhaps feeling left out by the constant communication own-goals by Facebook, Apple set up the mother of all self-owns. It’s hard to think of a more massive communication fuck up, honestly. Again, because this topic is so big, so important, and so sensitive. Apple probably should have had an event, or at the very least a large-scale pre-brief with journalists and bloggers to talk through these issues.
[…]
Second, this is all more than a little ironic given the whole “backdoor” debate Apple forcefully stood up against when government agencies sought to force Apple to build in a way to get into iPhones. Tim Cook was adament that Apple had no way to do this, and should not build it. If they didn’t exactly just create a way, they created a huge loophole that officials are going to test like velociraptors against an electric fence. Until they find the weakness… That’s what Apple set up here. The thing they stood up against! Apple can say all the right things. They also have to abide by laws. And laws are man-made things. Which change.
Apple commit to challenging requests to expand their CSAM detection to other material. So did UK ISPs, but they lost in court and did it anyway. Will Apple leave a market if put in the same position?
How would Apple not be able to add things to the hash list/ change which list they use? NMEC would need to publish some root hash of their list and Apple would have to bind it into their client software in a way even they couldn’t change. Thats a tall order.
It is also deeply disappointing to see so many tech journalists make inferences for Apple when all of the pressure should be on Apple to answer the questions directly and on the record, instead of collecting concerns on background
Matthew Panzarino (tweet, TechCrunch, MacRumors):
I spoke to Erik Neuenschwander, head of Privacy at Apple, about the new features launching for its devices.
[…]
The voucher generation is actually exactly what enables us not to have to begin processing all users’ content on our servers, which we’ve never done for iCloud Photos.
[…]
Well first, that is launching only for U.S., iCloud accounts, and so the hypotheticals seem to bring up generic countries or other countries that aren’t the U.S. when they speak in that way, and the therefore it seems to be the case that people agree U.S. law doesn’t offer these kinds of capabilities to our government.
But even in the case where we’re talking about some attempt to change the system, it has a number of protections built in that make it not very useful for trying to identify individuals holding specifically objectionable images. The hash list is built into the operating system, we have one global operating system and don’t have the ability to target updates to individual users and so hash lists will be shared by all users when the system is enabled.
He does not address Apple’s lack of ability to audit the hashes that it receives.
Update (2021-08-13): Nick Heer:
This note was appended one day after the Telegraph published its original report — that is, one day after it was cited by numerous other outlets. Unfortunately, none of those reports reflected the Telegraph’s correction and, because the Telegraph has a soft paywall and the title of the article remained “Apple scans photos to check for child abuse”, it is not obvious that there were any material changes to correct. Robinson’s Law strikes again.
Matthew Green (also: Edward Snowden):
People are telling me that Apple are “shocked” that they’re getting so much pushback from this proposal. They thought they could dump it last Friday and everyone would have accepted it by the end of the weekend.
Apple spent years educating the public on privacy for use as a marketing pitch and is now shocked that people care about privacy.
In a sense, it’s already too late. Apple hasn’t shipped the spyware yet, but Apple has already told the governments of the world that they will ship spyware in the operating system.
This is in stark contrast to what Apple said in the San Bernardino case.
Jokes aside, though, as engineers we regularly deal with complex systems that can be difficult for our users to understand. Having a hard time explaining how they work is one thing, but regardless of your position on this technology @Apple’s messaging has been unacceptable.
Their reluctance to clearly describe how the software works, their seeming inability to be straightforwards with the fact that it fundamentally detects CSAM using filters that they control and uploads it to them, is very concerning. This isn’t how you inspire trust.
“Encrypted” and “on device” and “hashed” are not magic words that magically grant privacy. You can’t say “nothing is learned about the content on the device” if you can take the vouchers it sends you and decrypt them–even if you are “sure” they are CSAM. That’s just incorrect.
Being better “compared to the industry standard way” does not mean the technology is automatically “private”. And when you say you’re better than the industry standard from the perspective of being auditable, don’t be in a place where you can’t verify you are doing any better.
You may be wondering why Apple includes this manual step of reviewing images before they are reported; the answer is U.S. v Ackerman. In this case, it was found that NCMEC is effectively a government actor due to the power that Congress has granted them. As a result, if NCMEC reviews a file, it is considered a 4th Amendment search; however, if Apple views the file and informs NCMEC of the content (conducting a private search that isn’t covered by the 4th Amendment), then NCMEC is free to view the file to confirm the accuracy of the report.
By manually reviewing the content prior to reporting, the search isn’t considered to be a violation of constitutional rights in the U.S., and thus can be used as evidence in court.
[…]
Based on how the system is designed, there doesn’t appear to be any need for the full image to be uploaded, only the Safety Voucher. Based on this design choice, it’s logical to conclude that the intention is to move beyond just iCloud into other areas.
[…]
Scanning images uploaded to iCloud for known CSAM is unlikely to have a notable impact. In a memo (discussed further below) to Apple employees from Marita Rodriguez, the Executive Director of Strategic Partnerships at NCMEC said, “…I hope you take solace in knowing that because of you many thousands of sexually exploited victimized children will be rescued…” - which sounds great, but is entirely unrealistic. This scanning system only looks for known CSAM that has been reported and added to the hash database; this system targets those collecting and trading CSAM. It’s not targeted to those producing new CSAM. While putting the criminals that traffic in this awful material in prison is a laudable goal, the impact is unlikely to resemble the goals NCMEC has expressed.
[…]
The fact that NCMEC hasn’t issued an apology and clarification is telling; they are doing little to work with privacy advocates to find solutions that meet these complex challenges, and instead attack and demean.
One can not reconcile these two things: 1.) Apple rolling out an automated, warrantless, opt-out surveillance tool to all US iCloud customers — and 2.) iPhone owners around the world having arbitrary data pushed to their devices by powerful nation-state adversaries who want them ruined.
The Pegasus story does not have a bookend. As it stands, it is very reasonable to assume that a hacker could push arbitrary data to your phone, including pictures. We have proof (and acknowledgement from Apple) that this is still happening. Because of the broken security of Apple devices, it is irresponsible to be rolling out an automated surveillance system, and frankly – exceedingly arrogant.
[…]
Apple’s CEO Tim Cook said at a Fortune event in 2017, when asked about its compliance with China’s censorship and problematic laws: “Each country in the world decides their laws and their regulations. And so your choice is: Do you participate, or do you stand on the sideline and yell at how things should be? You get in the arena, because nothing ever changes from the sideline.” Apple has been “in the arena” for well over a decade now, time for a scorecard.
But just because Apple has done its due diligence and made some careful choices in order to implement a tool to stop the spread of heinous material doesn’t mean that it’s off the hook. By making our phones run an algorithm that isn’t meant to serve us, but surveils us, it has crossed a line. Perhaps it was inevitable that the line would be crossed. Perhaps it’s inevitable that technology is leading us to a world where everything we say, do and see is being scanned by a machine-learning algorithm that will be as benevolent or malevolent as the society that implemented it.
Even if Apple’s heart is in the right place, my confidence that its philosophy will be able to withstand the future desires of law enforcement agencies and authoritarian governments is not as high as I want it to be. We can all be against CSAM and admire the clever way Apple has tried to balance these two conflicting needs, while still being worried about what it means for the future.
EFF (via Hacker News):
For example, the Five Eyes—an alliance of the intelligence services of Canada, New Zealand, Australia, the United Kingdom, and the United States—warned in 2018 that they will “pursue technological, enforcement, legislative or other measures to achieve lawful access solutions” if the companies didn’t voluntarily provide access to encrypted messages. More recently, the Five Eyes have pivoted from terrorism to the prevention of CSAM as the justification, but the demand for unencrypted access remains the same, and the Five Eyes are unlikely to be satisfied without changes to assist terrorism and criminal investigations too.
[…]
All it would take to widen the narrow backdoor that Apple is building is an expansion of the machine learning parameters to look for additional types of content, the adoption of the iPhoto hash matching to iMessage, or a tweak of the configuration flags to scan, not just children’s, but anyone’s accounts. Apple has a fully built system just waiting for external pressure to make the necessary changes.
You wouldn’t think a US company could be forced to scan all of it’s customers data, but Yahoo was. Don’t make the same mistake Apple.
Been there, didn’t do that, got the t-shirt.
Here’s an op-ed @alexstamos and I co-authored about the risks of Apple’s content scanning plan. It’s short and easy to read, and I’m hoping it makes the issues digestible to non-technical people.
[…]
My personal proposal to Apple is to limit this tech to photo sharing rather than whole libraries, and release their hash function design. And ideally wait until researchers have time to vet it before launching to 1bn users.
There’s a crucial difference between possessing photos and sharing photos. The former is expected to be private, the latter not. This is why iCloud and Facebook are not comparable.
This issue is nuanced and Apple’s decisions involve concessions. Personally, I think Apple have done well here. They probably could have handled the communication surrounding the announcement better, but the actual functionality and policy decisions are reasonable.
[…]
You have to assume that privacy issues are a key reason why Apple has historically been so lax in this department. It’s not that Apple has sympathy for the people spreading child pornography. Why right now? That is still unclear. Perhaps, behind closed doors, someone was threatening lawsuits or similar action if Apple didn’t step up to par soon. Either way, it’s crunch time.
[…]
The weakest link in the chain on the technical side of this infrastructure is the opaqueness of the hashed content database. By design, Apple doesn’t know what the hashes represent as Apple is not allowed to knowingly traffic illicit child abuse material. Effectively, the system works on third-party trust. Apple has to trust that the database provided by NCMEC — or whatever partner Apple works with in the future when this feature rolls out internationally — does only include hashes of known CSAM content.
All the conversations the community has been having are mirrored inside Apple; I think it’s an understandable worry that Apple is prepared to sell out all of its users despite knowing — and informing them — predators can avoid the system by turning off iCloud Photos. No wins here
Joseph Menn and Julia Love (Hacker News, MacRumors):
A backlash over Apple’s move to scan U.S. customer phones and computers for child sex abuse images has grown to include employees speaking out internally, a notable turn in a company famed for its secretive culture, as well as provoking intensified protests from leading technology policy groups.
Apple’s senior vice president of software engineering, Craig Federighi, has today defended the company’s controversial planned child safety features in a significant interview with The Wall Street Journal, revealing a number of new details about the safeguards built into Apple’s system for scanning users’ photos libraries for Child Sexual Abuse Material (CSAM).
I see the Apple PR line on photo scanning is that you don’t understand what’s going on. Your tiny brain cannot comprehend the splendor of this technology.
Apple Inc. has warned retail and online sales staff to be ready to field questions from consumers about the company’s upcoming features for limiting the spread of child pornography.
In a memo to employees this week, the company asked staff to review a frequently asked questions document about the new safeguards, which are meant to detect sexually explicit images of children. The tech giant also said it will address privacy concerns by having an independent auditor review the system.
Apple today shared a document that provides a more detailed overview of the child safety features that it first announced last week, including design principles, security and privacy requirements, and threat model considerations.
[…]
The document aims to address these concerns and reiterates some details that surfaced earlier in an interview with Apple’s software engineering chief Craig Federighi, including that Apple expects to set an initial match threshold of 30 known CSAM images before an iCloud account is flagged for manual review by the company.
[…]
Apple also said that the on-device database of known CSAM images contains only entries that were independently submitted by two or more child safety organizations operating in separate sovereign jurisdictions and not under the control of the same government.
[…]
Apple added that it will publish a support document on its website containing a root hash of the encrypted CSAM hash database included with each version of every Apple operating system that supports the feature.
This bit about multiple organizations is interesting, but it raises additional questions. Apple previously said that the feature will start out as US-only. So they’re only going to report images to NCMEC and only images that are in the intersection of NCMEC’s database and some other foreign database? That would seem to drastically reduce the chances of finding legitimate matches, unless the organizations are all working together to exchange data, which of course raises more questions. And, if you’re in the US, does that mean Apple could be reporting images to NCMEC that are not even in the US database, but rather in two separate foreign ones?
See also:
Previously:
Update (2021-08-18): Joseph Menn and Stephen Nellis:
Apple Inc said on Friday that it will hunt only for pictures that have been flagged by clearinghouses in multiple countries.
That shift and others intended to reassure privacy advocates were detailed to reporters in an unprecedented fourth background briefing since the initial announcement eight days prior of a plan to monitor customer devices.
I’m glad that Apple is feeling the heat and changing their policy. But this illustrates something important: in building this system, the only limiting principle is how much heat Apple can tolerate before it changes its policies.
I’m not sure this is actually a shift, as it was hinted at in the original documents Apple released.
Wait, so Apple wanted to ensure that the Child Sexual Abuse Material (CSAM) tech is understood to be totally separate from the iMessage photo scanning feature, and yet they’re calling it “Communication, Safety, And Messages”? 🥴👏
This whole thing would not be happening if Katie Cotton were still in charge of corporate communications.
I’m not even kidding here: Apple screwed up the messaging on this so completely that I wonder if a certain key person or two is on an extended vacation or personal leave and wasn’t around to oversee this.
It is also striking how difficult it is for even a media-trained executive to clearly articulate these features. In Stern’s interview, there are several moments when she has to pause the interview to explain, in layperson terms, what is happening with the assistance of some effective graphics. I appreciate Stern’s clarifications and I understand them to be accurate, but I wish those words came from Apple’s own representative without needing interpretation. I think Apple’s representatives are still using too much jargon.
Issues with the scope of things that CAN be done with some power cannot be resolved by voluntary choices made by the holder of that power. So long as they hold that power, they can revise their choices at any time, and they can be compelled to do things at any time.
Eva:
I’d like to take this moment to make it clear to poor Craig that no, I don’t misunderstand Apple’s plans to check photos in iCloud against NCMEC’s database of CSAM. It’s well-meaning but it’s also creating a mechanism that Apple will be forced to use for other things.
The company who makes my CPU, RAM, and hard drive don’t have any right or privilege to see my information, nor does the company who provides the locks on the door of my home. A smartphone is no different. This is not a radical position.
While I agree that this is a major privacy issue and that alone should be sufficient to call for a halt to this, I am surprised I’m not hearing more property & property rights arguments: this is Apple assigning work to user owned devices for jobs which do not benefit the user.
Apple truly screwed this up in a way that is almost beyond comprehension. All their effort on establishing an image of respecting privacy out the window.
I seriously hope they reconsider the entire thing, but knowing Apple there’s no chance at all.
Eva:
In their new FAQ, Apple says they will refuse govt requests to use their CSAM-scanning tech to scan for other forms of content. How exactly will they refuse? Will they fight it in court? Will they pull out of the country entirely? This is not a time to get vague.
Central to its case is for us to trust Apple not to use this same mechanism for other purposes. When we can’t even trust Apple to tell us what it has changed on our own Macs, we should be rightly suspicious. If it is to work at all, trust must work both ways: if Apple wants our trust, it has to trust us with the knowledge of what’s in a macOS update.
All I want to do here is convey what I think is a strong case against co-opting personal devices for law enforcement purposes, so that people who have done nothing wrong and don’t have anything to hide can see where we’re coming from when as a tech community we push back on these things.
[…]
Apple will certainly comply rather than withdraw from the markets, as they have done so far in China. It is likely that no more powerful tool for surveillance authoritarianism has ever been conceived by humans.
Member of the German parliament, Manuel Höferlin, who serves as the chairman of the Digital Agenda committee in Germany, has penned a letter to Apple CEO Tim Cook, pleading Apple to abandon its plan to scan iPhone users' photo libraries for CSAM (child sexual abuse material) images later this year.
Sign the petition and email Apple leadership to tell them to drop these plans and recommit to never opening any sort of backdoor to monitor our communications.
Update (2021-08-21): Malcolm Owen (via Kosta Eleftheriou, MacRumors):
“It’s the reality. If you put back doors in a system, anybody can use a back door. And so you have to make sure the system itself is robust and durable; otherwise you can see what happens in the security world,” said Cook.
Update (2021-09-08): Ben Lovejoy (Hacker News):
Apple confirmed to me that it has been scanning outgoing and incoming iCloud Mail for CSAM attachments since 2019. Email is not encrypted, so scanning attachments as mail passes through Apple servers would be a trivial task.
Apple also indicated that it was doing some limited scanning of other data, but would not tell me what that was, except to suggest that it was on a tiny scale. It did tell me that the “other data” does not include iCloud backups.
Christina Warren returns to the show to discuss Apple’s controversial child safety initiatives, the tumultuous summer of Safari 15 beta UI designs, and a bit more on MagSafe battery packs.
Gordon Kelly (via Hacker News):
iPhone users have put up with a lot in recent months but the company’s new CSAM detection system has proved to be a lightning rod of controversy that stands out from all the rest. And if you were thinking of quitting your iPhone over it, a shocking new report might just push you over the edge.
John Koetsier (via Hacker News):
Apple fraud executive Eric Friedman told colleague Herve Sibert that Apple is the greatest platform for distributing child pornography. The comment sheds light on why Apple is now pursing a controversial program and automating checks for child porn on customers’ phones and in their messages.
In preceding messages, Friedman writes about a presentation the two managers have been working on to be shown to Eddy Cue later that morning. Friedman shows a slide describing features within iOS that have revealed fraud and safety issues. The two relevant concerns are reports of child grooming in social features — like iMessages and in-app chat — and in App Store reviews, of all places. Subsequent messages indicate that this is partly what Friedman was referring to.
Edward Snowden (via Hacker News):
You might have noticed that I haven’t mentioned which problem it is that Apple is purporting to solve. Why? Because it doesn’t matter.
Having read thousands upon thousands of remarks on this growing scandal, it has become clear to me that many understand it doesn’t matter, but few if any have been willing to actually say it. Speaking candidly, if that’s still allowed, that’s the way it always goes when someone of institutional significance launches a campaign to defend an indefensible intrusion into our private spaces. They make a mad dash to the supposed high ground, from which they speak in low, solemn tones about their moral mission before fervently invoking the dread spectre of the Four Horsemen of the Infopocalypse, warning that only a dubious amulet—or suspicious software update—can save us from the most threatening members of our species.
[…]
Apple’s new system, regardless of how anyone tries to justify it, will permanently redefine what belongs to you, and what belongs to them.
[…]
I can’t think of any other company that has so proudly, and so publicly, distributed spyware to its own devices—and I can’t think of a threat more dangerous to a product’s security than the mischief of its own maker. There is no fundamental technological limit to how far the precedent Apple is establishing can be pushed, meaning the only restraint is Apple’s all-too-flexible company policy, something governments understand all too well.
Previously:
