Photos Data Loss With macOS 13.3 Beta
Josh Hill (via Marquelle D. McKean):
During a Europe trip in May 2022, I uploaded over 6,000 photos and hundreds of videos to the cloud. Upon editing and deleting some of the photos, I encountered an issue with Photos.app, which ultimately led to the complete wiping out of my entire cloud library. Despite my efforts to recover the lost data using tools such as Disk Drill and contacting Apple support, no useful recoverable files could be found. Unfortunately, Apple support refused to escalate the issue to the engineering team due to the use of a beta version of macOS Ventura.
The loss of my lifetime memories and tens of thousands of dollars’ worth of intellectual property is one of the most devastating experiences of my life. I believe it is crucial to highlight the importance of backing up all data, including cloud content, to prevent such a catastrophic loss.
As a result of this incident, Koingo Software is investing 100% of our resources into finding a solution; all standard business operations have been halted for the time being.
Cloud syncing is not a backup. Apple doesn’t provide a way to directly download a backup of your photos, so you need to have Photos set to Download Originals to this Mac, which for most people means that they need to fit on your internal SSD. Then you can back them up yourself with Time Machine or to another cloud provider.
The other issue here is using your real data with a beta OS version. It’s easy to say that one shouldn’t do that, and I try not to. But I can’t really blame a developer for doing so because we want to identify bugs before they hit our customers, and many bugs don’t show up when using a clean test system. There’s no way to, say, use different Apple IDs for calendars and photos. And it’s not as though release versions are free of data loss bugs, either. So I think the main problem was not realizing that his local backup needed to include the photo library.
By all reasonable standards, this is a person we would expect to have understanding of how these things work. You don’t become the CEO of a small software company that’s been in business for nearly 30 years by being ignorant about computers. (Okay, maybe some people do, but they generally don’t stay in business for long after that.)
If someone like Josh can get something so straightforward and fundamental so very wrong, is it any wonder “ordinary” users make the same mistake, over and over again?
Previously:
- iOS Betas Tied to Apple ID
- Switching on iCloud Photos
- iOS 15 Messages Bug Deletes Saved Photos
- Privacy for Apple Employees
- What Time Machine Doesn’t Back Up
- Why Public Betas?
- Mail Data Loss in macOS 10.15
- iCloud Data Loss With macOS 10.15 and iOS 13 Betas
- Done With iCloud Photo Library
Update (2023-03-21): Brian Stucki:
The greatest digital fear. I “Optimize for Storage” on MacBook and iPhone but I have a Mac mini in the data center set to “Download Originals to this Mac” and copy to an external hard drive. It does the same for full copies of iCloud Desktops and Documents. Automatic everything.
I expect big cloud providers have more redundancies than most of us do, which makes it all the more disappointing that none of this stuff comes with a guarantee. That is, even if Hill were only running production operating systems, Apple — like every major provider of consumer cloud services — does not obligate itself to perform data restoration. iCloud Photo Library may only officially be a syncing service, but it is easy to think of it as something more robust, especially when it is described as “safe”, where your photos are “always available”, “without worrying about space on your devices”. Not sure about you, but after reading Hill’s experience, I am less concerned about disk space and more concerned about whether those photos are actually as safe as they ought to be.
I still think that a backup that’s not in your direct control is not really a backup. But Apple could certainly do more here. Calendars and contacts are synced, but multiple versions are stored in the cloud, and you can roll them back. This should be extended to other types of data.
Apple today seeded the release candidate version of macOS Ventura 13.3 to developers for testing purposes, with the new software update coming a week after the release of the fourth macOS 13.3 beta.
I hope the bug is fixed, but I’ve seen no reason to believe that it is.
Previously:
Update (2023-03-23): Ezekiel Elin:
I’ve never seen any issues but 22E252 does run a Photos library upgrade
The Apple world is now used to beta being alpha quality without any questioning.
No Beta should incur data loss, they are almost production ready. And if data loss happens, it should be escalated quickly because of the oversight of a critical bug.
Update (2023-03-28): Josh Hill:
Regular every day phone-in support basically said there was nothing they could do — especially because we had used a beta. They wouldn’t even submit a ticket to the engineering team and told me to submit a comment using the Feedback Assistant.app tool instead (which often feels like the lowest priority format simply for remitting bugs).
[…]
I wrote in to upper management and explained the situation, and made a desperate plea to anyone in the tech community who could help provide suggestions.
[…]
After a few days, Apple’s backend engineers were finally able to recover what now seems to be all of the lost data, despite what the phone-in agent stated.
[…]
As it turns out, the “feature” is unlikely related to the beta at all after talking with Apple. The support agent I spoke with even told me he had the same problem.
The problem was triggered by creating a new local library and setting it as the system photo library.
If you designate a new library as the System Photo Library and then turn on iCloud Photos, the photos and videos in the new library will merge with those already in your iCloud Photos. Additionally, all photos and videos from iCloud will redownload to the device.
However, instead it deleted all the photos from all devices, bypassing “Recently Deleted.” It sounds like this is not fixed in macOS 13.3.
Previously:
Update (2023-04-03): Nick Heer:
I also do not love that Hill removed the original article and excluded it from the Wayback Machine because I think these things are important to document, even if they are later corrected[…]
15 Comments RSS · Twitter · Mastodon
from whoever posted this on TidBITs:
"I don’t want to sound like I’m passing judgment here, but this seems to me like a severe overreaction to failing to accept the consequences of trusting valuable data to a beta OS. I fully sympathize with the anger at having lost so much data, but calling Apple irresponsible (which is basically what the email does) for releasing a beta that may lead to data loss bothers me."
it's just incredible to me that anyone would excuse this. especially since Apple has not had a good track record in software quality for a very long time now.
If you're going on a trip and the damage of losing data is that large, you bring backup camera's, backup battery packs, backup cables and bags, heck even backup laptops or iPads. Also you never rely on cloud storage, the places you go may not even have internet access. That is not how you do data.
And now, this person is claiming tens of thousands worth of damage by using a beta(!) version of an operating system, and halts all company operations (including customer support) because of it and asks for donations. That is not how you do a software company.
What can possibly be so important in a foto library, that it literally can destroy your company??? It's hard to lose your personal photos, I've experienced this myself and it sucks...big time! But I just can't imagine that your images can be so important to your business that it goes down if the images are lost. And if so...how come you don't know that before?
@Person Yup. I enjoy reading Tidbits and participating in Tidbits Talk but the tone of the thread stinks of techno-elitism. It's, like, the whole reason we endorse Apple products that they are, in general, of superior fit and finish to the competition, and eminently suitable for use by ordinary people, with lives. Sure, it's less excusable in someone running a business on beta software with some expected degree of technical competence, but in the end software developers are users too and it still sucks donkey balls. Nobody should endure data loss, but especially when "the cloud" is involved and there's realistically no prospect of preventing it from happening when you have no means of backing up to a computer, or even if you have that, from destructively propagating to other devices. Think of every parent or grandparent with an iPad and iPhone, or even the MacBook user who's away from home with limited internal storage. This shit really does need to work right, and when it goes wrong, the state should be restorable from the past. Like, I dunno, Time Machine did in 2006? If we are going to be expected to trust the cloud, then it should provide the same affordances expected of local storage, simple as.
It't too bad the article has so little details. As a victim of data loss in Photos.app, albeit not related to iCloud syncing, I would like to know the steps that led to this disaster.
I agree that using beta software with important data is somewhat careless. On the other hand it's worrying, that Apple hasn't caught such a catastrophic bug earlier in the release cycle.
@Milo Yeah, they are already up to macOS 13.3 Developer Beta 4, which is more betas than some releases get. And this is basically a maintenance release, not a major new version.
The thing about iCloud Photo Library is not just that there are no guarantees about your cloud data being backed up. It’s that it makes it incredibly difficult for *you* to back up the data *yourself.* For me, this is well over the line from “easy to use” to “paternalistically preventing you from exercising control over your own data.” At least Apple’s contacts and calendar apps (on the Mac) allow you to export a single backup file with all of your data in it.
>I hope the bug is fixed, but I’ve seen no reason to believe that it is.
I'm confused from Josh's description what that bug actually is.
> Calendars and contacts are synced, but multiple versions are stored in the cloud, and you can roll them back. This should be extended to other types of data.
But photos have a "Recently Deleted" section (essentially a temporary trash). That's not quite the same as storing multiple versions, but it's not nothing. The UI says, "Photos and videos show the days remaining before deletion. After that time, items will be permanently deleted. This may take up to 40 days."
So, is the bug here that not only did the Photos app delete many more photos than Josh intended to delete, but on top of that, it also immediately erased them from the trash?
@Sören The description is not very specific, but, yes, I think that’s what Josh is saying. Recently Deleted protects against user errors, not bugs or cloud storage failures.
How do you really backup photos that are managed by iCloud?
For example, I can have Time Machine backups set up for my Photos library, but if iCloud starts deleting photos randomly I probably won't notice until a long time after (maybe even a few years), at which point Time Machine probably won't have a backup available from such a long time back.
@Bogdan Yes, that is a problem with Photos. Maybe you could use IntegrityChecker to alert you if files disappear. Personally, I use Lightroom (no syncing) and have it share storage with an EagleFiler library that will tell me if a photo disappears or is unexpectedly modified.
Koingo posted an update that the photo library was restored by Apple, and it has more details — https://www.koingosw.com/blog/?action=view&id=595. There was a triggering event to the data loss — they created a new Photo Library and set it as the System Photo Library. Not exactly regular usage undone by a fatal beta OS bug.
On the other hand, the Apple Support article about designating System Photo Libraries, says that photos in a new library will merge with those already in your iCloud Photos — https://support.apple.com/en-in/HT204414 — and it's not unreasonable for Koingo to do whatever they did with photo libraries.
But Koingo also says "We are well-trained from experience to keep backups of any files not stored in the cloud" and they really should know better. A regular Time Machine user using the defaults would have been better off.
[…] meaningless without a guarantee. I am not reassured by this resolution if the bug Hill documented still seems to exist in the shipping version of MacOS 13.3, and without a promise that similar recovery steps would be […]
Given the steps which triggered this issue are exactly the steps recommended by Apple themselves in their own support docs (as mentioned by Dips above - https://support.apple.com/en-in/HT204414), and also the fact that I have done this exact same thing with no deleted data resulting from it, I'd suspect this is a recently introduced bug.
In order to truly have a local, offline backup of your iCloud Photo Library on an external ssd, you pretty much have to follow those steps to create a new library, designate it as the system library, and then activate iCloud Photos. When I did this some time in 2022, it worked as designed.
Something has gone haywire and I am concerned that we don't know now whether to trust this method or not. Ironic, considering for some people this might be the only way to actually back up their photos in the first place.
[…] Via Michael Tsai: […]