Mail Data Loss in macOS 10.15
I’m working on more posts about the Catalina release, but I wanted to start with a short warning. I’ve heard a bunch of reports of data loss in Apple Mail. Thankfully, none seem to be caused by my apps. (Ironically, one of the bugs I’ve encountered is the inability to delete messages via AppleScript.) And, in fact, most of the damage has occurred without my Mail plug-in even being installed. Nevertheless, people contact me because it’s not unreasonable to wonder if third-party software is to blame, and I also hear from people who want a second opinion because what Apple support told them didn’t make sense.
What I’m hearing:
Updating Mail’s data store from Mojave to Catalina sometimes says that it succeeded, but large numbers of messages turn out to be missing or incomplete.
Moving messages between mailboxes, both via drag-and-drop and AppleScript, can result in a blank message (only headers) on the Mac. If the message was moved to a server mailbox, other devices see the message as deleted. And eventually this syncs back to the first Mac, where the message disappears as well.
I don’t know whether these are due to Mail bugs or to other factors such as problems on the Mac or with the mail server. But my advice is to hold off on updating to Catalina for now. These sorts of issues are pernicious because:
You may not notice that anything is wrong unless you are looking at the particular mailbox or messages that are affected.
Because the data is synced to the server, problems can propagate to other Macs and iOS devices.
Making a backup is difficult because, even if you set the preference, Mail no longer automatically fully downloads all messages. So the backup of the local data will necessarily be incomplete. (See EagleFiler’s Importing Attachments instructions for more about this. I’m happy to have most of my mail archived outside of Mail.)
- Restoring a backup is difficult because Mail data is constantly changing. There is no straightforward way to merge restored data in with messages received since the last backup, and also with the live data on the server.
Of course, it’s good to make backups anyway.
Apple advisors are apparently telling customers that if Mail data gets lost on Catalina, it can’t be recovered from a Time Machine backup that was made using Mojave. This didn’t make sense to me, and I’ve verified that it’s not the case. You can use Time Machine to get at previous versions of the folders in Mail’s data store, and then use the File ‣ Import Mailboxes… command to selectively import them into Catalina Mail. Since they import as new, local mailboxes, this shouldn’t affect messages that are on the server.
I also think that the advice to restore the whole Mac to Mojave makes no sense because as soon as you launch Mail it’s going to delete all the messages that were deleted on the server. In order to actually restore them, you have to make copies of the messages that might have already been deleted. That’s what Import Mailboxes does.
Update (2019-10-11): See also: Howard Oakley, TidBITS, Hacker News.
Update (2019-10-12): See also: MacRumors, iMore.
Update (2019-10-13): See also: AppleInsider.
Update (2019-10-14): Commenter Shiner says that the lost messages from “updating Mail’s data store” happens for local mailboxes every time Mail’s database is rebuilt, not just when migrating from Mojave, and reported this to Apple in August. It’s “trivial to reproduce” and lost 160,000 out of 200,000 messages in a test.
There’s some evidence that the moving messages issue is related to Microsoft Exchange accounts.
A doctor friend of mine lost about 20.000 emails when he upgraded to iOS 13. It was a POP3 account.
See also: TechRadar.
Update (2019-10-15): I’ve received four reports that messages moved to an iCloud mailbox (that were not originally in that iCloud account) simply disappear instead of showing up there. At first, we thought this was a Catalina issue, but it also seems to be happening with High Sierra now. My guess is that it’s an issue with certain of the iCloud servers.
Update (2020-01-29): macOS 10.15.3 fixes one of the bugs but not the others.
Update (2020-05-27): macOS 10.15.5 has the same version of Mail as macOS 10.15.4, which did not include any further fixes, as far as I know.
Update (2020-07-16): macOS 10.15.6 has a new version of Mail. I’ve received several reports that moving iCloud messages no longer makes them disappear, but there is some reason to believe that this may be due to a fix on the iCloud server rather than in Mail itself, meaning that the problem may still occur with other mail providers. If you were seeing messages disappear on previous versions of macOS 10.15 and have updated to 10.15.6, please post in the comments about which type of server you have and whether you’re still seeing the bug.
Update (2020-08-03): One commenter reports that the message moving bug is not fixed in macOS 10.15.6.
Update (2020-08-09): I received a report of the message moving bug on the Big Sur beta.
Update (2020-08-18): I received a report that Mail in macOS 10.15.6 is deleting messages from multiple servers without downloading them.
Update (2020-09-08): I received two more reports that moving IMAP messages in macOS 10.15.6 instead deletes them. Several other users are reporting that dragged messages don’t get deleted but don’t move, either.
Update (2020-09-11): One user reported losing the entire contents of two IMAP mailboxes when updating from macOS 10.14 to macOS 10.15.6. I also received a report from a user who is losing messages when dragging and dropping them on macOS 10.15.6.
Update (2020-09-28): One reader reports that the IMAP moving bug is not fixed in macOS 10.15.7.
Update (2020-10-07): I have received additional reports that the IMAP moving bug is not fixed in macOS 10.15.7 or in the current macOS 11 beta.
Update (2020-12-02): I have received additional reports that IMAP messages disappear when dragged and dropped between mailboxes (particularly when stored in different locations) on macOS 11.0.1.
Update (2021-05-25): I’ve received multiple reports that this still happens on macOS 11.3.1.
Update (2021-05-28): I’ve received a report that this still happens on macOS 11.4.
Update (2022-04-12): I’ve received a report that this still happens on macOS 12.3.1.
Update (2022-04-22): A reader reports losing messages moved server-to-local on macOS 12.3.1.
Update (2022-05-19): I’ve received another report of losing Exchange messages when moving them on macOS 12.3.1, and I saw lots of messages disappear when moving them from local to IMAP on macOS 12.4.
Update (2022-08-11): I’m still receiving reports of messages disappearing when dragged and dropped on macOS 12.4.
Update (2023-01-02): One commenter reports that moving messages from IMAP to local turns them blank on macOS 13.1.
Update (2023-01-18): A reader reports thousands of messages lost from On My Mac mailboxes with macOS 12.6.2.
Update (2023-01-31): Commenter Andy Scott is seeing data loss when moving messages between Exchange and local mailboxes on macOS 13.2.
Update (2023-06-19): Scott Palmer is seeing messages lost when moved out of an IMAP account on macOS 13.4.
Update (2023-06-28): I just encountered the bug again myself with macOS 13.4.1. When re-applying a rule that would move messages to an IMAP mailbox that they were already in, it just deleted them.
Update (2023-11-15): Some readers and customers have reported still seeing the data loss bug when moving messages on macOS 14.1.1.
