Mail Data Loss in macOS 10.15

I experienced data loss when upgrading from iOS 12 to 13. I had some drafts that were not synced with the server. When I upgraded the tabs for the drafts remained but their contents was missing. I wonder if the Mail apps on iOS and MacOS share code.

@Benedict Yes, I forgot to mention that I also heard similar reports from users updating to iOS 13 (before Catalina was out). I don’t recall this happening with any prior OS update. I think they share a lot of code.

[…] Tsai, the developer of the long-standing SpamSieve plug-in for Mail on the Mac, has written a blog post warning about potential Mail-related data loss when upgrading to macOS 10.15 Catalina. In particular, he’s hearing from numerous customers that […]

Thanks for the warning — I’d be interested in hearing what strategies you use to archive your messages outside of mail.

@Morgan (1) I regularly archive messages to EagleFiler, which I wrote partially for this purpose. It converts Mail’s data store to standard mbox files and checksums them so their integrity can be verified later. (2) For messages that I don’t actually need to access across multiple devices, I have Mail rules that move them to local mailboxes (“On My Mac”). This ensures that there’s a complete local copy (with all the attachments) that will make it into the regular backups of my Mac. (This only works when the messages are processed by rules as they arrive. If you manually move messages to local mailboxes later, even if via rules, that does not force Mail to fully download them; it can actually lead to data loss as Mail saves the incomplete message locally and deletes the complete copy that’s on the server.)

The main issue I and others have noticed is that Mail.app shows blank Sent folder/mailbox from GMail/GSuite accounts. Here is a thread on the issue from other users:

https://support.google.com/mail/thread/16312735?hl=en

There's also Mail Archiver X, which can archive emails either directly from IMAP or from the local Mail.app's database.

https://www.mothsoftware.com

[…] Tsai, the developer of EagleFiler and the SpamSieve plug-in for Apple Mail on Mac, has written a blog post warning macOS users about potential data loss in Mail when upgrading to macOS Catalina 10.15.0 […]

I lost a lot of email messages when upgrading to Catalina. I would not have noticed except that some of my email accounts - which, apart from my Apple ID accounts - are all POP accounts so are not on any remote servers, seemed to have a surprisingly small number of messages after upgrading. I have a total of more than 100,000 messages going back about 15 years. On checking I found that some of my accounts were missing thousands of messages. Which was obviously very worrying.

Luckily I had backed up before upgrading and I painstakingly imported the missing emails from my backup. It took many hours and wasn't altogether successful because some sent emails are now logged not by recipient but by sender (me) which is not very useful. But I think/ hope I managed to find almost all the missing emails.

I am not a Mac expert but I noticed that after upgrading Catalina had created a new folder: user/library/mail/V7 which was not in Mojave, where my messages are now stored.

Perhaps this has something to do with the problem?

I should add that I had backed up not via Time Machine, which I have always found problematic, but via SuperDuper so I had a complete backup from which I could copy my missing messages.

[…] blog post by Michael Tsai via MacRumors is today suggesting that macOS Catalina seems to be experiencing a […]

When the users move from v6 to v7 would the size of the folder be very different than before?

Import Mailboxes isn't a good idea because you would end up with even more duplicate emails than you have anyways in Mail.

The problems with deleting emails with AppleScript isn't really new. You can't even check if the deletion went okay or not.

[…] via mjtsai […]

There is a bug in Mail under iPadOS 13.1.2. If you send an email to yourself (say to read on a different device), it will appear in your Inbox and in your Sent box, as expected. If you move the copy in your Sent box to the Trash and then Delete it, the copy in your Inbox is deleted, too (and from your other device). That didn’t happen under iOS 12.4.

[…] blog post by Michael Tsai via MacRumors is today suggesting that macOS Catalina seems to be experiencing a […]

@Sam Each major new version of Mail migrates to a new folder, i.e. V6 on Mojave and V7 on Catalina. So that’s normal, and it’s good because it keeps the different versions of the data store separate.

SuperDuper is great, but I think it doesn’t fully support Catalina yet.

@Beatrix I think the file format hasn’t changed much this year (whereas a few years back the way attachments were stored changed). So in theory the sizes should be similar. However, I have also seen (going back a long time) that sometimes when the data store is rebuilt it throws out all the local copies of messages that are on the server. So in that case the V7 would temporarily be much smaller.

Yes, duplicates are a problem (#4), but that can be fixed. It’s much better to have duplicates than to lose your mail.

Deleting e-mails via AppleScript has always had issues, but it’s gone from working something like 99% of the time to failing 99% of the time.

Since updating to Catalina (and the new version of Spamsieve) my emails are no longer removed from my inbox, though they also appear in the Spam folder. I have a MacBook pro 2016 using Apple Mail, with Gmail, as the account/server. Any thoughts?

@Eric I haven’t heard of that issue before. Please contact SpamSieve support.

I routinely go through the pain of manually exporting all of my e-mail off Mail.app onto .mbox to prevent this kind of thing, since ever since they started messing up .emlx and storing attachments separately that the SQLite metadata stores have had a number of issues.

I suspect that in many cases mail messages are indexed by something else than the Message-ID header (which is in itself prone to duplicates) and that the indexes get confused, but overall (and having used Mail.app since the NeXT days), whatever has happened this time around is an(other) egregious failure of QA at Apple. There simply is no excuse for the amount of bugs that have shipped in macOS (and iOS), and for someone like me (with an e-mail folder tree that goes back to 1998) this means I won't be upgrading to Catalina until they sort it out.

(Ironically, I am more likely to move to an iPad Pro full-time for my personal stuff and set up a Windows or Linux box to work - I have a lot more trust on either of them to deliver stable releases right now...)

Not only data loss but the whole e mail program does not work on a mac book 2018. On a mac book pro early 2013 everything is almost ok. Sad because the mac book from 2018 is for work. Deleting te boxen en re install does not seem to work. disaster!

@Rui Unless it’s been fixed in 10.15, the built-in export from Mail to mbox is problematic because it doesn’t properly escape the “From ” lines (so message boundaries will get messed up if you import into anything other than Mail, which reads the boundaries from sidecar metadata) and it doesn’t download attachments that are not already cached locally. EagleFiler avoids this by generating the mbox itself and checking that all the attachments are there, and it also saves the metadata that mbox doesn’t support.

[…]     Source:マイケル・ツァイ氏のブログ via MacRumors […]

[…] dans cette même application. Il a expliqué la situation détaillée dans un article posté sur son blog : apparemment, un nombre non négligeable d’utilisateurs a rapporté qu’une mise à jour des […]

[…] a blog post by SpamSieve plug-in developer Michael Tsai, Mail is seen to be causing problems when users […]

Have you waited for one or two days after the upgrade? Have you tried re-indexing with Spotlight?
I had problems with my 60.000 items mailbox almost during every MacOS upgrade before; and the symptom was missing emails. In the end, I suspected it has to do with Spotlight: The list of mails in mail is created using a Spotlight query, and upgrading mail.app leads to re-indexing all mails in Spotlight - which takes time (and there is *no kind if indication from MacOS which is really a pain*). During that indexing, a lot of mails will simply be missing. In particular tough cases I launched re-indexing with Spotlight (forgot how to do this, but you‘ll find it easily) and that helped (and you can see whether it is finished by looking into Activity Monitor).
Can‘t say yet anything about Catalina, but it would be interesting to know if anyone who already dared to do this can solve it in this way?

@Peter I think Mail uses Spotlight for searching and for certain views like smart mailboxes, but it uses its SQLite database to browse normal mailboxes. So, if the messages are there, that should work without Spotlight.

@Michael yeah, I know about the "From" thing (I maintain https://github.com/rcarmo/imapbackup), but I was under the impression mbox export would download attachments (since I have that option set to "always") even if they weren't extracted from the MIME info (or even downloaded already).

Can I take an EagleFiler archive and reconstitute a full IMAP tree from it?

@Rui It’s just nested folders of mbox files, so I don’t see why not.

I upgraded and then saw this thread! I have not opened the Mail program yet, and was just about to re-install Mojave. But before I do, I wonder if I use POP accounts (still) on my own company‘s server, whether I’m still at risk of losing emails.
Any clues?

@Sarah I have not heard of any problems with POP accounts, although that could simply be because most people don’t use POP these days. At least with POP you can make complete backups and don’t have to worry about corruption getting propagated to other devices. Commenter Sam has lost messages in POP accounts during migration. I’ve not heard of any POP messages lost moving them between mailboxes.

@Michael Tsai Thanks. I probably should go back to Mojave anyway, yes? I had another issue when I went to turn Time Machine back on after installing Catalina, I got a message saying that I had two Macintosh HDs and had to rename one . . . Have you seen this?

@Sarah Personally, without a strong reason to upgrade now, I would go back. You may even be able to do that easily/quickly by restoring the prior APFS snapshot by booting into Recovery. I’ve not seen that Time Machine error.

@MichealTsai @PeterResele From my experience with the innards of Mail (and I have a fair bit after working on plugins for 15 years), both regular mailboxes and spotlight mailboxes are populated by performing a spotlight search to find library ids (ie, the integer part of the .emlx filename) and then looking up those messages in internal sqlite database. Mail also uses spotlight for doing the message counts for mailboxes.

Given this, rebuilding spotlight indexes may resolve the missing messages issues for some but mileage may vary.

As an aside while I personally think this is inane because Mail has a highly reliable, performant sqlite database, I think Mail uses spotlight to prevent a discrepancy in which a person will do a system wide spotlight search for a message will turn up the same result as a search inside Mail, and a few other reasons.

@Scott I think we’ve both been writing plug-ins for Mail since before it was using SQLite and before Spotlight existed. :-) Yours certainly hook into it more deeply, though. It’s an intriguing theory. I don’t know what to say, because it doesn’t square with my experience successfully using Mail with Spotlight completely turned off, the fact that Mail’s SQLite database has fields and triggers to maintain message counts, or the fact that I’ve fixed messed up counts by editing the SQLite database. I’ve also wondered why they don’t they don’t use Spotlight only for the full content searches, given the performance of SQLite.

Over on the MacRumors forums, some people suspect that the data loss problem may be specific to Outlook accounts. Do you happen to know if the folks who reported the issue to you also have Outlook accounts?

[…] macOS Apple Mail 插件開發者 Michael Tsai 的網誌，他警告升級 macOS 10.15 Catalina 的用戶要小心備份電郵，因為從 Mojave 升級到 […]

I'm wondering that all the smart people here dont know how to use Time Machine properly. You have to be in Mail and start TimeMachine from there. Now you can restore single Mails or folders directly. No importing Mailboxes from the Library.
Please correct your post.

[…] El más reciente, reportado por Michael Tsai, desarrollador de EagleFiler y SpamSieve, un complemento para la app Mail de macOS, asegura que el gestor de correo electrónico por defecto de Apple puede provocar pérdida de información si se actualiza a macOS Catalina, según ha comunicado a través de su blog personal. […]

Just a report here for other users:

(1) I have not noted any loss of email, but I noticed today that one message did not have its attachment downloaded.
(2) Search stopped working -- it's really annoying. I have had to go to the office.com outlook app to search for
emails, nothing is found on mac mail (it just shows that it is "searching"). I am currently rebuilding one of my mailboxes
to see if it gets fixed.

I updated a few days ago and things were mostly fine, Mail wise. Turned on this morning and all I have is Headers, no content in any email. Had old saved MailData, moved that in, no difference. It's all gone. I am lost for words.

[…] وحدد مايكل تساي Michael Tsai، الذي يطور إضافات EagleFiler و SpamSieve لتطبيق آبل للبريد الإلكتروني Apple Mail على نظام التشغيل MacOS، بعض المشكلات التي ظهرت في تدوينة. […]

[…] upgrading to macOS Catalina. If you depend on Mail for your work or in your personal life, please, read this before upgrading. […]

I have an open ticket on this going back to August (and was probably the first to raise the alarm on this one.)

Where you will see this issue is with "On My Mac" mail. If the Mail application launches with the "Mail Message Import" box (that you can't ignore), when Mail does it's "reimport" of all your mail, you will most likely have a number of "On My Mac" messages just flat out deleted. This is new behavior in 10.15 and wasn't taken seriously enough (IMO) when I reported the problem.

It's trivial to reproduce (if you have backups!) by deleting the "Envelope Index" files (Google around for how to do this -- it's been a long-standing recommended way to fix "search" issues *within* Mail), . It's not related to moving the Mail store from "V6" to "V7", and shouldn't (at least in any of my testing) affect IMAP mailboxes (as those just refresh by pulling mail back down from the server.)

I have test mailboxes with 200,000 .emlx files that end up with only 40,000 .emlx files in them after Mail does it's reimport. If you have "On My Mac" mail, you should test this yourself and immediately file feedback on it!

@Peter It does seem like Outlook/Exchange is overrepresented among the people with the moving messages problem. But there have also been multiple reports (for a different blank messages issue) with Gmail. And the importing issue seems to affect IMAP accounts and even local mailboxes.

@Jörg Have you tried using Time Machine from within Mail on macOS 10.15? Every user I’ve talked to said that this caused Mail to crash, as it did for me. It’s also not clear to me that Mail Time Machine can restore mailboxes that were backed up from previous versions of Mail.

@Shriner That’s really bad news if it’s related to rebuilding the Envelope Index database rather than just migration. There are so many problems that can only be fixed by rebuilding, which has historically been very safe. Even if IMAP messages can be pulled down from the server again, losing the local files in a rebuild would discard information that doesn’t sync to the IMAP server, such as colors.

[…] Developer Michael Tsai shares reports on users with Mail problems in macOS Catalina specifically missing messages or incomplete messages. In some cases you will see headers only and a blank message. Tsai recommends not updating to macOS Catalina yet, but if you already did he shares one way to fix this issue. […]

Yeah, I've never seen this in all my testing that it's related to migration *unless* the Migration forces the reimport (which, I believe, shouldn't happen intentionally.). I only noticed this because one migration I did forced this reimport and I didn't realize I had lost mail for a week or so after that -- the mailboxes (as seen in Mail) will still show the same number of messages -- it's only when you go to open them that you find they open blank (and then you can see the .emlx files are gone...)

My gut feeling is that people who have upgraded to 10.15 have already lost a lot of e-mail, but don't realize this yet.

[…] y SpamSieve, complementos para la ‘app’ Mail de macOS, ha difundido a través de su blog que ha recibido varios informes de usuarios de macOS Catalina que afirman haber sufrido una […]

Michael, This is Sam again. Just to confirm, further to your answer to Sarah about POP accounts, my lost messages were from POP accounts.

Hello all! Alse a simple POP3 account make Mail to crash. In my case I have multiple accounts (mostly with POP3) but the troubles come only with one that is very huge. Maybe the problem should happens with very huge mailboxes.

[…] part, elle contient quelques bugs très gênants, dont l’un d’eux rapportés par le blogger Michael Tsai, concernant les e-mails et plus spécifiquement, l’app […]

@Sam Thanks for the correction. I’ve updated my comment.

Michael, thanks for this, very helpful.

I'm not sure if you are aware of another even more serious mail issue that a number of us are hitting, which causes Mail to be completely unusable.

The error message you get is: "Mail cannot save information about your mailboxes because there isn't enough space in your home folder" ... the popup with this message has only one option, Quit. The popup happens some period of time after mail loads, sometimes almost immediately, sometimes after 1-2 hours.

This is an error that has appeared in the past with Mail but the fixes that worked in the past (removing index envelope data and the like) do not fix the issue this time, nor does reinstalling Catalina.

Here's the issue on Apple discussion forum: https://discussions.apple.com/thread/250717871

[…] Learn Extra […]

Thanks for posting. I look forward to seeing if today’s update fixes things.

Same issue here. Just lost a couple hundred thousand mails, randomly, all over the place, from the last 20 years.

Am restoring, thanks to Mike Bombich, using Carbon Copy Cloner, from nightly backup, which I had first mirrored to yet another disk.

It's running since a night or so, and the mails are coming back, with their folder structure. Just take one by one the content of each subfolder in ~/Library/Mail, and import it using Mail's import function.

This problem went unnoticed for a day over here, since the mails appear to be there. It is not dependent on what type of mail server you had been using - I had been using all POP3, IMAP, Exchange over the years. The mails that were lost were all local to the machine, i.e., long time deleted from the (in many cases long time not longer existing) mail server.

So it is not just Online folders, it is everything, everywhere, with no apparent pattern.

Beware. And Apple - WTF?

[…] […]

[…] plug-in developer Michael Tsai has posted a warning on his blog to reveal that after installing macOS Catalina, some users could end up with missing or incomplete […]

[…] warning arrives by way of SpamSieve plug-in developer Michael Tsai. His post is based on feedback provided by customers, though the Mail problems aren’t related to […]

OK, here is an update on this topic:

I've first of all gone back to backup and restored all of them.

This morning, all ok.

This evening, WTF? Again skeleton mails from today.

I think I've found some (frightening) systematic:

Don't move a mail that has an attachment out of an online folder. It'll just kill the mail content. You can copy it into another online folder, even from like exchange to imap, but if you move it out, it seems to become a skeleton.

There's more inconsistency in that; sometimes it doesn't even perform the copy / move operation.

So, summary: DO NOT USE APPLE MAIL for the moment.

I have a 2018 Macbook Pro and I'm having search issues with the mail app: the query does not find all of the relevant emails. I've tried rebuilding and reindexing spotlight with no success. The only thing that I found that worked was to manually look for the email and view it. Only then, it would show up in the search query.

I transfer all my emails to local. Since upgrading to Catalina, all emails that have an attachment are not transferred properly to local (all you see is the sender and subject line) and lost. It does not damage emails without attachments. This is incredibly annoying and disruptive.

When I upgraded to Catalina I experienced a new problem now of not being able to email back to myself, which I do with many emails. They all bounce as a Permanent Error. I have complained to Apple for years that Mail was behind the times and had lots of bugs. Now we get this new set of problems. Search is just not a valid structure, it is hit and miss. Signatures are so awkward to set up. So much is not clear from the help menu. Things just do not work they way they say. Now with Catalina I see sliding lines of text and mail folder that I have no idea what they are to do. They could have put some info help boxes into this it seems.

And the date/time of the incoming email cannot be placed on the left side like I had it. It is locked on the right column. Nothing in View will give this option. I need it on the left side.

There are no ways to color icons either that might help users.

I have not noticed any lost emails with Catalina (may not have discovered it yet). I had this problem long ago with Mojave back in the spring and tried to alert Apple. I lost 15 years worth of emails when I migrated to my new MacBook Pro in January. The MBP had a lot of problem root software they had to find and take out. In the process I lost the emails when he set up a test user account and I then went back to my main user account. The tech told me I had put them on my Comcast ISP in the Mail mail boxes and they deleted them but I know I had them On My Mac mailbox using IMAP. They would not restore from Time machine either, so they told me.

Another problem has came up with the migration to the MBP, either Mail or Comcast has been deleting all my emails over one month old. At first I thought this was a preferences setting on the MBP, which had been put in by the migration but it even exists on Comast ISP with the MBP settings ok now. I called Comast and complained and they said they pulled all my old emails from an archive file and put back in my ISP mailbox but they all show as coming from me with my name, so that is not much of a help.

Long ago I found a big problem trying to delete incorrect emails from Previous Recipients. They just will not go away and keep showing up in the TO field while not in the archive list. AND for a long time Mail has been deleting email addresses of people I may not have mailed in a year or so. They are just gone, removed, I did not remove them.

This is a sorry application for such a pretty good computer company, imo. Now I am new to these forums and don't know how to get replies or other info...but if someone wanted to help me send an email to me at citizen
@comcast.net. How does one email Michael Tsai directly?

Thanks,
Bob

The problem I have experienced since updating is that emails that I know have been sent to me, and I can verify were sent, are not appearing in my inbox. Yet, if I check with the sender and know the Subject, I can find them when I search Mail. Effectively, the incoming emails are - apparently randomly - invisible unless I know they have been sent and I know how to search for them. They are in the inbox, in the right place, and they are visible upon a search, but disappear again when the mailbox is revisited. This is a disaster, and I understand from colleagues they have experienced similar issues. Furthermore, the whole Mail system is unaccountably incredibly slow now, every time I log into it.

Man, nothing to add, but yikes! Fingers crossed this gets worked out. Anyone remember when Microsoft bought Danger and then promptly destroyed all the cloud data of Sidekick users? This is kind of feeling a bit like that. On a scale of "okay, we can fix this" to "Holy goodness, yikes!"

[…] Mail Data Loss in Mac OS 10.15 […]

My problem with the catalina upgrade on my 2018 macbook air is with the undo delete message option. It no longer works, and i actually use it a lot. Also, with some messages, but not all, the scroll down option does not work. Little annoying glitches! and I haven't seen the option of 10.15.1 yet. Also apple has not responded to my questions. I guess they have bigger problems than mine.

Would switching to another email client like Readdle Spark solve the mail problem?

Yes, I too have experienced similar frustrating issues with Mail in Catalina. One additional problem, I typically move old mail from my sent folder to an archive of mail in organized folders on icloud. So, as others have said copying mail to icloud is just failing like crazy usually (for me) with a timeout error. The mail that failed to copy to the icloud folder has gone from its original folder and now shows up in a new "On My Mac" recovered mail folder. But the date for all of the mail items in that recovered mail folder in column view is now today's day regardless of when it was sent!!! The date in the mailheader is the original correct date. So if I select date with the new sorting dropdown menu the newly moved items are sorted with todays sent mail.
I guess that I just cannot move any mail until Apple decides to fix this mess.

Hi all,
My upgrade to Catalina on my late 2015 iMac went reasonably smoothly, until I logged into the 2nd User Profile on the machine, and saw an error message about having to import email mailboxes. Didn't get to note down the message, however, a brief synopsis of the issue & resolution is:
Ran upgrade to Catalina on Sat Oct 19.
Emails from a POP3 mailbox were randomly missing content in the Mac Mail client, from a time period of Oct 18 back to April 2018.
By necessity, the emails had been deleted from the mail server - lack of provisioned space.
The mailbox had been in existence for about 12 months, but the last 7 months or so of email had only headers, no content.
The emails from the last couple of days were valid with headers & content.
Upon further investigation, and rebuilding the mailbox a couple of times, the emails without content disappeared entirely. I could verify this, as MS Outlook had been setup and had downloaded a copy of each mail item into its own mail store. However it did *not* contain all the sent emails, and was not gong to be solution.
Found the local mail store on the filesystem, went into a Time Machine backup point just prior to the upgrade being run. Restored the entire /Users//Library/Mail/V6 (Mojave) to a temporary directory in Catalina.
Imported *all* the mail into a new folder in Mail application. Was given the option to select a number of different folders, but chose bring all mail items in.
Found all the missing content, dragged & dropped into correct folders, including Sent emails for missing time period within the Mail app.
Seemed to have fixed the problem, however will require further checking over the next day or so.
HTH
Doug

I've been seeing "strange" Mail issues (okay, mostly using GMail backed mail accounts) even before I considered Mojave... that was on !0.12 & 10.13... where my laptop was still on 10.12 and my desktop on 10.13, I did on several occasions had to "rebuild" my mail database as I experienced (especially older) mails without their attachments or just empty.

Lately, after I've "upgraded" the desktop to new hardware + Mojave, I've been encountering (on both the laptop & the desktop) emails that are wrongly put in the same mail thread ;( some old non-related included in the same one ;(

Also discovered that the majority of my local mailboxes (hundreds of hierarchical mailboxes, >500,000 messages) had only partially made the conversion to the V7 mail store. There were missing messages and no body or attachments for most of the messages that were there. I had no problems with the four Gmail accounts syncing. I restored the V6 folder from right before the upgrade to 10.15 using Time Machine in the Finder, then imported the local mailbox folders into Mail piecemeal starting with the top level hierarchical folders. If you select a mailbox folder for import directly, you will also get the UUID folder names it contains, so put the mailbox in a folder and select the containing folder if you want to import individual mailboxes. Consolidation with old folders is going to be fun.

Some comments relative to the main post - no Exchange accounts were involved. There were none defined in Internet Accounts at the time of the upgrade.
Search wasn't working at all, but rebuilding the spotlight index didn't help in this case (there were missing message files - i.e. no underlying emlx files and attachments), and rebuilding the Envelope Index did make things worse (fewer messages listed and body no longer showing up for most messages that it had been visible in list preview before rebuild).

I had issues when upgrading to High Sierra with losing some old archived email, but the Catalina update really destroyed my email. 20 years of emails, archived in folders by year, and most of them are blank. Apple was unable to help. While I have backups, they are imperfect since the issues started with the last OS. I am ready to throw in the towel and setup all new email addresses.

These are the test I've done with results;

Like most users have reported, I too lost a huge amount of emails by updating from Mojave to Catalina. I had a solid TimeMachine backup so I simply formatted my drive then installed back Mojave and restore from TM. That worked perfectly and all my mail was back the way it should be. I have an extra MacBook Pro 13" so I decided to do some more testing. I first created a new folder on my Desktop and exported all my mailbox via the export function. Then I did a clean install of Catalina on my 13", open the mail app and imported all my mailbox via the import function. This allowed me to re-create a perfect copy of mac mail Mojave to mac mail Catalina. Then I made a brand new TimeMachine backup of my 13" Catalina install. Now I'm thinking I have a TimeMachine Catalina backup with all my mail so that should be perfect! Well no it was not. I erase my 13" and proceed to install a clean copy of Catalina, then use my perfect TimeMachine backup to restore thinking that my mail should be back the way i left it. Believe it or not, upon restart and opening the mail app, most of my mail only show headers and nothing in the message. This is a terrible bug and looking at the release notes of up-coming updates, it looks like Apple is not even addressing this issue! This is terrible really! Bad Apple :(

Did the last update fix any of these issues?

@Chris No. I will update the post if I hear of any fixes.

[…] people do not update there Mac to macOS Catalina. If they are upgraded again he put a solution on how to fix the Mail problem. However, you can use the time machine to get you into the previous mail version. Again if the time […]

Is there any hint how catalina mbox files can be imported to Mojave? I have 120GB of mbox on Catalina mail, went back to Mojave, but all mail mbox files have same problem "no valid mbox files were found" when I try to import to Mojave mail app.

@Miloš Despite the “.mbox” in the folder names, Apple Mail’s data store is not in mbox format. So if you are using the Import Mailboxes command, you should tell it to “Import data from” “Apple Mail” rather than “Files in mbox format.”

Hello,
I have lost all the content of an outbox mail of one of my gmail account when upgrading to Catalina.
I was in contact with Apple international support via telephone and they have confirmed me that this issue is known to them as a bug in Catalina, which will be fixed in the future.
I suppose that more and more applying to them would accelerate probably their speed.

@Michael Tsai Of course I tried "apple mail" option too but lots of errors in console with newer ending story of import. What about "Mail Library cannot be read because its format requires a newer version of macOS." is there solution to convince Mojave version to read Mail folder which was already in use by Catalina?
I am trying to solve 120GB emails which are present in Mail folder but not shown in Catalina, so I moved to Mojave and trying to get those emails in app...

@Miloš I had thought Mojave Mail would be able to import from Catalina Mail directly. If not, you could use my EagleFiler app to convert the files. Make a new EagleFiler library and then drag and drop all the Catalina account (UUID) folders inside the V7 folder into the EagleFiler window to import the whole mailbox hierarchy. This will convert all the mailboxes to mbox format. Then you can tell Mojave Mail to import the mbox files stored in the EagleFiler library.

I updated to Catalina yesterday. All of the folders for my iCloud email have gone missing. I also have a gmail account that appears to be working normally. Unfortunately, the iCloud account was being temporarily used for all of my business launch. My iPhone still has all of the folders and e-mail.

is the issue gone with 10.15.1 ?

Yes, would be great to get confirmation from Micheal on whether this is fixed or not with 10.15.1? Thanks

@abc @dfe I’ve heard that some Mail bugs are fixed in macOS 10.15.1, but not the ones mentioned in this post.

Mails for a whole year were wiped out and every time I try using time machine to retrieve them Mail crashes. There seems to be no way of getting them back and this is very disheartening.

I did a clean install of Catalina 10.15.1 today (Nov 1) and the bug is still not fixed.
All of my inbox messages (POP mail) from the past two years vanished sometime after I did the install. And for others, message headers are showing "no recipient" "no subject" and content is blank.

Would a clean install and a fresh sync of email from the server avoid these mail problems? Instead of trying to upgrade an existing email setup? Thanks.

The problem persists in 10.15.1. Almost 50% of my old emails have now disappeared (40Gb of info). Most of the emails only show headers and nothing in the message. When rebuilding the mailboxes, the emails just evaporate. I've looked into the finder structure and the emails are completely gone. I can only recover from backups, but many emails will be gone forever. Bad.

How can Apple release this with such a bug? It means you're taking a risk updating to Catalina, and one more serious than usual, because the data loss can propagate to your other devices.

It all seems odd - all that marketing for the new and great macOS, yet Apple must surely know that 'upgraders' are risking losing data! Unbelievable!

Might this blog thread make a note if and when the problem is resolved?

It is ridiculous that Apple would allow this to go ob for nearly a month. We are losing all corporate emails that have attachments (85%) as the email are blank once pulled onto our servers and personal computers. We do not store email on our server, they store locally and this a significant issue for our business.

We have worked on every possible fix and there is NO WORK AROUND to recover blank emails nor to stop the emails from converting to blank documents.

downloaded Catalina two days ago, went to check email and a blank email page opens without any listing of any emails. Mailboxes shows the various mailboxes but there is no information or display when you try them. Not sure where to turn for any fixes. My email is cloud based. Luckily email still shows on iPad mini and iPhone X S.

I think you guys have just helped me solve the big issue I have been trying to fix for the last three days. I upgraded to Catalina last week and have since lost 3 very important drafts in my Apple mail...they show up as drafts (which they are) and I can see the the preview with the first three lines of text in the drafts folder on my iphone Apple Mail App but when I open them, they are blank. None of these drafts show up in Apple Mail on my Mac, nor in Gmail. They simply never synced it appears. Sadly, one of the drafts was SUPER important to me, and now it appears corrupted and unsalveagable. I spent an hour on the phone with both Apple and Google support but now I think I understand the problem. I will hold on to the draft in in the draft box and hope that once there is a bug fix it comes back to life, but I am losing hope. I really need that draft. Any advice?

I experience the same problem. When I move or copy emails received from my outlook exchange account in another folder in Apple Mail, they appear to be totally empty, despite the fact that the sender, subject and date/hour appear correctly. You have to open the mails to realize the problem. The problem has appeared since the upgrade to Catalina. Other accounts (Gmail) are not affected by this problem. Only mails with attachments are affected. Other mails can be moved or copied without problem. Another problem is the non-synchronisation of the outlook exchange account. Mails read or deleted in Apple Mail do not change status on the Exchange server. As a consequence they appear to be unread or undeleted on my i-phone. This second problem seem to affect all emails in the outlook exchange account, with or without attachement. My solution (well not a real one, more a safety trick) as of today is to keep all mails from my outlook account in the inbox and postpone the cleaning until the bug fix is released.

This also happens with POP accounts in Apple Mail in High Sierra (10.13)(I don't use iCloud, so it's not just an iCloud problem with High Sierra). I first noticed it this summer: I couldn't find a message that I was fairly sure I had saved (all my saved messages are in a folder named Archive). A few weeks later, I looked for a different message that I KNEW I saved and that wasn't there either. When I looked at my Archive folder, it only had about 600 messages; there should be 2,000 messages. Restoring from Time Machine backups didn't work (every time I imported the TM mailboxes into Mail, the number of messages didn't change). The messages started disappearing around June 2019 (this is when the message counts in the TM mailboxes started going down).

What I finally did was use EagleFiler and imported every mailbox from every TM backup I had (yes, it took a lot of time, but I was able to restore every message, since I had TM backups going back to April). Now, I export my "Saved Messages" folder from Apple Mail to a .mbox file every few weeks and add it to my EagleFiler library, so I don't have to worry about losing messages.

THANK YOU, Michael, for creating EagleFiler, you saved my bacon.

If you really need to move a mail from the Inbox (from an Excchange account) to another Apple mail folder, you can copy the message on the Desktop, then open the message from the Desktop and copy it to the target folder. Doing it, the message is copied correctly. Do not forget to delete the message from the Inbox if you wanted to move it and not just to copy it. Obviously, such a procedure is far from being user friendly. At the very least, it shows to the Apple developers a step-by-step procedure that does not transform Exchange messages into blank messages. I leave it to them to make it automatic;-)

macOS 10.15.2 Beta 1 is out and Apple Mail is Version Mail 13.0 (3608.2). macOS 10.15.1 had Version 13.0 (3601.0.10). I don't know if anything has been fixed since I don't have any Mails in my Beta-VM but I wanted you to know.

Question: when does the data store upgrade happen - on installation of Catalina or on initial launch of Mail app? I installed Catalina a few weeks ago but happened to read this thread before opening Mail - wondering if the damage is already done, or if I can wait to open Mail once Apple issues a fix. (reading mail via iOS, fyi) Thanks!

@Tim On launching Mail.

[…] Mail trên macOS, đã phác thảo một số vấn đề đã được khắc phục trong một bài viết trên blog – ông khuyên nên giữ bản nâng cấp Catalina, nếu bạn dựa vào Apple Mail, cho […]

I made a clean install of Catalina (booting from usb stick containing 10.15.1) deleting Macintosh HD with disk-utility, then installing the operating system) and recovered my profiles via Time Machine. Locally stored emails and emails in pop accounts appeared blank except for the email header. I tried this several times. Then I made a clean install of Mojave and recovered my profile from the same time machine back up, and everything is fine. In other words, the problem is not a potentially corrupted back-up, but OS X 10.15.1
Each process about six hours, so I’m leaving it until fall 2020. CONGRATS APPLE!

Michael do you know if Apple is officially aware of the problem?

@John I’ve heard second-hand that they are. They haven’t responded to my bug report.

@downgrader What kind of mail accounts were you using that you were able to recover mail? I am looking specifically at the issue with Exchange mail.

Hi All,
I have been spending about 9 hours with an "APPLE expert" by the customer care service in Italy to try diagnose and fix the problems induced by Catalina on the mail client. No way, after every iteration it got worse. By now I feel like saying it looks very much related to the interaction with Exchange servers. Something got really screwed up. Back to the old story between Microsoft and Apple. I reported to the "expert" this is really something that should not happen; it makes no sense to release a new operating system without these very basic tests. I'm really disappointed.

@Massimo Was it an upgrade or a flattened install? So far it looks like it's all Catalina upgrades?

Michael, if I upgrade to Catalina and I don't have any interaction with Exchange Servers but only Gmail and Fastweb (an Italian server) accounts, do you think I could have problems?
Another question: if I no more use Mail but another mal client (i.e. Spark), could I avoid mail problems upgrading to Catalina?
Thank you.

Just to keep everyone informed: macOS 10.15.2 Beta 2 comes with a slightly fixed version of Mail.

10.15.1: Mail 13.0 (3601.0.10)
10.15.2 Beta 1: Mail 13.0 (3608.2)
10.15.2 Beta 2: Mail 13.0 (3608.40.2.2.1)

I don't know if anything has been fixed since I don't have any Mails in my Beta-VM but I wanted you to know.

@MANARA Yes, I have seen people have problems who are not using Exchange. Using another mail client would avoid them, however.

I can neither confirm nor deny that the "envelope index rebuild mail deletion bug" is *NOT FIXED* in any of the 10.15.2 betas yet.

So I can not confirm or deny that you *shouldn't expect it to be fixed in 10.15.2 at all* at this point because Apple said that a "lack of duplicate reports" is not moving this higher up the regression fix priority list.

I can neither confirm nor deny that we are not pleased about this by any means.

I *can* say -- that if you have tested this and you have reproduced the problem and you have not filed an official bug report about your results -- this is why it's not fixed yet. :-(

@Shriner Why didn’t they name Catalina GLOMAR?

Shriner, have you filed and official bug report yourself?

How will we know when upgrading to Catalina is safe? Has apple admitted the bug and given a way to track whne it will be fixed?

@Michael Apple has not publicly acknowledged it, but apparently they have replied to Shriner’s bug report. I will update this post if I hear that the bugs are fixed.

John Hart -- of course. During the 10.15.0 beta process (mid betas) we discovered this and filed it. We also pay for AppleCare Technical Services and filed it there, too. This is why we are most unhappy.

The only thing we haven't done is wrote Tim Cook directly. We are considering that. :-/

Im wondering if anyone can help me ...since upgrading to catalina , any time i begin to compose a message using mac mail but using my out bound webmail address , i receive this message below, it freezes my mail and I have to force quit.... the actual email server works when I log in through the website and sends the email zero
problem

The message “Re: ” could not be moved to the mailbox “WEBSITE Email” ...

Mail was unable to open this mailbox on the server “mail.lngphoto.com”.

The server returned the error: Mailbox does not exist, or must be subscribed to.

@Lisa Greenfield

At this point I would not trust anything Mac mail using Catalina. If you're able to revert to Mojave without any lost of data, that be the road I would take. It's shameful what Apple is doing here. Tempering with precious emails in that way is clearly unacceptable. It's almost impossible that Apple did not catch that issue while they were beta testing Catalina! Yet, they do not seemed to care!

I wonder just how widespread these issues might be given Apple's response to Shriner. I migrated two 2016 15" MacBook Pros to two new 2019 16"MacBook Pros that had Catalina 15.1 preinstalled last weekend without any apparent Apple Mail related issues. It's been 3-4 days since I did the migration to the new Macs and have done routine spot checks of my mail & attachments without finding any issues. I DID import all current mail (~26,000 emails on one machine) into EagleFiler, after confirming download of all attachments, as a precaution.

Note that neither new Mac did a mail "reimport" to my knowledge and I have not attempted to rebuild envelope indexes. I have three imap accounts (Gmail, iCloud, and one associated with a personal website) although the vast majority of mail is in the iCloud account. I also have an extensive "on my mac" folder hierarchy. I do not have an exchange account or utilize Outlook. So far, so good but I'm still spot checking and hanging on to the EagleFiler databases!

I'm fed up with this. I'm importing all my mails in EagleFiler. Hey, how can I change the font size in EagleFiler's Mails and Lists?

The way I'm importing is to work through a local dovecot that I've dockerized.

@eMaX There are settings for the various fonts in EagleFiler’s Preferences window.

It’s almost impossible that Apple did not catch that issue while they were beta testing Catalina! Yet, they do not seemed to care!

For whatever it’s worth, I’m not running into the issue.

My guess is this particular issue was non-trivial for Apple to reproduce at the time, and didn’t see quite enough users affected to make that high-priority enough to delay the entire release. (Which, it seems, wouldn’t have helped anyway, as they still haven’t been able to fix it by 10.15.2.)

Hi,

The problems seem not to be limited to the Upgrade. I still receive Emails with 10.15.1., can read them when in Inbox but when moved to a Folder and open it again, the content is missing. :-(

That can cause serious problems when the email is needed later for proof.

Frank

@Frank That’s correct. One problem is the missing messages during the upgrade or when the database needs to be rebuilt. But messages can also be lost or blanked when moving between mailboxes (either manually or via rules).

Hello,

Disabling Mail Preferences --> Rules appears to solve the issue of receiving seemingly random empty messages into my Exchange accounts (IMAP), which I had migrated from an earlier version of OS X.

Hope this is useful.

SP

I'm also hitting the problem with 10.15.1 and corporate Exchange email account.
I'm observing following problems:
- when I copy messages from Exchange Inbox to a local folder, the messages with attachments are not copied properly, only header is seen. It seems reliably reproducible. Copying messages with attachments from an IMAP Inbox works OK apparently.
- when message is moved from Exchange Inbox to a local folder by an Inbox rule, even messages without attachments are not moved properly.
- Mail.app is constantly showing "Moving Messages" with the numbers slowly jumping up. No idea what is it doing, and restarting Mail doesn't help, it starts counting from zero again.

I've contacted AppleCare, showed them the problem, they collected info from my machine, and now I'm awaiting a reply.

I also encountered problems with email after upgrading to Catalina. On the phone with Apple for hours, the guy telling me they'd had a lot of problems with this update, and they hadn't really tested it properly. I rolled back to Mojave. Now wondering whether they are ever going to address the issue. Any recommendations for an email client I can use instead of Mac Mail, where I can import all my emails and settings, anyone?

Yes, I can confirm that after weeks, I still experience *serious* Mail issues (made the mistake of installing Catalina from Day One and then hoped for updates, all installed). It's now "better" than at the beginning, but still totally bad.

In addition to the points mentioned previously (pasted below), I also have to manually delete everything in Mail.app (iPhone), so any spam I delete in Mail (MacBook), I must delete twice there plus once in the app…

from Frank:
"The problems seem not to be limited to the Upgrade. I still receive Emails with 10.15.1., can read them when in Inbox but when moved to a Folder and open it again, the content is missing."

from Robert H:
"I'm also hitting the problem with 10.15.1 and corporate Exchange email account.
I'm observing following problems:
- when I copy messages from Exchange Inbox to a local folder, the messages with attachments are not copied properly, only header is seen. It seems reliably reproducible. Copying messages with attachments from an IMAP Inbox works OK apparently.
- when message is moved from Exchange Inbox to a local folder by an Inbox rule, even messages without attachments are not moved properly.
- Mail.app is constantly showing "Moving Messages" with the numbers slowly jumping up. No idea what is it doing, and restarting Mail doesn't help, it starts counting from zero again."

Is there any indication that MacOS Catalina 10.15.2 Beta 3 will address the critical problem of lost emails bug(s)?
Like others I will finally migrate my 'archive' of emails to FileEagle librarys once I see the path to reconstructing what Catalina has destroyed.

First headway in ages: I saved my Mail folder (Library) and imported it in Mail on a different Macbook -- eh voilà, everything there, with folder structures AND attachments!

Kakis,

Can you be more specific? Did you get Catalina to work? or you copied the entire mail (V6) library and import to another mac NOT running Catalina?

@ John Hart: From Catalina to Catalina. But the from-Mail was completely screwed (see my first comment above). I went from my old MacBook Pro which was already on its last legs, had Catalina, and experienced all system updates for the past 6 years to the Library folder (Finder: press Option > Go > Library) and simply copied the Mail folder on an external drive. I then went to my brand-new MacBook Pro which I set up from scratch (NO migration), opened Mail.app there, and imported the Mail folder from the drive (File > Import Mailboxes: Apple Mail). Hey presto, took a while (220K emails) but it looks nice now. :-)

Kakis so there is no need for exporting Mailboxes in .mbox format? Will this technique work when going from Mojave to Catalina? I used to only copy Mail Library folder on older systems, but it stopped working few OS systems back. Are you saying I can copy my Library folder to external HD and import this very folder in Catalina and all my mailboxes, rules , etc will be copied?

Just got bit. After upgrading, all was fine. Then I tried fixing an issue with a smart mailbox displaying that there were messages in the mailbox when it was empty. Boom, hosed. My accounts are all POP (except iCloud and Gmail, which I don't really use but we're all forced to have). As others have said, only the large boxes are affected. I can't say how large they need to be, but my boxes with Amazon receipts, my wife's emails (going back to when we first met in 2002), and base inboxes full of unfiled messages are toast. If I go into Time Machine, they show up, but I'm not able to restore -- even from a TM disk I just started a few days ago. Diving into both the current V7 folder and ones from TM backups only nets me partial success. So far, the most recent message I can find in there are from three weeks ago. Both in TM and on Crashplan. Where is Mail storing more recent messages in the meantime? They're there somewhere. Also, new message since this happened are fine. So far.

@ Kakis: Did Mail rebuild the Envelope Index database? I can also copy my Library folder (V7) from my second, older Macbook and it looks and works fine. However, I'm not able to perform a single search. And as soon as I rebuild the database, all the data of folders containing +2k emails gets lost.

Can anyone else confirm a cutoff at around 2000 emails per folder?

I found a workaround for my system! After having bought a 2019 16" model last week, I upgraded my 2014 Macbook Pro from Mojave to Catalina and migrated all data to the 19" model. Mail worked fine for the 2014 model and was simultaneously heavily screwed up for the 2019 model as mentioned above - perhaps the one big difference being that after migration the latter entirely rebuilt its Mail database, whereas my old Macbook apparently didn't. I tested that a few times, deleting the current V7 folder (which would go from 6GB to "Import Mailboxes", and then selecting "Apple Mail" rather than only chosing particular mailboxes. After importing the intact V7 folder from my older Macbook this way, all messages showed up correctly and were searchable. The import does jumble up the order of folders and mailboxes, but that can easily be sorted out manually. I then imported signatures and rules and have a working Mail App without any data loss so far. Hope it stays that way and hope this is helpful for someone out there!

And btw, just to be safe I installed EagleFiler which does an excellent job in archiving all or any mailboxes and folders.

I don't know what to say other than what I wrote before (I'm not a computer person). As I said, I had thousands of emails, some folders also had more than 2k (to answer one query above). All I did was copy the Mail folder in my library -- but at that time, the old laptop was already in Catalina (so there weren't any V7 folders any longer). The main thing is that when I updated my old MacBook Pro to Catalina, all hell broke loose and Mail was rubbish with all attachments gone/not displaying -- and when I coped the Mail folder from that faulty laptop onto the new MacBook Pro (clean, no migration but with Catalina), everything was fine.

@Bernd -- there's no rhyme or reason about what mailboxes will have deleted mail as far as I can tell. The mailbox I sent to Apple has 1000 emlx files in it, but only 7 after the Envelope Index-deletion-reimport.

10.15.2 is out. Any news on the subject?

@Tasos: I'd love to know this too. Does 10.15.2 fix the issue?

10.15.2 does *NOT* fix this. If you have not filed this bug with Apple, what are you waiting for?

@Shriner, I am having the same issue with all of my POP and IMAP accounts except for my iCloud account and I have filed FB7490433 with Apple. I don't use gmail, just POP and IMAP mailboxes from my hosting provider.

I note that prior to my Catalina migration my Mail folder had 5,792,038,039 bytes (5.82 GB on disk) for 101,388 items and after running the Mail app after migration I had 2,871,906,047 bytes (2.8 GB on disk) for 37,633 items in that folder. All of the message envelope meta data appears to be there, but as I go back in time a few weeks on some folders, a few months on others, the message bodies are all blank.

I was able to import all of the mailboxes from the good backup into AirMail 3 (and into Mail Steward as backup) and all of the data was there.

I’m posting just to underscore a point that Shriner made above but that I don’t think has really been emphasized enough (both here and elsewhere with this issue): the issue ONLY affects local “on my Mac” mailboxes.

After careful preparation – 3 separate backups, and pruning and archiving (to mbox files) of my mail – I just upgraded to Catalina 10.15.2 yesterday. I was somewhat surprised when I crossed my fingers, held my breath, and launched Mail fo the first time....

It just launched. There was absolutely no updating or conversion of any of my mail. It was no different than launching Mail on Mojave.

The key which I see now, but wasn’t clear to me then: I only have IMAP (and Exchange) mail folders, with nothing stored locally on my Mac.

I realize for many people this potential bug is still a deal-breaker,but for anyone out there who I has been waiting to upgrade and either (A) like me, only has IMAP mail, or (B) can move things around so that they only have IMAP mail – then for those folks there is no reason not to upgrade now.

@Sam There are multiple issues. The one about upgrading and rebuilding mailboxes seems to only apply to On My Mac mailboxes. However, other users are having IMAP and Exchange messages disappear when moving them between mailboxes (even on 10.15.2). This doesn't seem to affect most users, but since we don’t know who will be affected I can’t recommend updating in general.

@Michael. Thanks. What you say makes sense.

It does make me wonder, though. It sounds like the "on my mac" problem has a clear source in Catalina: rebuilding the mail data store and/or the envelope index in Catalina Mail leads to corruption and a loss of messages. Shriner has found a reproducible bug here and reported it to Apple.

The IMAP and exchange problems don't have a clear source or explanation, and I wonder if they are even caused directly by Catalina. I know I've had Exchange problems in the past and they were almost always caused by a conflict between the huge variety of Exchange servers out there (including very old versions), and the interaction with them by the mail client.

Still, I'm going to watch my IMAP and Exchange folders carefully. I routinely move messages from an Exchange Inbox to a series of IMAP folders, so if the moving message bug is alive, I may still be exposed to it.

@Sam I think it’s Catalina because some of the people who reported these issues to me reverted to Mojave and the problem went away.

This is probably and edge-case question, but I only use IMAP with Fastmail. (My university uses Exchange, but I forward everything to Fastmail.) Does anyone know if the problem appears with IMAP on Fastmail?

The "On My Mac" Mail deletion bug? Still not fixed in 10.15.3 b1. We are besides ourselves here about this. Grr.

No, I rarely use "On My Mac. I store everything in Fastmail folders. I thought I read above that the deletion bug also appears with IMAP.

@Sherman The “moving a message to another mailbox instead deletes it” bug can happen with IMAP. I don’t know whether it’s limited to certain servers. I have definitely seen it with iCloud and Gmail. Not sure about FastMail.

With the update to 10.15.2 in Apple Mail I lost about 400.000 mails, which are still displayed with subject, sender and recipient but have no content anymore. Both in the preview and after opening the messages are "empty".

I noticed the whole thing after certain messages, which I normally access via search, were no longer displayed.

I then re-indexed the mailboxes via Onyx without success. Even more messages seem to have completely disappeared from the mailbox folders.

A recovery of the Library/Mail/V7 folder from a TimeMachine backup from 2 weeks ago was unsuccessful because 400,000 elements from the backup were "copied" back into the folder and Apple Mail subsequently indexed the mailboxes (also about 400,000 messages), but these are still all empty in the mail.

Does anyone also have this problem and a solution?

@Patzi. If you have a successful back up of your "on my mac" mail -- restore it to a different location (like your desktop). You should then be able to successful *import* the mail (File --> Import Mailboxes).

That works.

But please file a bug with Apple.

Great advice everybody - thank you!!

This is my issue:

I bought a new MacBook Pro 16". I use Apple Mail consisting of an iCloud address and 2 gmail accounts. I have close to 1.5 million e-mails. I had issues migrating from my older MacBook Pro 15" to this new 16". The older 15" used Mojave, the new one, Catalina. About 1/3 of my e-mails did not migrate to the new computer and many that did, only show headers - no body. Apple customer service really has no good answers. One idea was to upgrade the 15" to Catalina, then try to migrate. But I did try that before and after upgrading to Catalina, Mail froze during the set-up (kept on trying to download e-mail). It was fixed by reindexing and going back to Mojave, so I never got far enough to try a migration from a Catalina 15" to the Catalina 16".

Do you recommend reverting to Mojave on my 16" or trying some way to import the e-mails from the 15" via Time Capsule, external HD, a 3rd party box import program, etc.?

Thank you for bringing this critical information to the forum.

This really sucks. In my case I could still reproduce the problem under Catalina 10.15.2. Every time the reindixing is done more than half of my messages disappear or are only available without body.

@Shriner: thanks for the recommendation. I decided to wait with reimporting until this problem has been fixed by Apple.

Missing mail after Catalina update. Nearly 90,000 historic emails were lost, in my case, reducing the Mail folder size from 35.5GB to 2.7GB. Fortunately I have found an old backup with much of the historic messages. However, key messages remain missing including copies of customer invoices sent out by email. Destroying people's email archives has the potential to put companies, especially smaller folk, out of business. The problem is only obvious once one starts actively looking for those old messages. But, by then, it may be too late ... old backups have been replaced by new backups that lack the older messages. Even if one can recover messages from a backup, the time and effort ( and $$ ) involved in doing so is significant. Shame on you Apple. Protecting user's files/data has to be the #1 objective in any OS update.

How did we find out: we are missing our records of emails (the messages) sent out via Apple Mail!

Oops, emailed customer invoices.

Again -- if you have not *personally filed a bug report* -- you are not going to get this problem fixed any faster.

Apple needs to get more bug reports in on this ASAP.

If you use Time Machine you can recover your lost mail. Apple replaced the V6 folder with a V7 folder in Library/Mail. I have no idea how they screwed it up. Doesn't really matter at this point. We just want our mail back.

Open Time Machine and look in Library/Mail for the last backup with the V6 file. Restore it. This will put a copy in your current Library/Mail folder.

Take it out and put it on your desktop. Open Mail and select menu item "File/Import Mailboxes". Choose the V6 file on the desktop. This will put all of your mail up to that date in a folder called "import" in the Mail sidebar. You can drag and drop to selectively replace your lost mail. A bit tedious but better than losing everything.

Happened to me too. Drag and drop was losing attachments. I tried rebuild. Then it said it had to rebuild the index and import everything. 45 minutes later, my emails are massively corrupted. Lots with header only, no body. Many missing. Bunch of random stuff in "Recovered", mostly stuff that I had deleted. Worst problem with my Mac that I can remember in 20 years. Apple, please get on the ball!!

@Sherman — it was interesting to see your reference to FastMail, since that's what I use too. I'm actually delaying purchasing a new Mac because of this, since I'm doing alright on Mojave on my existing Mac, but I'm sure the new one will force a Catalina move. Since I only have FastMail and iCloud email, I'm thinking to just deconfigure iCloud email and go to a single IMAP account, which it appears from this thread shouldn't be a problem running on a single mailbox. I'm also thinking that if I really want to keep both active, maybe I should move to Outlook 2019 or 365, or to Thunderbird, or the FastMail web interface (which isn't great but is good enough), and get out of this quicksand entirely (since all I need to do at most is re-cache my email). It's insane that this is still an active problem by 10.15.3b1.

@dpz I have used both Postbox and MailMate (currently MailMate) since Catalina and have no problems. (FWIW, mileage varies on the Fastmail web interface. I rarely use it but when I do I find it fast and uncluttered.)

Just opened a "feedback" on the IMAP-move-to-other-account-issue. Feedback-Assistent lists "no similar feedback" :(
Will reproduce the on-my-mac-issue soon an open a feedback for this too.

> Feedback-Assistent lists "no similar feedback" :(

I wonder how "similar feedback" works. Could be a manual process, could be a very primitive keyword-based algorithm, could be machine learning. Either way, probably a lot of false positives/negatives.

I updated from Mojave to Catalina beta a while back and my mail was fine.
When I got a new Mac and reinstalled from a TM Backup via recovery the mailboxes where reimported and that is when I lost about 10.000 mail that were in a local mailbox.
After trying many things I installed a TM backup from before Catalina and grabbed the mailbox file of the local mailbox from a TM backup from the old mac running on Catalina.
I added this to the library of my reinstalled Mojave, deleted the envelope files and after reimport in Mojave I got my mails back from my TM backup that was run unter Catalina.

I will strick to Mojave until this is sorted out.

Best,
L

I just had a thought that I could run Mojave in a VMware Fusion VM for my Apple Mail until they fix their issues. Doubt I'll bother doing that, though.

Although my issue has nothing to do with Catalina, it does touch on the Time Machine
Mailbox recovery strategies mentioned in this thread. My issue was that I deleted a good
number of emails from my IMAP Inbox before archiving them, and then had to recover them
from Time Machine. At the time of the backup, I was running High Sierra on my machine Rockweed.
Now I am running Mojave. (I have learned to be very conservative in moving to new versions of OSX)
- recover V5 folder from Icarus, which contains my mail from last august:
/Volumes/Icarus/Backups.backupdb/Rockweed/2019-08-26-004934/Saltmarsh/Users/kellogg/Library/Mail/V5
copy to cernboxKellogg/Mail_recovery_2019-aug (note that files in cernboxKellogg are mirrored via the cloud
and available on all my machines)
- with Mail, Import to Import-2 on Rockweed as Apple Mail (since import as mbox claims it can find no mbox)
but when I try to move messages to another local mailbox (2017), I fail, with crazy message that it could not find
Mailbox "Happy New Year from Amherst", which is the title of one of the emails I am trying to move
notice that the mailbox structure is complex, and also that this backup was made under High Sierra, not Mojave
- try the import on maccmskel4, which is running Sierra
for INBOX-mac.mbox I can import it without problems, and I can move all the mails without attachments to 2017
I can view and download the attachments, but cannot move them, getting the warning that the body cannot be loaded
For INBOX-cern.mbox the behavior is exactly as on rockweed: I cannot move any email after importing it
getting the warning that the body cannot be loaded
rebuilding INBOX-cern does not help
- conclusion at this point: OSX versions are not the primary issue (even tried on ElCapitan with same results)
- EagleFiler import succeeds with copy on cernboxKellogg: cernboxKellogg/Mail_recovery_2019-aug/INBOX-cern.mbox (18300 records)
in the EagleFiler window, I see the emails listed very much like in Mail
the # column gives the number of attachments - if I click on the attachment, the entire message (including attachments)
opens in Apple Mail
- Since the Eagle_Library/Files/INBOX-cern flat file obviously contains all the attachments, in-line,
try importing it using Mail/Import as mBox to Import-4
get 18175 messages and a warning that some could not be imported - there are no attachments
Now I get somewhere:
- rename Import-2/INBOX-cern.mbox to 2017-cern and then drag 2017-cern to top level of On my Mac Mailboxes - success!
all the attachments remain visible (this procedure works on all my machines)
- I now delete all the non-2017 mails, and repeat the entire procedure for INBOX-mac and for 2018 and 2019
- only slight issue remaining is that I am not able to find a way of merging the -cern and -mac folders

Conclusions:

- something poisons the structure of the Mailboxes recovered from Time Machine
in such a way that neither EagleFiler nor Mail can handle them properly,
particularly their attachments
- the mail and attachments are recoverable, but their Mailboxes are extremely fragile
Basically: look, but don't touch

@Richard From what you’ve written above, it sounds like EagleFiler handled the attachments properly. But feel free to contact me if you want to discuss further. It’s possible that Time Machine made an accurate backup but that not all of the attachments were cached locally, so they weren’t there to back up. You can avoid that problem in the future by having a rule that moves or copies the messages to an “On My Mac” mailbox or by periodically archiving the messages to EagleFiler.

So, I posted a note to apple communities asking 1. if Apple had acknowledged the bug, and 2. if not, how would I know when it is fixed?

I got this masterpiece of stupidity in response: Apple will neither acknowledge or discuss fixing a bug until it is fixed (assuming there is a bug.)

Sigh.

I've been fighting with this problem for a while and just did an interesting experiment. I sent myself two messages with attachments. One I downloaded to a local folder by dragging it to that folder. The message and attachment survived. The second I moved to the local folder via the Message - Move to drop-down menu. That message only retained its header (no contents or attachment). Has this been observed by others, and does it point to the problem?

Thanks.

Mark

@Mark I’m not sure what the pattern is. I have seen cases where all the different methods (drag and drop, menu, rule, AppleScript) worked and also cases where all of them damaged the message or made it disappear entirely.

OK. I would love to hear if what Doug did on Oct 21 fixed his issues. I updated to Catalina on Nov 30, by Dec 2, I was instructed by the mail client that my mailboxes had issues and before I could finish reading the little pop up window mail crashed. On relaunch mail rebuilt my mail boxes and deleted random emails. A lot of sent emails and all current emails between November 27 and Nov 30. The current ones were obvious, but the others only came to light when I was trying to recover some procedures I had emailed to someone back in May 2019. Those were just gone. After discovering that I started digging aound on the internet and on my machine. Discovered elmx files on timemachine existed, but completely gone on my computer. Called Apple and got no where fast. Spent that last two weekends rebuilding a machine with Mojave and migrating data between it and my current Catalina install... I have recovered from TimeMachine directly to the mojave install and that fails. I have restored the mail and my Library files to an external hard drive for safe keeping. Each time I recover my email from that hard drive and try to put it onto the mojave install the same emails disappear when Mail is relaunched. I am convinced it is because the mail server is an Exchange server, but the sent emails were sent from my computer mail client, not the server... it really doesn't make sense to me that the emails are just deleted off my hard drive by the mail client. I have the email in timemachine and on my external drive, but have been unsuccessful in getting them loaded back into Catalina or Mojave mail. Apple hasn't been able to offer a solution so I am curious if what Doug did with the manual import actually worked.

@Erin As far as I’m aware, the deletion at Mail launch is not related to Exchange. It also happens for locally stored messages.

@Erin -- from everything I've been able to show Apple, a manual "File --> Import Mailboxes" from a backup should be fine.

[…] We’re still hesitant to recommend upgrading to Catalina, particularly if you use Mail, since reports of email data loss continue. Luckily, downgrading a new Mac from Catalina to 10.14 Mojave is possible for models other than the […]

I had a huge mail archive. i normally use POP3 (53Gb). After the upgrade (yesterday) i had visualisation (and search) issue. Rebuilding the mailbox.... 6Gb. Lost 47Gb of mail. I now trying to recovery from back-up.

@Massimo -- please file a bug with Apple about this -- we need more bug reports to get action!

Something good just happened, but I have no idea why or if happened for others. A massive number (all?) of my formerly blank emails I had downloaded to my computer from my campus server now have their attachments back! The other good news, which makes me think this state of affairs may last, is that when I delete a message from the server using my computer it also deletes from what my iPhone sees on the server. I wonder if there may have been a fix at the Exchange side of thing. Anyway, I'm happy for now and will (cautiously) downloads the many messages I've been leaving on the server for safekeeping.

Cheers.

Mark

This is stunningly awful. I haven't filed a bug report because I never actually had a problem -- I abandoned Mail when I first heard reports. I'm now living quite happily in MailMate.

[…] Si eres de esos que aún no ha sido lobotomizados lo suficiente para abandonar el uso del correo electrónico, en macOS Catalina 💩 encontrarás un sinfín de problemas relacionados con Mail.app. […]

Where was this obvious issue before last Wed. Went from 15.1 to 15.2 and all of my saved folders with emails for various reasons. On top of this the conversion, it was totally screwed up the interfaces with the provider.

Went to apple support, the first two efforts were absolutely worthless [ very un apple support]. Hours spent bouncing between Provider and Apple. Apple wanted a reboot on Time Machine. That was and still is a mixed bag of bad issues. I will never let anyone lower than tier 2 to go though a time machine reboot.

Finally got a senior adviser. After 45 mins he said this needed engineering. He ran a number of screen shots and zip programs. we short of got thing working only going forward. It is still not stable. My guess we will have to back before 15.2 and go from to square one.

What really bothers me the most, Apple knows there is an issue and refuses to acknowledge it. Then have me running all over the place, causing more issues.

Hello,
Same problem here with Catalina 10.15.2.
Unaware of all these problem with Apple Mail, and because I stupidly erased my mails (pop) while trying to change Pref. Syst./Internet Accounts (that's already strange: deleting an account will delete all related pop messages in Mail…), I decided to used my last Time Machine Backup.
Not working.
Ok, nothing to fear, I restore completely my MacBook Pro 16" with the last backup. Everything is working fine, but Apple Mail: wants to import messages, ok, and that's done, all messages are blank, just headers (and the lthe corresponding folder in Preferences/Mail is shrunk to 5% of its initial size: 54 GB to 2.5 GB...).
Not Happy.

that's already strange: deleting an account will delete all related pop messages in Mail

Yeah, that is so bad UX and it's been that way in Mail forever. Wonder how many out there have been bitten by this? There are so many more reasons to delete an account that just wanting to delete every message stored. Maybe you switch e-mail provider and just want to keep everything as an archive.

I can't talk about any public beta two of 10.15.3 to see if it fixes the issue with deleting the Envelope Index files/reimporting/reindexing "On My Mac" mail.

I would *like* to tell you to try that when it comes out to see if it fixes this problem on your end.

I would like to tell you it fixed my problems.

I really would.

All I can tell you -- it's a good day today. That I'm sure about.

@Shriner, can you not tell us whether or not this fix in the 10.15.3 beta simply repairs the index problem, or does it actually restore the lost e-mails?

I presume that it is not the latter, since, as others have noted, there is more free disk space after the Catalina upgrade, suggesting the actual deletion of e-mails and attachments, and not just an indexing issue.

I lost thousands of e-mails, with Gbs of additional free space after the upgrade.

I am anticipating having to go back to a TM backup from just prior to the Catalina upgrade, and importing e-mails, folder by folder, into the plethora of folder trees that I have, and then dealing with the duplication of e-mails between those imported, and the subset of e-mails that have, so far, survived the upgrade.

BTW, I did file *two* bug reports with Apple on this, since October.

Thank you for not telling us about this...

@Marc -- it will not restore e-mails if Mail.app went through a process (for whatever reason) of reindexing your existing "On My Mac" mail. Those messages are gone for good and would need to be restored from a backup if that happened.

If you have a backup, you should be able to find the backed-up "On My Mac" folder in ~/Library/Mail/V6 and restore that and then "import mailboxes" successfully.

@Shriner, thanks for the clarification/confirmation. It is as I expected, given the observations of increased free disk space post upgrade.

All of the e-mails that I lost were in On My Mac folders. I generally do not keep e-mails in the cloud, with the exception of a few folders for portable and timely access on my iPhone when away from my MBP.

As a second question, I have noted some AppleScript tools online that purport to identify and delete duplicate e-mails. For example:

http://jollyroger.kicks-ass.org/software/

or:

https://www.macupdate.com/app/mac/53180/remove-duplicate-messages

Any recommendations for such a tool based upon any experience?

I would guess that such a tool will be helpful to others in similar situations.

I will await the release of the 10.15.3 stable update and see what Apple has to finally say about this, before proceeding. This process of importing from TM and removing duplicate e-mails will be tedious and time consuming, thus only want to do it once. :-)

Thanks!

@Shriner, thanks for the weather report, very happy to hear about it being a good day as I'm about to have to commit to Catalina, and I just couldn't pull the trigger on moving to another client yet.

Can you also (not) tell us about any (alleged) resolution of the problem with moving messages between mailboxes? Is it a good day today as well?

@dpz -- I only personally move messages from my Gmail account to "on my mac" mailbox locations (that one way -- not the reverse).

So I'm not sure what that problem is directly nor what I'd be looking for to test against.

I would recommend watching for the public beta of 10.15.3 b2 and testing it?

@Marc -- no experience with "duplicate e-mail detectors" here, either. But, what I'd probably say would conceptually work would be that if you know the date you upgraded to Catalina, you could restore your "on my mac" mail from right before that upgrade, import it (which still works fine), and then just purge mail from the leftover "On My Mac" mailboxes from that date and before? That's what I did here. No duplicates remain doing that.

I am seeing some Exchange-related misbehaviour: all of a sudden, about 48 hours, my Catalina iMac stopped being able to fetch or synchronise email with my institute's Exchange server. New messages appear if I rebuild the mail boxes.

It was working absolutely fine before.

This occurs on both a wired iMac at my office and my home MBP on broadband. iOS mail and the Outlook web client work fine.

Moving lots of email out of the Inbox (from about 10k down to 1.7k) doesn't help.

Other accounts (Gmail, icloud) work fine on all machines.

This does not appear to be Catalina specific, however. Refs:
https://apple.stackexchange.com/questions/350372/apple-mail-always-need
https://discussions.apple.com/thread/8086520
https://discussions.apple.com/thread/8152487

Is it, nonetheless, related? Any ideas?

Hi, I just tested the actual Dev-Beta (10.15.3 beta 2) with Mail 13.0 (3608.60.0.2.2) and I can no longer reproduce the mail-loss after rebuilding the envelope-index.
Can someone else confirm this?

I guess i was both sad and happy to find this article. I removed my envelope index files and one of my inbox lost about 110,000 emails. I have a super duper backup where the emails are intact which i can boot to, however i cant export the mailbox from apple mail it just crashes.

Can i replace my broken mail setup with the version from the super duper backup. However i have looked and i cant figure out how to do this. Can i just copy the V7 folder in its entirety ?

Secondly i did actually keep my envelope files before the issue. Can i restore them to fix the it ?

Also like another user here i actually deleted all my pop3 emails from my server as a way to debug the issue :(

Any help would be great. Alex

@Mathew -- yes, it seems fixed here as well.

@Alex -- what you should be able to do from your backup would be to find the "On My Mac" mail folders on your backup drive and "File --> Import Mailboxes" from Mail. The act of *manually importing them* -- is fine. Just don't delete the Envelope Index files as a method to "reimport" them until 10.15.3 is out

Apple fixing it or not, having lost and hence having had to restore from backup 200k mails, I no longer trust Apple worth my mails. I’ve moved away from Apple Mail. My company uses outlook anyway. I’ve set up DEVONthink Pro so that it imports my mails from Outlook. All my mail act on rules I’ve implemented in DEVONthink and I’ve linked them all together with a simple Apple script that I call from my DEVONthink toolbar. Effectively, I’ve now a very simple Auto File procedure that takes my Mails out of outlook (or Apple Mail, but as I said, I’m not using it anymore) and into my folder structure in DEVONthink.

Search and retrieval is anyway far superior in DEVONthink, so there’s no point going back even if Apple finally fixed it. They never really acknowledged the problem in the first place, hence why should I trust them.

@shriner thanks for the advice so how do i select specific mailboxes using this method or do i have to basically import the whole V7 Folder and deal with the duplicates ?

@emax i too was thinking of going down the devonthink route to have a backup system aswell. Though other options i found that had decent reviews were

https://www.mothsoftware.com/index.php
https://mailsteward.com/

OK,

So I updated to Catalina ... had loads of problems and had to re-format fusion drive - a common problem apparently.

I re-insatalled from time machine

I could see about 100,000 inbox emails in my mail - but they have no content. About the same number in Sent emails.

After many conversations and trying a few things with Apple - been advised that the only way is to import the emails from the Library / Mail folder for Time Machine manually - which I have now done.

I now have double the amount of emails - the naff ones that can't be read in the main Inbox at the top ... and the same emails that can be read in (Import under 'On My Mac')

I have discussed this further with Apple and they have insisted it is not an apple problem - but I want to be able to get the emails for the bottom import folder into the correct Inbox folders at the top .... as searching emails is really difficult - having to dig down into folders and searching in each folder.

To make things worse I have about 6 or 7 different business emails which each have their own email folder in the top section and now in the bottom section .... But if they are run the correct inbox at the top - you can easily search all at once.

Any advice would be greatly appreciated...

MailMate, folks. Try it. Superior configurability (with a little coding in BBEdit), outstanding search. Easily exports to DevonThink or EagleFiler if you want that. It can be made to fit your workflow like a glove. Keyboard navigation is excellent and also highly configurable.

Apple continues to work on Apple Mail in the new developer beat of Catalina (10.15.3 beta 3) with Mail 13.0 (3608.60.0.2.6).
Mail-loss after rebuilding the envelope-index is still fixed.

Yes, I am hearing multiple reports that the macOS 10.15.3 beta fixes the rebuilding bug, but some users are still seeing messages that disappear when moved between mailboxes or that end up duplicated instead of moved.

Will mail 13 be also available under Mojave when it comes out?
I cannot go up to Catalina and my email suffers from 'eAlzheimer' since upgrade from High Sierra and despite multiple envelope rebuilds
thanks for your feedback

RE >>> Apple continues to work on Apple Mail in the new developer beat of Catalina (10.15.3 beta 3) with Mail 13.0 (3608.60.0.2.6).
Mail-loss after rebuilding the envelope-index is still fixed. >>>

is there an estimated time for this ?

as mentioned above I have all emails - but they are all in an import folder below - not in my inbox - which makes searching for emails a pain.

I may try and drag them all over to inbox - but if there is a fix - is it worth waiting ?

@Anonymous: no, Apple Mail version 13 is tied to macOS Catalina
@TREV: only Apple knows how much time they need till they release 10.15.3 as 'stable' to the public. But I would wait till at least some people here can confirm that the now known bugs are fixed.

So -- for the purposes of the bug that I know is fixed -- the problem was with "On My Mac" mail being deleted if "Envelope Index" files were deleted (or Mail.app threw up the dialog that the Mail index needed to be rebuilt from the first launch of Mail after upgrading to Catalina). This is fixed in the current 10.15.3 betas. I'm confident in stating that *specific* bug -- seems fixed.

I've not heard (here) about a way to replicate mail being deleted when "moved between mailboxes". I'm happy to try to replicate it on my end with some test accounts and open a case, but I'd need more detail like -- what kind of mailboxes -- and in what direction -- is the mail being moved? Exchange? Gmail? Moving between "IMAP" mailboxes? Pulling messages down from an "Inbox" to an "On my mac" mailbox? Moving message *back up* to an IMAP mailbox from an "On My Mac" mailbox? Is this all POP (for which I'm surprised people are still using that protocol these days...)? Exactly what is the setup where you see that specific problem?

To answer a couple of specific questions above (and I don't work for Apple -- just a sys admin at a large University so I see a lot of stuff...)

Mail is not going to be updated in Mojave (old OSes only get security fixes once a new OS comes out -- Applications are never majorly updated on N-1 or N-2 OS versions.

If you are trying to restore deleted "On My Mac" mail from a TimeMachine backup, I would recommend restoring the entire backed up ~/Library/Mail folder to another folder on your desktop. After you do that, you can look inside the folder for the mail structure. "On My Mac" mail would be within it's own folder with a long string name like: 0D214F03-E8D2-4DAD-8F4F-6285203B4475 or something and you can look inside that to identify the mail folders within. That would be the folder you would "File --> Import Mailboxes" only. No need to reimport *everything* to solve that specific problem (as you'll get duplicates of stuff you don't need.)

And nobody knows when 10.15.3 will be released apart from Apple. When the betas stop coming, you'll know it's close. *In general* (but not always) when beta build numbers end with letters closer to "a" -- then it's close to release. Beta 3 is 19D75a -- so it *may* be closer to release than I would have guessed considering there have been only 3 betas. Could be this week? Could be next week? Could be mid-Feb. Nobody knows.

@Shriner I’ve not been able to replicate the problems moving messages, either. But according to the reports I’ve received, it happens with IMAP (including Gmail) and Exchange mailboxes, moving both into and out of the inbox, and moving within the same account as well as to a different account.

According to this article:

https://appleinsider.com/articles/20/01/28/apple-releases-macos-10153-ios-1331-ipados-1331-tvos-1331-watchos-612

macOS 10.15.3 is released, albeit, it is not yet showing up for me as an update.

I am not seeing any explicit indications of a resolution of the mail bug in the coverage, at least not yet.

[…] https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/ […]

Sorry for the second rapid post here, but the 10.15.3 update is now showing for me, with the following details only:

This update:
• Optimizes gamma handling of low gray levels on Pro Display XDR for SDR workflows when using macOS
• Improves multi-stream video editing performance for HEVC and H.264 encoded 4K video on the 16-inch MacBook Pro (2019)

along with the following security updates:

https://support.apple.com/en-us/HT210919

So, still no official acknowledgement from Apple of the mail issue...

[…] Tsai, who has been tracking this issue, […]

I just updated my test-vm to 10.15.3 (stable), but interestingly the Mail Version is 13.0 (3608.60.0.2.5), wich is one build older than what shipped with 10.15.3-beta3 last friday.

Can anyone here confirm if is possible to take a backed up V7 folder and replace the broken V7 folder in its entirety ?

I understand anything not the old backup would be lost.

Thanks. Alex

@Shriner Here is my experience with loss of messages.

My account is on Gmail / IMAP. I can send all the settings if needed but the most relevant may be that the "Archive Mailbox" is [GMail]/All Mail. I have numerous rules for filing messages. This basic configuration was stable for many years until I upgraded to Catalina.

When a new message comes in, it is just fine no matter what mailbox it has been filed into. If the message was not filed by a rule (i.e. remained in the Inbox), moving the message to a different folder works fine. But if the message was already filed by a rule, as soon as it is moved to a different mailbox, the content is lost. Mail displays the sender and subject, but nothing else. "View Raw Source" displays an empty window.

John

P.S. This includes deleting the message. If it had been filed somewhere else, deletion loses the message. Undo does not help in any of these circumstances.

@Alex That should work.

I imported all emails in 10.5.3, but moving emails to a different folder in the "On my Mac" is only partially working, some messages move fine, while others only show empty content. Rebuilding the mailbox make them disappear completely. I've explored the V6 folder, and it seems that all "xxxxxx.emlx" files are imported/moved fine, but the "xxxxxx.partial.emlx" files are sometimes not imported and are the ones that disappear when moved from one folder to another. What's the difference between the emlx and the partial.emlx? Is this what is happening to other users?

@Pablo The partial ones are IMAP and Exchange messages that contain attachments (which are stored separately from the .partial.emlx file). This seems to be a different/new bug from the ones I’ve seen moving messages.

Hello everyone.

Since Apple doesn't seem to fix the problem, I have two questions and it would be wonderful if anyone could help:

1. Does Spark Mail have the any issue at all?

2. How can I migrate to Spark. I wonder if I cancel my Mail® accounts it will delete / free the space occupied by the mail messages 🤔...?

Thanks in advance,

Problem with mail.
With MAIL in the dock, I used to click on the icon once and it would open. Ever since I upgraded to Catalina and then the latest version I have to double+click the icon to open MAIL. Anyone else have this problem? I talked to Apple and the advisor said they were aware of this problem.
OK,,,,,, Fix it Apple!

@michael tsai — Regarding your reply to @Alex.
@shriner — Regarding your request for details.

My experience is as follows. My V7 was created on a brand new 16” MBPro running 10.5.2, by using the 16” ’s Migration Assistant feature and using Time Machine containing my V4 (Sierra) as the Source. The resulting V7 did not contain the same number of .emlx files contained in the V4. When I initiated the 16” Catalina Mail App, I was missing text of the emails in both the upper level “Inbox” Mailbox (POP3) and the lower level On My Mac individually named Mailboxes.

@michaeltsai—Thus, I believe the V7 folder is not reliably created from the prior version V# folder, and any future “backed up” V7 folder will itself be broken. Thus, @Alex will not be able to use a backed up v7 folder (which I believe will be broken itself) to fix his current broken v7 folder.

@shriner— do you have any experience similar to mine?

@HoustonMJ I took the question to be whether it’s possible to replace all the app data at once, which it is. Of course, you won’t end up with any data that wasn’t in the original backup. If you use macOS 10.15.3, you shouldn’t end up with fewer .emlx files.

Hi All,

In light of the last few comments, I want to verify that there appears to be lingering issues with Apple mail subsequent to the 10.15.3 update.

That is, not all data loss related Apple Mail bugs/issues that have been observed and reported are resolved by this update, especially those with the "On My Mac" local storage folders, which is where the vast majority of my mail is stored. I keep only current Inbox messages or those requiring expedient attention in a "cloud" based IMAP Inbox folder tree, whether that be a personal me.com account, or my business account, which is via Office 365.

If correct, which would be concerning, I am hesitant to begin the tedious and lengthy process of restoring hundreds of folders/sub-folders from a pre-Catalina TM backup, after the 10.15.3 update, which I have installed.

@Mathew's comment above might open the door to suggest that Apple did not ship the beta3 version of Apple Mail in the 10.15.3 release, which might perhaps explain lingering issues, and that perhaps there will be a future 10.15.4 update to Catalina that may yet resolve other mail issues.

I am also, for the first time in years, contemplating moving to a different e-mail client application on both macOS and iOS, given these issues, especially if they are not fully resolved, and there has been no ownership by Apple of them.

I already have an Office 365 account for other business related reasons, as noted above, so moving to Outlook as a macOS/iOS client for both personal and business accounts would be one pathway, albeit, I have seen data loss reports in past years regarding Outlook. I would, of course, still need to restore the pre-Catalina e-mails from TM, but only really want to do that once, whichever pathway relative to an e-mail client I take.

Any comments would be appreciated.

@Marc Yes, there are still data loss issues when moving messages. If you have messages in On My Mac mailboxes and don’t move them, I would expect them to be fine.

I am using Spamsieve, which requires an On My Mac mailbox “Spam” to operate and move messages around. Does this mean Spamsieve is not compatible with 10.15 currently?

@Dave SpamSieve can use any mailbox that you want. Most SpamSieve users are already on 10.15, and for 99+% of them there are no issues. If there is an issue with your setup, it should be immediately obvious when you test it out with a copy of a message, e.g. you drag and drop a message and it disappears or Mail rules fail to move messages. In that case, some people have found that Catalina Mail is better at moving messages to a mailbox on the same server as the inbox rather than to another account or a local mailbox. Some other workarounds are described here. SpamSieve can also be set not to move messages when training, if you happen to be one of the people affected by the move-messages bug.

@Michael, thank for your comments.

Unfortunately, I cannot be assured that messages in the local On My Mac folder tree would not have to be moved from one folder to another. Thus, raising integrity concerns, if that is still an issue.

So, it would seem that one path is waiting for some indication from folks here, that perhaps an upcoming 10.15.4 beta release does make progress on resolving these issues, or does not.

The alternative would be, if either the above does not show signs of resolution, or Apple simply takes too long to resolve this (for some definition of "too long"), to consider moving to a different client application as noted.

@Marc FWIW, I don’t think I’ve seen any reports of problems moving messages between two mailboxes that are both On My Mac.

@Michael, perhaps I misunderstood your prior comment to me regarding having e-mails in the On My Mac folders and not moving them, in which case they would be ok. The logical inference would be that moving e-mails between folders might not be ok, albeit, perhaps that is not confirmed, per your follow up comment.

If there is any reasonable question, at this point, in light of the underlying issues, of lingering data loss, I am not prepared to begin the process of restoring/importing the old TM backup e-mails. As they say, the only good thing about banging your head against the wall, is that it feels so good when you stop.

I can't be selective in which operations may or may not be safe from a data integrity perspective, and honestly, shame on Apple for engendering those concerns and doubts.

@Marc Moving between IMAP and On My Mac, I have reports of problems. Moving within On My Mac, I have not, but that doesn’t mean that there are none. As I wrote to Dave, it seems like some people are affected (frequent problems) and others are not (none), and so if you figure out that you’re in the latter group you’re probably OK. But this is all guesswork because Apple has the code and bug reports and isn’t talking (even about the bug that seems to be fixed).

@Michael, my comment above describes issues with moving messages between two "On My Mac" mail folders.

@John Thanks for clarifying. So the On My Mac mailboxes are also dangerous, but only if the message had been previously filed by a rule?

@Michael, yes exactly. If the message has NOT been filed by a rule, I can move it across "On My Mac" mailboxes at will without any problems. But if it has been filed by a rule, it is lost permanently if it is moved from the filed location.

The rules themselves are very simple "Move Message to Mailbox:" ...

@John If this is easily reproducible for you, please file a feedback with Apple.

My general practice is that *everything* gets moved to an IMAP server once it arrives—most incoming mail comes in through an Exchange server at work, and I manually folder it to a different IMAP server after I've reviewed it... so moving between an Exchange inbox and a different set of IMAP folders.

From reading the reports above it wasn't clear it me if others have reported problems with this style of setup; most seem to concern "On My Mac" foldering.

Is anyone using a similar setup to me, and have you experienced problems with Catalina?

@John, for clarification, are the rules that you are referring to, rules that you created in Apple Mail, or rules created on the gmail web site?

I ask because I have a number of rules on iCloud.com, that move me.com mails to IMAP sub-folders of the Inbox for e-mails lists or similar differentiation for organization. I did this on iCloud.com so that I get the same behavior on my iPhone as I do on my Mac, relative to folder movement.

I only have a few rules in Apple Mail, that based upon keywords or similar, color the subject line, but they do not move e-mails to a different folder. I don't see this, of course, on my iPhone.

@John

I have filed this bug with Apple in November : FB7449043. Have no response to it whatsover (hasn't even been marked as a dupe so it appears that it has gone in to their ignore bin despite the DATALOSS in the subject line.

Here is basically the text of the bug:

DATALOSS: Messages automatically moved by rule on receipt lose data on subsequent moves

Mail Version 13.0 (3601.0.10). On macOS 10.15.1 (19B88)

When a message is received and automatically moved by a message rule to a local (On My Mac) mailbox, subsequent moves of the message (by drag and drop or by menu move options) result in the total loss of message data.

Steps to reproduce:

1. Create a rule to move a message to a local mailbox (Mailbox 1)
2. Receive a message and confirm that message has been automatically moved to the local mailbox (Mailbox 1) and has its contents (headers/body etc)
3. Move the message (via drag and drop or the menu move actions) to different mailbox (Mailbox 2)
4. View the message in Mailbox 2.

Expected result: The message in Mailbox 2 has it headers and body.
Actual result: The message in Mailbox 2 is completely devoid on content -- viewing raw source shows no data at all.

Note:
• Reproducible 100% of time
• No plugins are installed/active.
• Occurs regardless of type of receiving server (IMAP, EWS, Gmail,iCloud).
• Occurs also if the rule copies (rather than moves) the message.
• Only occurs if destination mailbox of rule (ie Mailbox 1) is local -- if Mailbox 1 is on a server, subsequent moves are OK.
• Mailbox 2 can be any mailbox -- (ie may be local or on a a server).
• Messages received to the inbox and subsequently filed by manual application of rules are NOT affected.
• Messages received to the inbox and subsequently moved by manual moves of message are NOT affected.

I did more research at the time and determined that the message moved by the rule was lacking a specific bit setting in the message flags when compared to messages moved manually.
If I manually edited the sqlite database for Mail to turn on this bit for a message. (as a plugin developer I do a lot of spelunking), the message was moved without dataloss. So in my estimation, the rule application is not properly setting a specific bit on the message that needs to be set.

I communicated this observation in my bug report -- which if correct should lead to a quick fix. But that was 2 1/2 months ago.

@John

Just to followup -- I cannot seem to reproduce this bug now I just update to 10.15.3 so it may be that Apple did indeed fix it.

@Scott Thanks for this information! I think this is just one aspect of the problem, though. Some users are seeing messages disappear when a rule moves them from the inbox (on a server) to another server mailbox. And I have also seen reports of messages deleted when dragged and dropped between mailboxes on two different servers. I have not been able to reproduce these myself, though.

Update (2020-02-18): I have received a report from a customer with macOS 10.15.3 that seems to indicate your bug is not fully fixed, either.

I'm currently running Mojave and very reluctant to upgrade to Catalina because of the mail issues. However, I have installed Catalina on a VMware Fusion VM to do some testing and have found no issues with my Fastmail IMAP account. All my rules are server side so emails get moved into the appropriate folder at Fastmail's end. I don't use local mailboxes on my Mac - just the ones in my Fastmail account. I've done some manual moves of a large selection of emails back and forth between a number of different mailboxes and all seems OK.

Just to confirm - this is a brand new install of Catalina setting email up from scratch, not an upgrade from a previous version of MacOS.

@Marc, these are Apple Mail rules.

@Michael, I did file feedback with Apple.

@Scott, I am pleased to see that you filed such a detailed report to Apple, and that you have observed a possible fix in 10.15.3.

I will update this weekend and confirm whether I also see the issue has been fixed.

@John, thanks for the clarification.

I filed a third feedback report, since the fall, on this with Apple today, also providing a link to this post and comments. Hopefully somebody there will take the time to read the comments here.

I confirm that the issue I reported above has been fixed as of 10.15.3.

I just read the blog and im not sure whether this is related to my issue but whenever i move some of my mail to its relavant folder since im using the mail app, i lose all its content and all i have is its subject title. Can you halp solvong this because i dont understand how it started and how to stop it.

I have a friend who is currently using MacOS Mojave 10.14.6 She has IMAP mailboxes, but also quite a bit of older messages stored in folders in On My Mac. Not using any rules, and not very likely to be moving messages between On My Mac folders. No longer has any POP3 accounts. Is an update (not clean install) 10.15.3 release still a risky proposition, simply because she has a lot of old messages in On My Mac? It looks from what I see here that Apple has made some progress, but there are still some remaining issues, especially with moving messages in On My Mac.

Here are two possible solutions to the problems we are all having, which I've copied from the Apple discussion forums. If anybody's brave enough to try them, I'd be mightily interested to know the outcome:

I have solved the Catalina email loss problem and solved the email indexing problem! YEAHHHHHH!
I had over 500,000 emails, with more than 30 POP accounts and 15 IMAP accounts. I had over 200 folders and sub-folders. A disaster. I had updated from Mojave to Catalina and I realized the problem when I went to do a search without results. I had tried them all, but the emails disappeared. I contacted Apple's specialist support, but they didn't know how to answer, although many users complained about the problem. I will illustrate the solution;

1) obviously I had a manual backup of the Mac library on Mojave. The important thing is to have a backup where the emails previously worked. regardless of whether catalina or mojave. In catalina, only the name of the V6 changes to V7 but 99% is the same. So you can recover and run everything, even from a mojave backup to catalina

2) I restarted the Mac in safe mode (press shift button)

3) I entered the mac library and replaced all the contents of the ACCOUNT folder (library> account) with the content saved in the backup

4) I entered in Preferences (library> account) and I replaced these two files with those saved in the same position in backup hard disk:
com.apple.mail-shared.plist
com.apple.MailMigratorService.plis

5) I went into SyncedPreferences (Library> SyncedPreferences) and I replaced these two files (if there are none it does nothing) with those saved in the backup:
com.apple.mail-com.apple.mail.vipsenders.plist
com.apple.mail.plist

6) I went into Containers (Library> Containers) and replaced the folder saved in the backup:
com.apple.mail

7) Now go into Library> Mail> V7 and delete everything. From the backup disk, first I copy and paste the Maildata folder (before Catalina automatically rebuilds it - it is the first folder to copy) then I copy and paste from the backup disk, all the other folders in blocks of 10 at a time (so that if Catalina blocks me saying "this folder / files already exists, I can start over from the last point).

8) When I have finished replicating all of the V7's internal from the external hard drive to Catalina, I restart the mac in recovery mode (Option-Command-R or Shift-Option-Command-R to start up from macOS Recovery over the Internet ) and do an SOS on both Macintosh HD discs. Then reboot

9) Now with the active connection, I open MAIL and wait for all emails to be indexed. When Mail re-opens all the e-mails and folder trees, I will have a time of about 30 -40 minutes, where all the e-mails, the body of the e-mails can be seen correctly before catalina deletes everything. The first thing I do is to scroll (even quickly) all the folders to the end, so that the indexing of Core-Spotlight can load the contents-preview messages.

10) Attention! At this stage, indexing and email search still does not work. Whatever you are looking for, it still does not work.

11) I make sure there are no folders with no body messages, so I click on each folder, sub-folder, sent mail, individual sent mail folders, one by one, (I did it for 200 folders, pop, imap, pec , sent, archived, icloud etc) and click on> MAIL> BOX> REBUILD (be careful you have to do it one by one and wait for it to finish rebuilding).

12) I suggest storing the number of messages (eg 14.234) so ​​that when it rebuilds, it returns 14,234 messages.

13) Do it folder by folder and you will see how everything will stabilize and also spotlight and core-spotlight, it will give you the results.

If you have solved it, send me a like on my linkedin profile: https://it.linkedin.com/in/massi-vigilante-741837147

I summarize:
1) boot safe mode and connect disk backup
2) replace the folder library> accounts, the files in preferences, the files in syncedpreferences, the files in containers from the hard from the external hard disk in users library
3) delete the contents of the Mail> V7 and starting from the Maildata folder, copy the contents of the v7 from the backup to the internal library
4) restart in internet recovery mode, do sos
5) open mail after restarting the PC wait for the emails to be indexed
6) scroll all the folders with the mouse to load the content
7) folder after folder, reconstruct (mail> box> reconstruct)
8) wait for it to rebuild and then move to the next folder
Reply Helpful (1) Thread reply - more options

Several people have reported lost or blank e-mail messages after updating macOS to Catalina. I lost 50,000 messages. Here's my solution, which requires a back-up of messages made before the Catalina update.

Restore the folder V6 from User/Library/Mail to the desktop.
Open Mail if it is not already open.
Create a new mailbox where you will put recovered messages so that you can review them before storing them with other messages.
Select the command "File" - "Import Mailboxes  … ", then select "Apple Mail" as the format.
Look in the folder V6 on your desktop for a mailbox that you want to restore, select it, then click "Continue". The import will start. A message will tell you when the import is done.
After the messages have been imported, the messages will be in a new mailbox called "Import". Within that folder, they will be either in the simple folder structure that you would expect or in a multi-level folder hierarchy. If they are in a simple folder structure, click on a message in the message list, press command-A to select all the messages, and drag them to the folder that you created in step 3. If they are in a multi-level hierarchy, option-click on the triangle next to the folder called "Import" to expand the hierarchy. Command-click on all folders named "Messages". Click on one of the messages in the message list. Press command-A to select all messages in the message list. Click and drag the messages to the new mailbox that you created in step 3.
Progress in moving the messages will be shown at the bottom of the folder list. After all the messages have been moved, delete the "Import" folder by right-clicking on it and selecting "Delete Mailbox  … "
Repeat steps 3 to 7 for other mailboxes.

Methods that did not work for me:
Re-building mailboxes.
Deleting envelope files in folder V7 and forcing Mail to re-index V7 messages.
Temporarily removing folder V7 and forcing Mail to re-convert messages from folder V6.
Restoring a Time Machine back-up from within Mail. Mail crashes.

So it looks like the beta cycle for 10.15.4 has started:

https://appleinsider.com/articles/20/02/05/apple-releases-first-betas-for-macos-10154-ios-134-ipados-134-tvos-134-watchos-62

Just sayin... ;-)

I had the same problem with content missing after moving messages between local/offline mailboxes. Issues seems to be resolved after upgrading to 10.15.3 yesterday. Messages that lost content are still missing that content, but now moving messages doesn't cause the body to disappear any longer.

Boy there is a big jump. I just installed the new macOS-Beta (10.15.4 beta 1) and Apple Mail is now Version 13.4 (3608.80.13.2.1).
macOS 10.15.3 (stable) has Version is 13.0 (3608.60.0.2.5) so here could come a cure for a lot of bugs.

Sadly I am not able to reproduce any bugs in my VM but at least we can see Apple will ship a greatly improved version of Mail in 10.15.4.

Based on recent history, I would not be so quick to assume "greatly changed" implies "greatly improved".

This is the longest time that this thread has been quiet since it started :-)

Any feedback fom macOS 10.15.4 public beta 1?

[…] https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/ […]

Just a short Info macOS 10.15.4 dev-beta 2 comes with Apple Mail Version 13.4 (3608.80.19)

Maybe relevant for others. My issue was that my local mailboxes where empty after importing from a mail export of my previous machine.
I could finally solve this by importing each local-mail subfolder (first level) separately.
It seems in my case that importing the main local-mailbox did not lead to recursively import subfolders and was only creating them empty.
when importing each one by one their recursive content was imported.

Many, many THANKS to John Biggins for this fantastic information that really helped me. I lost about 75000 emails when upgrading from Mojave (10.14.6) to Catalina (10.15.3). My previous macbook pro had to be taken into the macstore for repair for the 4th time (2 logic boards replaced, the top case was replaced twice and the touch bar had an intermittent connection) and rather than fix my machine, they gave me a new 16” machine, which I love, but I was forced into upgrading to Catalina. :-/
Just wanted to make a few notes to help make this clearer for others that may be going through this. I have noted below according to the steps in the first part of his message.

1) i had a time machine backup of 10.14.6
3) when he mentions ACCOUNTS, he is talking about USER account, so what made more sense to me was this: Macintosh HD > Users > YOUR USER ACCOUNT > Library - you don’t need to replace all of the content from this folder - only what is relevant to Mail
4) Macintosh HD > Users > YOUR USER ACCOUNT > Library > Preferences If you dont’ have these files, don’t worry about it
com.apple.mail-shared.plist
com.apple.MailMigratorService.plis
5) Macintosh HD > Users > YOUR USER ACCOUNT > Library > SyncedPreferences If you dont’ have these files, don’t worry about it
com.apple.mail-com.apple.mail.vipsenders.plist
com.apple.mail.plist
6) Macintosh HD > Users > YOUR USER ACCOUNT > Library > Containersl
7) if you’re coming from Mojave, you will copy from V6 to V7 in Catalina I copied the Maildata folder and then everything else at the same time. took about 10 minutes to copy
8) I wasn’t sure what he meant about SOS, so I just ran First Aid disk repair - on both the Macintosh HD and Macintosh HD - Data disks
9) I didn’t understand what was meant about Catalina “deleting everything” and luckily didn’t run into this.
11) On my machine, nothing really happens when you hit Rebuild - the emails from that folder might blink. on the folders that had thousands of emails, if I clicked on an email, and then another email, and then selected the first one again, the sender would disappear and the body of the email would disappear. this means that it is rebuilding. this will all come back. Also, the rebuild command in the menu may grey out. when it’s ready again, it will appear as selectable again
12) I opened mail on a back up computer and checked folder by folder to confirm that all of the emails were, in fact, there.

Hi there

My issue is not losing data but before I upgraded to Catalina, I set up read receipts on my emails. Now I cannot delete it. I have been through terminal and done delete apple.mail.com UserHeaders but that does not work. Any suggestions?

Again a short Info: macOS 10.15.4 dev-beta 3 comes with Apple Mail Version 13.4 (3608.80.23), so a slight bump this time.

Can anyone that is running 10.15.4 beta 2 and 3, care to not comment on the behavior that they are not observing relative to the bugs? :-)

Thanks!

Tried with 10.15.4 beta 1 nothing changes... I found (almost) all my mail on my old MacMini12 but lot of them are only header. I dont know what to do anything could lead to permanent data loss :(

I also today found Mail is only giving me Headers in some emails and on other the full Header and message.
Cant figure what to do I pad and iPhone emails are all ok.
Any ideas anyone.

Today I created a Smart Mailbox to view selected messages that were stored in a mailbox in On My Mac.

I then manually moved the messages from the Smart Mailbox to a new mailbox in On My Mac (and, in case it matters, also created new Rules for that mailbox).

When I opened the messages in the new mailbox, they had lost their content. The headers are there but the body of the messages is blank.

When I exported the new mailbox, then imported it back into Mail, the content returned. (The status of the messages, such as read/unread, replied to, etc., is lost though.)

So, apparently the content is only hidden, and maybe someone can figure out how to display it again without exporting and importing.

I forgot to add that I am running 10.15.3.

@DavidB That’s good to hear, but for the other cases I’ve seen the content really was gone, with the files in Mail’s data store corrupted.

@Michael Tsai,

Yes -- I know my case may not be the same as others, although how would you know the files were corrupted?

@DavidB Because EagleFiler reported errors importing them, and so I checked in a text editor and found that they were indeed messed up.

Migrated to Catalina and have had the usual problems outlined above. Still have the e-mails on hard drive so imported them, but then found out that I had lost folders so tried Time Machine, but then Mail crashes and cannot use the backup. However, now have most e-mails back, so just crawling around to see what I had lost in the upgrade...one thing I've gained with Catalina, though, is a load of flashy applications that I don't need and don't want...such as 'News' and 'Stocks'...

@CTom It crashes if you invoke Time Machine from within Mail, but you can still use Time Machine from Finder to restore mail from a backup.

If I perform a fresh Catalina install and pull down mail from Exchange, am I going to have issues with mail disappearing when they move folders?

I move mail all the time, usually via an AppleScript I wrote years ago. I assume that no matter how mail is moved (by hand, via script, by local/server rules, SpamSieve mark-as-good), the problem exists?

@Bob The only way to know for sure is to try it. Most people are not affected, but some are. The method of moving the messages doesn’t seem to matter.

I've turned apple mail to offline. I still use exchange. I'm using ostbox for email and mailsteward to back up (long time user, need to give eaglefiler the time it deserves).

Does this mean it's reasonably safe for me to upgrade to catalina? Still not sure I'm gonna do it, but I am curious.

@msdisme Yes, that should be safe as long as you update to 10.15.3 or later, so that you get the fix for the bug with locally stored messages.

Just an FYI that 10.15.4 release is now available for upgrading.

@Marc Unfortunately, the messages-disappear-when-dragged bug does not seem to be fixed in macOS 10.15.4.

@Michael, I am seeing a variety of "flaky" behavior still, especially when viewing messages that have been moved between IMAP server folders on my MacBook Pro, in contrast to viewing the same messages on my iPhone, which I also upgraded today.

Synchronization issues abound in the IMAP Inbox and sub-folders that are on the servers. I have had other message content issues with moving mail from IMAP server folders to local On My Mac folders.

This is getting disheartening, in all honesty, given how long it has been going on.

I am about ready to give up on Apple Mail, and migrate to Outlook. I am finding myself, more and more frequently, needing to go back to e-mails from a while ago, that disappeared from Apple Mail local folders during the Catalina upgrade. I have yet to attempt re-importing the old mail from a TM backup, given how time consuming a process that will be, which makes accessing those older e-mails a pain.

Any thoughts from your end that might alter my views?

Thanks for maintaining this post.

@Marc

I concur with Marc, thank you for maintaining this post Michael.

My institution gave up on supporting Mac Mail because of this same issue. I still use Mail, but am spending more time on chasing missing attachments than on my research. Using Outlook and Mail simultaneously is a recipe for madness, not productivity. Are people bailingand moving to Outlook in the short term? ...is that even possible from Mac Mail? ...is there some .mbox > .pst converter out there that actually works?

@Bathead, I have not yet done a lot of research, but here is one approach:

https://medium.com/@johnsonanna0444/want-to-migrate-apple-mail-to-outlook-for-mac-this-is-how-professionals-do-it-6464cf7150a8

It seems to me that another option, presuming that you have the emails that you want to migrate in local Apple Mail folders, is to move them to IMAP folders on the server for the relevant account, for example, iCloud. This presumes that the emails and any attachments are intact.

Once they are in the IMAP folder on the server, you can then access them in Outlook, and then move them to local folders in Outlook.

Bear in mind, that at least for an iCloud account, you would need to create app specific passwords for security to use third party apps, like Outlook, presuming that you have 2FA enabled.

> Using Outlook and Mail simultaneously is a recipe for madness, not productivity.

For whatever that's worth, I use Mail for Mac, Outlook for Mac, Outlook for Windows (on two separate machines), and Outlook for iOS simultaneously on the same company account. Doesn't seem to cause issues.

While it's gotten better over the years, there are still things about Outlook for Mac that just don't feel right, and also things compared to the Windows version that are missing. For reading, I think I generally prefer Mail (and have been lucky, I guess?); for writing, there are some compatibility issues with the way Mail sends formatted mails, so I avoid that in official communication.

> there are still things about Outlook for Mac that just don't feel right

You mean like having Undo not undo your most recent action but instead randomly pick some other of your past N actions, often so far back in history that you can't see what actually was undone?

It should be fine to use Outlook and Mail together. There are also plenty of other Mac mail clients: MailMate, GyazMail, Airmail, Thunderbird, Postbox.

What bugs are still unpatched?

Can someone give me detailed Information how I can recreate the bug of corrupting massages when moving between accounts/mailboxes? I would like to try and recreate this on my VM.

@Mathew For some users, dragging and dropping a message from one mailbox to another (e.g. from the inbox to another mailbox on the same account or from an On My Mac mailbox to a server mailbox) makes the message disappear. And the same thing happens for other methods of moving messages (rules, AppleScript). I have multiple reports of this still happening on macOS 10.15.4. Moving to an On My Mac mailbox seems to work better.

Has anyone verified if the message move bug still exists in the final version of 10.15.4?

I have been wanting to move from MacOS Mojave to Catalina. Part of my e-mail history is a large group of messages that date from when I was using POP e-mail - "On My Mac". I currently use iMAP exclusively, and have I have multiple IMAP folders.

Michael said above: Yes, that should be safe as long as you update to 10.15.3 or later, so that you get the fix for the bug with locally stored messages.

If I update from Mojave 10.14.6 to Catalina 10.15.4 AND I don't attempt to move messages between folders on my Mac, am I likely to still have problems? I currently use an iPad Pro when receiving, sending or moving mail to folders.

@Gordon Obviously, I can’t guarantee anything, but I think that would be safe. Make backups, anyway.

@Gordon - note that I haven't read that the problem is moving messages between folders, per se. The problem is moving messages between mailboxes (i.e., accounts). Others should correct me if I'm misinterpreting that.

I'm setting up a VM to play with Postbox and Thunderbird. I'm done with Catalina Mail.

@dpz The problem affects moving messages between two mailboxes (folders) in the same account.

Thanks @Michael. I was interpreting "mailboxes" as servers, since I use the term "folders" when I mean folders. This is far worse. Clearly not a priority for Apple.

I tried all of the mail programs that @Michael recommended above and have chosen MailMate. The most important thing to me is to have "Smart Mailboxes" that include several email accounts. I also needed a flexible rules engine. Thanks @Michael for your recommendations. I am sad that Apple clearly doesn't consider this a problem. I have used Apple Mail for many years.

FWIW, I just filed what is my third or fourth bug report on this issue.

I would urge others to do the same, so that Apple is aware of this issue and gets feedback from multiple sources.

I included a link to this thread in my report.

I wanted to add a bit of info to this thread. I have not personally updated to Catalina, due solely to this bug—I just can't risk losing old email messages—but I've been monitoring this thread since October to see how things are progressing.

Thursday evening I decided to do something I've never done before, write to Tim Cook: "Tim, I’m sure you and the rest of the Apple senior leadership are slammed dealing with the COVID-19 outbreak. But I’m writing due to an ongoing, and very serious, bug in Catalina that doesn’t seem to be getting engineering attention. Catalina, even as of the 10.15.4 release this week, has a disastrous Apple Mail bug that can result in permanent loss of messages stored in remote servers and locally. Here, for example, is a discussion on a seasoned Apple developer blog about the problems that have been ongoing since October 2019 when Catalina was released: [link to this thread]. I hope you can make sure engineering is aware of the issue and is dealing with it, because it’s a show stopper for many of us."

I was shocked to get a call the very next day from a member of the Apple Corporate Executive Relations team who spoke to me about the problem and promised to look into it with engineering, and get back to me today. Today he called back and let me know that he couldn't share anything specific but indicated that now that the issue had been raised with Tim, Engineering would have to look into it with high priority and report back to Tim about its status. He said they normally would have put me in contact with someone from engineering to work through diagnosing the problem, but since I'm not on Catalina yet that's of course not an option. I made sure he was aware that multiple bug reports had been filed against this issue and he said they would find these as they look into the problem.

So I'm not sure what this means, of course, or whether it'll lead to any action. But I figure it can't hurt, and I was impressed by the speedy and high-touch response I got.

@Keith, wow, that’s great!

I was coming here to post that Apple has released the 10.15.5 betas to devs, and am wrestling with waiting to give one more stable release a try or move to Outlook.

Your post may give me the basis for waiting.

Thanks!

Sitting here looking at the update notification for at least the 100th time, I wondered if there had been any movement on this issue. Quite surprised to find that it's still a thing. I have an IMAP account with hundreds of thousands (million(s)?) of messages as I rarely delete things. It's just too useful and easy to be able to go back in time.

I wouldn't want to lose that. In fact I really need to find a piece of software / script to take an image of it in an open, encrypted format.

Glad to hear the issue seems to have been successfully escalated by you, Keith. This needs to get fixed before the next major release. Please keep us up to date - at least some of us are listening.

*Clicks "Remind me tomorrow" again*

> In fact I really need to find a piece of software / script to take an image of it in an open, encrypted format.

I think Michael Tsai might have one? (Not sure how open the format is.)

@Sören Yes, EagleFiler uses the standard mbox format, optionally encrypted using Apple’s .sparsebundle format.

And it's Postbox FTW after ruminating on Postbox vs Thunderbird for about a week. Bye Felicia.

Thanks for the all the guidance here!

Keith,

I also really appreciate your making that effort to get this bug some attention, as I am in the exact same situation that you are. Please do keep us posted!

Frank

https://support.apple.com/guide/mail/if-emails-arent-displayed-or-look-strange-mlhlp1159/13.0/mac/10.15

Does "display mail as plain text" make blanked emails readable?

Btw: if some here has Tim Cook on the phone: in previous version of OSX you could sort each folder in finder differently and the shared folder was usable, while today (10.14 and 10.15) a new file added to the shared folder is paradoxically not accessible from other profiles on my same iMac.

@de roth No, you can even see from the file on disk that the content has been deleted.

@de roth, to confirm what @Michael stated above, I lost around 30 gigabytes (yes, gigabytes) of mail and attachments on my HD after upgrading to Catalina. This is not just an indexing or formatting issue. The content is actually gone.

Thanks - Keith Edwards for your work on this contacting Tim Cook. Hopefully there'll be some resolution to this at some point.

So I have been testing macOS 10.15.4 for about a week now.
I have one POP3 account an one IMAP account set up in Apple Mail. Both running on a Kerio Connect server.
I can't find any misbehavior so far. Moving messages from POP or IMAP inbox to local (On my mac) mailboxes and between them works. Tried it by drag an drop by mouse, by TouchBar and with mail-rules. No mails lost, no content corrupted. And I have moved quite a bit, way over 100 mails.

So can anyone give more specific input on the open bugs? Maybe it is no longer an Apple Mail bug but for example a Exchange our Outlook-Mail-Server bug?

Kind regards, Mathew

@Mathew The message moving bug seems to only affect a small percentage of users, but for them it is frequent. So if you haven’t noticed any problems, you are probably one of the lucky ones. For the unlucky ones, it affects a variety of servers (iCloud, Gmail, Yahoo, various IMAP, Exchange), so I don’t think it’s due to a server bug.

@Mathew and @Michael, the other point I might make, is that if this was a server bug and not application specific bugs, you would reasonably expect to observe the problematic behavior in other mail applications (e.g. Outlook, Thunderbird, Postbox, etc.) connected to those same problematic servers.

That would include Apple Mail on pre-Catalina versions of macOS.

I don't think that we have seen such reports here, to my recollection. Thus, the common denominator would seem to specifically be Apple Mail on Catalina.

@Mathew
I made a clean install of 10.15.1 and restored my profiles from my TimeMachine Backup (as of 10.14.6). All downloaded emails (pop3 and those manually copied from my IMAP account to my local mailboxes were blanked. On both OS X profiles. I tried two times.
I am to lazy and too pessimistic to try it out once more with 10.15.4. Anyone did?

FYI:

Apple has released a supplemental update for 10.15.4. Nothing in the release notes about Mail.

https://9to5mac.com/2020/04/08/apple-releases-macos-10-15-4-supplemental-update/

Wanted to confirm that there was no change in the Apple Mail version with the 10.15.4 supplemental update.

It remains at 13.4 (3608.80.23.2.2)

All I know is that I have interminable, unpredictable, never-ending problems with Apple mail, using Mojave 10.14.6 (and iPhone iOS 13.x).

IMAP accounts/gmail in particular seem particularly obstinate to staying connected; continually, repeatedly asking for password, security alerts, etc. particularly on the iMac.

No matter what I do and how many times I respond: YES that was me! and continually re-enter password through safari, update / reconfirm alt or recovery email / phone, it keeps looping back and continually requiring the password to gmail accounts.

I used Howard Oakley's Silent Knight to install recent updates that OS X wouldn't/couldn't despite system prefs being configured to install security updates -- not sure, but I think that's when it started doing this whole repeat asking for passwords in apple mail for all IMAP/gmail accounts.

Really really really frustrating and annoying -- I can't figure out how to fix. What a freeking waste of my time on this repeating, resurfacing bug.

Thanks, if anyone has the fix, please do advise.

I am naive enough to think that surely Apple knows of the issues with Mail on Catalina and will send an update with correction. Is that naive? I too have gotten as far as the notice to read more as it brings in my messages. "Continue" is greyed out. Surely?

Same issue. Transfer of email between folders blanks out the body content and attachments. Random. Frustrating to the point of giving up on Apple Mail. Rebuilding mailboxes also results in lost content.

Has anyone used Ultdata for Mac to recover lost email data after a Catalina update? Tenorshare.com think it works... https://www.tenorshare.com/mac-os/mail-problems-in-catalina-and-fixes.html

@john Well, that page seems to be promoting their own app. I am skeptical because some of the other information on that page makes it look like they have no idea what they’re talking about.

@John, my initial read is that the information provided there is at best incomplete, and at worst, misleading and may even make the situation with Mail worse. It suggests a lack of technical understanding by the author.

I say that because several of the methods presented presume that the underlying data files are still present on the local drive. I know from personal experience, and others have posted similar information here, that the mail files and attachments are literally deleted from the local hard drive as a result of the Catalina mail bugs. I literally lost over 30 Gb of e-mail related data, as after the Catalina upgrade, I had 30 Gb of additional free space on my hard drive. If the mail data files were still physically present, and it was just a matter of recovering those files, or rebuilding the indexes to those files, that would not be the case. That reinforces the notion that unless you have a Time Machine or other backup of your Mail data files from just before the Catalina upgrade, you will likely have no way to recover the lost e-mails. Their application will be of no help in that scenario.

In addition, their recommendation for rebuilding the mailboxes may/will actually make the situation worse, resulting in the deletion of additional mails that may have survived the initial Catalina upgrade process.

Bottom line, I would not use it.

Thanks for the advice. I'll stay well clear :)

So...I have it on good authority that the current build for Apple Mail under 10.15.5 Beta 3 is still version 13.4, build number 3608.80.23.2.2.

No change from the currently released version of Mail included with 10.15.4, with the supplemental update from April 8.

Thus, there should be no expectation of any additional bug fixes once 10.15.5 comes out, unless there is some last minute update to Mail, before it is released.

Not good news...

So does all this mean I shouldn't upgrade to Catalina because if I do, I'll likely lose a ton of Apple Mail email content ?

@Philip, FWIW, if it was me, I would not upgrade yet, presuming that you are using Apple Mail and not another mail client, of which there are numerous options, some of which have been noted in the past here.

I would also be sure that you are diligent with frequent backups, Time Machine and otherwise.

I am still holding out hope that Apple will resolve the remaining Mail issues soon, but if they do not, within the timeline of one or two more incremental updates, I am going to move to Outlook, as I already pay for an Office/MS 365 subscription for business use. It gets increasingly frustrating when I need to go back to past e-mails that have been lost to the upgrade and take time to locate them in a backup from prior to the upgrade. I am not willing to risk restoring those e-mails back to Apple Mail via an import, for fear of losing them yet again.

I have already started to "play around" with Outlook on my MBP and iPhone, just to get used to the pros and cons of making the shift.

If you "need" to upgrade to Catalina in the near future, for some reason, I would give serious consideration to moving to a different mail client before doing so.

Hi,

if I would remove the synced email I have (eg.g gmail) from mail and then upgrade to catalina and then set it up again (i.e. sync to server) would I then be ok?

If I also have an archived mailfolder, can I make a copy of that before the update and the reimport it to mail in catalina after the update?

If that would work it could be a way forward. Hope for a good answer on this.

Best regards,

Rob

@Philip Please see above.

@Rob The importing/upgrading bug has been fixed, so that should “just work” now if you update directly to macOS 10.15.4. The message moving bug has not been fixed, and there’s no way to work around it, as far as I know, if you’re one of the ones affected.

Continuing problem - mail I send from my Exchange account via Apple Mail does not sync to Exchange server. Rebuilding the account causes the email to disappear from the Apple Mail Sent folder. Deleting and reinstalling the account works temporarily. Going to have to move to Microsoft Outlook.

FYI: No change in the Apple Mail version/build for 10.15.5 Beta 4.

Hi Marc,

Am also considering switching over to Outlook. If I download all my emails from the server (going back years) do you think they will survive the update to Catalina?

Thanks,
John

@John

Nothing is ever guaranteed, but if you have all of your e-mails stored on an IMAP server and keep them there, as opposed to using any local storage, and sync them to a new e-mail client (e.g. Outlook), they should be ok.

If you intend to then move those e-mails from the server to local storage in Outlook, I have not done that yet on my end, but in theory, those should not be affected by the upgrade to Catalina, since the bugs appear to be within Apple Mail in isolation. I won't do that myself until I am fully ready to move to Outlook and abandon Apple Mail. For me, that is a more complicated process, as I will need to first import mail from a pre-Catalina Apple Mail backup, and then go through the migration of those folders to Outlook, while checking for possible duplicates of e-mails in Apple Mail, since some small proportion of the pre-Catalina mails survived the upgrade.

I would be sure that you have backups (more than one) of your e-mails as they exist prior to the switch, just to have a fall back position, if needed.

Be aware that there are some tradeoffs with respect to iCloud accounts when going with Outlook, based upon my preliminary experience. This is not a full list, but some highlights:

1. Outlook's calendar on macOS does not fully support iCloud calendar. You can "subscribe" to the iCloud calendar URL, in a read-only mode, but cannot create new appointments. You would either need to stay with using Apple Calendar, or consider migrating all of your calendar data to Outlook and then using that moving forward.

2. Curiously, the iOS version of Outlook does appear to support iCloud calendar in read/write mode, which is different behavior than on macOS.

3. Similar issues with Apple's Contacts in terms of read/write access, so you may want to consider switching to Outlook's contact system.

4. Presuming that you are using an iOS device, consider moving to Outlook there as well, which is what I will do, if I make the move on my Mac.

5. There are some general behavioral differences, but if perhaps you are used to using Outlook for work, then you should be comfortable.

Hope that helps.

Thanks, Mark.

I have all my emails on servers and download them to POP accounts on my Mac. I can download them all from the server to Outlook. It won't, of course, download my 'sent' emails or arrange them into folders as they are in Mac Mail.

Seems like an awful lot of bother.

Why on earth can't Apple sort this out?

I wonder if Keith Edwards has got anywhere after his email to Tim Cook..?

@John:

I haven't heard anything else after the original follow-up, although to be fair the person I spoke with said I wouldn't get any more info unless engineering needed to contact me about the issue.

Very disappointing to see that this doesn't appear to be addressed in the latest beta.

Keith,

Would you be willing to reach back out to your contact person anyway, despite what he told you? It certainly couldn't hurt, and it would keep this on his radar screen. I would certainly appreciate it if you did that.

Also, I think I'm going to do exactly what you did - write to Tim Cook and express my serious concerns about this and why nothing has happened.

Frank

I wonder if anyone can answer me this. The only emails I *really* need to keep for legal/tax purposes are the last 5 yrs online receipts. If I lose them I'm fu**ed, if the tax authorities investigate me.
If I highlight them in Mail and drag them all into a folder and then drag that folder to an external hard drive, after upgrading to Catalina, will I be able to open those emails from the external hard drive?

@john That would be a great use for EagleFiler, as it will make sure that the full message content (all attachments) are saved and that the data doesn’t get damaged during long-term storage.

If you want to do it with Mail, the best way is to drag and drop the messages directly to a folder on your hard drive, which will save them as .eml files. Dragging them to another mailbox in Mail first is bad because (even not on Catalina) it can lose attachments (and delete them from the server). Make sure that you test-open the .eml files to check that all the attachments are there; Mail won’t warn you if they’re missing.

For reasons that I don’t understand, when using drag and drop, Mail will sometimes save the files in .emlx format instead .eml, which is not good. If that happens, try again, or drag the messages individually. I do not recommend using the Export Mailbox command, as it generates malformed mbox files.

Thanks, Michael.

Have since spoken to a guy at Apple about this - another Mac technician entirely unaware of any Catalina Mail problems. He told me that opening the emails from a separate folder will still require communication with the Mail app to open them. So if Mail's corrupted by the Catalina update then they won't open. His advice: stick with Mojave for the foreseeable future. Not ideal.

However, what I did find was that, after saving emails to the folder on my desktop, I could choose to open them in Microsoft Outlook. I guess this might work after a Catalina update.

@john The .eml files can be opened by Apple Mail as well as most other e-mail clients. I don’t think you have to worry about opening them with Catalina Mail because the messages are detached from its data store. It’s just viewing them as individual files. You can also view them in Finder using Quick Look. Of course, staying with Mojave is safe, too.

Thanks, Michael.

So what is the best way to back up all my iCloud email before I clean install to Catalina ?

Hello Fellow Sufferers,

I have a failing MBP, so I finally ordered a 2020 MBP 13 in, which is arriving later this month, with Catalina on it. I didn't know about this whole Mail issue, sadly, and I hadn't upgraded since El Capitan. I read several places that I should upgrade on this laptop to Catalina before migrating to the new laptop. Again, sadly, I did. The latest version, 10.15.4.

I don't seem to have a problem with distorted or empty messages, at least that I've discovered so far. I have the missing 4 GBs of emails problem. I opened Mail after upgrading and it was empty. No accounts, no emails. With help from a friend, I discovered the V7 folder in Library/Mail and it looks like all my emails are there. I have a Time Machine backup from right before I upgraded. (I have a SuperDuper backup and a Backblaze one, too, lol, because bad things have happened many times between It Just Works Apple and me.

I set up new accounts just fine, with the few emails still on the servers. I realized that I could make folders in the On My Mac section and import emails into them from V7 folder. I had many folders in the On My Mac section. I haven't tried importing any of the POP accounts yet, active or inactive. What I don't understand is this: If I import stuff from V7, when I migrate to the new laptop, will it be migrating what's newly replicated and arranged in my Mail app, or will it be just replicating from that V7 file and I'll have two versions of everything on the new laptop? If I import a mailbox from V7 to my Mail app, should I delete the source folder for that mailbox that's in V7 because the system will be updating as soon as I make a mailbox and THAT copy in V7 is the one I want? I hope that made sense. Import an On My Mac folder called Games from V7 folder into a folder that I name "Previous Games" in Mail app. Delete the the old folder in V7 that I just imported so I don't end up with both folders on the new laptop later this month.

I guess I do the same with the mboxs for the current POP accounts? Or if you think all of these might be corrupted in some way, import everything from Time Machine. Of course, I would keep a copy of the original V7 folder that the upgrade created, just in case. Any thoughts or warnings? I'm more tech-savvy than the average human, but much, much less tech savvy than many of you, I'm sure. Thank you so much for any thoughts.

@Carolyn People usually talk about importing from V6. V7 is the Catalina folder, so it normally doesn't make sense to import from the folder that’s already in use. I haven’t seen that before. That sounds more like a case where the Envelope Index needs to be rebuilt. And that you may eventually get duplicates (the next time a new database is created) if you’ve now created two copies of each mailbox in V7. Personally, I would start over with an empty V7 and import everything. When you migrate to a new Mac, nothing should change if you already have a V7. But, yes, keep copies of everything (and the backups) just in case.

I used to rely on Time Machine backups for some of the IMAP folder content but it seems right now Mail.app no longer downloads any of this content to disk.

@Robert Using Time Machine to backup IMAP content has not worked for a long time, since Mail may not be caching all of the data locally. You could either make a rule to move IMAP messages to an On My Mac mailbox so that they are fully saved locally or archive the messages using EagleFiler, which will ensure that the full message data has been downloaded. Either way, you’ll end up complete mail files that Time Machine can see. (Moving IMAP messages to an On My Mac mailbox manually—rather via a rule when they first come in—is not reliable and can cause parts of the messages to be permanently lost.)

@Michael Thanks so much for your help!

Totally beyond me working out how Eaglefiler works. Drag & Drop it aint! I've deleted it, fully defeated!

@john The instructions are here. You can drag and drop a message from the Mail window to the EagleFiler window or select the messages or mailboxes in Mail and press the F1 key.

Thanks, Michael. The Eaglefiler instructions make no sense to me whatsoever. I open a new Eaglefiler library, highlight a mailbox in Mail, press the f1 key and nothing happens. Or I try and drag the c.2,700 emails over and only one arrives.I just don't have time to drag every single email over that I need to save. I'm no technophobe but Eaglefiler has stumped me !

@john If nothing happened at all, it sounds like your keyboard is configured so that you need to hold the fn key when you press F1. You could also change the hotkey in the preferences. Drag and drop only works for single messages due to a bug in Mail.

Ah! that worked - thank you!

I have a somewhat similar situation to Carolyn above. I needed to upgrade my MacBook Air and I ordered a new MacBook Pro 13". I never upgraded my two macs(MacBook Air, Mac Mini) because someone told me about this bug back in October. I have kept my iOS devices updated to the latest versions with no apparent issues. The issue with the mail bug slipped my mind until today. It appears its still an issue and now I'm worried that I'll get this new mac and be unable to use anything related to mail on it because of this bug. I'm trying to get some idea of what I can expect if I do have an issue and how to best deal with it.

I've got three email accounts hooked up. One is a gmail account, one is an outlook.com exchange account and one is an IMAP account for a local ISP that I keep paying for email access from because so much is still tied to that address (I've had it for over 20 years) and I keep finding more every time I think I've migrated off. I don't have any pop3 accounts anymore (and email from the ones I had is already archived elsewhere). I'm a digital packrat and I don't delete legitimate (ie, not spam or fraud or something) email. I'm trying to get some idea of the scope of the problem. With those accounts, what can I expect?

I understand that apparently most people don't see a problem, but if I do, how can I check to see if its happening so I can try to mitigate any issues right away. If I do have an issue, will it affect the email stored on the server? I've seen references to mail lost due to the bug propagating to other computers because of mail sync, which makes me think it will. Are there any recommended mitigation/recovery steps if it does happen? Is it something that will keep happening or is it a "lose your mail once and you're done" thing I have two different sets of backups of both macs and I also think I'll just take both my current macs completely offline then power them off before I try to add my mail accounts to the new mac.

I'm starting to regret ordering that mac. Maybe I should have just bought a windows laptop instead.

Thanks!

@Ben As far as I know, the bug that deleted messages at rest (and propagated the deletion) has been fixed. The remaining bug is that (for some users) moving a message from one mailbox to another will delete it. Messages that had previously been moved by a rule seem to be at greater risk. So any old mail that is just sitting there should be fine. But messages that you drag and drop (or move in other ways) may be impacted at the time of the move. I don’t know of a 100% accurate way to test whether you are one of the unlucky users who are affected, but moving some test messages can give you an idea. Try moving messages that had previously been filed by a rule, try the different combinations (within the same account, between accounts, between server and local, etc.) and see what happens. For users who are affected, it nearly all messages moved in a certain way seem to disappear. If you’re using SpamSieve, it has a backup feature that keeps a copy of each message outside of Mail, so that you can recover any that are lost.

I was helping someone with this missing mail problem and in the end created a simple python script to help migrate from Apple Mail to Thunderbird. Hopefully this will help some readers out.

Start by creating a mail export ideally from your pre-Catalina version of Apple Mail (it /may/ work for an export with 'empty' mail from Catalina's Apple Mail, it's not clear if/how those exports are broken).

Make a copy of that export and operate on the copy.

The script is discussed [in an issue](https://github.com/thundernest/import-export-tools-ng/issues/28) on the ImportExportTools NG plugin for Thunderbird. It can currently be [downloaded here](https://raw.githubusercontent.com/obar/import-export-tools-ng/master/scripts/applembox2sbd.py) with [usage instructions here](https://github.com/obar/import-export-tools-ng#importing-from-apple-mail-mbox-format) and soon will be found in [the ImportExportTools NG repository](https://github.com/thundernest/import-export-tools-ng/) (instructions and script).

This shit is still happening. Mac Tech Support is useless. They don't get back to you. Engineers don't get back to them, etc. Rinse, cycle, repeat. NONE of this would have happened on Steve Jobs' watch. This is the worst probem I have ever had with an Apple product. I'm selling my stock. I've had it!

I discovered this problem immediately after using migration assistant.
Rather than recovering the old V6 folder from time machine, is it OK to just use target disk mode and drag the old V6 folder to my new mac, and then import to mail in Catalina?

@Jack Which version of macOS were you updating to where you had this problem after Migration Assistant? Yes, copying the files directly and then importing them should be fine.

Hi Michael, I was migrating from a MBP running Mojave 10.14.6 to a new MBP (Catalina).

@Jack I mean which version of Catalina? My understanding was that the migration bug is already fixed in macOS 10.15.4, so I was surprised to hear that you ran into it.

Michael,sorry for the confusion. I migrated "out of the box" and was then prompted for the 10.15.4 update, which explains my problem.
Thanks for your help and for this resource!

@ Michael Tsai -- Is the "bug" to which you refer on 5/20,4:19 pm ("the bug that deleted messages at rest (and propagated the deletion) has been fixed") -- the same "bug" to which you refer in your sentence at 5/24, 1:21 pm ( "My understanding was that the migration bug is already fixed in macOS 10.15.4") ?

@HoustonMJ Yes. I’m not sure how many bugs there are in the code. But from the user perspective there’s the issue of messages that you’re not touching getting deleted during a migration or database rebuild, and the other issue of messages disappearing or going blank when you move them. I’m pretty sure that the first one really is fixed. The second one is not fixed, although I’ve seen some reports that it no longer happens in all of the same circumstances where it once did. So, perhaps, partially fixed.

@MichaelTsai -- Thank you for your clarification (5/24; 8:57 pm).

Can you set forth your current thinking as to courses of action that those of us who are using a pre-Catalina OSs can take as part of our moving to a new Apple computer with Catalina installed as the OS (and with the inability of using Mojave on the new computer)?

What specific actions, mail clients [in lieu of Catalina Apple Mail], email archive/filing apps [for example Eagle Filer], and otherwise, do you propose? Thank you.

@HoustonMJ

1. Before updating to Catalina, make a backup of your Mac.

2. Ideally, archive the messages to EagleFiler to make sure that you get full copies of the messages, including attachments. (Time Machine and other backup apps won’t warn you about the ones that aren’t cached locally.)

3. Update all the way to macOS 10.15.4 before launching Apple Mail on Catalina.

4. For most users, Apple Mail on 10.15.4 will be fine. But you should test to make sure that you are in this group.

5. If moving messages seems to work for you, and you’re using SpamSieve, enable the backup feature, just in case. You may also want to regularly archive messages to keep a copies (or simply move them) outside of Mail. (I do this, anyway, because it bogs down with lots of mail and in case something goes wrong on the mail server.)

6. If you don’t want to use Mail, I like GyazMail and MailMate. Other popular options are Airmail, Postbox, Outlook, and Thunderbird.

[…] had with data loss after upgrading to Catalina. We’ll have to wait for Michael Tsai’s users to weigh in with the results of their testing (scroll to the bottom of the […]

[…] for WWDC is a noticeable focus on quality: fewer bugs, less waiting, better fit and finish, and no catastrophes when upgrading. The tick-tock cycle of feature-heavy releases followed by refinement versions is a […]

Hi All,

Just a quick note to confirm that with the 10.15.5 update released today, Apple Mail is unchanged from 10.15.4, at Version 13.4 (3608.80.23.2.2).

Just to let people know that I've today updated to the latest version of Catalina (10.15.5), and all existing emails appear to be in place, including the ones I had lost the last time I tried updating in October last year (they were reinstated when I reverted to Mojave then).

@john That’s good to hear. I think the upgrading bug was fixed in macOS 10.15.3.

I have had this issue since 10.15.3 (I am now on 10.15.5) - applying rules now simply makes some messages COMPLETELY VANISH! I ended up deleting my white/trash lists (which were getting lengthy), and added one simple rule to move messages ending in stacksocial.com to my normal Inbox from the Junk folder. Once I applied it, half of the entries in the Junk folder vanished. I know because I searched for the one that I wanted to add to the rule if successful.

@john What version did you upgrade from?

[…] been resolved in an increasingly longer list of macOS releases. It’s not as critical as Mail.app deleting emails but it’s still not something you want to have happening every few […]

@akim - I upgraded from the latest version of Mojave. And now I look, yes, there are some emails missing. Bummer. I have them backed up on Eaglefiler, thankfully.

@john

Just to confirm your methodology.

You upgraded from Mojave to Catalina, and installed all Catalina updates to version 10.15.5, BEFORE opening Apple Mail at any point during the upgrade process?

Did you by chance check the size of your Apple Mail folder tree, before upgrading and then after, to see if there was a reduction in the size of the tree?

Or perhaps more coarsely, check for any changes in free disk space before opening Apple Mail and then after, to gauge how much mail content may have been lost?

Thanks.

Hi @marc.

Actually, I've found the email which I thought was missing! I had trashed it prior to the upgrade. So it was still in the 'bin'.

So: so far for me the upgrade has gone smoothly, with no emails lost.

Hi @john

Thanks for the clarification. I was getting very concerned that there were lingering upgrade issues, given your report.

Hope that it continues to go well for you.

[…] Maar in mijn ogen de meest alarmerende fout in het programma was dat er mailberichten zoekraakten. […]

I'm helping an old friend to find all her missing emails. It started when she bought a brand new MacBook for a few weeks ago, and the Apple Authorized reseller promised to help her with transferring all data from her old 2012 MacBook. And af course almost all messages was gone in Catalina. Around 20.000! What the reseller did was just migrating from the old OS in one big jump.
And now they claim that there were no emails from the beginning (!?)
I managed to take out the old hard disk and did a simple data recovery with a freeware. And found all the emails!
What's the safest way now to import the old mailboxes to Catalina Mail app?

@Sven First, update the new Mac to the latest version of Catalina. Then you can use the File ‣ Import Mailboxes… command.

Thanks Michael !!

Hi,
I wrote at the end of May. I just wanted to say that my problem wasn't an envelope one, which you mentioned. We'd already tried removing the envelope files and it didn't help. (I didn't say anything in response to that suggestion because I was exhausted from dealing with this problem.) I have the huge data loss problem. I upgraded from El Capitan. Unfortunately I hadn't seen all the warnings. All of my emails were missing, including MANY On My Mac folders. We tried all sorts of things. For other reasons, I got a new laptop that has Catalina on it, of course. So, I imported my Time Machine backup from BEFORE the Catalina upgrade on the old laptop. I have the same problem as before (it's 10.5.4). The files sit there in the Library/Mail folder, but even if I import them, I have a big mess. For example, I can't rename any of the folders. And none of the emails in the folders have a "From" line. And when importing all the previous emails in the main section (meaning, not On My Mac, but regular accounts) they don't end up in the accounts, they end up in On My Mac. I've seen a couple people report that using Import didn't actually Import all of their emails. They ended up using PostBox, which successfully brought them all over, as far as they can tell.

I moved to Catalina, kicking and screaming as my 2018 Mac Mini running Mojave kept crashing when idle. Apple engineering told me to upgrade, implying that this was never going to be fixed.

Running Mojave I had about 10 local folders under "On My Mac".
After installing Catalina the folders are invisible. I currently have a ticket open with Apple on this.

A customer had Catalina 10.15.4 and two of three new email accounts hosted by the same provider added without problems. The third would not add. Apple of course blamed the email provider. On my machine, also running 10.15.4 I added all three without issue.
The client upgraded to 10.15.5 and was able to install the third account.

I have two other apps that install mail bundles and both bundles fail (SmallCubed MailSuite, and DEVONthink) with "…because the developer cannot be verified".

Apple has lost its way - sadly, the famous attention to detail has vanished.

Hi all,

A quick note to point out that through 10.15.6 beta 2, the version of Apple Mail included has not changed from 10.15.5 release.

It is still Version 13.4 (3608.80.23.2.2).

[…] references Mail’s data loss problems and Finder’s bizarre instability and file system misreporting as he […]

[…] stable, most disruptive-of-fluency macOS releases in recent memory. There are egregious issues of data loss in Apple Mail. Mail also simply won’t scroll — scroll! — certain HTML emails anymore. […]

Ugh! This Mail problem is a nightmare! Two months on the phone with Apple. I'm in a black hole with them. Clearly they are just spinning their wheels with me as they know this is a HUGE issue and they don't have a fix for it. I have disappearing mailboxes. reappearing deleted emails, cannot move emails with attachments between my import On My Mac and my iCloud side. If someone could recommend another email program that I could import my mesages into I would surely appreciate it! I miss Steve!

As a developer, I would need to move to Catalina at some point. I tried out MailMate using Dovecot (serving only to localhost) to store all my old "On My Mac" Mail messages (as MailMate is IMAP only). Very happy with this move.

[…] Tsai, who has been tracking this issue, […]

[…] hesitant to advocate upgrading to Catalina, significantly in the event you use Mail, since reports of email data loss continue. Fortunately, downgrading a brand new Mac from Catalina to 10.14 Mojave is feasible for fashions […]

I don’t see in the comments here a bug report number that is for the still outstanding data loss issue when moving messages between mailboxes. Does anyone have radar number(s) for that unfixed issue?

@David FB7558932 and FB7449043, and a related one for moving messages is FB7388651. Other Mail bugs in Catalina: FB7035263, FB7500485, FB6142266. I know others have filed bugs (even before mine) for the primary issue, but I don’t have their numbers.

I had a conversation today with someone in Apple's corporate office (I believe) about this whole matter.

This person followed up on my complaints that nothing was being done about this serious matter. They told me that, "I can assure you that we are aware of it"; however, they were non-committal about if or when it is going to be fixed. This individual learned from engineering that there was one major aspect of this that was already fixed (consistent with comments on this blog) but other aspects were not yet fixed. Reading between the lines, it sounded like they are having a hard time understanding the situations when this occurs, whether there are certain mail servers that are more likely to result in Mac Mail losing mail, etc. I think this person was trying to convey that they didn't know when or if this would be fixed, and that a fix wasn't imminent because they haven't isolated the problem. It's possible that it will eventually be fixed, and also possible that it won't.

So, while I wish I had received a different response, I appreciated the honesty (and I did think it was an honest response as I could tell that they had looked into it) and was at least glad to know not to keep waiting. I haven't yet upgraded to Catalina, waiting for a fix, and now I know to get myself a different e-mail program to use in preparation to upgrade.

I also gave feedback that it would be really nice to get better communication from Apple about these types of things, one aspect of which would be to issue "release notes" that provide more comprehensive information about what has been fixed.

So, there you have it. Hope that is helpful.

@Frank Thanks for the update. Interestingly, the most common problem server that I hear about is iCloud. So in theory Apple has full access to compare how macOS 10.14 and 10.15 communicate with it to see where the problem is.

Will anybody be able to test the updated Mail client in the first MacOS 11 Big Sur beta? I'm not clear on what restrictions are associated with the developer beta in so much as being able to comment on the bug status would be concerned. Thoughts?

@Gregory I think Apple doesn’t restrict discussions about beta versions anymore. I have Big Sur installed but was not seeing the remaining bug myself, so someone else would need to report on this.

Looking at he comment trail the following.
I have issues with content disappearing from my email, moving emails from my Inbox to other mailboxes.
I run Catalina 10.15.5.
In the comment trail it looks like this is a known issue for quite a while. It seems like this issue is not solved in 10.15 and scheduled for 10.16. Is this a correct interpretation? And if 'yes' when is 10. 16 scheduled?
I am thinking of moving to Outlook since this bug is pretty annoying.

@Paul Apple hasn’t said when the fix is scheduled. It’s not clear whether there will be a macOS 10.15.7. macOS 11.0 is already in beta, but I’ve not heard yet whether it fixes the problem.

With a variety of problems in 10.5.5, I decided to try 10.5.6b2 because in all reality, the difference between an Apple beta and release since MacOS X is just the calendar date.

Anyway, I downloaded the beta enabler for 15.5.6 beta 2 and let it do its thing. I come back and I’m on macOS 11 AND Mail crashes at launch. https://twitter.com/cerniuk/status/1275774386193170433?s=21

To get mail to work, I had to remove the V8 folder under the Mail where it keeps its data (3GB) and it rebuilt it as V2 as if it was importing the data (from somewhere) but did it way way too fast. I never told Mail that the V8 folder was on the desktop. Hmmm.

In some ways macOS 11 beta is more stable than max is 10.5.5, in other ways not... but based upon this experience be ready to lose a lot of mail data when you migrate to macOS 11.

[…] for WWDC is a noticeable focus on quality: fewer bugs, less waiting, better fit and finish, and no catastrophes when upgrading. The tick-tock cycle of feature-heavy releases followed by refinement versions is a […]

Hi all,

Been a bit since any traffic here.

Today, Apple released macOS Catalina 10.15.6.

The version of Mail that was in 10.15.5 was Version 13.4 (3608.80.23.2.2).

The version of Mail that is now in 10.15.6 is Version 13.4 (3608.120.23.2.1).

So, there has been a small incremental update in the app with this new macOS update. No documentation of any changes are included, so it is not clear what has actually changed.

[…] about fixes for problems users have experienced with Mail. However, the most recent comment in Michael Tsai’s long-running thread warning about data loss notes that Mail’s build number has changed, so it’s possible that Apple made some […]

Since the integrity of email is essential to me, I decided to move to MS Outlook…………
It puzzles me it takes so long for Apple to fix this mail problem…………I expected a higher standard

I decided a few weeks ago to give up on Mail and use Thunderbird. Not because of this problem, but because of many other Mail problems (example: I have never been able to learn how searching mail is supposed to work, and it fails on at least two thirds of my simple attempts, such as a From: address). I use a POP server, so may be immune from the disappearing mail problem.

The terrible aspect of this is that I will lose SpamSeive, which, as we all know, is as close to error free as we are likely to see for a very long time. I decided to hope that the Thunderbird Bayesian filter works well enough. I plan to use the tool by Gladwev to make the switch, one account at a time (extract, then import). Unless I learn of an alternative. (Since Michael mentioned it, I'll look at Eagle Filer.) I don't believe Thunderbird direct import is working.

Scrolling backward to read Michael's contributions, I read Frank's of June 18, 2020 2:51 PM.

Thank you Frank! I just bought a Mac Pro (late 2019) to replace my Mac Pro (late 2010). I will be forced onto Catalina. I have been talking to the folks at Apple second level Enterprise support, and they believe use of Migration Assistant will be painless.

After reading what Frank wrote (and having read other reports here) I have decided to let the new Mac sit empty until I convert all my mail to Thunderbird and see that working smoothly.

@Joaquin Were you not using POP, it would be possible to use another IMAP client to filter mail via SpamSieve while still using Thunderbird to read it. If you send me an e-mail, I can add you to a list to notify if/when there’s a better way to use it with Thunderbird. You could certainly use EagleFiler to extract the messages from Mail, and then have Thunderbird directly import from the EagleFiler archive. I would trust this over Mail’s own export, which has at times messed up the mbox format and (still) doesn’t warn about missing attachments (though that wouldn’t be an issue for POP.

Thank you, Michael.

Two days ago, I updated from High Sierra to Catalina. Mail worked flawlessly under Hs (apart from search, which was flakey at best), but in C, I find that moving messages out of Spam/Junk folders to inbox or any other folder deletes them irretrievably - they don't seem to exist anywhere any more. That's bloody annoying.

I've tried all the 'fixes' I can find on various sites, and none of them make any difference to the behaviour. All I can conclude is that Mail is broken.

@Phil Which version of macOS 10.15.x re you using?

I submitted a request to Apple months ago to look into this bug, but of course did not hear back. I'm responsible for many Mac models across my 2 businesses, not to mention at home. All are on 10.14.6 because I can't afford to lose important emails. I have iCloud, Gmail and Exchange accounts sync'd to many different mailboxes inside Mail and am frozen without a path forward. I would consider switching mail clients, but I use Daylite CRM in both of my businesses and it only links to emails from Mail. This is a terrible situation with no seeming path forward.

I should add that I had backed up not via Time Machine, which I have always found problematic, but via SuperDuper so I had a complete backup from which I could copy my missing messages.

[…] der Verlust dieser Daten gar nicht erst auffällt. Der Entwickler Michael Tsai schreibt in seinem Blog, dass ein großer Teil von macOS Catalina Nutzern betroffen sei weshalb er im gleichen Zug vor dem […]

Does anybody have new information about Mail in macOS Big Sur? Does the moving-messages-bug still appear?

@Christoph It’s not possible to prove a negative, but so far I have seen no reports of the bug occurring on Big Sur.

I upgraded from 10.14 to 10.15.6 Catalina on 7/29/20. Since then I have lost a couple of mails completely as I dragged them from one mailbox to another. They absolutely disappeared from my hard drive, as best I could tell with UNIX search tools. I was able to find one of them going to my backup TimeMachine drive, and by searching for a text phrase that I knew was unique the missing email. But if I wasn't paying attention, I would have lost it forever without even knowing what I was missing from my archive.

Michael - you claimed in another post earlier today that Big Sur "fixes the mail table view". Is that the underlying cause of the disappearing emails discussed here? If so, do we have knowledge from anyone on the inside that affirms it has in fact been addressed?

@Jack In Catalina, the message list in “classic” mode has fixed columns that can’t be configured or re-arranged. In Big Sur, it’s back to being a standard macOS table view, like in Mojave and earlier. This has nothing to do with the data loss bugs. I’m not aware of anyone on the inside saying those have been addressed. Commenter Jedidiah above says that he just lost some moved messages with macOS 10.15.6, and I have other reports of large numbers of Exchange messages at rest disappearing.

First I'd like to say thank you Michael for keeping comments open on this post. Your blog seems to be one of the few places were you can get some concrete information about the mysterious "moving-dissapearing email" bug.

I'm still running Mojave on two Macs because of a Fujitsu Scanner I use and because of the Mail.app bug. Fujitsu has recently released a 64bit version of ScanSnap Mangager, therefore theoretically I could now upgrade to Catalina. After being bitten by a nasty bug in Photos.app that deleted a couple of hundred pictures from my library in one of the early Mojave releases, I am now extremely cautious with macOS upgrades.

Here is my question. How easy is it to test or reproduce the email-moving-bug? I have a clean Catalina install on an external boot drive. I could configure my email account on this Catalina install and try moving a couple of emails and see what happens. Would this test be sufficient to know if my email is safe from the moving email bug?

@Dan From what I’ve heard, it’s easy to reproduce. For those affected by the bug, moving any message (or perhaps any message that had previously been moved by a rule) will make it disappear. If that succeeds, you are likely one of the lucky unaffected users (due unknown particulars of your Mail setup or server). I can’t promise that you’ll be safe, but I don’t recall hearing from anyone who was intermittently affected by the bug.

Thanks @Michael for the quick reply. Clearly every setup and mail provider is a different case. But if it's indeed so easy to reproduce the bug I will give a try. Off course then again a clean install might be less prone to the bug than an upgraded system. Anyways thanks for your opinion.

Same here by mere - empty mails and mails which cant be moved...
OSX Catalina 10.15.6 (19G73)

Intermittent fault with emails waiting to be sent, stuck in Apple Mail Outbox.

Eventually have to Force Quit Mail. Outbox nowhere to be seen on app reboot. Emails have evaporated. Not in drafts/sent/outbox empty.

Have rebuilt. Twice. Nothing.

Gone into ~Library etc. Nada.

Rebooted Macbook. Can send fresh emails as normal but still no sign of missing emails that were stuck in 'sending 5 items purgatory'.

Repaired permissions.
Checked mail settings - all OK.
Used Data recovery - nothing.

Fault doesn't happen often enough to be constant, consequently I've no way of reproducing it, it's so random. But it always seems to be on a complicated (not necessarily large) email that's taken me a while to pull together. 🙄

Two hours this evening on phone to Apple Support - eventually senior advisor. Nothing.

Also, any open drafts I'm working on when eg system crashes, often reappear after reboot as just a blank email. Empty 'To' field, empty 'Subject', zero content. So I don't even know what it is I've lost in order to try and recreate it.

2018 Macbook
1TB data
Plenty of free storage
32GB RAM
Running Catalina - 10.15.6 (19G73)

Any ideas? Before I commit violence...

Happens on several email accounts, btw.

icloud
zoho
34sp
yahoo

It's no respecter of server. 🙄

(Nothing stored on any of the servers, nor other devices)

I recently wrote a mail to federighi@apple.com, complaining about the state of this issue, and referencing this very page. I then received a reply by "craig" (or whoever answers this mail account at Apple) asking: "Are you personally experiencing this issue?", to which I replied that I didn't as I was still too afraid to upgrade to Catalina because of the comments on the Internet about this very issue.

So it might be worth it for anyone who _is_ actually experiencing the issue (and can maybe even reproduce it somewhat reliably) to write a short mail to federighi@apple.com stating this.

If you don't receive a reply in a couple of days, you might also try writing to tcook@apple.com instead, which is what I initially did when I first asked Apple about the state of this issue back in January when there was no fix in 10.15.2, to which I got a reply from the federighi@apple.com address, asking me which reports I was referring to. After I referred them to this very page, I didn't hear back anymore in January. So no actual idea, but maybe this triggered the fix for the first problem in 10.15.3 …

Missing (corrupted) imported Mail Messages ... continues in 10.15.6

Due to persistent kernel panics (10.15.6), I did a clean install. This included using CarbonCopyCloner (5.1.21) to move the Mail folder from a Time Machine (HFS+) backup disk to the iMac's (2015 I7 w 32GB) external 2TB boot disk (APFS). Then I ran CCC a second time to confirm the integrity of the copied files/folder. It passed without any apparent errors.

Then, using the Mail import function, I imported the >150,000 messages from the new Mail folder copy on the SSD.

The result: hundreds of messages devoid of text and attachments. But they all had the correct elmx file name.

Significantly, however: the corrupted Mail problem did not result from the Mail import function. Rather, the messages were clearly corrupted during the original copy process from the HFS+ backup to the APFS SSD target disk.

That is, the Mail folder copy, created with CCC contained corrupted Mail message documents/files. But the original Mail folder/directory still on the Time Machine disk did not.

How do I know? The corruption destroyed each message's contents but not the file names. So I could do a file by file comparison.

The corrupted files appear to be those that had some form of attachment: image, pdf, pages document. The corruption, from what I could tell, did not occur to messages comprised only of text.

Next I did a manual drag/drop copy of the same Mail folder from the Time Machine backup to the new SSD startup disk.

Good grief: no corrupted files whatsoever!

To draw some preliminary conclusions I'd want to repeat the processes several times to see if the results are consistent.

But that would be too time consuming for me just now.

Questions remain. Does CCC run a fundamentally different copy routine than manual drag/drop entails? Is there an issue related to moving files across the HFS+ - APFS divide, et cetera. Why does the problem only occur, from the few corrupted messages I checked, to those messages that contain an attachment of some sort - a linking problem/corruption?

Again I do not have the time to trouble shoot this fully ...

Mike Bombich @ CCC just wrote me:

Don't use CarbonCopyCloner to copy files from a Time Machine backup. Rather use the Finder. Otherwise unexpected problems may result.

Thanks to everyone for contributing experiences and recommendations. According to Michael, the mass deletion of thousands of e-mails upon migration isn't so much an issue with the current version of MacOS Catalina. But for some, moving messages between folders can cause a disappearance. Most recent posts are about IMAP mail, not surprisingly. I am using POP3 with many rules for sorting incoming mail. So obviously, moving happens. Any current information on the risk of moving a POP message to another folder? Thanks.

@Gordon I don’t think there was ever a problem moving POP/local messages.

any update if the bug is still there in big sur ?

@me I’ve received one report of messages disappearing when moved on Big Sur (a few betas ago).

I talked to an Apple Business Services rep yesterday and expressed my frustration over this bug. I told him I have 2 companies running Mojave and I'm afraid to upgrade all of our computers. He told me he thought the bug was fixed in Catalina, but I am also supposed to hear back from their systems engineer. We'll see... I just can't believe this has been going on so long as it is a terrible bug.

I have been avoiding Catalina because of this (there are other problems, but they might be able to be worked around). I don't even need the bug to be fixed immediately; what is causing it? Is there any way to know whether you will be affected? If there is some way to know who it affects and who it doesn't, then at least you can determine if you can upgrade or not. Right now, even if it only ever affected 10 people in the entire world, without *knowing* what causes it, I have to assume it will also bite me...

Michael Tsai : Your comment yesterday has me a bit confused so I am hoping that you could clarify things for me. Reading through all of the comments it seems that there were two initial bugs in the Mail app in Catalina:

1)Upgrading from Mojave resulting in losing many- if not all emails
2)Moving messages between Mailboxes resulted in the emails being empty and losing attachments

The first one seems to have been fixed (from earlier reports) so I will focus on the second issue.

My confusion is when the bug is an issue when moving messages around and whether it affects both IMAP and POP users.

Can you please answer the following questions from what you know?

1Are emails lost/corrupted when one moves messages between folders that reside on an IMAP server?
2)Are emails lost/corrupted when one moves messages from an IMAP server to a local(on the mac) folder/mailbox?
3)Are emails lost/corrupted when one moves messages that exist in a POP3 Inbox to other local(on the mac) folders/mailboxes - either manually or via defined rules?

My main email account is POP3 and I have rules that automatically move many messages into local folders/mailboxes on my Mac as they are downloaded into my Inbox. I also manually move many messages around locally on my Mac (from my Inbox as well as folder/mailbox to folder/mailbox.

I have been holding off on upgrading to Catalina because I thought that this bug affected both IMAP and POP3, but if it is only IMAP then I should be in good shape and could move ahead with the upgrade.

Thanks for your help and advice!

@Doug 1. Yes. 2. Yes. 3. I don’t think so. I’ve only heard of it affecting IMAP and Exchange accounts.

This may be a dumb question, but here goes...

I recently made the mistake of upgrading to Catalina, and am now at 10.15.6. For some time prior to that I had been using rules to archive my POP3 mail in 2 mailboxes On My Mac, on an older Mac running 10.13.6. There were about 4,500 messages in one of these and only about 1,500 in the other one. After I set up my newer Mac about a month ago and upgraded to Catalina, I discovered that all messages prior to July 1, 2020, were missing from both mailboxes on the newer Mac. I suspect this is not an indexing problem, but the messages are really gone. Fortunately, they are still complete on the old Mac, and most had been archived in MailSteward, and the MailSteward databases have been successfully copied to the new Mac. So, the question: if I deleted v7 from the Mail Data folder on the newer Mac, and replaced it with v5 from the older Mac, would Catalina recognize it and in effect, restore the missing messages to my 2 mailboxes?

@Ken Probably, or you could just ask Mail to import from the V5 folder.

This may be a dumb question, but here goes:

If I remove the v7 Mail Data folder from Catalina 10.15.6 and replace it with the v5 Mail Data folder from 10.13.6, will Catalina be able to read it and recover lost messages in it?

The back story is that I have been running 10.13.6 on another Mac, and had no email problems. I used outgoing and incoming rules to store most POP3 email into 2 mailboxes On My Mac folders; one had about 4,500, the other only about 1,500. Then I began using a newer Mac, and upgraded it to 10.15.6. I discovered that both of the above On My Mac mailboxes on the newer Mac only contained mail beginning in July 2020. I suspect this is not an indexing problem, and that the messages are really missing in Catalina. Fortunately I had archived all messages in those mailboxes with MailSteward, and have moved the MailSteward database over to the new Mac, so have recovered everything. But I still would like to get the missing mail back into Mail on my new Mac. I asked this question on Apple support communities and the only response I got was basically, "Try it and see," along with also deleting ~/Library/Containers/com.apple.mail and ~/Library/Preferences/com.apple.mail.plist.

@Ken I think it will work, but it would be less error-prone to simply use the Import Mailboxes command and point it at the V5 folder.

Thanks Michael. Hadn't thought of that, although it did occur to me that I could export them from MailSteward and import them back into Mail.

Thanks for all of the continuing good information. Do the last 2 or 3 posts mean there is still a possibility of losing hundreds/thousands of messages, even POP3, when updating MacOS from High Sierra or Mojave to Catalina? I had been under the impression that was no longer an issue.

@Gordon As far as I know, updating to Catalina and moving POP messages have no issues in macOS 10.15.6.

I thought I would share some recent (9/1/2020) upgrade/migration experiences with the group. I few years ago I helped several friends transition from PCs to Macs and of course now I am their main source for tech support!

Migrated from a 15” Mid 2014 Macbook Pro running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6

Migrated from a 13” Early 2015 Macbook Pro running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6

Updated a 13” Early 2015 Macbook Pro running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6

Updated a 27” iMac Mid 2015 running macOS Mojave 10.14.6 to new 16” Macbook Pro running macOS Catalina 10.15.6

First two computers had users that had IMAP email accounts including Google and iCloud among others.

Second two computers are mine and I have a main POP3 account but also have Google and iCloud accounts that are IMAP.

Results:
In all cases emails were preserved in their entirety and moving messages between mailboxes/folders leaves the emails intact. The only issue noticed to date is that one of the users that migrated to Catalina from Mojave has noticed some indexing problems with Mail where emails are not being returned for specific searches even though they can be found individually.

I am not saying that based on my experience people should migrate/upgrade, I simply wanted to add my experience to the “collective.” It is disturbing that there seems to be no rhyme or reason as to who is affected by this particular bug.

Like many others, I'm still using Mojave because I cannot afford losing my mail history in IMAP folders. I will probably skip Catalina altogether and hope that Apple will be able to fix this bug in Big Sur. Does anybody have e-mail addresses of important people at Apple, so we can tell them about this nasty problem (again) and urge them to finally fix this long-standing bug?

I finally updated to Catalina on my trusty 2012 Retina MacBook Pro because I need to run Xcode 12. I happily exported all archived (local) mailboxes to EagleFiler and am currently trying out MailMate for active email. Did a clean install of Catalina on a new SSD (upgraded to 1 TB) and moved my data via Migration Assistant (unfortunately, SuperDuper! had failed to verify some internal partition).

Upgraded from 10.14.6 to 10.15.5 some weeks ago and am affected by this persisting bug since then.
Unfortunately, have to confirm that it still has not been resolved on the current 10.15.7 release.

If you are willing not to loose data from your IMAP Accounts, avoid Catalina by all means!

Which exact bug are you experiencing? Only the one that's triggered by moving messages? Does it happen randomly, or is it reproducible all the time for you?

Hey,

So this may be unrelated but I’ve had an issue for months where each time I opened Mail on Catalina, it would download thousands of emails from my dovecot server.

I could see it in activity manager using up roughly 100mb of bandwidth each time and though I don’t think I lost any emails, I figured out what was causing it.

A nested folder named “archive”. For whatever reason, Apple mail doesn’t like that, even though I had a separate folder named “Archive” at the top level. Anyways, renaming it “_archive” fixed it, and since this thread is getting a lot of attention, I wanted to add that to some repository of knowledge.

@Yoni Thanks, I’ve heard of others with that re-downloading problem but had not heard of a solution.

I have finally given up on Mail this week, I have wasted too much resolving issues. A couple of days ago Mail stopped receiving email from one of my accounts (worked fine on iOS devices), I tried rebuilding the mailbox and all obvious fixes. I have moved to Outlook, and must admit it was daunting at first, but after a couple of days am finding it a joy to use, feature rich and stable (and this from a Mac user since 1989)... It is disappointing that Apple do not have a stable and feature rich mail client, but perhaps it has given up the battle with Microsoft on this one...

Had Mail issue since probably 2 weeks ago. Could be because of the upgrade to 10.15.7 or O365 server side, not sure. Was all ok before. Originally it's my O365 work email stop receiving emails. Reopened the Mail client sometimes pull in the latest email, but not reliable at all.

Apple Support is USELESS. Asked me to collect a lot of information by their Capture_Data app and simply asked another rep called back and checked the connections doctor that i checked 100 times and called it a Server side issue. While Outlook and Spark getting emails fine. That O365 has around 70k emails over the years.

After many struggle nights, I even had to recover the whole mac from Time Machine backup because I lost some emails when working on another email address (this is somehow related to this article). Since I was working on the emails, I was moving an old email account from POP3 (which was set in POP3 when I started to use that for another work email) to IMAP. The export mailbox didn't seem to capture all the messages and I lost a lot of records (hence the need to recover from TM. TM seems to save the old message back, and the IMAP server seem to have email count correct, but now the IMAP can't download all the messages, it's stops around 18 months ago. While the server side has 8+ years worth of messages. This time both Outlook and Mail have the same issue.

It could be the messages got a bit corrupted during MacOS upgrade (as this article suspected) or server side issue, or TM didn't get all the messages backed up, or something else.

Still not sure if the original O365 issue was fixed or not as I'm still fixing all the headaches regarding emails.
Lastly, I just want to add Mac Mail still has the best search function. Outlook (while it's surprisingly stable and did not stop/break the download for no reason), its search is buggy in my exp. It can't find an email I got couple days ago (or it's still indexing...not sure). I hope Apple fixes Mail, as I am really ok with the "features" I get from it. Most of the Outlook features I can use other app.

Btw, another minor issue I found from Mail is the "Siri remind me this email" command many times link me to the wrong email. So the mailbox is probably corrupted in someway (or Mail is just broken). I've "Rebuild" the mailboxes TOO MANY times...

Just want to add I was using 10.15.6. So my issue did not start from Catalina upgrade

Can anyone comment on whether the "deletion upon moving a message between folders" issue remains in Catalina 10.15.7 or for that matter the most recent public beta of Mac OS Big Sur?

Thanks

Another update after yesterday's googling and trial. My other web mail is on Ali Cloud IMAP service. For some reason it looks like it only sync the most recent 5000 messages in a single folder. So my 2nd issue is more a server side issue for sure. Put them in different folder fix the issue.

@Gordon The issue remains in macOS 10.15.7 and persists in Big Sur Developer Beta 10.

Witnessed this bug for the first time myself today when moving an email from my outlook.com inbox (Exchange) to my iCloud inbox (IMAP). About 5 seconds after moving the message it just disappeared from the iCloud inbox with no interaction. I have been unable to restore it from anywhere.

I finally gave up on Catalina after one year, in particular, since mail started crashing every couple of minutes, in addition to mail with attachements disppearing when moving to my Mac. Installing Mojave was nontrivial and required a boot disk. The migration assistant did not work, but I finally got it done, including importing almost all of my mail in the V7 folder from Catalina. Using mail in Mojave is a pleasure - it was worth the effort.

Has the Apple Mail data loss been fixed in the latest macOS 11.0.1 (20B5012d) Big Sur public BETA 7? Thanks.

Has the Apple Mail data loss been fixed in the latest macOS 11.0.1 (20B5022a) Big Sur Release Candidate (RC)? Thanks.

I just found out about this bug the hard way. I lost an important email - can't find it anywhere. I am able to reproduce the bug as follows:

1. create a new mailbox and select location "On My Mac".
2. manually move (I used the pulldown menu "Move to >" command) a mail message (that you don't care to lose) from your iCloud inbox to the newly created mailbox on your Mac.
3. you can verify the move and view the message if you like.
4. close the message but keep it selected and move it again to an existing mailbox (for instance Junk - I used the pulldown menu with specific "Move To Junk" command).
5. Poof the message should now be gone.

I plan to report this to Apple. I believe the bug is triggered by first moving a message(s) to a mailbox local to your Mac hard drive and then moving it again to a cloud mailbox. Maybe there are other conditions at play as I have many mailboxes and accounts but this simple action reproduces the problem for me.

Thanks for maintaining this thread as it has helped me understand what happened in my case of the missing emails.

Finally, I have not tried any remedies other than to search for the emails using keywords specific to the headers/text. No results.

I should have added that I'm on a MBPro running Catalina 10.15.7.

@johnnybrandom Thanks for writing out those steps. I’m seeing that same bug on 10.15.7. The destination mailbox briefly shows a number indicating that the message is there, and then it disappears. (I’m also hearing of messages disappearing when moved from server to local or between two servers, but I haven’t been able to reproduce that.)

@mjtsai - thanks for confirming the bug. I did report this to Apple using their Feedback Assistant yesterday. I've had a mixed bag with reporting bugs using Feedback Assistant - one previous report I filed did get a response from Apple engineers and a second did not. Hopefully this one will get some attention. Thanks again.

Just purchased a new MBP with Catalina. Everything was working fine. Woke up this morning and 3 IMAP accounts in MacMail were empty.... Went to the web host server and they're empty there now too. I literally just spent weeks directing all of my websites/service providers to these email addresses so I can have one easy place to look. Now they're all gone. I didn't move anything, sync anything, or do anything different. Last night at 10PM I got emails into those accounts, this morning everything is gone. I have an important email I need to respond to that came in late last night.... Any suggestions?

@johnnybrandom: If you do not receive a response through the Feedback Assistant, you might also try writing to tcook@apple.com about this issue. I did so in January after reading the reports here, and got a reply from federighi@apple.com asking me to point them to the reports which I mentioned in my mail. I referred them to this page and never heard back after. So I wrote to federighi@apple.com again in August asking about the status. I promptly got a reply asking me if I was personally experiencing the bug, which I am not, because I am still too scared to upgrade to Catalina. Maybe things will work out better when you tell them that you are actually experiencing the issue and are able to easily reproduce it.

Hopefully they manage to resolve this before the end of security updates for Mojave. I really don't want to move to a different e-mail client.

Has the Apple Mail data loss been fixed in the latest macOS 11.0.1 (20B28) Big Sur Release Candidate (RC) 2? Thanks.

Thank you all for posting updates about the disappearing emails issue. I upgraded to Catalina 10.15.7 a couple of weeks ago.

My experience with Catalina's Mail.app is also kind of rocky. While cleaning up my Gmail Inbox and deleting lot's of mails by dragging them to trash, Mail.app crashed multiple times on me.

Most comments here are about moving mails from IMAP to local mailboxes and between mailboxes on different IMAP servers. From my experience above I'm now wondering if moving between mailboxes on the same IMAP account is also problematic?

@Dan Moving messages between mailboxes on the same IMAP server was unreliable on previous versions. I don’t think I’ve heard a report of that specific issue on macOS 10.15.7 yet.

Like @Tex I'm wondering if this bug is fixed in Big Sur?

@Matthias It was not fixed as of a previous Big Sur beta. I’m not sure about the current version.

Just an FYI that Big Sur was released today:

https://apps.apple.com/us/app/macos-big-sur/id1526878132

This is version 11.0.1

The official Apple page is:

https://www.apple.com/macos/big-sur/

It would appear that Big Sur did not resolve this issue...

I actually never deleted any of the mail i got. Anyway, im still in Mojave, and i kind of want to update to the last release of catalina to test the waters. I have some time machine backups, my question is, if mail delete my emails i can recover them from time machine? Or do i need to set or activate something firet in mail and then do a time machine backup? Im a little confuse with the local and server mailboxes. And one last question, lets say i update to catalina but instead of using mail i use canary, that will avoid me the problem? Thabks

That's disappointing to see.

I have not updated to BS yet, and will wait for the .1 release before doing so. I have held off moving to Outlook, hoping that Big Sur would fix this issue.

I am not clear on why Apple has not fixed this. I can't imagine that they are ignoring this ongoing issue that has been present for a year. Not to make excuses for them, but perhaps they could not resolve it sufficiently and did not want to delay the release of BS for this. Thus, I am still hoping that we might see something in the .1 update.

That being said, whatever the dynamics, if there is no clear indication that by the time of the .1 release that it has been fixed, I will fully move to Outlook, for mail, calendar and contacts, which is frustrating. Given the current differential in the functionality of Outlook on macOS versus iOS, based upon my prior testing, a full move seems the best route for continuity. I had some prior comments earlier this year on some of these issues with Outlook, for anyone considering this path:

https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/#comment-3218657

@Mayo I would not recommend Time Machine because it may not back up all of the Mail data unless the messages are in local mailboxes. But moving old messages to local mailboxes (even on Mojave) can lose attachment data if it wasn’t already fully downloaded. So I would back them up in place using EagleFiler or using another mail client that can do full downloads.

After updating to Catalina, I think there’s no longer any danger to old messages so long as you aren’t moving them around. Or, using another mail client would also avoid the problem.

@Marc I don’t know what’s going on with Apple either. They never even replied to the bug reports that I and some others filed about this (starting more than a year ago). On the other hand, some say that they got replies saying that people at the top are aware.

@Michael, I had submitted 3 or 4 bug reports earlier this year myself, and never heard back from them.

Just an FYI, for folks that may be considering moving to Outlook.

I was playing around with Outlook again this morning, to find out that the "new" Outlook, which is available now as a toggled option, does not support iCloud/IMAP e-mail accounts, nor does it support local mail folders.

Apparently, the new version only supports online Office365/Outlook.com and Google e-mail accounts.

Thus, one needs to revert to the older version, at least for now.

There is an article here that covers this:

https://www.forbes.com/sites/barrycollins/2020/10/15/new-outlook-for-mac-launchesand-cuts-off-some-email-accounts/

and there is a page here, that explains known limitations:

https://support.microsoft.com/en-us/office/the-new-outlook-for-mac-6283be54-e74d-434e-babb-b70cefc77439

Lastly, there is a feature vote page that provides a means for users (current or prospective) to make their wishes known, including addressing the two issues that I note above:

https://outlook.uservoice.com/forums/924856

@Marc I have also written about that here.

@Michael, thanks for pointing that out, I missed it.

Seems strange that MS would roll this out as a release version, one that does not at least have feature parity with the older version. It is really a beta...

I've been following this thread since Catalina was released and the bug with moving messages between IMAP mailboxes has kept me from upgrading. But I'm going to be moving to Big Sur one way or another in the near future. So my question is:

Does the bug with messages disappearing and/or being corrupted by moving between mailboxes ALWAYS happen on those computers where it happens? That is, if I try this on Big Sur and do not have the problem with one or two test messages, does it seem that my system is unaffected? OR is it intermittent on a given computer and sometimes it happens and sometimes it does not?

@David My experience is that for a given type of operation it either always happens or never happens. So the key is to test all the different types to make sure that you aren’t affected. For example: moving (for each account) IMAP-to-local, local-to-IMAP, IMAP-to-Exchange, messages fresh in the inbox to different destinations, messages previously moved by an automatically applied rule to different destinations. (And please make backups just in case.)

@Michael that's helpful to know, thanks - any hints about whether the problem is machine/installation specific or mail account specific?
eg, if I test it on on one machine as suggested above and don't have problems, does that imply I won't have problems if I upgrade my main mac?

@MattR I have seen the same mail account work and delete messages on different Macs. However, it could be due to a differing use pattern rather than the files on the Macs. You don’t know for sure until you try it for real.

I am using Mail Version 13.4 (3608.120.23.2.4) on a late 2013 Macbook Pro Retina 15" wit macOS Catalina Version 10.15.7 (19H2).
For some time I had the issue that my older mails didn't show any content, if I tried to open them. On one of my mail accounts (POP3 | Provider: gmx.de) I could solve this problem with "Choose Mailbox > Rebuild", so I followed the same procedure with another mail account (also POP3 @gmx.de) and it deleted all of my messages before August 1st 2020. Or at least mail doesn't display these mails any longer.

Michael, thank you very much for your response. I know it is not 100% save, but, just to be clear, you think there will be no problem with my older mails as long as i dont move them to local mailboxes? As i said before, i really dont touch those, just from time to time i tend to search for them, you know, to remember what i was doing 10 years ago. The only mail i move is the new ones i got, sometimes i got spam and i need to move it to the spam folder. You think moving only the new will be save or that also can trigger the bug and delete all my emails? Thanks in advance, again, for your kind response.

@Mayo I think only e-mails that are moved are potentially in danger. I’ve mostly seen messages disappear when moving from a local mailbox to a server one and vice-versa, although a few people have reported the problem moving within the same server.

I'm late to this party. I upgraded my MacBook Air to Catalina in the fall and have been working with Apple ever since to resume access to my Mail program (eventually a success) and to retrieve all my downloaded mail folders/emails (years of cataloged info) a continued fail despite a backup drive.

I REALLY want that information back. Any thoughts?

Apple's latest suggestion is to upgrade again to Big Sur but I can't bear any other computer trauma at this moment.

Just an FYI, that Big Sur 11.1 is now available.

There are no indications, none, of anything related to Mail fixes in the What's New:

https://apps.apple.com/us/app/macos-big-sur/id1526878132

I will update from Catalina to Big Sur now, with this release. However, this is likely to be the make or break release for me in deciding to move to Outlook.

If Apple has not resolved the Mail issues at this point, given the additional time post the initial .0 release, I have no hope Obi Wan...

I look forward to experiences from others here, given the lack of predictable behavior under differing circumstances.

A quick follow up, which is that I am now on BS 11.1. The upgrade went smoothly, having been on the most recent version of Catalina, just before the update today.

The version of Apple Mail that was on BS 11.0.1, the prior first patch release, is Mail v14 (3654.20.0.2.21) according to a friend.

The version of Apple Mail that is on BS 11.1, is Version 14.0 (3654.40.0.2.32).

So, there is a "small-ish" incremental update for 11.1.

Let's see what happens...

Thanks, Marc! I am very much looking forward to the reports, as I am anxiously waiting to move from 10.14.6 to 11.1.

@David, don't celebrate too soon...

Looks like lingering issues with Mail on 11.1:

https://mjtsai.com/blog/2020/12/14/macos-11-1/

I will say, qualitatively, that BS seems to be faster than Catalina, so one gain.

Also, for those following along, there were a number of MS Office updates yesterday, including for the "new" Outlook macOS UI. MS has apparently added support for iCloud accounts, which is incremental progress. However, as far as I can tell, there is still no support for local folders, or other IMAP based accounts. Thus, if you wish to move to Outlook and need local folder functionality, as opposed to purely online storage, be sure to stay with the older UI for now, which is a toggle in the UI. More info here:

https://9to5mac.com/2020/12/15/microsoft-office-apple-silicon-icloud/

There are other limitations for the new Outlook, and folks can keep track here:

https://support.microsoft.com/en-us/office/the-new-outlook-for-mac-6283be54-e74d-434e-babb-b70cefc77439

as well as pending feature requests here:

https://outlook.uservoice.com/forums/924856

Just wanted to report that i’m having this issue on macOS 11.1. noticed missing messages from a date range for my icloud mail account in an IMAP server mailbox. restored that mailbox from Time Machine, imported it to Mail, found the missing messages, dragged them into the server mailbox and... they vanished. repeated the same recover from TM steps and this time, the “missing” messages were not in the restored mailbox!? restored again from a day earlier and now they were there. used “copy to mailbox” command this time and...messages appeared in server mailbox then vanished.

frustrating.

Note re my above post: it seems that only specific messages are affected. i’m able to move other messages from icloud server to local and local to server with no issues. but I cannot determine a commonality amount the affected messages (date range seems possible but that makes no sense and i don’t have enough messages to test.)

I have two questions, which may help me decide whether or not I should upgrade to Big Sur:
#1: Does the mail loss problem occur only when messages are moved between *different* mail servers (or between IMAP and local mail storage), or do mails get corrupted also when moving them within the *same* IMAP server?
#2: If moving causes the problem, would it be possible to work around the bug by first *copying* messages to the new location and then deleting them in the original location?
Thanks for any insights.

@gue 1. It most commonly occurs when moving between different storage locations, but some people have reported the problem even within the same server. 2. You can try but from what I’ve seen the copy doesn’t actually produce a copy.

I waited over a year for apple to fix this issue and moved from macOS 10.14 toward macOS 11.1 now. As I'd many local archives due to size limitations of various IMAP servers, I now wrote a script to download and detach large mail attachments from the servers and to upload the local archives. This way I can avoid using local archives and therefore avoid the bug at hand and additionally I've all my mails on the IMAP servers again (which is also handy). Maybe it's of interest to you: https://github.com/AlexanderWillner/MailboxCleanup

@Alex To my knowledge, the remaining bug is with moving messages to/from server mailboxes. Local archives should be fine.

The problem still is happening with Big Sur.

Oddly, I haven't had this problem until January 2, 2021 -- the first time after the New Year that I checked my email on my computer. I had the latest Catalina OS at the time. I updated to Big Sur with the hopes of a magic bullet, but to no avail. When moving the mail from the Inbox to a folder, I have different things happen at different times. Sometimes, the attachments are gone. Sometimes everything (From, Subject, Body, Attachment) is gone...and everything in between. Thanks for this post, since I couldn't find any other reliable, detailed discussion online about this.

@Marc, thanks, no celebration, but I am taking the plunge. I have backed up to Time Machine and Backblaze, and made a SuperDuper clone. I am booting from the clone now. If it works as expected I will reboot and upgrade to 11.1.

Update: The upgrade was smooth enough, and so far everything looks more or less fine in Mail. All the mailbox message counts look correct except that a few smart mailboxes containing thousands of messages have 1-6 messages more than they did before. All the totals in "account info" for each account looks correct too. The only issue so far is that I have multiple Drafts/Junk/Trash located "On My Mac", most of them empty. I wonder if I can safely just delete those?

Hi All,

For those who may have missed this story, and may be considering moving to Outlook, or perhaps have already done so, Michael has coverage here:

https://mjtsai.com/blog/2021/01/04/project-monarch/

and Windows Central has coverage here:

https://www.windowscentral.com/project-monarch-outlook-web-universal-email-client-microsoft

The move to a unified, cross-platform, web-based Outlook application may have functional implications for Mac based Outlook users in the future, perhaps including support for third party IMAP accounts, and offline (local storage) of e-mail and associated content.

Given the time frame before "Monarch" would be available for general use, it is reasonable to consider that the functional specs could evolve from what may be implied today. Something to watch very closely, if you are considering moving away from Apple Mail.

This will certainly give me pause to make the move now, given the confusion over the current functional limitations of the "new" Mac Outlook application for Big Sur, and what may be less incentive now for MS to allocate resources to develop full functionality for that version of the application, since it would be deprecated once Monarch is released.

I'm considering switching to Big Sur, but with regular backups of all my IMAP folders. Does anybody know of a way to check the validity of the mail messages on an IMAP server? Maybe an AppleScript that scans the contents of all folders on the server? I would then use this script for a sanity check before creating a new backup, and also for testing whether my mails are still OK or should be restored from the backup. I could create such a script myself, but I don't know which kind of "damage" the script should check for. Has anybody tried something like this?

@gue I don’t think a script can check for damage because the issue is that Mail tells the server to delete the messages. The server then thinks you intended to delete them. So there’s no “damage” on the server; it just no longer has your messages.

Hello,

Have stayed on Mojave 10.14.6 since this problem was identified, and am blown away that it still hasn't been solved. I have a massive IMAP email account that has 100s of thousands, if not millions, of emails since the mid-naughts hosted on my own service. Being MacOS/iOS centric I can no longer wait to upgrade. So I'm going to have to find a new Desktop client at minimum, and ideally the same on iOS. Such a shame. Does anyone have recommendations for a client for someone in my position (100% IMAP, want the same thing as Apple Mail but it works / doesn't eat things) for "almost free" (2.99/year would be ok)

I use Thunderbird, which is still being actively developed.

I’m in nearly the exact same position, and have also decided that I need to give up on Apple fixing Mail and move to something else so I can upgrade MacOS. Earlier this month I started migrating to MailMate.

http://mailmate-app.com/

I can’t recommend it highly enough. You can use it like Mail, but it is so much more powerful and customisable. It’s built around IMAP, so will be perfect for you, and should just work. I have all my old emails (from the mid-1990s!) in local folders in Mail, so migration’s taking a bit longer. But if you have everything on IMAP already, it will be quick, and you can run both for a while if needed.

Pricing doesn’t require a subscription, and there’s a free one month trial so you can give it a whirl. I now don’t know why I didn’t move years ago, MailMate is really excellent.

I'm losing e-mail content every time I move e-mails from in-box to another mail-box. Running Catalina. Seems to be e-mails received via Microsoft Exchange accounts... but they're the most important accounts! I've rebooted, rebuilt mail boxes and generally pratted about but the content isn't reappearing.

I've started to look into a new mail client, have a mixture of pop and imap accounts, so not sure MailMate is right.

Meanwhile, any news on whether upgrading to BigSur would solve the problem?

Regarding Adam's question, I am in the same boat - 1.1 million messages stored on Gmail via IMAP. I tried the various programs recommended in this forum and elsewhere and settled on PostBox. I think it's a personal thing as to which one you like, so I suggest giving several of the best ones a trial run. I chose Postbox not because it was the best at handling such a large archive (it isn't), but because I liked the feature set and look-and-feel the best. Personal choice.

Postbox works "OK" but can be slow at times with such a large archive. It also gets really slow to use when you have very large mailboxes - like 100K-200K messages in a single mailbox. Breaking those into smaller (20K messages) mailboxes seems to help, as does telling Gmail to make them invisible to Postbox. (In other words, 20 mailboxes with 10K messages each is much better than 1 mailbox with 200K messages).

The Postbox people were very helpful. When I told them that I had a million messages, they basically said to either (a) "don't use Postbox" or (b) take many of those message offline. I only partially followed the latter advice; I took the very large mailboxes offline since I didn't need them to be accessible and I broke up the others into smaller mailboxes. I currently have 700K messages in Postbox, spread across maybe 40 mailboxes and its OK much of the time and slow (5-seconds of spinning beachballs) a small proportion of the time. I found that manually indexing all 700K messages helps a lot. Took a lot of tinkering but I'm at a place where I've stopped using Mac Mail completely, so I can finally upgrade beyond Mojave. Oh, and Postbox slowdowns aren't always solely because of Postbox -- I've found that it gets slow during a Time Machine backup or when other things are going on on my computer.

The other thing I noticed is that I find myself using Gmail in my browser to do searches. Postbox search sometimes works but sometimes doesn't, and I haven't yet taken the time to figure out what I'm doing wrong.

Hope that helps.

Hi All,

Just thought that I would note, FWIW, that I have been testing PostBox for a few days now, under their trial. I am not (yet) advocating it as an alternative to Apple Mail, but having looked at other alternatives, since my prior comments above regarding the direction of Outlook (https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/#comment-3378775), it seems to fit, at least functionally.

In the course of reviewing other apps, some key differentiations that I found. I will keep these as key functional points to consider, without tying them to specific apps.

1. Does the client support only IMAP, or does it also support local folders. Many of the popular clients are IMAP only, which means that if you have a lot of e-mail in local storage folders in Apple Mail, which I do, they will not work for you. This would generally also infer the lack of POP account support, if you need that. If the client is IMAP only, consider what your online storage requirements may be, since all IMAP folder content will be online, and what that might cost you, if you are not already paying for that storage.

2. Does the client require you to use their servers for functionality, including push notifications. If so, what does that mean in terms of ongoing costs. This can also be a key privacy and security consideration, including for your business accounts. So think about this, especially if they may share usage and other data with third parties. Be sure to fully understand their data access and use policies.

3. Is the client a "pure" e-mail client, or does it have its own calendar and/or contact functionality. Consider this if you want to simply use a drop-in replacement for Apple Mail only, and still use Apple's contacts and calendar apps for that integrated functionality.

4. Do they provide a mobile app. I will keep using Apple Mail on my iPhone, since that appears to be stable at present and it integrates with Apple's contacts and calendar on the iPhone. However, if you want a consistent UX with macOS, this may be a consideration for you. If so, can you continue to use the Apple contact and calendar apps, or do you need to use the third party apps, and how does that affect general phone usage. The sandboxing on iOS places some read and/or write restrictions on access to Apple's apps and their data, and this was an issue for me using Outlook on iOS.

Hopefully, this might help folks to some level.

Marc,

That's a really excellent summary of the issues to look out for!

One footnote to point 1 (as I was in a similar situation as you) is that it's relatively trivial to run an IMAP server that can only be accessed from the Mac it's running on (essentially providing an 'account' for local email storage). I realise this isn't for everyone: it's fiddly, requires extra storage (as both the IMAP server and email client will store a copy of your mail), and requires a certain level of technical ability. But for those who are interested in this, I followed the easy-to-follow instructions here:

https://xdeb.org/post/2014/03/07/running-dovecot-as-a-local-only-imap-server-on-os-x/

Jolin,

Thanks for your kind comment.

On running a local mail server, I would agree that it is not for the faint of heart. I ran RH and Fedora Linux on Dell Laptops for a number of years, starting with RH 8, up through the low teen versions of Fedora, before moving to macOS full time. The local mail server installation that I had back then took a fair amount of tuning and maintenance, especially when there were OS version updates, which was not trivial with Fedora, given the rapid release cycle.

Those are technical headaches and time requirements that I am glad to be rid of. :-)

Hi All,

Just a quick note that with today's Big Sur 11.2 update, there was a small incremental update to Mail from Version 14.0 (3654.40.0.2.32) under 11.1, to Version 14.0 (3654.60.0.2.21) under 11.2.

There is no explicit reference to Mail in the release notes at all.

I've been losing email content when moving emails to local folders on Apple Mail since at least last summer (2020). I've always updated my system and am now on 11.1, but it keeps happening. It's strange because it doesn't happen to all emails. It seems random, but when it happens everything but the email header disappears, which is a major problem for me since I'm an attorney and must be able to access emails going back several years.

I've been searching for a solution without success but have noticed 3 things that could be helpful to some:
1. The content does not come back if you try to move the email back to your Inbox. Hopefully not, but it seems as if the content has been fully deleted.
2. The email cannot be printed or saved as a PDF. You get the "Generating Content to Print" message and it hangs until you cancel.
3. If you try to find the email with the missing content in your user/Library/Mail/V8 directory, it simply does not exist. It's like the email got lost between your email server and the local folder and is now lost in the matrix.

If anyone has found a solution to getting those emails back, please post. I have resorted to not moving any emails to local folders while this continues to happen so I now have thousands of emails in my Inbox which isn't workable in the long term. I may be mistaken but sadly it doesn't appear that Apple is doing much about it at this point.

@Andrew Do the messages show the full content before you move them to the local mailbox? If so, they may be in a Time Machine backup of your account’s folder under V8.

Going forward, if you need to make a local copy of the e-mails, you could try telling Mail to export the whole mailbox or use EagleFiler to archive the mailbox or select messages. This would give you a safe copy outside of Mail, and you could also import it back into Mail as a local mailbox if desired.

Thanks, Michael. Those are good ideas for recovering lost emails and any more I may lose in the future, but I wish Apple would fix the problem because it seems fairly widespread. It also doesn't seem to be an issue caused by any particular email host since I use several different ones (gmail, MS Exchange, yahoo, and aplus.net) and it happens with all of them. So it's clearly an Apple Mail issue. Anyway, I just updated to 11.2 and am hoping this update takes cares of it. I've tested it by moving a few emails to local folders and they seem to be there so far so we'll see if that changes. I'll let everyone know.

Andrew I have the same issue as you. All is the same. I updated from 11.1 to 11.2 and the content is still missing when I move message from inbox to my local folder on Mac. No change unfortunately.

I have upgraded to Big Sur and lost all my gmail emails. All other email accounts were fine. Only gmail affected. But I did lose 15 thousand emails.

Same issue here, bought an M1 Macbook Air (Big Sur 11.1) and replicated my old machine via a Time Machine backup (outgoing machine was still pre-Catalina). Like a lot of users in these comments, for years I've used 'On My Mac' folders to file emails. Everything with Mail seemed fine from late Dec 2020, but from 12 Jan 2021 selected emails moved from inbox to On My Mac folders now lose all body content and attachments - mail is hosted on an Exchange account (Office 365). Have turned off Mail on any other synced devices (e.g. iPhone) - this has not helped.

I seem to be lucky in that the data loss is constrained to emails moved from 12 Jan 2021, not to historical emails in On My Mac folders. Although flicking through I'm seeing a blank email here and there, so hoping that this loss won't be ongoing in the background.

I've used Macs for just on 25 years and have done so as I've always felt in control of my own machines and content. This situation is really disheartening, my laptop is now playing 'Russian roulette' with important data... If anyone has tips on 3rd-party mail clients able to host 'on my mac' folders and with a similar interface to Mail please share your thoughts - without a fix from Apple very soon I'm feeling a move away from Mail is imminent!

@David,

For some general topics to consider in reviewing third party e-mail clients, see my comment above:

https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/#comment-3389890

That being said, the ability to host local (On My Mac) e-mail folders, which generally means avoiding IMAP only e-mail clients, is a key functional requirement for me.

As I have noted here previously, I lost over 30 Gb of e-mails during the Catalina upgrade.

I have given up on Apple fixing the Mail bugs after a year and a half, have made the decision to move to Postbox (https://www.postbox-inc.com), and am now in the process of moving all of my e-mail to that application. Postbox is effectively a drop-in replacement for Apple Mail, while still allowing for the use of Apple's Contacts and Calendar apps.

As I have two sets of local e-mail folder trees, one live in Apple Mail on Big Sur currently, and one in an archive that I luckily have from October 2019, prior to the Catalina upgrade, I am using Michael's EagleFiler to merge both sets of folder trees, and then using the associated Remove Duplicate E-Mail script to remove the post-merge duplicates of e-mails. I should be done with this process by the end of the weekend, and ready to go into next week being rid of Apple Mail.

@Marc

Thanks very much for the comprehensive reply and sorry to hear about your mammoth data loss. I hope the shift and merge goes as smoothly as possible and much of the loss is reinstated from that archive.

It feels like a retrograde step to be moving away from the latest version of Mail, but I'm getting the feeling that it's the only workable option. Postbox does have great reviews from what I've seen and I think I'll head that way also as local ('on my mac') folders are crucial to the way I work also. Thanks again for the pointers and background!

Have there been any reports of Mail deleting e-mails in response to actions taken through other clients when Mail is connected via IMAP?

Say I have Mail open. I then, through my webmail client, move messages around. Mail, after a short delay, will update its copy of messages. Does anyone know if Mail has a chance of deleting these messages that get moved around on the server?

@Nathan From what I’ve seen, the problem only occurs when Mail itself moves the messages.

I've narrowed my issue down to emails with attachments only. Emails without attachments can be moved to local folders without a problem, but when I move an email with an attachment it first just shows the header without any content or any attachments in that folder. Then, as I've discovered, if you try to "Rebuild" the mailbox, the entire message will disappear from that folder, as if it never existed. It's not deleted or archived-it's just gone. Even if you look for the email in your [user]/Library/Mail directory, it's not there (before or after rebuild). Thus, backing up doesn't really help unless you can locate and recover it from your earlier Inbox directory.

I now have more than 1500 emails with attachments sitting in my Inbox which is a real problem for the way I've been handling emails for more than 11 years. If this can't be fixed soon, I may have to look into switching to a different email client such as Outlook (which I eagerly got away from 11 years ago) or Postbox.

Hi All,

Just an FYI, that I had to call Apple Support today, because of lingering iCloud Mail service issues, that are related to the outage that was reported last night. These are server side issues, not Apple Mail macOS client application issues.

In the course of the discussion, they escalated my call. Once the primary discussion on the issue ended, I asked if there was an awareness of the Apple Mail issue that has been discussed here since the fall of 2019. The person that I was speaking with was not aware of it.

She had sent me an e-mail to use to reply to, with any updates on the iCloud server side issues. I used that e-mail to send her the URL to this blog post while we were on the phone, with a request that she review the post and/or pass the URL on to others that can escalate and raise this issue internally for further discussion.

We will see where it might go from here. Hopefully, at some point, it will get to the right people and be addressed.

I have this issue quite frequently, mostly with Gmail accounts. I used to believe that it was mostly happening when moving mails with attachments or moving mails between folders on different accounts. But today I observed mail loss while simply moving mails (without attachments) to a subfolder (nested folder). The two mails I lost were moved individually (dragged) to that folder and were not visible within that mailbox. I closed down Mail and checked Gmail on the web and found those two emails to be in the exact location where I dragged them to. All seemed fine. I then relaunched Mail and the mails were deleted from Gmail as well and nowhere to be found.
Conclusion: Mail appears to be fundamentally broken. :-(

Adding to my previous post: Same issues with Mail on Catalina as on Big Sur 11.2.3. Mails keep vanishing.

I can consistently reproduce mail data loss on 11.2.3 (20D91) doing the following:

1) Selecting the "All Junk" Mailbox (I have several accounts with Junk folders)
-> Message count says 1217
2) Create a new Rule which triggers when all of the conditions are met with condition 1) being "from" contains a word appearing in the from Name of a few emails and condition 2) "from" ends with .net (domain of those emails) and moves the emails to a local Mailbox folder
3) After saving the rule it asks whether to apply to currently selected Mailbox, click yes
4) There is some activity in ~/Library/Mail/V8/MailData
5) Some mails vanish from “All Junk” and the Message count goes down to 1185
6) In the local Mailbox folder there are only 29 messages
7) -> a loss of 3 mails
8) The lost emails are deleted from IMAP servers as well

When I did this the first time I had 13000 mails in “All Junk” and 11000 mails got lost!!!

It is so outrageous that Apple sells crap like this for money that I am really wondering how to get media coverage for this issue.

Hi Chris,

Thanks for the report. It would be great if you could report to Apple. so that they can fix it:

 Feedback Assistant
https://feedbackassistant.apple.com

Thanks again!

I see that it's not only me with this horrendous Mail app. I run Big Sur 11.2.3, and waiting for an update that will finally debug it. I wonder if Apple will sometime regain the good name as reliable software developers or will let us poor users simply cry out loud and do NOTHING o help. I regret the moment I up-dated to Big Sur, I was a bitMac fan (not anymore)I would re-name it to Big...
I am seriously thinking to step back to Microsoft.
Sorry for emptying my heart folks!

[…] Der Programmierer Michael Tsai teilt Datensätze über Benutzer mit Mail-Problemen in macOS Catalina, insbesondere fehlende oder unvollständige Nachrichten. Manchmal sehen Sie nur Überschriften und auch eine leere Nachricht. Tsai rät, noch kein Upgrade auf macOS Catalina durchzuführen, wenn Sie dies derzeit getan haben teilt eine Methode, um dieses Problem zu beheben […]

I have been having issues with mailboxes on an IMAP server. Turns out even though it was setup with IMAP, as soon as any files got moved it somehow deleted the files off the server and only stored local copies. Sent mail never made it to the Sent folder on the server. Trying to move to a new computer became a nightmare... I moved the Mail v7 files to the new computer and imported, they went into a local folder, but any email with an attachment wouldn't move into the new synced IMAP folders. Anything with an email attachment wouldn't export to eml or mbox.

Someone made a tool on github that will take your mbox folders from the V# mail folder and convert all the emails to eml. It will attach any attachments that are there, otherwise it will "attach" empty files with the name of the attachment. Not ideal to loose attachments, but apple mail already lost them, not like I can open them in there either. use the --ignoreErrors flag. I converted one folder at a time, Sent, Archive, Trash, Inbox, e.t.c. Now all of my emails are in .eml and can do what I want with them.

Here is the link to that tool
https://github.com/qqilihq/partial-emlx-converter

@Nick Of course, I recommend my EagleFiler app for converting Mail’s partial .emlx files to standard mbox or .eml files.

@Michael, Oh I assumed based off the size of this comment thread and skimming through many of the comments that it wasn't working all the way with attachments or something... idk. Your app certainly would have helped me out, I will keep that in mind if that happens again! (Definitely looks simpler than how I did it)

Has this serious data loss bug been fixed in the latest macOS 11.3 (20E232) Big Sur released yesterday?

@MaX I’m not sure yet. I am still seeing the same old other Mail bugs where messages are not always moved properly.

I'm running 11.3.1 and I finally took the plunge to Apple Mail, even after reading this thread. I was using Postbox or MailMate before. I wanted a simpler client, because I don't have large email needs. Both of my accounts are hosted with Fastmail, that may be an important fact. I haven't had any problems, yet.

The data loss issue is present in 11.3.1 as well (Mail.app version 14.0). I lost probably a hundred emails this morning when moving mail between folders.

I have just had my first personal experience with this issue. My mother reported that two e-mails vanished when she moved them from a Junk folder (which I believe is local) to an IMAP inbox.

Any reports from 11.4? Mail has been bumped from 3654.80.0.2.43 to 3654.100.0.2.22.

@David Yes, I have reports (the update from 2021-05-28 and others) of this occurring on macOS 11.4.

Has there been any narrowing down of the conditions under which this bug is triggered? From local to IMAP? From IMAP to local? Between IMAP folders on the same account/server? Between IMAP servers? Is Exchange affected? With or without attachments?

@David Moving within the same IMAP/Exchange account seems to not trigger the bug. It occurs with or without attachments.

Thanks! It is amazing that a significant data loss issue is still unresolved almost two years later. Besides the phone call described above, has there been any feedback or acknowledgement from Apple?

Does the bug affect both IMAP to local and local to IMAP?

How about between IMAP/Exchange accounts?

@David I have heard from several people that Apple is aware of it and that it has been escalated internally. However, I’ve been hearing that for over a year. No acknowledgement on the feedbacks that I filed.

Yes, the bug affects moving to/from all types of accounts.

@Michael Just for renewed reassurance, this is still an IMAP-only bug which does not apply to POP, correct?

@Steve There was a bug (now fixed) that would delete POP messages when migrating from a previous version of Mail. I don’t think POP messages have ever been subject to the “moving messages” bug (except if you try to move them to an IMAP or Exchange account).

@Michael Yes, I recall discussion of the POP-related migration problem. Thanks!

@Michael, Thanks so much for hosting this discussion.

On a new macOS 11.4 Big Sur install, I added one IMAP account (Gmail) and one Exchange account (Outlook.com) in Mail.app 14.0 (3654.100.0.2.22).

Then I repeatedly moved messages between the IMAP account, Exchange account, and a local mailbox, trying every possible permutation.

Yet, I was unable to reproduce the missing body/attachment bug, even after rebuilding the mailboxes.

My main concern is losing messages or attachments when moving from IMAP accounts to local mailboxes. Do you know if this issue has persisted for any users?

@Miles I’m not exactly sure what you mean by persisted. From what I’ve seen, a given Mail setup is either very obviously affected for all operations of a particular type or it is not affected. So my guess is that you are in the clear.

@Michael Thank you for your reply. By "Do you know if this issue has persisted for any users?" I meant whether you knew of any user affected by the mail loss bug in 11.4 when moving email from IMAP accounts to local mailboxes. From what you shared, I gather that some users are still impacted as of 11.4.

I was able to reproduce the iCloud Junk folder bug described by johnnybrandom[1], but not the Exchange issue outlined by Robert H[2], nor the rules-based trouble explained by Scott Morrison[3] (tested with IMAP).

[1] https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/#comment-3353163
[2] https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/#comment-3119393
[3] https://mjtsai.com/blog/2019/10/11/mail-data-loss-in-macos-10-15/#comment-3160952

@Miles Yes, I have at least one report of a message disappearing when moved from IMAP to local on macOS 11.4.

@Michael:
Do you think I'm safe if I use Big Sur to only read and write mails, but not move them? Then I'd just need to keep a machine with Mojave around in case I needed to move mails.

Do you know if copying from IMAP to local (and deleting on the server afterwards) instead of moving will also trigger the bug?

Have you received any reports on Monterey yet?

@tajmahal Yes, that should be safe unless you have rules that move the messages. Copying instead of moving does not work, from what I’ve seen. No Monterey reports yet.

@Michael: Why would copying instead of moving not work? Will copying from IMAP to local by itself already trigger the bug and lead to the messages being deleted on the server?!

Thanks for the hint with the rules! I'd probably have overlooked these.

@tajmahal I don’t recall the details at the moment. I think maybe the copy works but then deleting one message deletes both of them, or something. But maybe that’s only when copying to another server mailbox. Perhaps it’s worth trying on a test message to see how the current version works for your use case.

It has taken me months and months to find this site.
I have experienced the nasty loss of emails many have mentioned when moving mail to local folders.
It is the greatest annoyance I have experienced in my computing life.
I will now consider:
1. not moving any mails out of inbox.
2. trying another mail client- Postbox seems to have been mentioned favorably here- I do not look forward to migrating hundreds of local folders.

I am BLOWN AWAY that this issue has not been resolved...

I am currently on Big Sur- 11.5.2 and the issue definitely persists

I have several email accounts but primarily use an Exchange account where I see this loss when moving emails to local folders.
It is somewhat erratic- but there is also some consistency.
I don't know a more accurate way to describe it

I am currently on Big Sur and use Postbox. But I am not quite happy with Postbox and consider moving back to Apple Mail, if it turns out to be safe for me. Since I read that the bug either exists for a certain user or not, I want to find out whether I am affected or not.
I came up with a testing procedure that hopefully triggers the problem if I am affected. Of course, I hoped that the bug would not occur and I therefore can assume that I can safely return to Apple Mail.
My testing procedure looks like this: I use 7 mail accounts (iCloud, Gmail, and a company server), all with IMAP. I sent a testing mail with an attachment (PNG picture) to the first mail addresses. To test moves within the same IMAP server, I moved the message around between various IMAP folders. When this succeeded, I moved the message to the Inbox of the second IMAP account, then between folders within that account, then to the next mail account, and so on. After the last mail account, I moved the message to a local mail folder and again between various local folders.
For each move, I checked whether the message has disappeared from the source folder and then correctly showed up in the destination folder (including the attached picture). The good news is that all these tests succeeded. After about 30 moves, the message was still intact.
QUESTION: Is this test enough to ensure that I am safe? Or would anybody suggest additional experiments?
I'm hoping to come up with a "standardized testing procedure" that could be used by others to find out whether they are affected by the bug.
Thanks for ideas and suggestions. gue

@Günther That seems like a good test to me.

To Günther Blaschek

I think that's a good start.
But I don't think you can feel "safe" after your procedure.
The "bug" might not appear with one email, but might with another.
As I attempted to describe in my post a few days ago, there is some regularity to the bug but also some inconsistency.
For instance emails from a certain sender that I have filed away into "On My Mac" created mailboxes for a period of time, may all of a sudden show the bug.
Interestingly it seems like there is more of a likelihood emails I receive from Mexico will display the bug.
It is truly bewildering and infuriating.

Hello,

I've tested Monterey beta 6 with Exchange and Mail.app 15.0 I am unable to reproduce the bug of losing the body of a message during message move. I've tried moving messages from a folder on the server to a local folder and local sub-folder. It works as expected preserving the body of each message. I've tried moving one at a time and two at a time. It still works. I've tried moving messages between a server Inbox sub-folder to a local sub-folder. It works as expected.

The only thing approaching a disappearing message body was the list view of messages intermittently did not have the body appear briefly. Clicking on the message in the list view reveals the entire message in the view pane. The list view eventually updates and shows the summary lines of the message. This seems to be a separate non-destructive bug that is merely annoying.

I am not willing to definitely say the move delete message bug is fixed as these are only a handful of moves on a simple folder structure during a test instead of daily use. However, it is encouraging results.

@Josh Were you seeing the bug before Monterey? Many people never did. But it would be interesting if you were affected before and then saw the problem go away.

I had one user report the issue. We told our users to avoid using Mail.app based on your blog.

Here's a bit of a wrinkle I've recently noticed. I don't know if it has been noted previously.
As a tedious workaround to this problem, before I move an email that I want to save to a folder I have created "On My Mac", I will first do a "Copy to" that folder.
If I don't lose the content of that email, I can then delete the email from the main Inbox.
However, sometimes I will do a "Copy to" and it appears the content has been lost in the "Column" view.
But if I double click on the moved/copied to mail the content will appear.

I should add that if the "Copy to" fails and loses the content of the email, I will then do a "Move to" or "Copy to" an "Archive" folder where it will usually "survive".

I hope I have been clear.
As I stated previously this is a nasty bug...and I still haven't been willing to scrap Apple Mail altogether.

I've recently upgraded straight from macOS Mojave to Big Sur (11.6) and can't reproduce these issues. Moving emails with attachments between my Exchange Online account and local folders works OK and so does moving between folders in the Exchange account.

I'm pleased, because I wasn't looking forward to changing mail clients.

Mike Z- did you have the issues in Mojave?

any update on macos montery ?

@xxx I’ve not heard any reports of the problem with Monterey, although this may be because my customers have already transitioned to setups that avoid the bug. I have also not heard back from Apple about my bug reports indicating that they are fixed.

@Moses Kravitz - the issues mentioned here started with Catalina, I believe.

thanks a lot. i also moved to a different solution but still thinking of moving back as soon as apple mail is fixed....

Thanks for the information. Keep this thread alive! Looking forward to news about macOS 12 Monterey and the data loss bug.

long time no update - is there any confidence that the issue is fixed with macOS 12 Montery ?

I made a quick trip to Monterey. From what I can tell the issue is not resolved. I don't know if this will shed any light but there are several sources I get emails from that originate in Mexico. They are the most numerous, but not exclusively, the casualties of this BUG

I can confirm that the problem still exists in Monterey (12.1). After switching to Postbox and an experiment with Big Sur (see my previous post from September 4, 2021), I switched back to Apple Mail. I did not see the problem for a while. Yesterday I upgraded from Big Sur to Monterey 12.1. Still no problem, but today I decided to archive older mails from 2018 and 2019 (I regularly clean up my mailboxes at the end of the year). The process involves moving the mail messages to local mail folders in "On My Mac". Fortunately, I *copied* the messages instead of moving them. Then I found that some message had no contents. I could not see any pattern; the bug affected messages with pictures and attachments as well as short plain messages without any attachments. Other messages (the majority) were not affected at all.
During my previous experiments, I had also set up a local IMAP server with dovecot. I now repeated the same process and moved all these messages to mail folder on my local IMAP server. This seems to have worked without any problem. No loss of mail contents.
My lesson from this is: Do not move messages to local mail folders in "On My Mac". For me, moving messages within IMAP folders has worked fine so far. But this may be just me and my iMac…

@Günther Blaschek: Are you sure copying instead of moving did no harm? I asked Michael about this above and he replied "Copying instead of moving does not work, from what I’ve seen". I assumed he meant copying would break the original messages as well. But maybe I misunderstood that, and he wanted to say that while copying produces the same broken messages in the target location as moving, it doesn't harm the messages at the source location.

@tajmahal My comments about copying were in relation to the message disappearing entirely, not messages that are visible but with no content.

@tajmahal: Sorry if this was not clear. I did these experiments by copying instead of moving, as I feared losing the contents of messages. By copying, I still had the original messages.
And I was right: The contents of *copied* messages was lost when I copied them to local mail folders in "On My Mac". But copying to the local IMAP server (by dovecot) worked.
But please be careful: This was not a thorough scientific analysis. It is possible that different things happen on computers of other Mac users.

@Michael: I'm still not sure I understood you properly. When copying, is it just that some messages are not being copied to the destination location? Or does copying cause the actual original messages at the source location to disappear?

@Günther Blaschek: I understood you. It was just that Michael seemed to have different findings from what you are reporting.

Also, I think nobody knows yet how the "lost message" and "lost message content" issues are related. But both still seem to occur in recent versions of Apple Mail.

Also, I think it's odd that iOS is apparently not affected at all. I'd have expected Apple to use the same implementation at least for the low level mail handling stuff which is probably responsible for the issue on the Mac.

@tajmahal It doesn’t happen for most people, but I have seen cases where copying causes the original message to disappear (not always right away).

Hi All,

Been a bit since I last posted here. Happy New Year to all.

Having now been using Postbox for some time, as I had previously noted, while generally happy, I have come across an issue, for which I have submitted a bug report to Postbox. However, given their support model, I am not likely to hear back from them until a future release, if they can confirm and resolve the issue. If not, then presumably, silence. The alternative is to pay them for a per incident support issue, which, at present, I am not inclined to do, since I paid for a subscription.

The issue is that, while I have not observed any data loss, as I and many here had previously observed with Apple Mail, there appear to be synch problems when moving (not copying) an e-mail from an IMAP folder to a local folder on my Mac. It does not happen all the time, nor predictably due to attachments or other factors, but enough such that I periodically notice the e-mails are still in IMAP folders when I use Apple Mail on iOS, and when I directly log into the web sites for Office 365 and for iCloud. I then have to manually "clean-up" the discrepancy.

Thus, while arguably not as bad as the Apple Mail bug vis-a-vis data loss, it raises the possibility that there is some common implementation/API issue when moving e-mails from IMAP folders to local folders using a client that supports such functionality. Since it happens for both iCloud and business accounts on my end, it is not specific to an iCloud server-side issue.

Part of the challenge, is that one needs to be using a mail client that supports POP3/Local Storage, which many of the mail clients do not. Many popular macOS clients are IMAP only. As a result, anyone using an IMAP only client is not likely to observe this issue, much less the data loss issue in Apple Mail.

So, first and foremost, I wanted to make folks aware of this.

Second, given this issue, over the past few days, I have been giving more serious consideration to moving to an IMAP only model, and paying for additional storage for iCloud, where I can go up to 2 TB. Office 365, irrespective of the account type, only goes to 50Gb per mailbox (account), which is not likely to be enough to store all of my e-mail. So I would, in time, move all of my e-mail, personal and business, from the local On My Mac folders, to iCloud, to have it all in one place and just use the relevant account to send/reply e-mails from.

From what I can tell, there is no downside to the IMAP only approach, other than possibly not having access to mail in the absence of an internet connection, or server-side issues, where only e-mail headers might be stored on the relevant client device. However, there are some options to download complete offline copies, if I am actively working on something. On my Mac, at least, I could feasibly download all folders and content, to have, in effect, local copies.

To that end, I have been reviewing several IMAP only clients, which I had avoided in the past, given my need for local storage. These include Spark, Spike, Canary, and Edison. I also looked at GyazMail, which does support POP3, but has not been updated since last May, and AirMail, which also supports POP3, but the reviews, especially regarding their support, are rather mixed.

Part of the consideration is an e-mail client that has both macOS and iOS versions for consistency, but does not require the use of a built-in ecosystem for calendar and contacts (e.g. such as Outlook), thus, allowing me to continue to use Apple's calendar and contacts.

A number of these IMAP only clients may have you use their servers for certain functionality, but at least with one exception (Edison), appear to have reasonable privacy/security policies regarding what they may store on their server, and not selling your data to third parties, especially if you do not use functions like delayed send, etc. Some appear to allow you to not use their servers at all explicitly.

Edison, on the other hand, explicitly indicates that their business model is effectively built upon selling your data, albeit "anonymized", to third parties, which for me, rules them out.

Thus, if anyone has any input on the clients above, pros and cons, that would be appreciated. I have looked at a number of reviews online and comments in the respective app stores for other feedback as well.

Thanks!

A follow up to my last post.

It would seem that a full migration of my local e-mail folders to IMAP, whether it be to iCloud, or even to Office 365, is going to be frought with issues based upon additional research and testing. Even if successful, it is likely to take far too long to be practical.

Trying to copy, within Postbox, even single folders from local storage to iCloud fails, much less a smallish folder tree (parent folder and a few sub-folders) to try to simplify the process. This may be, to some extent, a reverse of the problem that I filed the bug report with them for, which was for moving e-mails from IMAP to local folders.

Similar behavior is experienced with trying to copy folders to Office 365 just to test that pathway, albeit, it goes further than iCloud before failure.

Again, not clear if this is a Postbox issue, or an IMAP issue, and I will wait to hear back from Postbox. If it turns out to be a Postbox bug, then maybe there will be some hope to incrementally move to IMAP at some point.

My guess is that various rate limits on the IMAP server end, with both iCloud and Office 365, may also be at play here. These could be server timeouts, bandwidth issues, time based transfer limits, and so on, and may make the migration process highly problematic in either case.

So, for the time being, I will stay with my hybrid IMAP/local folder approach, and with Postbox as the client app.

Thanks for taking the time to review both posts.

Sorry, one additional update. It occurred to me to try a different email client that supports local and IMAP folders to see if this was indeed an IMAP bug in Postbox or perhaps confirm this as a server-side issue.

Thus, having used Thunderbird in years past on Linux, and given that it has a long history, thus should be reasonably stable, I tried it. The nice thing about Thunderbird is that I did not need to take the time to migrate/import the local folder tree from Postbox, and could just copy a small subset of the MBOX tree from Postbox to a desktop folder and point Thunderbird to that. This way, I did not risk the integrity of the Postbox tree, just in case.

In short, the same behavior with iCloud in not completing the movement of the emails, even from a single folder, from local to iCloud.

For Office 365, it would successfully move a single folder, including a "large" number of e-mails and attachments, but not a folder tree. In the case of one folder, it copied ~1,500 e-mails, and all seemed to be intact.

Thus, moving local folders to IMAP would not seem to be a bug in Postbox, and be a server side limitation, with iCloud being far more problematic.

As noted above, moving single folders, much less single folder contents in batches of emails at a time, would take far too long.

A question for those willing to offer comments. I see that Fastmail appears to support a "bulk" MBOX import:

https://www.fastmail.help/hc/en-us/articles/360058753594-Import-your-mail#importfiles

Has anyone used that functionality to migrate to Fastmail? It would be yet another mail provider to support a move to a full IMAP configuration, but if I were willing to pay for more storage on iCloud, a paid Fastmail account may be an alternative to consider longer term.

Thanks!

@Marc Thunderbird and Postbox are based on the same open-source code, so that is not an independent test.

Hi @Michael,

That's an important point to consider. It was my understanding that, over the years, there was a divergence in the code base, but perhaps not enough to make a functional difference in this particular context.

Is there a different e-mail app that supports IMAP and local MBOX folders that is stable enough that you would recommend here?

Of the two that I have found, GyazMail has not been updated for quite some time, and Airmail would seem to be problematic based upon a number of reviews.

Thanks!

@Marc GyazMail doesn’t get many updates, but I do like it and have used it with success on Monterey.

@Michael,

Ok, fair enough. I will give it a try. It seems to have a trial period, which is enough to at least test this functionality. Thanks!

Airmail has little issues here or there, as any software does, but I've found it to be very reliable. I can't say I've used the built-in Mail app so I don't know firsthand what the issues are there, but Airmail has never caused messages to unintentionally be deleted to my knowledge.

@Michael,

I tried GyazMail. Unfortunately, it does not directly support MBOX files as a local folder tree, and one must actually import the folders, which would take a lot of time. There are some idiosyncrasies in that process, in that it does not seem to support/recognize MBOX sub-folders, and I ended up having to manually import each MBOX folder and then reconstruct a small local folder tree for the app. Perhaps I am missing something in that process and the available documentation was wanting in that regard.

That being said, once a small folder tree was in place, I was not able to drag and drop either individual folders or the folder tree from the local folders to either IMAP account. It does seem to support moving e-mails from one folder to one folder in both directions however, This would mean migrating one folder at a time, first creating the folder tree on the IMAP side and then copying/moving e-mails one folder at a time. Again, perhaps I am missing something. I did not see any relevant preference settings in the app, nor were Google searches fruitful.

@Kevin,

Thanks for your comments on Airmail. At least for the purpose of this process, I may go ahead and try it and get a sense for the app. There have been reviews and comments suggesting IMAP stability issues with the app, failure to send, etc. which is why I am hesitant, since that is the focus here. There are rather mixed comments on their support as well. But, if it is a matter of trying to isolate app versus server side limitations in e-mail migration, it may provide some insights, depending upon the IMAP/local folder movement functionality that is supported.

Thanks!

I have had some issues in Catalina with "no content" emails - Sender, Recipient, Subject and nothing else. I have been able on some of them to "resurrect" the content in a few fortunate cases by choosing View, Message, Raw Source. If I see content there, and do it once or twice, and perhaps select a different email in between, I have gotten contents back. Don't know if this has already been mentioned in this thread.

long time no update here. is the issue in the meantime resolved so that is safe to move back to mail ?

@anti I updated the post 2 days ago. The problem does not seem to be resolved in macOS 12.3.1, but my experience is that the bug can usually be avoided if you don’t move messages cross account.

@Michael: Does "cross account" also include "server to local"? Or is this still considered unsafe?

@tajmahal Yes. My experience is that server-to-local is OK but that local-to-server sometimes doesn’t work.

Hello Michael,
Does server-to-local mean that it is now safe to move messages from the inbox to a local folder without loosing the content? However, you mention that "The problem does not seem to be resolved in macOS 12.3.1".
Thanks for clarifying

@Stefan Yes. But local-to-server and server-A-to-server-B still don’t work for some people.

@Michael,
I am still confused. @Günther Blaschek recently reported: "The contents of *copied* messages was lost when I copied them to local mail folders in "On My Mac". But copying to the local IMAP server (by dovecot) worked."
This suggests that moving or copying to a local mail folder in "On My Mac" is still causing blank messages to be stored.
Could you please clarify?
Thanks

@Stefan Sorry, I had forgotten about Günther’s report. I have not heard of others like that recently, nor seen that issue myself, but unless it was fixed since macOS 12.1, I guess you should be careful about server-to-local, too.

Just updated to Monterey 12.3.1. Issue has reappeared (3rd time having the issue over the last 2 years). When moving mails to local folders they become empty/blank. Last time I lost around +10k mail and this time around +5k mails - all business related. Spend many hours to find "workarounds", but seams

@Thomas,
Thanks for sharing your experience. I am surprised that after 2.5 years of complaints there is neither a solution nor any insight why this is happening. Very concerning

@Stefan: Please note that copying was not part of the problem; it was just a precautionary measure on my side. When messages had been corrupted during a move, I would have lost these messages. So I decided to copy the messages first. And when that had worked, I would have deleted the original messages. As it turned out, some copied messages were indeed corrupted, but I still had the original messages.
(I guess that moving messages to a different IMAP server or a local mail folder is internally done by first copying and then deleting the original messages, anyway; but this process would not check for corrupted items and the refuse to delete the originals, so I did this manually.)
Instead of using local "On My Mac" mail folders in Apple Mail, I had then set up a local IMAP server (see the posting by @Jolin from January 28, 2021, which includes a link to instructions). Since then, I have not seen any message corruption. Now I'm running Monterey 12.3.1, and all other operations (archiving messages, moving them manually to other IMAP folders or to my local IMAP server) have not caused any trouble. I just avoid using local "On My Mac" mail folders.

@Guenther
Thanks for pointing out the workarounds. It is good to know that setting up a local IMAP server circumvents the "On My Mac" mail folder issue. I have not see how this integrates with the Apple Mail GUI, but assume that it just show up as a another account. However, I am concerned about moving my existing "On My Mac" mail folders. What is the experience with these?

Just saw this bug (moving an email w/attachment from IMAP to "On My Mac" resulting in empty/blank message) under macOS 11.6.6 Big Sur and Mail 14.0 (3654.120.0.1.13). Searched the ~/Library/Mail directory with ack; the message was gone. Performing a rebuild on the mailbox made even the blank message disappear.

I am running MacOS 12.4 and Apple Mail 16.0

I have been experiencing this IMAP mail loss intermittent bug when moving a message from my Microsoft Exchange Inbox (via drag-n-drop or Rules) to:

1. Offline "On My Mac" folders - I have been using Rules for years to move incoming messages to "On My Mac" folders, so when I first noticed this problem I was on Mojave. I updated to Monterey for other reasons and the problem persisted, so I disabled my Rules. Am still seeing lost message bodies.

2. Online IMAP folders - which I created on the same Exchange server that the messages arrived on, based on some ideas in this thread. I was hoping that using the Exchange Online IMAP folders wouldn't have this problem, but it persists.

Curiously, I have not noticed this message loss for emails which arrive via my IMAP account hosted by Spectrum (@austin.rr.com)

DETAILS:

I use the three-column pane view in Apple Mail: i.e. Folders / Messages / Message content. When this intermittent happens, the Messages still show who it's from and the Subject line, but no Message body (preview), and the rightmost Message window is totally blank. If I select one of these messges in the middle pane and do a:

Mail > View > All Headers, then the rightmost pane has only the Date and "(No Subject)"

Mail > View > Raw Source, I get a pop-up with only the Subject field in the titlebar.

@Andy. I have found that deleting all existing mail rules solves these problems. I am going to leave it a day or two to find out whether it is a fix and then start to recreate some rules and see what happens.

@Mark I don’t think that deleting the rules is a fix because the problem also occurs on setups with no rules.

@Michael... you are correct. ;-) For me, it was better for some emails but when I tested more thoroughly I found that anything with an attachment would 'disappear' when drag and dropped into the local mail box. I then tested a range of email types and got mixed results. Oh well, back to testing more messages.

@Andy... I have done some more testing. I deleted my exchange server account from system preferences>internet accounts (once I made sure all my emails were on the exchange server via a web interface). Checked in mail to make sure it had gone from my mail app. Quit Mail and then re-added the account and everything seems to be working correctly again. I haven't tried adding my rules back yet but I start of with some simple ones and see how it goes.

Yesterday's Apple presentation included some significant changes to Mail in Mac OS Ventura. While I absolutely refuse to get my hopes up, given the years of inaction on this issue, maybe we will be pleasantly surprised...

I do hope they fix this problem in Ventura. I also hope they fix it in Monterey as I have a fully adequate 2016 MBP, and Ventura won't support me. Losing mail is an abject *fail* for *any* mail application.

In my June 5, 2022 post, I thought a possible workaround could be to move from using On My Mac folders to online IMAP folders on the same Exchange serv er, but my ongoing experience is that Apple Mail 16.0 corrupts a significant number of Mail messages when moving them between two IMAP folders on the same Exchange server. In a small test case, I moved (via drag-n-drop) 137 messages from my Exchange Inbox to an IMAP folder on the same Exchange server (over , and 14 mail messages were corrupted. 14/137. Surely a ~10% failure rate is debuggable.

any update if the issue is solved in macOS ventura ?

I would love to know whether or not this is fixed in Ventura as well. MacWorld claims that Ventura may be final and released within the next two weeks.

It is baffling that three years later this still requires discussion.

I am also waiting for reports on Ventura. Anyone with experience of data loss so far?

Waiting also to know if the macOS 13 Ventura fixed the Apple Mail data loss.

Same here. Really hoping Ventura’s new Mail app fixes this issue. Been using Thunderbird since 2019 while waiting for this to be fixed!

Moving messages between mailboxes, both via drag-and-drop and AppleScript, can result in a blank message (only headers) on the Mac.

Sorry to say, I just caught this, for the second time, on Ventura. Fortunately the messages weren't important, and they were simply being moved from an IMAP mailbox to a local one for faster and more convenient use before being deleted anyway. But alas, the problem still persists, and I won't rely on it going forward for that purpose. (Curiously, the same problem has never happened with mail moves between IMAP mailboxes, at least not for me.) I have set up and use isync/mbsync, instead, whenever I need local, offline access to IMAP mailboxes, or for backups or migrations.

Have these Apple Mail issues --and particularly the data loss-- been fixed in macOS 13.2 (22D49) Ventura? Thanks!

In Ventura 13.2 I am still experiencing the intermittent data loss when drag-and-dropping messages from my Microsoft Exchange Inbox to "On My Mac folders", just as in my June 17, 2022 post above. Upon a message move to a "On My Mac" folder, the Message pane shows occassion messages as missing their Headers, (No Subject) in the subject line, and empty message bodies. Sigh.

I have not updated macOS for many years now, because of these problems. I am still using Mojave 10.14.6 on Mac Mini 2018. If I simply update to Ventura,now ... April 2023 ... does anyone know whether that simple act alone will endanger any of my historical Apple iCloud emails? Is it best to complete erase the SSD, and do a "clean install" somehow? Is there any danger of "bricking" my Mini?? Thanks much.

@Chris My understanding is that the migration bug was fixed a long time ago, so I don’t think there’s any danger for e-mails at rest. The problems can occur when moving messages between mailboxes.

@Michael Thanks SO much.

Has anyone experienced data loss moving messages either between folders on the same iCloud account, or between local folders and folders on an iCloud account?

I finally got a chance to test the iCloud custom domain options and concluded that they are not entirely ideal, I can make it work. (I would have preferred more than five domains, more than three addresses per domain per user, the option to forward messages to specific addresses without hitting the inbox and the option to use an address as an alias to several other addresses, oh well.)

I am currently in the progress of migrating all users from other IMAP providers, and haven't hit any obvious issues except the iCloud 20 MB per e-mail limit.

Just to note that mail messages still occasionally disappear into thin air in the latest macOS Ventura 13.4 – rebuilt an ordinary IMAP mailbox that Ventura had garbled some messages in (seemingly they had combined two into one as when when rebuilt there were two messages next to each other – one with the content and one with the subject that belonged to another message). The message with the correct subject now had no content (which might or not have been correct), but a common trick to repair the body of a message is to move it to another mailbox and then move back again. However the message were no longer to be found anywhere on the computer after the move between the two IMAP mailboxes belong to the same account. So, as everyone knows, Apple still doesn't understand much about mail ... .

Hi All,

It has been a while since I last posted here, but I wanted to provide an update.

Between using EagleFiler and some other utilities to backup and migrate mail content, which included Mail Backup X and Mail Extractor Pro, I was able to, in time, successfully move all content to online IMAP folders and stop using local folders for e-mail storage. That shift allowed me to consider other email applications that were IMAP only and did not support local storage folders. It also eases the ability to switch from one e-mail application to another, without having to worry about migrating local folder content from one to the other.

I have been using Airmail for some time, given the large feature set on both macOS and iOS, but have noted ongoing issues with e-mail content synching between the macOS and iOS versions, and some other issues, which seem to go unresolved, despite communications. The intervals between app updates also appears to be unpredictable, sometimes going for months between updates. I suspect that the Airmail folks are just short on dev resources, despite their shift to a subscription model to generate recurring revenues.

Generally speaking, I want to use the same application on both macOS and iOS, where the publisher has both, for consistency.

As a result, having ruled out other applications, and notwithstanding my prior comments above, I recently started to use Edison E-Mail. While not perfect, and I have also opted out of the data mining aspect of the application, it is working well enough that I will continue to use it for now. I am also in the process of reporting what appear to be some subtle bugs and submitting some feature enhancement requests.

Time will tell, but so far, I am liking what I see with Edison.

Hi All,

Well, it has been a few more months since my last post above, where I noted that I started to use Edison as my e-mail client on macOS and iOS.

In that time, I began to note a variety of behaviors that have become problematic:

1. On iOS, you cannot move an e-mail from the Sent folder to a different folder for organization, after sending an e-mail. This appears to be a a conscious design decision, perhaps based upon the false perception of some technical limitation. No other iOS e-mail application has this limitation that I have found, and the Edison macOS application can do this without issue.

I am not clear on what the Edison folks expect from a workflow perspective on iOS to deal with this limitation, as you either have to go into a different e-mail application on iOS to move the e-mail from the Sent folder to a different folder, or wait until you are on your Mac to move the e-mail using their macOS application.

Edison support has confirmed this limitation, albeit, without an explanation as to why it exists.

2. There are numerous font and text format rendering issues on both macOS and iOS that result in a level of inconsistency in how the e-mail body is rendered in their applications, and then in other e-mail applications. This raises concerns as to how the recipients of e-mails will see the content rendered in their application, including which font will be used, how paragraphs are spaced, issues with bullet lists, and so forth.

It is not clear if or when these issues will be resolved.

I do want to note that Edison support has been very responsive to my communications, and their app update release cycles are relatively frequent. While the above issues remain unresolved to date, other issues that I have noted have been fixed. The one thing that would be great is when they do release a new version, is to provide a list of the issues that have been fixed and any new features, rather than, as numerous vendors do, provide a generic update message.

In light of the above issues and other idiosyncrasies in behaviors with Edison, a few weeks ago, I moved to Outlook on both iOS and macOS, since I do have an Office 365 subscription. While Outlook has its own interesting issues, it has been stable so far, font/text rendering issues are not present and the workflow is reasonably productive. As you may be aware, the new Outlook on macOS is still a work in progress and MS has a page here listing features that are in various stages of implementation:

https://support.microsoft.com/en-us/office/new-outlook-for-mac-features-86bd33bc-3c8d-4c91-843e-0aaa018793ba

Since my previous comments on Outlook here in years past, MS has made some substantial improvements in the functionality on both macOS and iOS.

Outlook on iOS has some extra steps needed to go between multiple e-mail accounts, each with it own folder tree interface, whereas others have a more integrated UI. The one key feature that I wish they would implement on iOS is collapsing folder trees, and I submitted an RFE for that. There are a few others like going to the next message after deleting an e-mail rather than going back to the folder that you are in, and there are some folder tree scrolling issues that are annoying.

It honestly seems strange to me that there is not really a single e-mail application on both OS's that provides a full feature set, is stable, has a good UI/UX, and has responsive tech support with reasonably frequent release cycles. There is clearly an opportunity out there for somebody that has the resources and desire to engage in the development of an e-mail application.

[…] If you had enabled the experimental message backup feature in SpamSieve 2, to protect against the Mail data loss bug, any backed up messages will now be available in the Log window. Messages from the old Save false […]

is there any update if the issue is solved with the sonoma update ?

Still an issue in Sonoma. Did a clean install on a new M3 iMac and decided to move / archive old IMAP email to a local "On my Mac" mailbox. Most, but not all, messages lost their bodies, and it isn't just a display issue because they're still empty after moving them back. Not good.

Did somebody ever enter in touch with Apple Mail staff to speak about this bug and press for Apple to fix it as quickly as possible? It seems unbelievable that this bug has been there since Catalina.

Just did some testing in macOS Sequoia 15 beta 2 with 5 different accounts (iCloud, Gmail, Exchange, AOL, and Spectrum; all configured as IMAP or Exchange). The Mail version is reported as 16.0 (3810.100.6) (as opposed to 16.0 (3774.600.62) in macOS Sonoma 14.5). After a few thousand emails and an hour or two, I have not been able to reproduce the disappearing body or attachment bug when dragging to local mailboxes under On My Mac. Anyone out there still seeing the issue in Sonoma 14.5 or the Sequoia beta?

is there any update if the issue is fixed with the latest macos ?

This is just to report that this is still a problem under macOS Sonoma -- I have had trouble under with two separate email accounts in the last month. In the past, I have had individual message bodies disappear when moving into folders on my mac. But now, (both MS / Exchange), I am unable to move messages into a separate folder without losing some proportion of them. It is reproducible in that I can restore an old copy of my inbox from a TM backup, import it into mail as a folder, try to move all the messages to another folder, and the bug recurs. All the messages are intact on import from TM, and each time I try this different messages end up without their bodies, so I don't think the starting mailboxes are corrupt. For the moment, I'm just living with the messages segregated in the imported mailbox, which seems to work fine.

It has been a couple of major versions since I last did a test, so I just attempted moving some messages from a variety of servers (FastMail IMAP, Gmail, and Microsoft 365) to an On My Mac mail folder. Small sample size, but I have definitely experienced issues in the past with moving small numbers of messages to local storage from IMAP (where contents of messages touched by Apple Mail appear to be blank.)

50 total messages moved, mix of plain text and HTML. All made it to local storage without issue. I then moved those messages from local to a remote IMAP folder and all were successfully moved with contents intact. I still want to hear from additional sources before I consider this issue resolved though. There have been enough bugs and the mail ecosystem is so disparate that it's easy to get burned.

thomasjpr: What Mac OS were you using for these tests?

@frank Sonoma, 15.0.1

@thomasjpr Did you mean Sequoia 15.0.1? Or Sonoma 14.x?

I'm on Sequoia 15.0.1 and still experiencing the intermittent data loss when drag-and-dropping messages from my Microsoft Exchange Inbox to "On My Mac folders", just as in my June 17, 2022 and January 31, 2023 posts above.

Upon a message move to a "On My Mac" folder, the Message pane shows occasional messages as missing their Headers, (No Subject) in the subject line, and zero-byte message bodies. Sometimes an immediate Command-Z restores the file and contents. Until I have time to debug this further, I'll continue to keep all critical emails in my IMAP online Inbox. Sigh^2.

@frank ugh, I meant Sequoia. Apologies!

Hello,
I may have found an explanation for empty mail messages on MacOS 12.7.6 (Monterey). Empty messages are not lost, but possibly stuck in the Envelope Index file while not moving into the target directory (textedit will locate the subject line). In the activity window there will be an indication of messages moving without any progress. This issue started approximately two weeks after a software upgrade from trouble-free Mojave. In my case stuck mails accumulated until mail crashed and started from scratch with importing all messages. Once this was complete, there were no empty messages in any of the folders on my Mac and the activity window showed no activity. I then started moving messages to folders on my Mac including some types that previously ended up being empty and none of these were empty.
Hope this is helpful

Hi All,

It has been a little over a year since I last posted here, but have been tracking the ongoing updates here since then.

Is the general consensus now that Apple Mail under Sequoia is stable in the scenario where one is only using online IMAP folders, and that any lingering data loss related bugs are when moving e-mail between IMAP and Local On My Mac Folders only?

I had moved to an IMAP only model some time ago to enable me to have the ability to try differing e-mail apps without the need to migrate local folders, which is a feature that many e-mails apps do not have anyway.

Since my last post, I moved from Outlook, given ongoing quirks in the UI/UX on both macOS and iOS, to Spark on both macOS and iOS, using their version 2 desktop on macOS, as I did not like the version 3 UI/UX at all.

It is not clear to me how long Readdle can, in reality, continue to develop and maintain two parallel macOS versions. With the new and yet to come Apple AI features now available in Apple Mail, that begins to remove any potential feature advantages of Spark or similar apps that are including relevant AI features using ChatGPT or similar.

Thus, I am ready to jump back into the Apple Mail fold after all these years, barring any data integrity issues that folks may be aware of.

Thanks!

@Marc, I'm in the same boat, as my Mac Mail alternative, PostBox, just announced they are shutting down. I also see no good alternatives and am considering jumping back into Mac Mail.

I don't know of any "general consensus" - all I know is from the few posts here, which seem to support what you're saying - that Mac Mail seems to work under Sequoia, with the possible exception of what @AndyScott reported a few days ago. I will probably jump back into Mac Mail if I don't see any other concerning data points, and will avoid using local On My Mac folders.

@Frank,

Thanks for your comments.

I had used PostBox back in late 2021 to early 2022, before deciding that there were sufficient issues with it to make the move to try other apps, which in turn compelled me to finally move to an IMAP only schema after a number of years of using local On My Mac folders. That gave me the flexibility to move from one app to another more easily, as I noted above, and be able to consider apps that did not support local folders.

I was aware of PostBox' recent demise given coverage in some of the technical new sources that I follow. In fact, that news gave me cause to try Thunderbird again, but found it to still have too many issues for me.

After using Outlook for quite a while, both the legacy version and the newer macOS version which has struggled to meet feature parity, I actually enjoyed using Spark version 2 on my Mac. However when I tried version 3 with an entirely new UI/UX, it lost quite a bit of functionality, while also enforcing a whole new e-mail workflow paradigm, and it was a challenge. This shift was by design and at least in part, due to their migration to an Electron based app development environment, which other software publishers have also done to facilitate more efficient cross-platform development. Using Electron, you also end up with a generic looking UI, rather than being OS specific, so you lose the macOS look and feel in the app. I suspect the reason that they are still supporting version 2 after quite a bit of time, is that there is sufficient user base disappointment with version 3 to justify the resource allocation to version 2, given the continued volume of use, which I presume they can monitor. It is not clear what their future plans may be, but presumably, as I noted, they cannot support both versions in parallel forever.

Andy's message, as you note, would support the notion that there are still lingering issues with the use of a hybrid IMAP/Local Folder schema, and prior comments would seem to suggest that an IMAP only schema is stable, which gave me the thought to come back to Apple Mail, as functionally, it checks all of the boxes that I need. It really always has, it was just frustrating that the bugs in Catalina five years ago now caused so many headaches.

As you note, hopefully there will be other comments with some level of confirmation of the current situation.

Thanks!