Friday, October 11, 2019 [Tweets] [Favorites]

Mail Data Loss in macOS 10.15

I’m working on more posts about the Catalina release, but I wanted to start with a short warning. I’ve heard a bunch of reports of data loss in Apple Mail. Thankfully, none seem to be caused by my apps. (Ironically, one of the bugs I’ve encountered is the inability to delete messages via AppleScript.) And, in fact, most of the damage has occurred without my Mail plug-in even being installed. Nevertheless, people contact me because it’s not unreasonable to wonder if third-party software is to blame, and I also hear from people who want a second opinion because what Apple support told them didn’t make sense.

What I’m hearing:

I don’t know whether these are due to Mail bugs or to other factors such as problems on the Mac or with the mail server. But my advice is to hold off on updating to Catalina for now. These sorts of issues are pernicious because:

  1. You may not notice that anything is wrong unless you are looking at the particular mailbox or messages that are affected.

  2. Because the data is synced to the server, problems can propagate to other Macs and iOS devices.

  3. Making a backup is difficult because, even if you set the preference, Mail no longer automatically fully downloads all messages. So the backup of the local data will necessarily be incomplete. (See EagleFiler’s Importing Attachments instructions for more about this. I’m happy to have most of my mail archived outside of Mail.)

  4. Restoring a backup is difficult because Mail data is constantly changing. There is no straightforward way to merge restored data in with messages received since the last backup, and also with the live data on the server.

Of course, it’s good to make backups anyway.

Apple advisors are apparently telling customers that if Mail data gets lost on Catalina, it can’t be recovered from a Time Machine backup that was made using Mojave. This didn’t make sense to me, and I’ve verified that it’s not the case. You can use Time Machine to get at previous versions of the folders in Mail’s data store, and then use the File ‣ Import Mailboxes… command to selectively import them into Catalina Mail. Since they import as new, local mailboxes, this shouldn’t affect messages that are on the server.

I also think that the advice to restore the whole Mac to Mojave makes no sense because as soon as you launch Mail it’s going to delete all the messages that were deleted on the server. In order to actually restore them, you have to make copies of the messages that might have already been deleted. That’s what Import Mailboxes does.

Update (2019-10-11): See also: Howard Oakley, TidBITS, Hacker News.

Update (2019-10-12): See also: MacRumors, iMore.

Update (2019-10-13): See also: AppleInsider.

Update (2019-10-14): Commenter Shiner says that the lost messages from “updating Mail’s data store” happens for local mailboxes every time Mail’s database is rebuilt, not just when migrating from Mojave, and reported this to Apple in August. It’s “trivial to reproduce” and lost 160,000 out of 200,000 messages in a test.

There’s some evidence that the moving messages issue is related to Microsoft Exchange accounts.

Peter Steinberger:

A doctor friend of mine lost about 20.000 emails when he upgraded to iOS 13. It was a POP3 account.

See also: TechRadar.

Update (2019-10-15): I’ve received four reports that messages moved to an iCloud mailbox (that were not originally in that iCloud account) simply disappear instead of showing up there. At first, we thought this was a Catalina issue, but it also seems to be happening with High Sierra now. My guess is that it’s an issue with certain of the iCloud servers.

Update (2020-01-29): macOS 10.15.3 fixes one of the bugs but not the others.

295 Comments

Benedict Cohen

I experienced data loss when upgrading from iOS 12 to 13. I had some drafts that were not synced with the server. When I upgraded the tabs for the drafts remained but their contents was missing. I wonder if the Mail apps on iOS and MacOS share code.

@Benedict Yes, I forgot to mention that I also heard similar reports from users updating to iOS 13 (before Catalina was out). I don’t recall this happening with any prior OS update. I think they share a lot of code.

[…] Tsai, the developer of the long-standing SpamSieve plug-in for Mail on the Mac, has written a blog post warning about potential Mail-related data loss when upgrading to macOS 10.15 Catalina. In particular, he’s hearing from numerous customers that […]

Thanks for the warning — I’d be interested in hearing what strategies you use to archive your messages outside of mail.

@Morgan (1) I regularly archive messages to EagleFiler, which I wrote partially for this purpose. It converts Mail’s data store to standard mbox files and checksums them so their integrity can be verified later. (2) For messages that I don’t actually need to access across multiple devices, I have Mail rules that move them to local mailboxes (“On My Mac”). This ensures that there’s a complete local copy (with all the attachments) that will make it into the regular backups of my Mac. (This only works when the messages are processed by rules as they arrive. If you manually move messages to local mailboxes later, even if via rules, that does not force Mail to fully download them; it can actually lead to data loss as Mail saves the incomplete message locally and deletes the complete copy that’s on the server.)

The main issue I and others have noticed is that Mail.app shows blank Sent folder/mailbox from GMail/GSuite accounts. Here is a thread on the issue from other users:

https://support.google.com/mail/thread/16312735?hl=en

There's also Mail Archiver X, which can archive emails either directly from IMAP or from the local Mail.app's database.

https://www.mothsoftware.com

[…] Tsai, the developer of EagleFiler and the SpamSieve plug-in for Apple Mail on Mac, has written a blog post warning macOS users about potential data loss in Mail when upgrading to macOS Catalina 10.15.0 […]

I lost a lot of email messages when upgrading to Catalina. I would not have noticed except that some of my email accounts - which, apart from my Apple ID accounts - are all POP accounts so are not on any remote servers, seemed to have a surprisingly small number of messages after upgrading. I have a total of more than 100,000 messages going back about 15 years. On checking I found that some of my accounts were missing thousands of messages. Which was obviously very worrying.

Luckily I had backed up before upgrading and I painstakingly imported the missing emails from my backup. It took many hours and wasn't altogether successful because some sent emails are now logged not by recipient but by sender (me) which is not very useful. But I think/ hope I managed to find almost all the missing emails.

I am not a Mac expert but I noticed that after upgrading Catalina had created a new folder: user/library/mail/V7 which was not in Mojave, where my messages are now stored.

Perhaps this has something to do with the problem?

I should add that I had backed up not via Time Machine, which I have always found problematic, but via SuperDuper so I had a complete backup from which I could copy my missing messages.

[…] blog post by Michael Tsai via MacRumors is today suggesting that macOS Catalina seems to be experiencing a […]

Beatrix Willius

When the users move from v6 to v7 would the size of the folder be very different than before?

Import Mailboxes isn't a good idea because you would end up with even more duplicate emails than you have anyways in Mail.

The problems with deleting emails with AppleScript isn't really new. You can't even check if the deletion went okay or not.

There is a bug in Mail under iPadOS 13.1.2. If you send an email to yourself (say to read on a different device), it will appear in your Inbox and in your Sent box, as expected. If you move the copy in your Sent box to the Trash and then Delete it, the copy in your Inbox is deleted, too (and from your other device). That didn’t happen under iOS 12.4.

[…] blog post by Michael Tsai via MacRumors is today suggesting that macOS Catalina seems to be experiencing a […]

@Sam Each major new version of Mail migrates to a new folder, i.e. V6 on Mojave and V7 on Catalina. So that’s normal, and it’s good because it keeps the different versions of the data store separate.

SuperDuper is great, but I think it doesn’t fully support Catalina yet.

@Beatrix I think the file format hasn’t changed much this year (whereas a few years back the way attachments were stored changed). So in theory the sizes should be similar. However, I have also seen (going back a long time) that sometimes when the data store is rebuilt it throws out all the local copies of messages that are on the server. So in that case the V7 would temporarily be much smaller.

Yes, duplicates are a problem (#4), but that can be fixed. It’s much better to have duplicates than to lose your mail.

Deleting e-mails via AppleScript has always had issues, but it’s gone from working something like 99% of the time to failing 99% of the time.

Since updating to Catalina (and the new version of Spamsieve) my emails are no longer removed from my inbox, though they also appear in the Spam folder. I have a MacBook pro 2016 using Apple Mail, with Gmail, as the account/server. Any thoughts?

@Eric I haven’t heard of that issue before. Please contact SpamSieve support.

I routinely go through the pain of manually exporting all of my e-mail off Mail.app onto .mbox to prevent this kind of thing, since ever since they started messing up .emlx and storing attachments separately that the SQLite metadata stores have had a number of issues.

I suspect that in many cases mail messages are indexed by something else than the Message-ID header (which is in itself prone to duplicates) and that the indexes get confused, but overall (and having used Mail.app since the NeXT days), whatever has happened this time around is an(other) egregious failure of QA at Apple. There simply is no excuse for the amount of bugs that have shipped in macOS (and iOS), and for someone like me (with an e-mail folder tree that goes back to 1998) this means I won't be upgrading to Catalina until they sort it out.

(Ironically, I am more likely to move to an iPad Pro full-time for my personal stuff and set up a Windows or Linux box to work - I have a lot more trust on either of them to deliver stable releases right now...)

Not only data loss but the whole e mail program does not work on a mac book 2018. On a mac book pro early 2013 everything is almost ok. Sad because the mac book from 2018 is for work. Deleting te boxen en re install does not seem to work. disaster!

@Rui Unless it’s been fixed in 10.15, the built-in export from Mail to mbox is problematic because it doesn’t properly escape the “From ” lines (so message boundaries will get messed up if you import into anything other than Mail, which reads the boundaries from sidecar metadata) and it doesn’t download attachments that are not already cached locally. EagleFiler avoids this by generating the mbox itself and checking that all the attachments are there, and it also saves the metadata that mbox doesn’t support.

[…] dans cette même application. Il a expliqué la situation détaillée dans un article posté sur son blog : apparemment, un nombre non négligeable d’utilisateurs a rapporté qu’une mise à jour des […]

[…] a blog post by SpamSieve plug-in developer Michael Tsai, Mail is seen to be causing problems when users […]

Have you waited for one or two days after the upgrade? Have you tried re-indexing with Spotlight?
I had problems with my 60.000 items mailbox almost during every MacOS upgrade before; and the symptom was missing emails. In the end, I suspected it has to do with Spotlight: The list of mails in mail is created using a Spotlight query, and upgrading mail.app leads to re-indexing all mails in Spotlight - which takes time (and there is *no kind if indication from MacOS which is really a pain*). During that indexing, a lot of mails will simply be missing. In particular tough cases I launched re-indexing with Spotlight (forgot how to do this, but you‘ll find it easily) and that helped (and you can see whether it is finished by looking into Activity Monitor).
Can‘t say yet anything about Catalina, but it would be interesting to know if anyone who already dared to do this can solve it in this way?

@Peter I think Mail uses Spotlight for searching and for certain views like smart mailboxes, but it uses its SQLite database to browse normal mailboxes. So, if the messages are there, that should work without Spotlight.

@Michael yeah, I know about the "From" thing (I maintain https://github.com/rcarmo/imapbackup), but I was under the impression mbox export would download attachments (since I have that option set to "always") even if they weren't extracted from the MIME info (or even downloaded already).

Can I take an EagleFiler archive and reconstitute a full IMAP tree from it?

@Rui It’s just nested folders of mbox files, so I don’t see why not.

I upgraded and then saw this thread! I have not opened the Mail program yet, and was just about to re-install Mojave. But before I do, I wonder if I use POP accounts (still) on my own company‘s server, whether I’m still at risk of losing emails.
Any clues?

@Sarah I have not heard of any problems with POP accounts, although that could simply be because most people don’t use POP these days. At least with POP you can make complete backups and don’t have to worry about corruption getting propagated to other devices. Commenter Sam has lost messages in POP accounts during migration. I’ve not heard of any POP messages lost moving them between mailboxes.

@Michael Tsai Thanks. I probably should go back to Mojave anyway, yes? I had another issue when I went to turn Time Machine back on after installing Catalina, I got a message saying that I had two Macintosh HDs and had to rename one . . . Have you seen this?

@Sarah Personally, without a strong reason to upgrade now, I would go back. You may even be able to do that easily/quickly by restoring the prior APFS snapshot by booting into Recovery. I’ve not seen that Time Machine error.

@MichealTsai @PeterResele From my experience with the innards of Mail (and I have a fair bit after working on plugins for 15 years), both regular mailboxes and spotlight mailboxes are populated by performing a spotlight search to find library ids (ie, the integer part of the .emlx filename) and then looking up those messages in internal sqlite database. Mail also uses spotlight for doing the message counts for mailboxes.

Given this, rebuilding spotlight indexes may resolve the missing messages issues for some but mileage may vary.

As an aside while I personally think this is inane because Mail has a highly reliable, performant sqlite database, I think Mail uses spotlight to prevent a discrepancy in which a person will do a system wide spotlight search for a message will turn up the same result as a search inside Mail, and a few other reasons.

@Scott I think we’ve both been writing plug-ins for Mail since before it was using SQLite and before Spotlight existed. :-) Yours certainly hook into it more deeply, though. It’s an intriguing theory. I don’t know what to say, because it doesn’t square with my experience successfully using Mail with Spotlight completely turned off, the fact that Mail’s SQLite database has fields and triggers to maintain message counts, or the fact that I’ve fixed messed up counts by editing the SQLite database. I’ve also wondered why they don’t they don’t use Spotlight only for the full content searches, given the performance of SQLite.

Over on the MacRumors forums, some people suspect that the data loss problem may be specific to Outlook accounts. Do you happen to know if the folks who reported the issue to you also have Outlook accounts?

[…] macOS Apple Mail 插件開發者 Michael Tsai 的網誌,他警告升級 macOS 10.15 Catalina 的用戶要小心備份電郵,因為從 Mojave 升級到 […]

Jörg Gudehus

I'm wondering that all the smart people here dont know how to use Time Machine properly. You have to be in Mail and start TimeMachine from there. Now you can restore single Mails or folders directly. No importing Mailboxes from the Library.
Please correct your post.

[…] El más reciente, reportado por Michael Tsai, desarrollador de EagleFiler y SpamSieve, un complemento para la app Mail de macOS, asegura que el gestor de correo electrónico por defecto de Apple puede provocar pérdida de información si se actualiza a macOS Catalina, según ha comunicado a través de su blog personal. […]

Victor Lazzarini

Just a report here for other users:

(1) I have not noted any loss of email, but I noticed today that one message did not have its attachment downloaded.
(2) Search stopped working -- it's really annoying. I have had to go to the office.com outlook app to search for
emails, nothing is found on mac mail (it just shows that it is "searching"). I am currently rebuilding one of my mailboxes
to see if it gets fixed.

I updated a few days ago and things were mostly fine, Mail wise. Turned on this morning and all I have is Headers, no content in any email. Had old saved MailData, moved that in, no difference. It's all gone. I am lost for words.

[…] وحدد مايكل تساي Michael Tsai، الذي يطور إضافات EagleFiler و SpamSieve لتطبيق آبل للبريد الإلكتروني Apple Mail على نظام التشغيل MacOS، بعض المشكلات التي ظهرت في تدوينة. […]

[…] upgrading to macOS Catalina. If you depend on Mail for your work or in your personal life, please, read this before upgrading. […]

I have an open ticket on this going back to August (and was probably the first to raise the alarm on this one.)

Where you will see this issue is with "On My Mac" mail. If the Mail application launches with the "Mail Message Import" box (that you can't ignore), when Mail does it's "reimport" of all your mail, you will most likely have a number of "On My Mac" messages just flat out deleted. This is new behavior in 10.15 and wasn't taken seriously enough (IMO) when I reported the problem.

It's trivial to reproduce (if you have backups!) by deleting the "Envelope Index" files (Google around for how to do this -- it's been a long-standing recommended way to fix "search" issues *within* Mail), . It's not related to moving the Mail store from "V6" to "V7", and shouldn't (at least in any of my testing) affect IMAP mailboxes (as those just refresh by pulling mail back down from the server.)

I have test mailboxes with 200,000 .emlx files that end up with only 40,000 .emlx files in them after Mail does it's reimport. If you have "On My Mac" mail, you should test this yourself and immediately file feedback on it!

@Peter It does seem like Outlook/Exchange is overrepresented among the people with the moving messages problem. But there have also been multiple reports (for a different blank messages issue) with Gmail. And the importing issue seems to affect IMAP accounts and even local mailboxes.

@Jörg Have you tried using Time Machine from within Mail on macOS 10.15? Every user I’ve talked to said that this caused Mail to crash, as it did for me. It’s also not clear to me that Mail Time Machine can restore mailboxes that were backed up from previous versions of Mail.

@Shriner That’s really bad news if it’s related to rebuilding the Envelope Index database rather than just migration. There are so many problems that can only be fixed by rebuilding, which has historically been very safe. Even if IMAP messages can be pulled down from the server again, losing the local files in a rebuild would discard information that doesn’t sync to the IMAP server, such as colors.

[…] Developer Michael Tsai shares reports on users with Mail problems in macOS Catalina specifically missing messages or incomplete messages. In some cases you will see headers only and a blank message. Tsai recommends not updating to macOS Catalina yet, but if you already did he shares one way to fix this issue. […]

Yeah, I've never seen this in all my testing that it's related to migration *unless* the Migration forces the reimport (which, I believe, shouldn't happen intentionally.). I only noticed this because one migration I did forced this reimport and I didn't realize I had lost mail for a week or so after that -- the mailboxes (as seen in Mail) will still show the same number of messages -- it's only when you go to open them that you find they open blank (and then you can see the .emlx files are gone...)

My gut feeling is that people who have upgraded to 10.15 have already lost a lot of e-mail, but don't realize this yet.

[…] y SpamSieve, complementos para la ‘app’ Mail de macOS, ha difundido a través de su blog que ha recibido varios informes de usuarios de macOS Catalina que afirman haber sufrido una […]

Michael, This is Sam again. Just to confirm, further to your answer to Sarah about POP accounts, my lost messages were from POP accounts.

Hello all! Alse a simple POP3 account make Mail to crash. In my case I have multiple accounts (mostly with POP3) but the troubles come only with one that is very huge. Maybe the problem should happens with very huge mailboxes.

[…] part, elle contient quelques bugs très gênants, dont l’un d’eux rapportés par le blogger Michael Tsai, concernant les e-mails et plus spécifiquement, l’app […]

@Sam Thanks for the correction. I’ve updated my comment.

Michael, thanks for this, very helpful.

I'm not sure if you are aware of another even more serious mail issue that a number of us are hitting, which causes Mail to be completely unusable.

The error message you get is: "Mail cannot save information about your mailboxes because there isn't enough space in your home folder" ... the popup with this message has only one option, Quit. The popup happens some period of time after mail loads, sometimes almost immediately, sometimes after 1-2 hours.

This is an error that has appeared in the past with Mail but the fixes that worked in the past (removing index envelope data and the like) do not fix the issue this time, nor does reinstalling Catalina.

Here's the issue on Apple discussion forum: https://discussions.apple.com/thread/250717871

Thanks for posting. I look forward to seeing if today’s update fixes things.

Same issue here. Just lost a couple hundred thousand mails, randomly, all over the place, from the last 20 years.

Am restoring, thanks to Mike Bombich, using Carbon Copy Cloner, from nightly backup, which I had first mirrored to yet another disk.

It's running since a night or so, and the mails are coming back, with their folder structure. Just take one by one the content of each subfolder in ~/Library/Mail, and import it using Mail's import function.

This problem went unnoticed for a day over here, since the mails appear to be there. It is not dependent on what type of mail server you had been using - I had been using all POP3, IMAP, Exchange over the years. The mails that were lost were all local to the machine, i.e., long time deleted from the (in many cases long time not longer existing) mail server.

So it is not just Online folders, it is everything, everywhere, with no apparent pattern.

Beware. And Apple - WTF?

[…] plug-in developer Michael Tsai has posted a warning on his blog to reveal that after installing macOS Catalina, some users could end up with missing or incomplete […]

[…] warning arrives by way of SpamSieve plug-in developer Michael Tsai. His post is based on feedback provided by customers, though the Mail problems aren’t related to […]

OK, here is an update on this topic:

I've first of all gone back to backup and restored all of them.

This morning, all ok.

This evening, WTF? Again skeleton mails from today.

I think I've found some (frightening) systematic:

Don't move a mail that has an attachment out of an online folder. It'll just kill the mail content. You can copy it into another online folder, even from like exchange to imap, but if you move it out, it seems to become a skeleton.

There's more inconsistency in that; sometimes it doesn't even perform the copy / move operation.

So, summary: DO NOT USE APPLE MAIL for the moment.

I have a 2018 Macbook Pro and I'm having search issues with the mail app: the query does not find all of the relevant emails. I've tried rebuilding and reindexing spotlight with no success. The only thing that I found that worked was to manually look for the email and view it. Only then, it would show up in the search query.

I transfer all my emails to local. Since upgrading to Catalina, all emails that have an attachment are not transferred properly to local (all you see is the sender and subject line) and lost. It does not damage emails without attachments. This is incredibly annoying and disruptive.

Anderson Robert

When I upgraded to Catalina I experienced a new problem now of not being able to email back to myself, which I do with many emails. They all bounce as a Permanent Error. I have complained to Apple for years that Mail was behind the times and had lots of bugs. Now we get this new set of problems. Search is just not a valid structure, it is hit and miss. Signatures are so awkward to set up. So much is not clear from the help menu. Things just do not work they way they say. Now with Catalina I see sliding lines of text and mail folder that I have no idea what they are to do. They could have put some info help boxes into this it seems.

And the date/time of the incoming email cannot be placed on the left side like I had it. It is locked on the right column. Nothing in View will give this option. I need it on the left side.

There are no ways to color icons either that might help users.

I have not noticed any lost emails with Catalina (may not have discovered it yet). I had this problem long ago with Mojave back in the spring and tried to alert Apple. I lost 15 years worth of emails when I migrated to my new MacBook Pro in January. The MBP had a lot of problem root software they had to find and take out. In the process I lost the emails when he set up a test user account and I then went back to my main user account. The tech told me I had put them on my Comcast ISP in the Mail mail boxes and they deleted them but I know I had them On My Mac mailbox using IMAP. They would not restore from Time machine either, so they told me.

Another problem has came up with the migration to the MBP, either Mail or Comcast has been deleting all my emails over one month old. At first I thought this was a preferences setting on the MBP, which had been put in by the migration but it even exists on Comast ISP with the MBP settings ok now. I called Comast and complained and they said they pulled all my old emails from an archive file and put back in my ISP mailbox but they all show as coming from me with my name, so that is not much of a help.

Long ago I found a big problem trying to delete incorrect emails from Previous Recipients. They just will not go away and keep showing up in the TO field while not in the archive list. AND for a long time Mail has been deleting email addresses of people I may not have mailed in a year or so. They are just gone, removed, I did not remove them.

This is a sorry application for such a pretty good computer company, imo. Now I am new to these forums and don't know how to get replies or other info...but if someone wanted to help me send an email to me at citizen
@comcast.net. How does one email Michael Tsai directly?

Thanks,
Bob

The problem I have experienced since updating is that emails that I know have been sent to me, and I can verify were sent, are not appearing in my inbox. Yet, if I check with the sender and know the Subject, I can find them when I search Mail. Effectively, the incoming emails are - apparently randomly - invisible unless I know they have been sent and I know how to search for them. They are in the inbox, in the right place, and they are visible upon a search, but disappear again when the mailbox is revisited. This is a disaster, and I understand from colleagues they have experienced similar issues. Furthermore, the whole Mail system is unaccountably incredibly slow now, every time I log into it.

Man, nothing to add, but yikes! Fingers crossed this gets worked out. Anyone remember when Microsoft bought Danger and then promptly destroyed all the cloud data of Sidekick users? This is kind of feeling a bit like that. On a scale of "okay, we can fix this" to "Holy goodness, yikes!"

My problem with the catalina upgrade on my 2018 macbook air is with the undo delete message option. It no longer works, and i actually use it a lot. Also, with some messages, but not all, the scroll down option does not work. Little annoying glitches! and I haven't seen the option of 10.15.1 yet. Also apple has not responded to my questions. I guess they have bigger problems than mine.

MANARA Riccardo

Would switching to another email client like Readdle Spark solve the mail problem?

Yes, I too have experienced similar frustrating issues with Mail in Catalina. One additional problem, I typically move old mail from my sent folder to an archive of mail in organized folders on icloud. So, as others have said copying mail to icloud is just failing like crazy usually (for me) with a timeout error. The mail that failed to copy to the icloud folder has gone from its original folder and now shows up in a new "On My Mac" recovered mail folder. But the date for all of the mail items in that recovered mail folder in column view is now today's day regardless of when it was sent!!! The date in the mailheader is the original correct date. So if I select date with the new sorting dropdown menu the newly moved items are sorted with todays sent mail.
I guess that I just cannot move any mail until Apple decides to fix this mess.

Hi all,
My upgrade to Catalina on my late 2015 iMac went reasonably smoothly, until I logged into the 2nd User Profile on the machine, and saw an error message about having to import email mailboxes. Didn't get to note down the message, however, a brief synopsis of the issue & resolution is:
Ran upgrade to Catalina on Sat Oct 19.
Emails from a POP3 mailbox were randomly missing content in the Mac Mail client, from a time period of Oct 18 back to April 2018.
By necessity, the emails had been deleted from the mail server - lack of provisioned space.
The mailbox had been in existence for about 12 months, but the last 7 months or so of email had only headers, no content.
The emails from the last couple of days were valid with headers & content.
Upon further investigation, and rebuilding the mailbox a couple of times, the emails without content disappeared entirely. I could verify this, as MS Outlook had been setup and had downloaded a copy of each mail item into its own mail store. However it did *not* contain all the sent emails, and was not gong to be solution.
Found the local mail store on the filesystem, went into a Time Machine backup point just prior to the upgrade being run. Restored the entire /Users//Library/Mail/V6 (Mojave) to a temporary directory in Catalina.
Imported *all* the mail into a new folder in Mail application. Was given the option to select a number of different folders, but chose bring all mail items in.
Found all the missing content, dragged & dropped into correct folders, including Sent emails for missing time period within the Mail app.
Seemed to have fixed the problem, however will require further checking over the next day or so.
HTH
Doug

I've been seeing "strange" Mail issues (okay, mostly using GMail backed mail accounts) even before I considered Mojave... that was on !0.12 & 10.13... where my laptop was still on 10.12 and my desktop on 10.13, I did on several occasions had to "rebuild" my mail database as I experienced (especially older) mails without their attachments or just empty.

Lately, after I've "upgraded" the desktop to new hardware + Mojave, I've been encountering (on both the laptop & the desktop) emails that are wrongly put in the same mail thread ;( some old non-related included in the same one ;(

Also discovered that the majority of my local mailboxes (hundreds of hierarchical mailboxes, >500,000 messages) had only partially made the conversion to the V7 mail store. There were missing messages and no body or attachments for most of the messages that were there. I had no problems with the four Gmail accounts syncing. I restored the V6 folder from right before the upgrade to 10.15 using Time Machine in the Finder, then imported the local mailbox folders into Mail piecemeal starting with the top level hierarchical folders. If you select a mailbox folder for import directly, you will also get the UUID folder names it contains, so put the mailbox in a folder and select the containing folder if you want to import individual mailboxes. Consolidation with old folders is going to be fun.

Some comments relative to the main post - no Exchange accounts were involved. There were none defined in Internet Accounts at the time of the upgrade.
Search wasn't working at all, but rebuilding the spotlight index didn't help in this case (there were missing message files - i.e. no underlying emlx files and attachments), and rebuilding the Envelope Index did make things worse (fewer messages listed and body no longer showing up for most messages that it had been visible in list preview before rebuild).

I had issues when upgrading to High Sierra with losing some old archived email, but the Catalina update really destroyed my email. 20 years of emails, archived in folders by year, and most of them are blank. Apple was unable to help. While I have backups, they are imperfect since the issues started with the last OS. I am ready to throw in the towel and setup all new email addresses.

These are the test I've done with results;

Like most users have reported, I too lost a huge amount of emails by updating from Mojave to Catalina. I had a solid TimeMachine backup so I simply formatted my drive then installed back Mojave and restore from TM. That worked perfectly and all my mail was back the way it should be. I have an extra MacBook Pro 13" so I decided to do some more testing. I first created a new folder on my Desktop and exported all my mailbox via the export function. Then I did a clean install of Catalina on my 13", open the mail app and imported all my mailbox via the import function. This allowed me to re-create a perfect copy of mac mail Mojave to mac mail Catalina. Then I made a brand new TimeMachine backup of my 13" Catalina install. Now I'm thinking I have a TimeMachine Catalina backup with all my mail so that should be perfect! Well no it was not. I erase my 13" and proceed to install a clean copy of Catalina, then use my perfect TimeMachine backup to restore thinking that my mail should be back the way i left it. Believe it or not, upon restart and opening the mail app, most of my mail only show headers and nothing in the message. This is a terrible bug and looking at the release notes of up-coming updates, it looks like Apple is not even addressing this issue! This is terrible really! Bad Apple :(

Did the last update fix any of these issues?

@Chris No. I will update the post if I hear of any fixes.

[…] people do not update there Mac to macOS Catalina. If they are upgraded again he put a solution on how to fix the Mail problem. However, you can use the time machine to get you into the previous mail version. Again if the time […]

Miloš Mitrović

Is there any hint how catalina mbox files can be imported to Mojave? I have 120GB of mbox on Catalina mail, went back to Mojave, but all mail mbox files have same problem "no valid mbox files were found" when I try to import to Mojave mail app.

@Miloš Despite the “.mbox” in the folder names, Apple Mail’s data store is not in mbox format. So if you are using the Import Mailboxes command, you should tell it to “Import data from” “Apple Mail” rather than “Files in mbox format.”

Hello,
I have lost all the content of an outbox mail of one of my gmail account when upgrading to Catalina.
I was in contact with Apple international support via telephone and they have confirmed me that this issue is known to them as a bug in Catalina, which will be fixed in the future.
I suppose that more and more applying to them would accelerate probably their speed.

Miloš Mitrović

@Michael Tsai Of course I tried "apple mail" option too but lots of errors in console with newer ending story of import. What about "Mail Library cannot be read because its format requires a newer version of macOS." is there solution to convince Mojave version to read Mail folder which was already in use by Catalina?
I am trying to solve 120GB emails which are present in Mail folder but not shown in Catalina, so I moved to Mojave and trying to get those emails in app...

@Miloš I had thought Mojave Mail would be able to import from Catalina Mail directly. If not, you could use my EagleFiler app to convert the files. Make a new EagleFiler library and then drag and drop all the Catalina account (UUID) folders inside the V7 folder into the EagleFiler window to import the whole mailbox hierarchy. This will convert all the mailboxes to mbox format. Then you can tell Mojave Mail to import the mbox files stored in the EagleFiler library.

I updated to Catalina yesterday. All of the folders for my iCloud email have gone missing. I also have a gmail account that appears to be working normally. Unfortunately, the iCloud account was being temporarily used for all of my business launch. My iPhone still has all of the folders and e-mail.

is the issue gone with 10.15.1 ?

Yes, would be great to get confirmation from Micheal on whether this is fixed or not with 10.15.1? Thanks

@abc @dfe I’ve heard that some Mail bugs are fixed in macOS 10.15.1, but not the ones mentioned in this post.

Mails for a whole year were wiped out and every time I try using time machine to retrieve them Mail crashes. There seems to be no way of getting them back and this is very disheartening.

I did a clean install of Catalina 10.15.1 today (Nov 1) and the bug is still not fixed.
All of my inbox messages (POP mail) from the past two years vanished sometime after I did the install. And for others, message headers are showing "no recipient" "no subject" and content is blank.

Would a clean install and a fresh sync of email from the server avoid these mail problems? Instead of trying to upgrade an existing email setup? Thanks.

The problem persists in 10.15.1. Almost 50% of my old emails have now disappeared (40Gb of info). Most of the emails only show headers and nothing in the message. When rebuilding the mailboxes, the emails just evaporate. I've looked into the finder structure and the emails are completely gone. I can only recover from backups, but many emails will be gone forever. Bad.

How can Apple release this with such a bug? It means you're taking a risk updating to Catalina, and one more serious than usual, because the data loss can propagate to your other devices.

It all seems odd - all that marketing for the new and great macOS, yet Apple must surely know that 'upgraders' are risking losing data! Unbelievable!

Might this blog thread make a note if and when the problem is resolved?

It is ridiculous that Apple would allow this to go ob for nearly a month. We are losing all corporate emails that have attachments (85%) as the email are blank once pulled onto our servers and personal computers. We do not store email on our server, they store locally and this a significant issue for our business.

We have worked on every possible fix and there is NO WORK AROUND to recover blank emails nor to stop the emails from converting to blank documents.

downloaded Catalina two days ago, went to check email and a blank email page opens without any listing of any emails. Mailboxes shows the various mailboxes but there is no information or display when you try them. Not sure where to turn for any fixes. My email is cloud based. Luckily email still shows on iPad mini and iPhone X S.

I think you guys have just helped me solve the big issue I have been trying to fix for the last three days. I upgraded to Catalina last week and have since lost 3 very important drafts in my Apple mail...they show up as drafts (which they are) and I can see the the preview with the first three lines of text in the drafts folder on my iphone Apple Mail App but when I open them, they are blank. None of these drafts show up in Apple Mail on my Mac, nor in Gmail. They simply never synced it appears. Sadly, one of the drafts was SUPER important to me, and now it appears corrupted and unsalveagable. I spent an hour on the phone with both Apple and Google support but now I think I understand the problem. I will hold on to the draft in in the draft box and hope that once there is a bug fix it comes back to life, but I am losing hope. I really need that draft. Any advice?

I experience the same problem. When I move or copy emails received from my outlook exchange account in another folder in Apple Mail, they appear to be totally empty, despite the fact that the sender, subject and date/hour appear correctly. You have to open the mails to realize the problem. The problem has appeared since the upgrade to Catalina. Other accounts (Gmail) are not affected by this problem. Only mails with attachments are affected. Other mails can be moved or copied without problem. Another problem is the non-synchronisation of the outlook exchange account. Mails read or deleted in Apple Mail do not change status on the Exchange server. As a consequence they appear to be unread or undeleted on my i-phone. This second problem seem to affect all emails in the outlook exchange account, with or without attachement. My solution (well not a real one, more a safety trick) as of today is to keep all mails from my outlook account in the inbox and postpone the cleaning until the bug fix is released.

This also happens with POP accounts in Apple Mail in High Sierra (10.13)(I don't use iCloud, so it's not just an iCloud problem with High Sierra). I first noticed it this summer: I couldn't find a message that I was fairly sure I had saved (all my saved messages are in a folder named Archive). A few weeks later, I looked for a different message that I KNEW I saved and that wasn't there either. When I looked at my Archive folder, it only had about 600 messages; there should be 2,000 messages. Restoring from Time Machine backups didn't work (every time I imported the TM mailboxes into Mail, the number of messages didn't change). The messages started disappearing around June 2019 (this is when the message counts in the TM mailboxes started going down).

What I finally did was use EagleFiler and imported every mailbox from every TM backup I had (yes, it took a lot of time, but I was able to restore every message, since I had TM backups going back to April). Now, I export my "Saved Messages" folder from Apple Mail to a .mbox file every few weeks and add it to my EagleFiler library, so I don't have to worry about losing messages.

THANK YOU, Michael, for creating EagleFiler, you saved my bacon.

If you really need to move a mail from the Inbox (from an Excchange account) to another Apple mail folder, you can copy the message on the Desktop, then open the message from the Desktop and copy it to the target folder. Doing it, the message is copied correctly. Do not forget to delete the message from the Inbox if you wanted to move it and not just to copy it. Obviously, such a procedure is far from being user friendly. At the very least, it shows to the Apple developers a step-by-step procedure that does not transform Exchange messages into blank messages. I leave it to them to make it automatic;-)

macOS 10.15.2 Beta 1 is out and Apple Mail is Version Mail 13.0 (3608.2). macOS 10.15.1 had Version 13.0 (3601.0.10). I don't know if anything has been fixed since I don't have any Mails in my Beta-VM but I wanted you to know.

Question: when does the data store upgrade happen - on installation of Catalina or on initial launch of Mail app? I installed Catalina a few weeks ago but happened to read this thread before opening Mail - wondering if the damage is already done, or if I can wait to open Mail once Apple issues a fix. (reading mail via iOS, fyi) Thanks!

@Tim On launching Mail.

[…] Mail trên macOS, đã phác thảo một số vấn đề đã được khắc phục trong một bài viết trên blog – ông khuyên nên giữ bản nâng cấp Catalina, nếu bạn dựa vào Apple Mail, cho […]

I made a clean install of Catalina (booting from usb stick containing 10.15.1) deleting Macintosh HD with disk-utility, then installing the operating system) and recovered my profiles via Time Machine. Locally stored emails and emails in pop accounts appeared blank except for the email header. I tried this several times. Then I made a clean install of Mojave and recovered my profile from the same time machine back up, and everything is fine. In other words, the problem is not a potentially corrupted back-up, but OS X 10.15.1
Each process about six hours, so I’m leaving it until fall 2020. CONGRATS APPLE!

Michael do you know if Apple is officially aware of the problem?

@John I’ve heard second-hand that they are. They haven’t responded to my bug report.

@downgrader What kind of mail accounts were you using that you were able to recover mail? I am looking specifically at the issue with Exchange mail.

Hi All,
I have been spending about 9 hours with an "APPLE expert" by the customer care service in Italy to try diagnose and fix the problems induced by Catalina on the mail client. No way, after every iteration it got worse. By now I feel like saying it looks very much related to the interaction with Exchange servers. Something got really screwed up. Back to the old story between Microsoft and Apple. I reported to the "expert" this is really something that should not happen; it makes no sense to release a new operating system without these very basic tests. I'm really disappointed.

@Massimo Was it an upgrade or a flattened install? So far it looks like it's all Catalina upgrades?

MANARA Riccardo

Michael, if I upgrade to Catalina and I don't have any interaction with Exchange Servers but only Gmail and Fastweb (an Italian server) accounts, do you think I could have problems?
Another question: if I no more use Mail but another mal client (i.e. Spark), could I avoid mail problems upgrading to Catalina?
Thank you.

Just to keep everyone informed: macOS 10.15.2 Beta 2 comes with a slightly fixed version of Mail.

10.15.1: Mail 13.0 (3601.0.10)
10.15.2 Beta 1: Mail 13.0 (3608.2)
10.15.2 Beta 2: Mail 13.0 (3608.40.2.2.1)

I don't know if anything has been fixed since I don't have any Mails in my Beta-VM but I wanted you to know.

@MANARA Yes, I have seen people have problems who are not using Exchange. Using another mail client would avoid them, however.

I can neither confirm nor deny that the "envelope index rebuild mail deletion bug" is *NOT FIXED* in any of the 10.15.2 betas yet.

So I can not confirm or deny that you *shouldn't expect it to be fixed in 10.15.2 at all* at this point because Apple said that a "lack of duplicate reports" is not moving this higher up the regression fix priority list.

I can neither confirm nor deny that we are not pleased about this by any means.

I *can* say -- that if you have tested this and you have reproduced the problem and you have not filed an official bug report about your results -- this is why it's not fixed yet. :-(

@Shriner Why didn’t they name Catalina GLOMAR?

Shriner, have you filed and official bug report yourself?

How will we know when upgrading to Catalina is safe? Has apple admitted the bug and given a way to track whne it will be fixed?

@Michael Apple has not publicly acknowledged it, but apparently they have replied to Shriner’s bug report. I will update this post if I hear that the bugs are fixed.

John Hart -- of course. During the 10.15.0 beta process (mid betas) we discovered this and filed it. We also pay for AppleCare Technical Services and filed it there, too. This is why we are most unhappy.

The only thing we haven't done is wrote Tim Cook directly. We are considering that. :-/

Lisa Greenfield

Im wondering if anyone can help me ...since upgrading to catalina , any time i begin to compose a message using mac mail but using my out bound webmail address , i receive this message below, it freezes my mail and I have to force quit.... the actual email server works when I log in through the website and sends the email zero
problem

The message “Re: ” could not be moved to the mailbox “WEBSITE Email” ...

Mail was unable to open this mailbox on the server “mail.lngphoto.com”.

The server returned the error: Mailbox does not exist, or must be subscribed to.

@Lisa Greenfield

At this point I would not trust anything Mac mail using Catalina. If you're able to revert to Mojave without any lost of data, that be the road I would take. It's shameful what Apple is doing here. Tempering with precious emails in that way is clearly unacceptable. It's almost impossible that Apple did not catch that issue while they were beta testing Catalina! Yet, they do not seemed to care!

I wonder just how widespread these issues might be given Apple's response to Shriner. I migrated two 2016 15" MacBook Pros to two new 2019 16"MacBook Pros that had Catalina 15.1 preinstalled last weekend without any apparent Apple Mail related issues. It's been 3-4 days since I did the migration to the new Macs and have done routine spot checks of my mail & attachments without finding any issues. I DID import all current mail (~26,000 emails on one machine) into EagleFiler, after confirming download of all attachments, as a precaution.

Note that neither new Mac did a mail "reimport" to my knowledge and I have not attempted to rebuild envelope indexes. I have three imap accounts (Gmail, iCloud, and one associated with a personal website) although the vast majority of mail is in the iCloud account. I also have an extensive "on my mac" folder hierarchy. I do not have an exchange account or utilize Outlook. So far, so good but I'm still spot checking and hanging on to the EagleFiler databases!

I'm fed up with this. I'm importing all my mails in EagleFiler. Hey, how can I change the font size in EagleFiler's Mails and Lists?

The way I'm importing is to work through a local dovecot that I've dockerized.

@eMaX There are settings for the various fonts in EagleFiler’s Preferences window.

Sören Nils Kuklau

It’s almost impossible that Apple did not catch that issue while they were beta testing Catalina! Yet, they do not seemed to care!

For whatever it’s worth, I’m not running into the issue.

My guess is this particular issue was non-trivial for Apple to reproduce at the time, and didn’t see quite enough users affected to make that high-priority enough to delay the entire release. (Which, it seems, wouldn’t have helped anyway, as they still haven’t been able to fix it by 10.15.2.)

Hi,

The problems seem not to be limited to the Upgrade. I still receive Emails with 10.15.1., can read them when in Inbox but when moved to a Folder and open it again, the content is missing. :-(

That can cause serious problems when the email is needed later for proof.

Frank

@Frank That’s correct. One problem is the missing messages during the upgrade or when the database needs to be rebuilt. But messages can also be lost or blanked when moving between mailboxes (either manually or via rules).

Hello,

Disabling Mail Preferences --> Rules appears to solve the issue of receiving seemingly random empty messages into my Exchange accounts (IMAP), which I had migrated from an earlier version of OS X.

Hope this is useful.

SP

I'm also hitting the problem with 10.15.1 and corporate Exchange email account.
I'm observing following problems:
- when I copy messages from Exchange Inbox to a local folder, the messages with attachments are not copied properly, only header is seen. It seems reliably reproducible. Copying messages with attachments from an IMAP Inbox works OK apparently.
- when message is moved from Exchange Inbox to a local folder by an Inbox rule, even messages without attachments are not moved properly.
- Mail.app is constantly showing "Moving Messages" with the numbers slowly jumping up. No idea what is it doing, and restarting Mail doesn't help, it starts counting from zero again.

I've contacted AppleCare, showed them the problem, they collected info from my machine, and now I'm awaiting a reply.

I also encountered problems with email after upgrading to Catalina. On the phone with Apple for hours, the guy telling me they'd had a lot of problems with this update, and they hadn't really tested it properly. I rolled back to Mojave. Now wondering whether they are ever going to address the issue. Any recommendations for an email client I can use instead of Mac Mail, where I can import all my emails and settings, anyone?

Yes, I can confirm that after weeks, I still experience *serious* Mail issues (made the mistake of installing Catalina from Day One and then hoped for updates, all installed). It's now "better" than at the beginning, but still totally bad.

In addition to the points mentioned previously (pasted below), I also have to manually delete everything in Mail.app (iPhone), so any spam I delete in Mail (MacBook), I must delete twice there plus once in the app…

from Frank:
"The problems seem not to be limited to the Upgrade. I still receive Emails with 10.15.1., can read them when in Inbox but when moved to a Folder and open it again, the content is missing."

from Robert H:
"I'm also hitting the problem with 10.15.1 and corporate Exchange email account.
I'm observing following problems:
- when I copy messages from Exchange Inbox to a local folder, the messages with attachments are not copied properly, only header is seen. It seems reliably reproducible. Copying messages with attachments from an IMAP Inbox works OK apparently.
- when message is moved from Exchange Inbox to a local folder by an Inbox rule, even messages without attachments are not moved properly.
- Mail.app is constantly showing "Moving Messages" with the numbers slowly jumping up. No idea what is it doing, and restarting Mail doesn't help, it starts counting from zero again."

Is there any indication that MacOS Catalina 10.15.2 Beta 3 will address the critical problem of lost emails bug(s)?
Like others I will finally migrate my 'archive' of emails to FileEagle librarys once I see the path to reconstructing what Catalina has destroyed.

First headway in ages: I saved my Mail folder (Library) and imported it in Mail on a different Macbook -- eh voilà, everything there, with folder structures AND attachments!

Kakis,

Can you be more specific? Did you get Catalina to work? or you copied the entire mail (V6) library and import to another mac NOT running Catalina?

@ John Hart: From Catalina to Catalina. But the from-Mail was completely screwed (see my first comment above). I went from my old MacBook Pro which was already on its last legs, had Catalina, and experienced all system updates for the past 6 years to the Library folder (Finder: press Option > Go > Library) and simply copied the Mail folder on an external drive. I then went to my brand-new MacBook Pro which I set up from scratch (NO migration), opened Mail.app there, and imported the Mail folder from the drive (File > Import Mailboxes: Apple Mail). Hey presto, took a while (220K emails) but it looks nice now. :-)

Kakis so there is no need for exporting Mailboxes in .mbox format? Will this technique work when going from Mojave to Catalina? I used to only copy Mail Library folder on older systems, but it stopped working few OS systems back. Are you saying I can copy my Library folder to external HD and import this very folder in Catalina and all my mailboxes, rules , etc will be copied?

Just got bit. After upgrading, all was fine. Then I tried fixing an issue with a smart mailbox displaying that there were messages in the mailbox when it was empty. Boom, hosed. My accounts are all POP (except iCloud and Gmail, which I don't really use but we're all forced to have). As others have said, only the large boxes are affected. I can't say how large they need to be, but my boxes with Amazon receipts, my wife's emails (going back to when we first met in 2002), and base inboxes full of unfiled messages are toast. If I go into Time Machine, they show up, but I'm not able to restore -- even from a TM disk I just started a few days ago. Diving into both the current V7 folder and ones from TM backups only nets me partial success. So far, the most recent message I can find in there are from three weeks ago. Both in TM and on Crashplan. Where is Mail storing more recent messages in the meantime? They're there somewhere. Also, new message since this happened are fine. So far.

@ Kakis: Did Mail rebuild the Envelope Index database? I can also copy my Library folder (V7) from my second, older Macbook and it looks and works fine. However, I'm not able to perform a single search. And as soon as I rebuild the database, all the data of folders containing +2k emails gets lost.

Can anyone else confirm a cutoff at around 2000 emails per folder?

I found a workaround for my system! After having bought a 2019 16" model last week, I upgraded my 2014 Macbook Pro from Mojave to Catalina and migrated all data to the 19" model. Mail worked fine for the 2014 model and was simultaneously heavily screwed up for the 2019 model as mentioned above - perhaps the one big difference being that after migration the latter entirely rebuilt its Mail database, whereas my old Macbook apparently didn't. I tested that a few times, deleting the current V7 folder (which would go from 6GB to "Import Mailboxes", and then selecting "Apple Mail" rather than only chosing particular mailboxes. After importing the intact V7 folder from my older Macbook this way, all messages showed up correctly and were searchable. The import does jumble up the order of folders and mailboxes, but that can easily be sorted out manually. I then imported signatures and rules and have a working Mail App without any data loss so far. Hope it stays that way and hope this is helpful for someone out there!

And btw, just to be safe I installed EagleFiler which does an excellent job in archiving all or any mailboxes and folders.

I don't know what to say other than what I wrote before (I'm not a computer person). As I said, I had thousands of emails, some folders also had more than 2k (to answer one query above). All I did was copy the Mail folder in my library -- but at that time, the old laptop was already in Catalina (so there weren't any V7 folders any longer). The main thing is that when I updated my old MacBook Pro to Catalina, all hell broke loose and Mail was rubbish with all attachments gone/not displaying -- and when I coped the Mail folder from that faulty laptop onto the new MacBook Pro (clean, no migration but with Catalina), everything was fine.

@Bernd -- there's no rhyme or reason about what mailboxes will have deleted mail as far as I can tell. The mailbox I sent to Apple has 1000 emlx files in it, but only 7 after the Envelope Index-deletion-reimport.

10.15.2 is out. Any news on the subject?

@Tasos: I'd love to know this too. Does 10.15.2 fix the issue?

10.15.2 does *NOT* fix this. If you have not filed this bug with Apple, what are you waiting for?

@Shriner, I am having the same issue with all of my POP and IMAP accounts except for my iCloud account and I have filed FB7490433 with Apple. I don't use gmail, just POP and IMAP mailboxes from my hosting provider.

I note that prior to my Catalina migration my Mail folder had 5,792,038,039 bytes (5.82 GB on disk) for 101,388 items and after running the Mail app after migration I had 2,871,906,047 bytes (2.8 GB on disk) for 37,633 items in that folder. All of the message envelope meta data appears to be there, but as I go back in time a few weeks on some folders, a few months on others, the message bodies are all blank.

I was able to import all of the mailboxes from the good backup into AirMail 3 (and into Mail Steward as backup) and all of the data was there.

I’m posting just to underscore a point that Shriner made above but that I don’t think has really been emphasized enough (both here and elsewhere with this issue): the issue ONLY affects local “on my Mac” mailboxes.

After careful preparation – 3 separate backups, and pruning and archiving (to mbox files) of my mail – I just upgraded to Catalina 10.15.2 yesterday. I was somewhat surprised when I crossed my fingers, held my breath, and launched Mail fo the first time....

It just launched. There was absolutely no updating or conversion of any of my mail. It was no different than launching Mail on Mojave.

The key which I see now, but wasn’t clear to me then: I only have IMAP (and Exchange) mail folders, with nothing stored locally on my Mac.

I realize for many people this potential bug is still a deal-breaker,but for anyone out there who I has been waiting to upgrade and either (A) like me, only has IMAP mail, or (B) can move things around so that they only have IMAP mail – then for those folks there is no reason not to upgrade now.

@Sam There are multiple issues. The one about upgrading and rebuilding mailboxes seems to only apply to On My Mac mailboxes. However, other users are having IMAP and Exchange messages disappear when moving them between mailboxes (even on 10.15.2). This doesn't seem to affect most users, but since we don’t know who will be affected I can’t recommend updating in general.

@Michael. Thanks. What you say makes sense.

It does make me wonder, though. It sounds like the "on my mac" problem has a clear source in Catalina: rebuilding the mail data store and/or the envelope index in Catalina Mail leads to corruption and a loss of messages. Shriner has found a reproducible bug here and reported it to Apple.

The IMAP and exchange problems don't have a clear source or explanation, and I wonder if they are even caused directly by Catalina. I know I've had Exchange problems in the past and they were almost always caused by a conflict between the huge variety of Exchange servers out there (including very old versions), and the interaction with them by the mail client.

Still, I'm going to watch my IMAP and Exchange folders carefully. I routinely move messages from an Exchange Inbox to a series of IMAP folders, so if the moving message bug is alive, I may still be exposed to it.

@Sam I think it’s Catalina because some of the people who reported these issues to me reverted to Mojave and the problem went away.

Sherman Wilcox

This is probably and edge-case question, but I only use IMAP with Fastmail. (My university uses Exchange, but I forward everything to Fastmail.) Does anyone know if the problem appears with IMAP on Fastmail?

The "On My Mac" Mail deletion bug? Still not fixed in 10.15.3 b1. We are besides ourselves here about this. Grr.

Sherman Wilcox

No, I rarely use "On My Mac. I store everything in Fastmail folders. I thought I read above that the deletion bug also appears with IMAP.

@Sherman The “moving a message to another mailbox instead deletes it” bug can happen with IMAP. I don’t know whether it’s limited to certain servers. I have definitely seen it with iCloud and Gmail. Not sure about FastMail.

With the update to 10.15.2 in Apple Mail I lost about 400.000 mails, which are still displayed with subject, sender and recipient but have no content anymore. Both in the preview and after opening the messages are "empty".

I noticed the whole thing after certain messages, which I normally access via search, were no longer displayed.

I then re-indexed the mailboxes via Onyx without success. Even more messages seem to have completely disappeared from the mailbox folders.

A recovery of the Library/Mail/V7 folder from a TimeMachine backup from 2 weeks ago was unsuccessful because 400,000 elements from the backup were "copied" back into the folder and Apple Mail subsequently indexed the mailboxes (also about 400,000 messages), but these are still all empty in the mail.

Does anyone also have this problem and a solution?

@Patzi. If you have a successful back up of your "on my mac" mail -- restore it to a different location (like your desktop). You should then be able to successful *import* the mail (File --> Import Mailboxes).

That works.

But please file a bug with Apple.

Great advice everybody - thank you!!

This is my issue:

I bought a new MacBook Pro 16". I use Apple Mail consisting of an iCloud address and 2 gmail accounts. I have close to 1.5 million e-mails. I had issues migrating from my older MacBook Pro 15" to this new 16". The older 15" used Mojave, the new one, Catalina. About 1/3 of my e-mails did not migrate to the new computer and many that did, only show headers - no body. Apple customer service really has no good answers. One idea was to upgrade the 15" to Catalina, then try to migrate. But I did try that before and after upgrading to Catalina, Mail froze during the set-up (kept on trying to download e-mail). It was fixed by reindexing and going back to Mojave, so I never got far enough to try a migration from a Catalina 15" to the Catalina 16".

Do you recommend reverting to Mojave on my 16" or trying some way to import the e-mails from the 15" via Time Capsule, external HD, a 3rd party box import program, etc.?

Thank you for bringing this critical information to the forum.

This really sucks. In my case I could still reproduce the problem under Catalina 10.15.2. Every time the reindixing is done more than half of my messages disappear or are only available without body.

@Shriner: thanks for the recommendation. I decided to wait with reimporting until this problem has been fixed by Apple.

Missing mail after Catalina update. Nearly 90,000 historic emails were lost, in my case, reducing the Mail folder size from 35.5GB to 2.7GB. Fortunately I have found an old backup with much of the historic messages. However, key messages remain missing including copies of customer invoices sent out by email. Destroying people's email archives has the potential to put companies, especially smaller folk, out of business. The problem is only obvious once one starts actively looking for those old messages. But, by then, it may be too late ... old backups have been replaced by new backups that lack the older messages. Even if one can recover messages from a backup, the time and effort ( and $$ ) involved in doing so is significant. Shame on you Apple. Protecting user's files/data has to be the #1 objective in any OS update.

How did we find out: we are missing our records of emails (the messages) sent out via Apple Mail!

Oops, emailed customer invoices.

Again -- if you have not *personally filed a bug report* -- you are not going to get this problem fixed any faster.

Apple needs to get more bug reports in on this ASAP.

If you use Time Machine you can recover your lost mail. Apple replaced the V6 folder with a V7 folder in Library/Mail. I have no idea how they screwed it up. Doesn't really matter at this point. We just want our mail back.

Open Time Machine and look in Library/Mail for the last backup with the V6 file. Restore it. This will put a copy in your current Library/Mail folder.

Take it out and put it on your desktop. Open Mail and select menu item "File/Import Mailboxes". Choose the V6 file on the desktop. This will put all of your mail up to that date in a folder called "import" in the Mail sidebar. You can drag and drop to selectively replace your lost mail. A bit tedious but better than losing everything.

Happened to me too. Drag and drop was losing attachments. I tried rebuild. Then it said it had to rebuild the index and import everything. 45 minutes later, my emails are massively corrupted. Lots with header only, no body. Many missing. Bunch of random stuff in "Recovered", mostly stuff that I had deleted. Worst problem with my Mac that I can remember in 20 years. Apple, please get on the ball!!

@Sherman — it was interesting to see your reference to FastMail, since that's what I use too. I'm actually delaying purchasing a new Mac because of this, since I'm doing alright on Mojave on my existing Mac, but I'm sure the new one will force a Catalina move. Since I only have FastMail and iCloud email, I'm thinking to just deconfigure iCloud email and go to a single IMAP account, which it appears from this thread shouldn't be a problem running on a single mailbox. I'm also thinking that if I really want to keep both active, maybe I should move to Outlook 2019 or 365, or to Thunderbird, or the FastMail web interface (which isn't great but is good enough), and get out of this quicksand entirely (since all I need to do at most is re-cache my email). It's insane that this is still an active problem by 10.15.3b1.

@dpz I have used both Postbox and MailMate (currently MailMate) since Catalina and have no problems. (FWIW, mileage varies on the Fastmail web interface. I rarely use it but when I do I find it fast and uncluttered.)

Just opened a "feedback" on the IMAP-move-to-other-account-issue. Feedback-Assistent lists "no similar feedback" :(
Will reproduce the on-my-mac-issue soon an open a feedback for this too.

Sören Nils Kuklau

> Feedback-Assistent lists "no similar feedback" :(

I wonder how "similar feedback" works. Could be a manual process, could be a very primitive keyword-based algorithm, could be machine learning. Either way, probably a lot of false positives/negatives.

I updated from Mojave to Catalina beta a while back and my mail was fine.
When I got a new Mac and reinstalled from a TM Backup via recovery the mailboxes where reimported and that is when I lost about 10.000 mail that were in a local mailbox.
After trying many things I installed a TM backup from before Catalina and grabbed the mailbox file of the local mailbox from a TM backup from the old mac running on Catalina.
I added this to the library of my reinstalled Mojave, deleted the envelope files and after reimport in Mojave I got my mails back from my TM backup that was run unter Catalina.

I will strick to Mojave until this is sorted out.

Best,
L

I just had a thought that I could run Mojave in a VMware Fusion VM for my Apple Mail until they fix their issues. Doubt I'll bother doing that, though.

Although my issue has nothing to do with Catalina, it does touch on the Time Machine
Mailbox recovery strategies mentioned in this thread. My issue was that I deleted a good
number of emails from my IMAP Inbox before archiving them, and then had to recover them
from Time Machine. At the time of the backup, I was running High Sierra on my machine Rockweed.
Now I am running Mojave. (I have learned to be very conservative in moving to new versions of OSX)
- recover V5 folder from Icarus, which contains my mail from last august:
/Volumes/Icarus/Backups.backupdb/Rockweed/2019-08-26-004934/Saltmarsh/Users/kellogg/Library/Mail/V5
copy to cernboxKellogg/Mail_recovery_2019-aug (note that files in cernboxKellogg are mirrored via the cloud
and available on all my machines)
- with Mail, Import to Import-2 on Rockweed as Apple Mail (since import as mbox claims it can find no mbox)
but when I try to move messages to another local mailbox (2017), I fail, with crazy message that it could not find
Mailbox "Happy New Year from Amherst", which is the title of one of the emails I am trying to move
notice that the mailbox structure is complex, and also that this backup was made under High Sierra, not Mojave
- try the import on maccmskel4, which is running Sierra
for INBOX-mac.mbox I can import it without problems, and I can move all the mails without attachments to 2017
I can view and download the attachments, but cannot move them, getting the warning that the body cannot be loaded
For INBOX-cern.mbox the behavior is exactly as on rockweed: I cannot move any email after importing it
getting the warning that the body cannot be loaded
rebuilding INBOX-cern does not help
- conclusion at this point: OSX versions are not the primary issue (even tried on ElCapitan with same results)
- EagleFiler import succeeds with copy on cernboxKellogg: cernboxKellogg/Mail_recovery_2019-aug/INBOX-cern.mbox (18300 records)
in the EagleFiler window, I see the emails listed very much like in Mail
the # column gives the number of attachments - if I click on the attachment, the entire message (including attachments)
opens in Apple Mail
- Since the Eagle_Library/Files/INBOX-cern flat file obviously contains all the attachments, in-line,
try importing it using Mail/Import as mBox to Import-4
get 18175 messages and a warning that some could not be imported - there are no attachments
Now I get somewhere:
- rename Import-2/INBOX-cern.mbox to 2017-cern and then drag 2017-cern to top level of On my Mac Mailboxes - success!
all the attachments remain visible (this procedure works on all my machines)
- I now delete all the non-2017 mails, and repeat the entire procedure for INBOX-mac and for 2018 and 2019
- only slight issue remaining is that I am not able to find a way of merging the -cern and -mac folders

Conclusions:

- something poisons the structure of the Mailboxes recovered from Time Machine
in such a way that neither EagleFiler nor Mail can handle them properly,
particularly their attachments
- the mail and attachments are recoverable, but their Mailboxes are extremely fragile
Basically: look, but don't touch

@Richard From what you’ve written above, it sounds like EagleFiler handled the attachments properly. But feel free to contact me if you want to discuss further. It’s possible that Time Machine made an accurate backup but that not all of the attachments were cached locally, so they weren’t there to back up. You can avoid that problem in the future by having a rule that moves or copies the messages to an “On My Mac” mailbox or by periodically archiving the messages to EagleFiler.

So, I posted a note to apple communities asking 1. if Apple had acknowledged the bug, and 2. if not, how would I know when it is fixed?

I got this masterpiece of stupidity in response: Apple will neither acknowledge or discuss fixing a bug until it is fixed (assuming there is a bug.)

Sigh.

I've been fighting with this problem for a while and just did an interesting experiment. I sent myself two messages with attachments. One I downloaded to a local folder by dragging it to that folder. The message and attachment survived. The second I moved to the local folder via the Message - Move to drop-down menu. That message only retained its header (no contents or attachment). Has this been observed by others, and does it point to the problem?

Thanks.

Mark

@Mark I’m not sure what the pattern is. I have seen cases where all the different methods (drag and drop, menu, rule, AppleScript) worked and also cases where all of them damaged the message or made it disappear entirely.

OK. I would love to hear if what Doug did on Oct 21 fixed his issues. I updated to Catalina on Nov 30, by Dec 2, I was instructed by the mail client that my mailboxes had issues and before I could finish reading the little pop up window mail crashed. On relaunch mail rebuilt my mail boxes and deleted random emails. A lot of sent emails and all current emails between November 27 and Nov 30. The current ones were obvious, but the others only came to light when I was trying to recover some procedures I had emailed to someone back in May 2019. Those were just gone. After discovering that I started digging aound on the internet and on my machine. Discovered elmx files on timemachine existed, but completely gone on my computer. Called Apple and got no where fast. Spent that last two weekends rebuilding a machine with Mojave and migrating data between it and my current Catalina install... I have recovered from TimeMachine directly to the mojave install and that fails. I have restored the mail and my Library files to an external hard drive for safe keeping. Each time I recover my email from that hard drive and try to put it onto the mojave install the same emails disappear when Mail is relaunched. I am convinced it is because the mail server is an Exchange server, but the sent emails were sent from my computer mail client, not the server... it really doesn't make sense to me that the emails are just deleted off my hard drive by the mail client. I have the email in timemachine and on my external drive, but have been unsuccessful in getting them loaded back into Catalina or Mojave mail. Apple hasn't been able to offer a solution so I am curious if what Doug did with the manual import actually worked.

@Erin As far as I’m aware, the deletion at Mail launch is not related to Exchange. It also happens for locally stored messages.

@Erin -- from everything I've been able to show Apple, a manual "File --> Import Mailboxes" from a backup should be fine.

[…] We’re still hesitant to recommend upgrading to Catalina, particularly if you use Mail, since reports of email data loss continue. Luckily, downgrading a new Mac from Catalina to 10.14 Mojave is possible for models other than the […]

I had a huge mail archive. i normally use POP3 (53Gb). After the upgrade (yesterday) i had visualisation (and search) issue. Rebuilding the mailbox.... 6Gb. Lost 47Gb of mail. I now trying to recovery from back-up.

@Massimo -- please file a bug with Apple about this -- we need more bug reports to get action!

Something good just happened, but I have no idea why or if happened for others. A massive number (all?) of my formerly blank emails I had downloaded to my computer from my campus server now have their attachments back! The other good news, which makes me think this state of affairs may last, is that when I delete a message from the server using my computer it also deletes from what my iPhone sees on the server. I wonder if there may have been a fix at the Exchange side of thing. Anyway, I'm happy for now and will (cautiously) downloads the many messages I've been leaving on the server for safekeeping.

Cheers.

Mark

Sherman Wilcox

This is stunningly awful. I haven't filed a bug report because I never actually had a problem -- I abandoned Mail when I first heard reports. I'm now living quite happily in MailMate.

[…] Si eres de esos que aún no ha sido lobotomizados lo suficiente para abandonar el uso del correo electrónico, en macOS Catalina 💩 encontrarás un sinfín de problemas relacionados con Mail.app. […]

Where was this obvious issue before last Wed. Went from 15.1 to 15.2 and all of my saved folders with emails for various reasons. On top of this the conversion, it was totally screwed up the interfaces with the provider.

Went to apple support, the first two efforts were absolutely worthless [ very un apple support]. Hours spent bouncing between Provider and Apple. Apple wanted a reboot on Time Machine. That was and still is a mixed bag of bad issues. I will never let anyone lower than tier 2 to go though a time machine reboot.

Finally got a senior adviser. After 45 mins he said this needed engineering. He ran a number of screen shots and zip programs. we short of got thing working only going forward. It is still not stable. My guess we will have to back before 15.2 and go from to square one.

What really bothers me the most, Apple knows there is an issue and refuses to acknowledge it. Then have me running all over the place, causing more issues.

Hello,
Same problem here with Catalina 10.15.2.
Unaware of all these problem with Apple Mail, and because I stupidly erased my mails (pop) while trying to change Pref. Syst./Internet Accounts (that's already strange: deleting an account will delete all related pop messages in Mail…), I decided to used my last Time Machine Backup.
Not working.
Ok, nothing to fear, I restore completely my MacBook Pro 16" with the last backup. Everything is working fine, but Apple Mail: wants to import messages, ok, and that's done, all messages are blank, just headers (and the lthe corresponding folder in Preferences/Mail is shrunk to 5% of its initial size: 54 GB to 2.5 GB...).
Not Happy.

that's already strange: deleting an account will delete all related pop messages in Mail

Yeah, that is so bad UX and it's been that way in Mail forever. Wonder how many out there have been bitten by this? There are so many more reasons to delete an account that just wanting to delete every message stored. Maybe you switch e-mail provider and just want to keep everything as an archive.

I can't talk about any public beta two of 10.15.3 to see if it fixes the issue with deleting the Envelope Index files/reimporting/reindexing "On My Mac" mail.

I would *like* to tell you to try that when it comes out to see if it fixes this problem on your end.

I would like to tell you it fixed my problems.

I really would.

All I can tell you -- it's a good day today. That I'm sure about.

@Shriner, can you not tell us whether or not this fix in the 10.15.3 beta simply repairs the index problem, or does it actually restore the lost e-mails?

I presume that it is not the latter, since, as others have noted, there is more free disk space after the Catalina upgrade, suggesting the actual deletion of e-mails and attachments, and not just an indexing issue.

I lost thousands of e-mails, with Gbs of additional free space after the upgrade.

I am anticipating having to go back to a TM backup from just prior to the Catalina upgrade, and importing e-mails, folder by folder, into the plethora of folder trees that I have, and then dealing with the duplication of e-mails between those imported, and the subset of e-mails that have, so far, survived the upgrade.

BTW, I did file *two* bug reports with Apple on this, since October.

Thank you for not telling us about this...

@Marc -- it will not restore e-mails if Mail.app went through a process (for whatever reason) of reindexing your existing "On My Mac" mail. Those messages are gone for good and would need to be restored from a backup if that happened.

If you have a backup, you should be able to find the backed-up "On My Mac" folder in ~/Library/Mail/V6 and restore that and then "import mailboxes" successfully.

@Shriner, thanks for the clarification/confirmation. It is as I expected, given the observations of increased free disk space post upgrade.

All of the e-mails that I lost were in On My Mac folders. I generally do not keep e-mails in the cloud, with the exception of a few folders for portable and timely access on my iPhone when away from my MBP.

As a second question, I have noted some AppleScript tools online that purport to identify and delete duplicate e-mails. For example:

http://jollyroger.kicks-ass.org/software/

or:

https://www.macupdate.com/app/mac/53180/remove-duplicate-messages

Any recommendations for such a tool based upon any experience?

I would guess that such a tool will be helpful to others in similar situations.

I will await the release of the 10.15.3 stable update and see what Apple has to finally say about this, before proceeding. This process of importing from TM and removing duplicate e-mails will be tedious and time consuming, thus only want to do it once. :-)

Thanks!

@Shriner, thanks for the weather report, very happy to hear about it being a good day as I'm about to have to commit to Catalina, and I just couldn't pull the trigger on moving to another client yet.

Can you also (not) tell us about any (alleged) resolution of the problem with moving messages between mailboxes? Is it a good day today as well?

@dpz -- I only personally move messages from my Gmail account to "on my mac" mailbox locations (that one way -- not the reverse).

So I'm not sure what that problem is directly nor what I'd be looking for to test against.

I would recommend watching for the public beta of 10.15.3 b2 and testing it?

@Marc -- no experience with "duplicate e-mail detectors" here, either. But, what I'd probably say would conceptually work would be that if you know the date you upgraded to Catalina, you could restore your "on my mac" mail from right before that upgrade, import it (which still works fine), and then just purge mail from the leftover "On My Mac" mailboxes from that date and before? That's what I did here. No duplicates remain doing that.

I am seeing some Exchange-related misbehaviour: all of a sudden, about 48 hours, my Catalina iMac stopped being able to fetch or synchronise email with my institute's Exchange server. New messages appear if I rebuild the mail boxes.

It was working absolutely fine before.

This occurs on both a wired iMac at my office and my home MBP on broadband. iOS mail and the Outlook web client work fine.

Moving lots of email out of the Inbox (from about 10k down to 1.7k) doesn't help.

Other accounts (Gmail, icloud) work fine on all machines.

This does not appear to be Catalina specific, however. Refs:
https://apple.stackexchange.com/questions/350372/apple-mail-always-need
https://discussions.apple.com/thread/8086520
https://discussions.apple.com/thread/8152487

Is it, nonetheless, related? Any ideas?

Hi, I just tested the actual Dev-Beta (10.15.3 beta 2) with Mail 13.0 (3608.60.0.2.2) and I can no longer reproduce the mail-loss after rebuilding the envelope-index.
Can someone else confirm this?

I guess i was both sad and happy to find this article. I removed my envelope index files and one of my inbox lost about 110,000 emails. I have a super duper backup where the emails are intact which i can boot to, however i cant export the mailbox from apple mail it just crashes.

Can i replace my broken mail setup with the version from the super duper backup. However i have looked and i cant figure out how to do this. Can i just copy the V7 folder in its entirety ?

Secondly i did actually keep my envelope files before the issue. Can i restore them to fix the it ?

Also like another user here i actually deleted all my pop3 emails from my server as a way to debug the issue :(

Any help would be great. Alex

@Mathew -- yes, it seems fixed here as well.

@Alex -- what you should be able to do from your backup would be to find the "On My Mac" mail folders on your backup drive and "File --> Import Mailboxes" from Mail. The act of *manually importing them* -- is fine. Just don't delete the Envelope Index files as a method to "reimport" them until 10.15.3 is out

Apple fixing it or not, having lost and hence having had to restore from backup 200k mails, I no longer trust Apple worth my mails. I’ve moved away from Apple Mail. My company uses outlook anyway. I’ve set up DEVONthink Pro so that it imports my mails from Outlook. All my mail act on rules I’ve implemented in DEVONthink and I’ve linked them all together with a simple Apple script that I call from my DEVONthink toolbar. Effectively, I’ve now a very simple Auto File procedure that takes my Mails out of outlook (or Apple Mail, but as I said, I’m not using it anymore) and into my folder structure in DEVONthink.

Search and retrieval is anyway far superior in DEVONthink, so there’s no point going back even if Apple finally fixed it. They never really acknowledged the problem in the first place, hence why should I trust them.

@shriner thanks for the advice so how do i select specific mailboxes using this method or do i have to basically import the whole V7 Folder and deal with the duplicates ?

@emax i too was thinking of going down the devonthink route to have a backup system aswell. Though other options i found that had decent reviews were

https://www.mothsoftware.com/index.php
https://mailsteward.com/

OK,

So I updated to Catalina ... had loads of problems and had to re-format fusion drive - a common problem apparently.

I re-insatalled from time machine

I could see about 100,000 inbox emails in my mail - but they have no content. About the same number in Sent emails.

After many conversations and trying a few things with Apple - been advised that the only way is to import the emails from the Library / Mail folder for Time Machine manually - which I have now done.

I now have double the amount of emails - the naff ones that can't be read in the main Inbox at the top ... and the same emails that can be read in (Import under 'On My Mac')

I have discussed this further with Apple and they have insisted it is not an apple problem - but I want to be able to get the emails for the bottom import folder into the correct Inbox folders at the top .... as searching emails is really difficult - having to dig down into folders and searching in each folder.

To make things worse I have about 6 or 7 different business emails which each have their own email folder in the top section and now in the bottom section .... But if they are run the correct inbox at the top - you can easily search all at once.

Any advice would be greatly appreciated...

Sherman Wilcox

MailMate, folks. Try it. Superior configurability (with a little coding in BBEdit), outstanding search. Easily exports to DevonThink or EagleFiler if you want that. It can be made to fit your workflow like a glove. Keyboard navigation is excellent and also highly configurable.

Apple continues to work on Apple Mail in the new developer beat of Catalina (10.15.3 beta 3) with Mail 13.0 (3608.60.0.2.6).
Mail-loss after rebuilding the envelope-index is still fixed.

Yes, I am hearing multiple reports that the macOS 10.15.3 beta fixes the rebuilding bug, but some users are still seeing messages that disappear when moved between mailboxes or that end up duplicated instead of moved.

Will mail 13 be also available under Mojave when it comes out?
I cannot go up to Catalina and my email suffers from 'eAlzheimer' since upgrade from High Sierra and despite multiple envelope rebuilds
thanks for your feedback

RE >>> Apple continues to work on Apple Mail in the new developer beat of Catalina (10.15.3 beta 3) with Mail 13.0 (3608.60.0.2.6).
Mail-loss after rebuilding the envelope-index is still fixed. >>>

is there an estimated time for this ?

as mentioned above I have all emails - but they are all in an import folder below - not in my inbox - which makes searching for emails a pain.

I may try and drag them all over to inbox - but if there is a fix - is it worth waiting ?

@Anonymous: no, Apple Mail version 13 is tied to macOS Catalina
@TREV: only Apple knows how much time they need till they release 10.15.3 as 'stable' to the public. But I would wait till at least some people here can confirm that the now known bugs are fixed.

So -- for the purposes of the bug that I know is fixed -- the problem was with "On My Mac" mail being deleted if "Envelope Index" files were deleted (or Mail.app threw up the dialog that the Mail index needed to be rebuilt from the first launch of Mail after upgrading to Catalina). This is fixed in the current 10.15.3 betas. I'm confident in stating that *specific* bug -- seems fixed.

I've not heard (here) about a way to replicate mail being deleted when "moved between mailboxes". I'm happy to try to replicate it on my end with some test accounts and open a case, but I'd need more detail like -- what kind of mailboxes -- and in what direction -- is the mail being moved? Exchange? Gmail? Moving between "IMAP" mailboxes? Pulling messages down from an "Inbox" to an "On my mac" mailbox? Moving message *back up* to an IMAP mailbox from an "On My Mac" mailbox? Is this all POP (for which I'm surprised people are still using that protocol these days...)? Exactly what is the setup where you see that specific problem?

To answer a couple of specific questions above (and I don't work for Apple -- just a sys admin at a large University so I see a lot of stuff...)

Mail is not going to be updated in Mojave (old OSes only get security fixes once a new OS comes out -- Applications are never majorly updated on N-1 or N-2 OS versions.

If you are trying to restore deleted "On My Mac" mail from a TimeMachine backup, I would recommend restoring the entire backed up ~/Library/Mail folder to another folder on your desktop. After you do that, you can look inside the folder for the mail structure. "On My Mac" mail would be within it's own folder with a long string name like: 0D214F03-E8D2-4DAD-8F4F-6285203B4475 or something and you can look inside that to identify the mail folders within. That would be the folder you would "File --> Import Mailboxes" only. No need to reimport *everything* to solve that specific problem (as you'll get duplicates of stuff you don't need.)

And nobody knows when 10.15.3 will be released apart from Apple. When the betas stop coming, you'll know it's close. *In general* (but not always) when beta build numbers end with letters closer to "a" -- then it's close to release. Beta 3 is 19D75a -- so it *may* be closer to release than I would have guessed considering there have been only 3 betas. Could be this week? Could be next week? Could be mid-Feb. Nobody knows.

@Shriner I’ve not been able to replicate the problems moving messages, either. But according to the reports I’ve received, it happens with IMAP (including Gmail) and Exchange mailboxes, moving both into and out of the inbox, and moving within the same account as well as to a different account.

According to this article:

https://appleinsider.com/articles/20/01/28/apple-releases-macos-10153-ios-1331-ipados-1331-tvos-1331-watchos-612

macOS 10.15.3 is released, albeit, it is not yet showing up for me as an update.

I am not seeing any explicit indications of a resolution of the mail bug in the coverage, at least not yet.

Sorry for the second rapid post here, but the 10.15.3 update is now showing for me, with the following details only:

This update:
• Optimizes gamma handling of low gray levels on Pro Display XDR for SDR workflows when using macOS
• Improves multi-stream video editing performance for HEVC and H.264 encoded 4K video on the 16-inch MacBook Pro (2019)

along with the following security updates:

https://support.apple.com/en-us/HT210919

So, still no official acknowledgement from Apple of the mail issue...

I just updated my test-vm to 10.15.3 (stable), but interestingly the Mail Version is 13.0 (3608.60.0.2.5), wich is one build older than what shipped with 10.15.3-beta3 last friday.

Alex Mcmillan

Can anyone here confirm if is possible to take a backed up V7 folder and replace the broken V7 folder in its entirety ?

I understand anything not the old backup would be lost.

Thanks. Alex

@Shriner Here is my experience with loss of messages.

My account is on Gmail / IMAP. I can send all the settings if needed but the most relevant may be that the "Archive Mailbox" is [GMail]/All Mail. I have numerous rules for filing messages. This basic configuration was stable for many years until I upgraded to Catalina.

When a new message comes in, it is just fine no matter what mailbox it has been filed into. If the message was not filed by a rule (i.e. remained in the Inbox), moving the message to a different folder works fine. But if the message was already filed by a rule, as soon as it is moved to a different mailbox, the content is lost. Mail displays the sender and subject, but nothing else. "View Raw Source" displays an empty window.

John

P.S. This includes deleting the message. If it had been filed somewhere else, deletion loses the message. Undo does not help in any of these circumstances.

@Alex That should work.

I imported all emails in 10.5.3, but moving emails to a different folder in the "On my Mac" is only partially working, some messages move fine, while others only show empty content. Rebuilding the mailbox make them disappear completely. I've explored the V6 folder, and it seems that all "xxxxxx.emlx" files are imported/moved fine, but the "xxxxxx.partial.emlx" files are sometimes not imported and are the ones that disappear when moved from one folder to another. What's the difference between the emlx and the partial.emlx? Is this what is happening to other users?

@Pablo The partial ones are IMAP and Exchange messages that contain attachments (which are stored separately from the .partial.emlx file). This seems to be a different/new bug from the ones I’ve seen moving messages.

Hello everyone.

Since Apple doesn't seem to fix the problem, I have two questions and it would be wonderful if anyone could help:

1. Does Spark Mail have the any issue at all?

2. How can I migrate to Spark. I wonder if I cancel my Mail® accounts it will delete / free the space occupied by the mail messages 🤔...?

Thanks in advance,

William C. Sloss

Problem with mail.
With MAIL in the dock, I used to click on the icon once and it would open. Ever since I upgraded to Catalina and then the latest version I have to double+click the icon to open MAIL. Anyone else have this problem? I talked to Apple and the advisor said they were aware of this problem.
OK,,,,,, Fix it Apple!

@michael tsai — Regarding your reply to @Alex.
@shriner — Regarding your request for details.

My experience is as follows. My V7 was created on a brand new 16” MBPro running 10.5.2, by using the 16” ’s Migration Assistant feature and using Time Machine containing my V4 (Sierra) as the Source. The resulting V7 did not contain the same number of .emlx files contained in the V4. When I initiated the 16” Catalina Mail App, I was missing text of the emails in both the upper level “Inbox” Mailbox (POP3) and the lower level On My Mac individually named Mailboxes.

@michaeltsai—Thus, I believe the V7 folder is not reliably created from the prior version V# folder, and any future “backed up” V7 folder will itself be broken. Thus, @Alex will not be able to use a backed up v7 folder (which I believe will be broken itself) to fix his current broken v7 folder.

@shriner— do you have any experience similar to mine?

@HoustonMJ I took the question to be whether it’s possible to replace all the app data at once, which it is. Of course, you won’t end up with any data that wasn’t in the original backup. If you use macOS 10.15.3, you shouldn’t end up with fewer .emlx files.

Hi All,

In light of the last few comments, I want to verify that there appears to be lingering issues with Apple mail subsequent to the 10.15.3 update.

That is, not all data loss related Apple Mail bugs/issues that have been observed and reported are resolved by this update, especially those with the "On My Mac" local storage folders, which is where the vast majority of my mail is stored. I keep only current Inbox messages or those requiring expedient attention in a "cloud" based IMAP Inbox folder tree, whether that be a personal me.com account, or my business account, which is via Office 365.

If correct, which would be concerning, I am hesitant to begin the tedious and lengthy process of restoring hundreds of folders/sub-folders from a pre-Catalina TM backup, after the 10.15.3 update, which I have installed.

@Mathew's comment above might open the door to suggest that Apple did not ship the beta3 version of Apple Mail in the 10.15.3 release, which might perhaps explain lingering issues, and that perhaps there will be a future 10.15.4 update to Catalina that may yet resolve other mail issues.

I am also, for the first time in years, contemplating moving to a different e-mail client application on both macOS and iOS, given these issues, especially if they are not fully resolved, and there has been no ownership by Apple of them.

I already have an Office 365 account for other business related reasons, as noted above, so moving to Outlook as a macOS/iOS client for both personal and business accounts would be one pathway, albeit, I have seen data loss reports in past years regarding Outlook. I would, of course, still need to restore the pre-Catalina e-mails from TM, but only really want to do that once, whichever pathway relative to an e-mail client I take.

Any comments would be appreciated.

@Marc Yes, there are still data loss issues when moving messages. If you have messages in On My Mac mailboxes and don’t move them, I would expect them to be fine.

I am using Spamsieve, which requires an On My Mac mailbox “Spam” to operate and move messages around. Does this mean Spamsieve is not compatible with 10.15 currently?

@Dave SpamSieve can use any mailbox that you want. Most SpamSieve users are already on 10.15, and for 99+% of them there are no issues. If there is an issue with your setup, it should be immediately obvious when you test it out with a copy of a message, e.g. you drag and drop a message and it disappears or Mail rules fail to move messages. In that case, some people have found that Catalina Mail is better at moving messages to a mailbox on the same server as the inbox rather than to another account or a local mailbox. Some other workarounds are described here. SpamSieve can also be set not to move messages when training, if you happen to be one of the people affected by the move-messages bug.

@Michael, thank for your comments.

Unfortunately, I cannot be assured that messages in the local On My Mac folder tree would not have to be moved from one folder to another. Thus, raising integrity concerns, if that is still an issue.

So, it would seem that one path is waiting for some indication from folks here, that perhaps an upcoming 10.15.4 beta release does make progress on resolving these issues, or does not.

The alternative would be, if either the above does not show signs of resolution, or Apple simply takes too long to resolve this (for some definition of "too long"), to consider moving to a different client application as noted.

@Marc FWIW, I don’t think I’ve seen any reports of problems moving messages between two mailboxes that are both On My Mac.

@Michael, perhaps I misunderstood your prior comment to me regarding having e-mails in the On My Mac folders and not moving them, in which case they would be ok. The logical inference would be that moving e-mails between folders might not be ok, albeit, perhaps that is not confirmed, per your follow up comment.

If there is any reasonable question, at this point, in light of the underlying issues, of lingering data loss, I am not prepared to begin the process of restoring/importing the old TM backup e-mails. As they say, the only good thing about banging your head against the wall, is that it feels so good when you stop.

I can't be selective in which operations may or may not be safe from a data integrity perspective, and honestly, shame on Apple for engendering those concerns and doubts.

@Marc Moving between IMAP and On My Mac, I have reports of problems. Moving within On My Mac, I have not, but that doesn’t mean that there are none. As I wrote to Dave, it seems like some people are affected (frequent problems) and others are not (none), and so if you figure out that you’re in the latter group you’re probably OK. But this is all guesswork because Apple has the code and bug reports and isn’t talking (even about the bug that seems to be fixed).

@Michael, my comment above describes issues with moving messages between two "On My Mac" mail folders.

@John Thanks for clarifying. So the On My Mac mailboxes are also dangerous, but only if the message had been previously filed by a rule?

@Michael, yes exactly. If the message has NOT been filed by a rule, I can move it across "On My Mac" mailboxes at will without any problems. But if it has been filed by a rule, it is lost permanently if it is moved from the filed location.

The rules themselves are very simple "Move Message to Mailbox:" ...

@John If this is easily reproducible for you, please file a feedback with Apple.

My general practice is that *everything* gets moved to an IMAP server once it arrives—most incoming mail comes in through an Exchange server at work, and I manually folder it to a different IMAP server after I've reviewed it... so moving between an Exchange inbox and a different set of IMAP folders.

From reading the reports above it wasn't clear it me if others have reported problems with this style of setup; most seem to concern "On My Mac" foldering.

Is anyone using a similar setup to me, and have you experienced problems with Catalina?

@John, for clarification, are the rules that you are referring to, rules that you created in Apple Mail, or rules created on the gmail web site?

I ask because I have a number of rules on iCloud.com, that move me.com mails to IMAP sub-folders of the Inbox for e-mails lists or similar differentiation for organization. I did this on iCloud.com so that I get the same behavior on my iPhone as I do on my Mac, relative to folder movement.

I only have a few rules in Apple Mail, that based upon keywords or similar, color the subject line, but they do not move e-mails to a different folder. I don't see this, of course, on my iPhone.

@John

I have filed this bug with Apple in November : FB7449043. Have no response to it whatsover (hasn't even been marked as a dupe so it appears that it has gone in to their ignore bin despite the DATALOSS in the subject line.

Here is basically the text of the bug:

DATALOSS: Messages automatically moved by rule on receipt lose data on subsequent moves

Mail Version 13.0 (3601.0.10). On macOS 10.15.1 (19B88)

When a message is received and automatically moved by a message rule to a local (On My Mac) mailbox, subsequent moves of the message (by drag and drop or by menu move options) result in the total loss of message data.

Steps to reproduce:

1. Create a rule to move a message to a local mailbox (Mailbox 1)
2. Receive a message and confirm that message has been automatically moved to the local mailbox (Mailbox 1) and has its contents (headers/body etc)
3. Move the message (via drag and drop or the menu move actions) to different mailbox (Mailbox 2)
4. View the message in Mailbox 2.

Expected result: The message in Mailbox 2 has it headers and body.
Actual result: The message in Mailbox 2 is completely devoid on content -- viewing raw source shows no data at all.

Note:
• Reproducible 100% of time
• No plugins are installed/active.
• Occurs regardless of type of receiving server (IMAP, EWS, Gmail,iCloud).
• Occurs also if the rule copies (rather than moves) the message.
• Only occurs if destination mailbox of rule (ie Mailbox 1) is local -- if Mailbox 1 is on a server, subsequent moves are OK.
• Mailbox 2 can be any mailbox -- (ie may be local or on a a server).
• Messages received to the inbox and subsequently filed by manual application of rules are NOT affected.
• Messages received to the inbox and subsequently moved by manual moves of message are NOT affected.

I did more research at the time and determined that the message moved by the rule was lacking a specific bit setting in the message flags when compared to messages moved manually.
If I manually edited the sqlite database for Mail to turn on this bit for a message. (as a plugin developer I do a lot of spelunking), the message was moved without dataloss. So in my estimation, the rule application is not properly setting a specific bit on the message that needs to be set.

I communicated this observation in my bug report -- which if correct should lead to a quick fix. But that was 2 1/2 months ago.

@John

Just to followup -- I cannot seem to reproduce this bug now I just update to 10.15.3 so it may be that Apple did indeed fix it.

@Scott Thanks for this information! I think this is just one aspect of the problem, though. Some users are seeing messages disappear when a rule moves them from the inbox (on a server) to another server mailbox. And I have also seen reports of messages deleted when dragged and dropped between mailboxes on two different servers. I have not been able to reproduce these myself, though.

Update (2020-02-18): I have received a report from a customer with macOS 10.15.3 that seems to indicate your bug is not fully fixed, either.

I'm currently running Mojave and very reluctant to upgrade to Catalina because of the mail issues. However, I have installed Catalina on a VMware Fusion VM to do some testing and have found no issues with my Fastmail IMAP account. All my rules are server side so emails get moved into the appropriate folder at Fastmail's end. I don't use local mailboxes on my Mac - just the ones in my Fastmail account. I've done some manual moves of a large selection of emails back and forth between a number of different mailboxes and all seems OK.

Just to confirm - this is a brand new install of Catalina setting email up from scratch, not an upgrade from a previous version of MacOS.

@Marc, these are Apple Mail rules.

@Michael, I did file feedback with Apple.

@Scott, I am pleased to see that you filed such a detailed report to Apple, and that you have observed a possible fix in 10.15.3.

I will update this weekend and confirm whether I also see the issue has been fixed.

@John, thanks for the clarification.

I filed a third feedback report, since the fall, on this with Apple today, also providing a link to this post and comments. Hopefully somebody there will take the time to read the comments here.

I confirm that the issue I reported above has been fixed as of 10.15.3.

Mohammad Monadjem

I just read the blog and im not sure whether this is related to my issue but whenever i move some of my mail to its relavant folder since im using the mail app, i lose all its content and all i have is its subject title. Can you halp solvong this because i dont understand how it started and how to stop it.

Gordon Henderson

I have a friend who is currently using MacOS Mojave 10.14.6 She has IMAP mailboxes, but also quite a bit of older messages stored in folders in On My Mac. Not using any rules, and not very likely to be moving messages between On My Mac folders. No longer has any POP3 accounts. Is an update (not clean install) 10.15.3 release still a risky proposition, simply because she has a lot of old messages in On My Mac? It looks from what I see here that Apple has made some progress, but there are still some remaining issues, especially with moving messages in On My Mac.

Here are two possible solutions to the problems we are all having, which I've copied from the Apple discussion forums. If anybody's brave enough to try them, I'd be mightily interested to know the outcome:

I have solved the Catalina email loss problem and solved the email indexing problem! YEAHHHHHH!
I had over 500,000 emails, with more than 30 POP accounts and 15 IMAP accounts. I had over 200 folders and sub-folders. A disaster. I had updated from Mojave to Catalina and I realized the problem when I went to do a search without results. I had tried them all, but the emails disappeared. I contacted Apple's specialist support, but they didn't know how to answer, although many users complained about the problem. I will illustrate the solution;

1) obviously I had a manual backup of the Mac library on Mojave. The important thing is to have a backup where the emails previously worked. regardless of whether catalina or mojave. In catalina, only the name of the V6 changes to V7 but 99% is the same. So you can recover and run everything, even from a mojave backup to catalina

2) I restarted the Mac in safe mode (press shift button)

3) I entered the mac library and replaced all the contents of the ACCOUNT folder (library> account) with the content saved in the backup

4) I entered in Preferences (library> account) and I replaced these two files with those saved in the same position in backup hard disk:
com.apple.mail-shared.plist
com.apple.MailMigratorService.plis

5) I went into SyncedPreferences (Library> SyncedPreferences) and I replaced these two files (if there are none it does nothing) with those saved in the backup:
com.apple.mail-com.apple.mail.vipsenders.plist
com.apple.mail.plist

6) I went into Containers (Library> Containers) and replaced the folder saved in the backup:
com.apple.mail

7) Now go into Library> Mail> V7 and delete everything. From the backup disk, first I copy and paste the Maildata folder (before Catalina automatically rebuilds it - it is the first folder to copy) then I copy and paste from the backup disk, all the other folders in blocks of 10 at a time (so that if Catalina blocks me saying "this folder / files already exists, I can start over from the last point).

8) When I have finished replicating all of the V7's internal from the external hard drive to Catalina, I restart the mac in recovery mode (Option-Command-R or Shift-Option-Command-R to start up from macOS Recovery over the Internet ) and do an SOS on both Macintosh HD discs. Then reboot

9) Now with the active connection, I open MAIL and wait for all emails to be indexed. When Mail re-opens all the e-mails and folder trees, I will have a time of about 30 -40 minutes, where all the e-mails, the body of the e-mails can be seen correctly before catalina deletes everything. The first thing I do is to scroll (even quickly) all the folders to the end, so that the indexing of Core-Spotlight can load the contents-preview messages.

10) Attention! At this stage, indexing and email search still does not work. Whatever you are looking for, it still does not work.

11) I make sure there are no folders with no body messages, so I click on each folder, sub-folder, sent mail, individual sent mail folders, one by one, (I did it for 200 folders, pop, imap, pec , sent, archived, icloud etc) and click on> MAIL> BOX> REBUILD (be careful you have to do it one by one and wait for it to finish rebuilding).

12) I suggest storing the number of messages (eg 14.234) so ​​that when it rebuilds, it returns 14,234 messages.

13) Do it folder by folder and you will see how everything will stabilize and also spotlight and core-spotlight, it will give you the results.

If you have solved it, send me a like on my linkedin profile: https://it.linkedin.com/in/massi-vigilante-741837147

I summarize:
1) boot safe mode and connect disk backup
2) replace the folder library> accounts, the files in preferences, the files in syncedpreferences, the files in containers from the hard from the external hard disk in users library
3) delete the contents of the Mail> V7 and starting from the Maildata folder, copy the contents of the v7 from the backup to the internal library
4) restart in internet recovery mode, do sos
5) open mail after restarting the PC wait for the emails to be indexed
6) scroll all the folders with the mouse to load the content
7) folder after folder, reconstruct (mail> box> reconstruct)
8) wait for it to rebuild and then move to the next folder
Reply Helpful (1) Thread reply - more options

Several people have reported lost or blank e-mail messages after updating macOS to Catalina. I lost 50,000 messages. Here's my solution, which requires a back-up of messages made before the Catalina update.

Restore the folder V6 from User/Library/Mail to the desktop.
Open Mail if it is not already open.
Create a new mailbox where you will put recovered messages so that you can review them before storing them with other messages.
Select the command "File" - "Import Mailboxes  … ", then select "Apple Mail" as the format.
Look in the folder V6 on your desktop for a mailbox that you want to restore, select it, then click "Continue". The import will start. A message will tell you when the import is done.
After the messages have been imported, the messages will be in a new mailbox called "Import". Within that folder, they will be either in the simple folder structure that you would expect or in a multi-level folder hierarchy. If they are in a simple folder structure, click on a message in the message list, press command-A to select all the messages, and drag them to the folder that you created in step 3. If they are in a multi-level hierarchy, option-click on the triangle next to the folder called "Import" to expand the hierarchy. Command-click on all folders named "Messages". Click on one of the messages in the message list. Press command-A to select all messages in the message list. Click and drag the messages to the new mailbox that you created in step 3.
Progress in moving the messages will be shown at the bottom of the folder list. After all the messages have been moved, delete the "Import" folder by right-clicking on it and selecting "Delete Mailbox  … "
Repeat steps 3 to 7 for other mailboxes.

Methods that did not work for me:
Re-building mailboxes.
Deleting envelope files in folder V7 and forcing Mail to re-index V7 messages.
Temporarily removing folder V7 and forcing Mail to re-convert messages from folder V6.
Restoring a Time Machine back-up from within Mail. Mail crashes.

I had the same problem with content missing after moving messages between local/offline mailboxes. Issues seems to be resolved after upgrading to 10.15.3 yesterday. Messages that lost content are still missing that content, but now moving messages doesn't cause the body to disappear any longer.

Boy there is a big jump. I just installed the new macOS-Beta (10.15.4 beta 1) and Apple Mail is now Version 13.4 (3608.80.13.2.1).
macOS 10.15.3 (stable) has Version is 13.0 (3608.60.0.2.5) so here could come a cure for a lot of bugs.

Sadly I am not able to reproduce any bugs in my VM but at least we can see Apple will ship a greatly improved version of Mail in 10.15.4.

Based on recent history, I would not be so quick to assume "greatly changed" implies "greatly improved".

This is the longest time that this thread has been quiet since it started :-)

Any feedback fom macOS 10.15.4 public beta 1?

Just a short Info macOS 10.15.4 dev-beta 2 comes with Apple Mail Version 13.4 (3608.80.19)

Stephane Plaisance

Maybe relevant for others. My issue was that my local mailboxes where empty after importing from a mail export of my previous machine.
I could finally solve this by importing each local-mail subfolder (first level) separately.
It seems in my case that importing the main local-mailbox did not lead to recursively import subfolders and was only creating them empty.
when importing each one by one their recursive content was imported.

Many, many THANKS to John Biggins for this fantastic information that really helped me. I lost about 75000 emails when upgrading from Mojave (10.14.6) to Catalina (10.15.3). My previous macbook pro had to be taken into the macstore for repair for the 4th time (2 logic boards replaced, the top case was replaced twice and the touch bar had an intermittent connection) and rather than fix my machine, they gave me a new 16” machine, which I love, but I was forced into upgrading to Catalina. :-/
Just wanted to make a few notes to help make this clearer for others that may be going through this. I have noted below according to the steps in the first part of his message.

1) i had a time machine backup of 10.14.6
3) when he mentions ACCOUNTS, he is talking about USER account, so what made more sense to me was this: Macintosh HD > Users > YOUR USER ACCOUNT > Library - you don’t need to replace all of the content from this folder - only what is relevant to Mail
4) Macintosh HD > Users > YOUR USER ACCOUNT > Library > Preferences If you dont’ have these files, don’t worry about it
com.apple.mail-shared.plist
com.apple.MailMigratorService.plis
5) Macintosh HD > Users > YOUR USER ACCOUNT > Library > SyncedPreferences If you dont’ have these files, don’t worry about it
com.apple.mail-com.apple.mail.vipsenders.plist
com.apple.mail.plist
6) Macintosh HD > Users > YOUR USER ACCOUNT > Library > Containersl
7) if you’re coming from Mojave, you will copy from V6 to V7 in Catalina I copied the Maildata folder and then everything else at the same time. took about 10 minutes to copy
8) I wasn’t sure what he meant about SOS, so I just ran First Aid disk repair - on both the Macintosh HD and Macintosh HD - Data disks
9) I didn’t understand what was meant about Catalina “deleting everything” and luckily didn’t run into this.
11) On my machine, nothing really happens when you hit Rebuild - the emails from that folder might blink. on the folders that had thousands of emails, if I clicked on an email, and then another email, and then selected the first one again, the sender would disappear and the body of the email would disappear. this means that it is rebuilding. this will all come back. Also, the rebuild command in the menu may grey out. when it’s ready again, it will appear as selectable again
12) I opened mail on a back up computer and checked folder by folder to confirm that all of the emails were, in fact, there.

Hi there

My issue is not losing data but before I upgraded to Catalina, I set up read receipts on my emails. Now I cannot delete it. I have been through terminal and done delete apple.mail.com UserHeaders but that does not work. Any suggestions?

Again a short Info: macOS 10.15.4 dev-beta 3 comes with Apple Mail Version 13.4 (3608.80.23), so a slight bump this time.

Can anyone that is running 10.15.4 beta 2 and 3, care to not comment on the behavior that they are not observing relative to the bugs? :-)

Thanks!

Tried with 10.15.4 beta 1 nothing changes... I found (almost) all my mail on my old MacMini12 but lot of them are only header. I dont know what to do anything could lead to permanent data loss :(

I also today found Mail is only giving me Headers in some emails and on other the full Header and message.
Cant figure what to do I pad and iPhone emails are all ok.
Any ideas anyone.

Today I created a Smart Mailbox to view selected messages that were stored in a mailbox in On My Mac.

I then manually moved the messages from the Smart Mailbox to a new mailbox in On My Mac (and, in case it matters, also created new Rules for that mailbox).

When I opened the messages in the new mailbox, they had lost their content. The headers are there but the body of the messages is blank.

When I exported the new mailbox, then imported it back into Mail, the content returned. (The status of the messages, such as read/unread, replied to, etc., is lost though.)

So, apparently the content is only hidden, and maybe someone can figure out how to display it again without exporting and importing.

I forgot to add that I am running 10.15.3.

@DavidB That’s good to hear, but for the other cases I’ve seen the content really was gone, with the files in Mail’s data store corrupted.

@Michael Tsai,

Yes -- I know my case may not be the same as others, although how would you know the files were corrupted?

@DavidB Because EagleFiler reported errors importing them, and so I checked in a text editor and found that they were indeed messed up.

Migrated to Catalina and have had the usual problems outlined above. Still have the e-mails on hard drive so imported them, but then found out that I had lost folders so tried Time Machine, but then Mail crashes and cannot use the backup. However, now have most e-mails back, so just crawling around to see what I had lost in the upgrade...one thing I've gained with Catalina, though, is a load of flashy applications that I don't need and don't want...such as 'News' and 'Stocks'...

@CTom It crashes if you invoke Time Machine from within Mail, but you can still use Time Machine from Finder to restore mail from a backup.

If I perform a fresh Catalina install and pull down mail from Exchange, am I going to have issues with mail disappearing when they move folders?

I move mail all the time, usually via an AppleScript I wrote years ago. I assume that no matter how mail is moved (by hand, via script, by local/server rules, SpamSieve mark-as-good), the problem exists?

@Bob The only way to know for sure is to try it. Most people are not affected, but some are. The method of moving the messages doesn’t seem to matter.

I've turned apple mail to offline. I still use exchange. I'm using ostbox for email and mailsteward to back up (long time user, need to give eaglefiler the time it deserves).

Does this mean it's reasonably safe for me to upgrade to catalina? Still not sure I'm gonna do it, but I am curious.

@msdisme Yes, that should be safe as long as you update to 10.15.3 or later, so that you get the fix for the bug with locally stored messages.

Just an FYI that 10.15.4 release is now available for upgrading.

@Marc Unfortunately, the messages-disappear-when-dragged bug does not seem to be fixed in macOS 10.15.4.

@Michael, I am seeing a variety of "flaky" behavior still, especially when viewing messages that have been moved between IMAP server folders on my MacBook Pro, in contrast to viewing the same messages on my iPhone, which I also upgraded today.

Synchronization issues abound in the IMAP Inbox and sub-folders that are on the servers. I have had other message content issues with moving mail from IMAP server folders to local On My Mac folders.

This is getting disheartening, in all honesty, given how long it has been going on.

I am about ready to give up on Apple Mail, and migrate to Outlook. I am finding myself, more and more frequently, needing to go back to e-mails from a while ago, that disappeared from Apple Mail local folders during the Catalina upgrade. I have yet to attempt re-importing the old mail from a TM backup, given how time consuming a process that will be, which makes accessing those older e-mails a pain.

Any thoughts from your end that might alter my views?

Thanks for maintaining this post.

@Marc

I concur with Marc, thank you for maintaining this post Michael.

My institution gave up on supporting Mac Mail because of this same issue. I still use Mail, but am spending more time on chasing missing attachments than on my research. Using Outlook and Mail simultaneously is a recipe for madness, not productivity. Are people bailingand moving to Outlook in the short term? ...is that even possible from Mac Mail? ...is there some .mbox > .pst converter out there that actually works?

@Bathead, I have not yet done a lot of research, but here is one approach:

https://medium.com/@johnsonanna0444/want-to-migrate-apple-mail-to-outlook-for-mac-this-is-how-professionals-do-it-6464cf7150a8

It seems to me that another option, presuming that you have the emails that you want to migrate in local Apple Mail folders, is to move them to IMAP folders on the server for the relevant account, for example, iCloud. This presumes that the emails and any attachments are intact.

Once they are in the IMAP folder on the server, you can then access them in Outlook, and then move them to local folders in Outlook.

Bear in mind, that at least for an iCloud account, you would need to create app specific passwords for security to use third party apps, like Outlook, presuming that you have 2FA enabled.

Sören Nils Kuklau

> Using Outlook and Mail simultaneously is a recipe for madness, not productivity.

For whatever that's worth, I use Mail for Mac, Outlook for Mac, Outlook for Windows (on two separate machines), and Outlook for iOS simultaneously on the same company account. Doesn't seem to cause issues.

While it's gotten better over the years, there are still things about Outlook for Mac that just don't feel right, and also things compared to the Windows version that are missing. For reading, I think I generally prefer Mail (and have been lucky, I guess?); for writing, there are some compatibility issues with the way Mail sends formatted mails, so I avoid that in official communication.

> there are still things about Outlook for Mac that just don't feel right

You mean like having Undo not undo your most recent action but instead randomly pick some other of your past N actions, often so far back in history that you can't see what actually was undone?

It should be fine to use Outlook and Mail together. There are also plenty of other Mac mail clients: MailMate, GyazMail, Airmail, Thunderbird, Postbox.

What bugs are still unpatched?

Can someone give me detailed Information how I can recreate the bug of corrupting massages when moving between accounts/mailboxes? I would like to try and recreate this on my VM.

@Mathew For some users, dragging and dropping a message from one mailbox to another (e.g. from the inbox to another mailbox on the same account or from an On My Mac mailbox to a server mailbox) makes the message disappear. And the same thing happens for other methods of moving messages (rules, AppleScript). I have multiple reports of this still happening on macOS 10.15.4. Moving to an On My Mac mailbox seems to work better.

Has anyone verified if the message move bug still exists in the final version of 10.15.4?

I have been wanting to move from MacOS Mojave to Catalina. Part of my e-mail history is a large group of messages that date from when I was using POP e-mail - "On My Mac". I currently use iMAP exclusively, and have I have multiple IMAP folders.

Michael said above: Yes, that should be safe as long as you update to 10.15.3 or later, so that you get the fix for the bug with locally stored messages.

If I update from Mojave 10.14.6 to Catalina 10.15.4 AND I don't attempt to move messages between folders on my Mac, am I likely to still have problems? I currently use an iPad Pro when receiving, sending or moving mail to folders.

@Gordon Obviously, I can’t guarantee anything, but I think that would be safe. Make backups, anyway.

@Gordon - note that I haven't read that the problem is moving messages between folders, per se. The problem is moving messages between mailboxes (i.e., accounts). Others should correct me if I'm misinterpreting that.

I'm setting up a VM to play with Postbox and Thunderbird. I'm done with Catalina Mail.

@dpz The problem affects moving messages between two mailboxes (folders) in the same account.

Thanks @Michael. I was interpreting "mailboxes" as servers, since I use the term "folders" when I mean folders. This is far worse. Clearly not a priority for Apple.

I tried all of the mail programs that @Michael recommended above and have chosen MailMate. The most important thing to me is to have "Smart Mailboxes" that include several email accounts. I also needed a flexible rules engine. Thanks @Michael for your recommendations. I am sad that Apple clearly doesn't consider this a problem. I have used Apple Mail for many years.

FWIW, I just filed what is my third or fourth bug report on this issue.

I would urge others to do the same, so that Apple is aware of this issue and gets feedback from multiple sources.

I included a link to this thread in my report.

Keith Edwards

I wanted to add a bit of info to this thread. I have not personally updated to Catalina, due solely to this bug—I just can't risk losing old email messages—but I've been monitoring this thread since October to see how things are progressing.

Thursday evening I decided to do something I've never done before, write to Tim Cook: "Tim, I’m sure you and the rest of the Apple senior leadership are slammed dealing with the COVID-19 outbreak. But I’m writing due to an ongoing, and very serious, bug in Catalina that doesn’t seem to be getting engineering attention. Catalina, even as of the 10.15.4 release this week, has a disastrous Apple Mail bug that can result in permanent loss of messages stored in remote servers and locally. Here, for example, is a discussion on a seasoned Apple developer blog about the problems that have been ongoing since October 2019 when Catalina was released: [link to this thread]. I hope you can make sure engineering is aware of the issue and is dealing with it, because it’s a show stopper for many of us."

I was shocked to get a call the very next day from a member of the Apple Corporate Executive Relations team who spoke to me about the problem and promised to look into it with engineering, and get back to me today. Today he called back and let me know that he couldn't share anything specific but indicated that now that the issue had been raised with Tim, Engineering would have to look into it with high priority and report back to Tim about its status. He said they normally would have put me in contact with someone from engineering to work through diagnosing the problem, but since I'm not on Catalina yet that's of course not an option. I made sure he was aware that multiple bug reports had been filed against this issue and he said they would find these as they look into the problem.

So I'm not sure what this means, of course, or whether it'll lead to any action. But I figure it can't hurt, and I was impressed by the speedy and high-touch response I got.

@Keith, wow, that’s great!

I was coming here to post that Apple has released the 10.15.5 betas to devs, and am wrestling with waiting to give one more stable release a try or move to Outlook.

Your post may give me the basis for waiting.

Thanks!

Sitting here looking at the update notification for at least the 100th time, I wondered if there had been any movement on this issue. Quite surprised to find that it's still a thing. I have an IMAP account with hundreds of thousands (million(s)?) of messages as I rarely delete things. It's just too useful and easy to be able to go back in time.

I wouldn't want to lose that. In fact I really need to find a piece of software / script to take an image of it in an open, encrypted format.

Glad to hear the issue seems to have been successfully escalated by you, Keith. This needs to get fixed before the next major release. Please keep us up to date - at least some of us are listening.

*Clicks "Remind me tomorrow" again*

Sören Nils Kuklau

> In fact I really need to find a piece of software / script to take an image of it in an open, encrypted format.

I think Michael Tsai might have one? (Not sure how open the format is.)

@Sören Yes, EagleFiler uses the standard mbox format, optionally encrypted using Apple’s .sparsebundle format.

And it's Postbox FTW after ruminating on Postbox vs Thunderbird for about a week. Bye Felicia.

Thanks for the all the guidance here!

Stay up-to-date by subscribing to the Comments RSS Feed for this post.

Leave a Comment