Tuesday, July 23, 2019

xattr Flags and iCloud Drive

Howard Oakley:

If you work with xattrs, you’ve probably already seen this in xattrs whose name ends with a hash # then one or more characters: that’s actually the flags, not part of the name, what Apple refers to as a ‘property list’. To avoid confusion I won’t use that term here, but refer to them as xattr flags. A common example of this is com.apple.lastuseddate#PS, which is seen quite widely.

Flags can be upper or lower case letters C, N, P and S, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, whilst lower case clears (disables) that property.

[…]

The sought-for ‘whitelist’ is actually baked into the xattr flag code, where as of 2013 the following default flags are set for different types of xattr[…]

[…]

If you want a xattr preserved when it passes through iCloud, you therefore need to give it a name which ends in the xattr flag S, such as co.eclecticlight.MyTest#S.

This is like filename extensions all over again, cramming two pieces of data into the same field. If you have an app that uses xattrs (maybe even from before iCloud Drive was introduced), you need to migrate all your metadata (and forever check the old xattrs when reading a file) if you want to set the “flags” for proper iCloud treatment.

Previously:

Update (2019-07-24): Howard Oakley:

Thus the S flag preserved all the xattrs containing Skim’s annotations, but when that file was opened in Skim, it was unable to read them because it doesn’t strip the xattr flags from their names when reading xattrs. And that is the problem with this technique: although the xattrs are, in general, preserved, because most apps don’t expect to have to handle xattr flags appended to their names, the preserved xattrs aren’t used – after all that.

The solution therefore is for all apps which access xattrs by name to drop any xattr flags from the names before using them. This demonstrates the cost of this elegant kludge. As this isn’t handled transparently in calls such as getxattr() which access xattrs, every app is left to its own devices to handle xattr flags appended to xattr names, which is inefficient and encourages inconsistencies between apps.

Update (2019-07-25): Pierre Habouzit:

In general xattrs are a difficult challenge to sync because of the fact that it’s a thing that has no meaning for the end user, and the problem it’s trying to solve is that xattr serve very different purposes, some are metadata for the document (XATTR_FLAG_SYNCABLE), some are about metadata tied to the machine you’re on and make no sense on another machine (XATTR_FLAG_NO_EXPORT), or is privacy sensitive.

The two motivations FWIW are completely for good user experience for 3 reasons:

  • privacy
  • not blowing up your data plan because you have an xattr used for indexing e.g. that keeps being updated
  • avoiding fake “conflicts” in the sync UI that would be extremely confusing.

is it more work for developers? yes.

but this is to eventually get a better user experience. syncing everything by default, because the sync engine cannot understand what is a meaningful change and what isn’t, would pop up the user due to conflicts on xattrs ALL THE F-ing time.

so instead we rely on developers to tell us what is really important to sync for the document to be whole (and in general we prefer packages with metadata in a plist inside the document or in a Resource/ subdir of the bundle).

To be clear, I think the flags seem to do a good job of solving real problems. I just don’t think their existence was communicated very well, and the implementation—combining them with the name—is not very friendly to pre-existing data (or xattrs that need to change flags later).

Update (2019-07-26): Howard Oakley:

Jonathan Levin – who of course does know all about these, and has now added a short section about them to volume I of his reference books on Apple’s operating systems – points out that xattr flags only affect copy behaviour under the copyfile(3) API. That means that copies made using the Finder will respect them, and will strip xattrs where so instructed, but cp in Terminal doesn’t, and preserves all xattrs regardless of their flags. That can of course be a mixed blessing.

11 Comments RSS · Twitter

Pierre Habouzit

Look at <xattr_flags.h> and xattr_name_with_flags(3), xattr_name_without_flags(3), xattr_flags_from_name(3). These are API since macOS 10.13 and iOS 8.

HISTORY
These functions first appeared in Mac OS in 2013.

Pierre Habouzit

Also the implementation of all of that is open source:

https://opensource.apple.com/source/copyfile/copyfile-138/xattr_flags.h.auto.html
https://opensource.apple.com/source/copyfile/copyfile-138/xattr_flags.c.auto.html

And that's what the CNPS mean:

struct propertyListMapping {
	char enable;	// Character to enable
	char disable;	// Character to disable -- usually lower-case of enable
	xattr_operation_intent_t	value;
};
static const struct propertyListMapping
PropertyListMapTable[] = {
	{ 'C', 'c', XATTR_FLAG_CONTENT_DEPENDENT },
	{ 'P', 'p', XATTR_FLAG_NO_EXPORT },
	{ 'N', 'n', XATTR_FLAG_NEVER_PRESERVE },
	{ 'S', 's', XATTR_FLAG_SYNCABLE },
	{ 0, 0, 0 },
};

Wait, slow down for the OS X learning impaired, I am not following this discussion. I used to try to avoid problems with losing OS X metadata by backing up such data to disk images on servers that were formatted HFS Extended. Eventually I gave up and my switch to Linux and Windows user devices has made the point moot. To clarify, if I have a Mac, I then create documents and the app appends xattr flags (which seems to be unavoidable these days), and I then sync to Apple's own cloud drive service, I can loose crucial data!!!! The PDF example was particularly galling.

Is this really best practices for user data? Everything seems increasingly overly engineered and fragile to me, but I'm getting older (news at 11, old man yells at clouds) and a current OS X outsider, perhaps I am missing some subtlety to this problem? Anyone care to explain to me why this data fidelity issue is not very scary? Not to mention confusing in light of Apple's decision to move to a more robust file system that was supposed to be less prone to data loss. Two steps forward and one step back type of thing?

@Nathan The way I look at it is that iCloud Drive does not preserve xattrs by default, but there is a (not widely known) way (by renaming the xattrs) that an app can opt in.

Thank you for the clarification! Any reason iCloud is not built to simply handle the preservation of xattrs by default?

@Nathan Some xattrs don’t make sense to sync, for privacy or other reasons.

Thanks again Michael!

Remember "defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE" ?

I guess that was too good an idea... It sure would be nice to be able to turn off things like "com.apple.lastuseddate#PS" on NFS-mounted drives. «sigh»

Sören Nils Kuklau

Remember “defaults write com.apple.desktopservices DSDontWriteNetworkStores -bool TRUE” ?

I guess that was too good an idea

I never understood why people get so upset about .DS_Store files. There also never seems to be a good alternative suggestion. The Finder could instead keep a database in your home directory, but now it doesn’t travel with you, has to keep track of all kinds of disks that may not even exist any more, etc.

[…] Tsai has a brief page on xattr Flags and iCloud Drive with a similar […]

Leave a Comment