xattr Flags and iCloud Drive

Howard Oakley:

If you work with xattrs, you’ve probably already seen this in xattrs whose name ends with a hash # then one or more characters: that’s actually the flags, not part of the name, what Apple refers to as a ‘property list’. To avoid confusion I won’t use that term here, but refer to them as xattr flags. A common example of this is com.apple.lastuseddate#PS, which is seen quite widely.

Flags can be upper or lower case letters C, N, P and S, and invariably follow the # separator, which is presumably otherwise forbidden from use in a xattr’s name. Upper case sets (enables) that property, whilst lower case clears (disables) that property.

[…]

The sought-for ‘whitelist’ is actually baked into the xattr flag code, where as of 2013 the following default flags are set for different types of xattr[…]

[…]

If you want a xattr preserved when it passes through iCloud, you therefore need to give it a name which ends in the xattr flag S, such as co.eclecticlight.MyTest#S.

This is like filename extensions all over again, cramming two pieces of data into the same field. If you have an app that uses xattrs (maybe even from before iCloud Drive was introduced), you need to migrate all your metadata (and forever check the old xattrs when reading a file) if you want to set the “flags” for proper iCloud treatment.

Previously:

• iCloud Drive Can Strip Metadata From Your Documents

Update (2019-07-24): Howard Oakley:

Thus the S flag preserved all the xattrs containing Skim’s annotations, but when that file was opened in Skim, it was unable to read them because it doesn’t strip the xattr flags from their names when reading xattrs. And that is the problem with this technique: although the xattrs are, in general, preserved, because most apps don’t expect to have to handle xattr flags appended to their names, the preserved xattrs aren’t used – after all that.

The solution therefore is for all apps which access xattrs by name to drop any xattr flags from the names before using them. This demonstrates the cost of this elegant kludge. As this isn’t handled transparently in calls such as getxattr() which access xattrs, every app is left to its own devices to handle xattr flags appended to xattr names, which is inefficient and encourages inconsistencies between apps.

Update (2019-07-25): Pierre Habouzit:

In general xattrs are a difficult challenge to sync because of the fact that it’s a thing that has no meaning for the end user, and the problem it’s trying to solve is that xattr serve very different purposes, some are metadata for the document (XATTR_FLAG_SYNCABLE), some are about metadata tied to the machine you’re on and make no sense on another machine (XATTR_FLAG_NO_EXPORT), or is privacy sensitive.

The two motivations FWIW are completely for good user experience for 3 reasons:

• privacy
• not blowing up your data plan because you have an xattr used for indexing e.g. that keeps being updated
• avoiding fake “conflicts” in the sync UI that would be extremely confusing.

is it more work for developers? yes.

but this is to eventually get a better user experience. syncing everything by default, because the sync engine cannot understand what is a meaningful change and what isn’t, would pop up the user due to conflicts on xattrs ALL THE F-ing time.

so instead we rely on developers to tell us what is really important to sync for the document to be whole (and in general we prefer packages with metadata in a plist inside the document or in a Resource/ subdir of the bundle).

To be clear, I think the flags seem to do a good job of solving real problems. I just don’t think their existence was communicated very well, and the implementation—combining them with the name—is not very friendly to pre-existing data (or xattrs that need to change flags later).

Update (2019-07-26): Howard Oakley:

Jonathan Levin – who of course does know all about these, and has now added a short section about them to volume I of his reference books on Apple’s operating systems – points out that xattr flags only affect copy behaviour under the copyfile(3) API. That means that copies made using the Finder will respect them, and will strip xattrs where so instructed, but cp in Terminal doesn’t, and preserves all xattrs regardless of their flags. That can of course be a mixed blessing.

Extended Attributes iCloud Drive Mac macOS 10.14 Mojave Metadata Programming Syncing

11 Comments RSS · Twitter

struct propertyListMapping {
	char enable;	// Character to enable
	char disable;	// Character to disable -- usually lower-case of enable
	xattr_operation_intent_t	value;
};
static const struct propertyListMapping
PropertyListMapTable[] = {
	{ 'C', 'c', XATTR_FLAG_CONTENT_DEPENDENT },
	{ 'P', 'p', XATTR_FLAG_NO_EXPORT },
	{ 'N', 'n', XATTR_FLAG_NEVER_PRESERVE },
	{ 'S', 's', XATTR_FLAG_SYNCABLE },
	{ 0, 0, 0 },
};

Leave a Comment