Archive for March 4, 2014

Tuesday, March 4, 2014 [Tweets] [Favorites]

Arq 4.0

Stefan Reitshamer:

I’m really excited about this release! It’s got features that many people have been asking for, and it opens Arq up to a whole new range of options for storing backup data.

Glacier backups now use the S3 Glacier Lifecycle feature. Among other benefits, this allows Arq to prune old Glacier commits (that previously were immortal) and subject them to the budget. Unfortunately, Glacier vaults from previous versions of Arq cannot be transitioned; you have to delete them and create a new backup target (not in that order!).

You can now back up to other S3-compatible destinations such as as DreamObjects, which is about half the price of Amazon S3 and has fewer restrictions than the (even cheaper) Amazon Glacier. I plan to continue using Glacier and S3 because the performance has been great and (in theory, see below) the reliability is unmatched. But it’s nice to have alternative services to switch to or use in parallel.

Arq now supports backups via SFTP, which is something I’ve wanted a backup app to do for as long as I can remember. I have an account with DreamHost, and they offer 50 GB of SFTP space for personal backups. This is a convenient, free space I can use for my most important backups. It avoids the delays and expense of restoring from Glacier. DreamHost Personal Backup is great as a secondary backup target, but it is not itself backed up so you should still use AWS or another service for your primary.

You can also use SFTP to make a local backup or archive on a NAS or other Mac that you have an account on.

Aside from the new storage options, the other big new feature is that you can now have multiple backup targets. This lets you have multiple backups going to different cloud services. You can also spread your files across multiple targets, e.g. if you want your Documents folder to have a different backup schedule than your Aperture or iTunes library. Each target can also have a separate budget, which lets you keep a longer history for certain folders. You can also pause a backup target (by setting its schedule to manual) in order to give priority to other targets (since Arq seems to only back up to one target at a time). Alas, the targets cannot be renamed or reordered, and you cannot copy file exclusion patterns from one target to another.

I’ve been seriously using Arq since version 2, and version 3 was one of my favorite apps. Version 4 so far seems to be better still. The app itself has been reliable (rarely crashing) and has not hogged the CPU (like other backup apps I’ve tried). However, I have had some problems with the reliability of Arq’s backups. It’s not clear whether this is due to a bug in Arq itself or problems with the cloud storage provider (AWS).

Twice in the last six months, I’ve found that backup snapshots (“commits”) older than a certain date had disappeared. Arq stores the commits in a linked list. If a commit object is lost, Arq, naturally, will no longer be able to find the trees and blobs in that commit. But it will also lose the link to the parent commit (previous backup snapshot) and, thus, all of the previous snapshots. In theory, much of the data is still on the server, but it’s no longer in an accessible form, and Arq will garbage collect it when it enforces the budget.

The developer, of course, takes this sort of thing very seriously. The first time I noticed missing backup snapshots, he told me that several other customers had reported the same problem around the same time. It seemed as though the problem was that Amazon S3 was reporting objects as missing (when doing the equivalent of an ls) even though it could successfully fetch their data when asked (the equivalent of stat or cat). So when Arq periodically verified its backups, it would delete objects related to the “missing” ones unnecessarily. An update to Arq was soon released to fix this.

At the time, I was using S3 Reduced Redundancy Storage for my backups. RRS storage is cheaper than regular S3 but offers only 99.99% durability compared with 99.999999999%. Since I have other backups besides Arq, I did not think I needed to pay for those extra 9’s. I thought it was acceptable to lose 1 in 10,000 objects, even though I have many more files than that. What I failed to appreciate was that the lost object might not be a file. It could instead be a commit object. In that case, losing that one object effectively means losing hundreds of thousands or even millions of other objects. These days, I think there is little reason to use RRS with Arq. You can store your backup data in Glacier, which is much cheaper than RRS yet has the same durability as S3. The backup metadata is stored in S3.

It’s not clear whether RRS was at fault, but I switched away from it just to be safe. Then, a few months later, I noticed that more old backups had disappeared. This time, other Arq users had apparently not encountered the same problem. It’s hard to know, though, because it is not obvious in the user interface that backups have been lost. You only notice it when you click a disclosure triangle to see the list of snapshots and see that the list is shorter than expected.

I never actually lost any current backups, but I was intending to use Arq as a historical archive as well, because sometimes I need access to old versions of files. In that sense, the cloud backup is much more than a backup; I do not have master local copies of all the versions.

It’s obviously very troubling to have a backup app or cloud storage provider lose my backups. But I continue to use and recommend Arq for several reasons. First, I have confidence in the product’s basic design and in Stefan, its developer. Second, Arq 4’s support for multiple backup targets offers a variety of ways to mitigate the problems caused by lost objects. Third, I have tried just about every backup product I could find over the years, and I have yet to find one that’s better. The closer I look, the more flaws and design limitations become visible. For example, Backblaze is highly regarded, yet it silently deletes backups of external drives that haven’t been connected in a while.

Backups are important enough that I make local ones (using SuperDuper and DropDMG) even though that’s more work than just relying on the cloud. I want to have copies of my data in my physical possession. There are also obvious benefits to making cloud backups, e.g. using Arq, so I do that as well. What I have more recently come to realize is that cloud backups are important enough that I shouldn’t rely on just one provider. Before Arq I used CrashPlan, and it, too, occasionally lost my data. The lesson here is that there is no perfect cloud provider. I should plan for failure and use multiple good providers. I am now using CrashPlan alongside Arq.

The second lesson I’m learning is that I value access to old versions of files but that there are few, if any, backup products that can provide this over the long term. The answer, I believe, is to structure the data so that the backup, rather than the backup history, contains the old versions. In other words, put the versions in band, where possible. For example, a single backup snapshot of a Git repository includes the complete, checksummed history for those files. I don’t need last year’s backup if I committed the file to Git last year and I have yesterday’s backup. Of course, my source code has been in version control from the beginning. But I am now using version control to track other types of files such as notes, recipes, my 1Password database, and my calendar and address book. This lets a newly created cloud backup contain versions from years ago.

The same logic holds for verifying the backup. It’s nice if the backup software can do this, but if your data has in band checksums you can verify the restored files independently. You can also verify your working files so that you can identify damage and know when you need to restore a clean copy from backup. You can verify files in Git using git-fsck. For files not in Git, I use EagleFiler and IntegrityChecker.

Unintentionally Hidden Dot-files

Rob Pike (via Hoà V. DINH):

Long ago, as the design of the Unix file system was being worked out, the entries . and .. appeared, to make navigation easier. […] When one typed ls, however, these files appeared, so either Ken or Dennis added a simple test to the program. It was in assembler then, but the code in question was equivalent to something like this:

if (name[0] == '.') continue;


I’m pretty sure the concept of a hidden file was an unintended consequence. It was certainly a mistake.

Add Security Access Groups for Accessibility APIs

Craig Hockenberry:

Disallowing an app from controlling another is a good idea (I sure don’t want an app selecting menu items for me!) and the App Sandbox Design Guide’s statements about accessibility make complete sense.

That being said, automatically moving windows around on my screen is something that helps me do my job and something I can explicitly control using Accessibility in System Preferences. As a user, this type of “controlling my app” means “making my work easier”.

He wants part of System Events’ AppleScript dictionary to have an access group so that it can be a scripting target. This would make it possible to target System Events from a sandboxed application using the entitlement rather than the broader one that’s likely to be rejected by App Review.

Unfortunately, access groups are not yet widely supported by Mac OS X’s built-in applications or by third-party ones. One app that does support access groups is iTunes, whose .sdef file is, curiously, not stored inside

BBEdit 10.5.9

There are two changes in this update that I really like:

When generating the Markers popup, leading whitespace from the marker name is now used to indent the menu item, so that type-to-select works correctly in the menu.


Control-Tab actually does not work for its intended purpose, which was to flip the sense of “Auto-Expand Tabs” on the fly when entering a tab character. To work around this, Option-Tab has been defined so that it always enters a literal Tab character; thus, if “Auto-Expand Tabs” is turned on, use Option-Tab to enter a tab character instead of spaces.

Read the release notes for each BBEdit update to see just how much behind-the-scenes work it takes to keep a top Mac app up-to-date and polished.

Misconceptions About Changing Email Addresses

Joe Kissell:

Having an email address in a domain you control and hosting your email at a provider you like can solve numerous problems and perhaps even improve your image.

Apple OpenSSL Verification Surprises

Hynek Schlawack (via Kyle Sluder):

Apple ships a patched version of OpenSSL with OS X. If no precautions are taken, their changes rob you of the power to choose your trusted CAs, and break the semantics of a callback that can be used for custom checks and verifications in client software.


The reason for this unexpected behavior is that Apple is trying to be helpful. Certificate validation and especially trust databases are a hassle and OpenSSL’s handling of them is rather user-hostile. So Apple patched a Trust Evaluation Agent (TEA) into their OpenSSL. It gives failed verifications a second chance using the system keyring as trust store.


Nick Heer:

Apple has rebranded iOS in the Car as the much more syllable-friendly “CarPlay”, and launched it in Geneva. This new version has a much different interface than that shown at WWDC, as can be seen on the CarPlay page on Apple’s website. Also of note: there are third-party apps which support CarPlay; it isn’t known yet whether third-party developers require a special agreement to enable CarPlay support.

This is too bad since the two apps I use most often in the car are both from third-party developers: Google Maps (App Store) and Downcast (App Store).

Anand Lal Shimpi:

Interacting with CarPlay can be done via buttons/knobs or directly by touch (if available). It’s important to note that CarPlay likely won’t replace the need for checking an expensive box on your car’s option list. The OEM still needs to provide the underlying hardware/interface, CarPlay simply leverages the display and communicates over Apple’s Lightning cable.

Marco Arment:

It also has the potential to fizzle out because Apple demands more control than their partners are comfortable with, like iAd, or their interests conflict too much with the partners’ interests without enough upside to the partners, like iTunes TV rentals.

John Gruber:

The risk seems clear: Apple isn’t building the hardware in the cars. Color me skeptical that this is going to work smoothly. Also, no third-party app support — yet. UPDATE: Actually, there are a handful of third-party apps — Beats Radio, iHeartRadio, Spotify, and Stitcher — but those are hand-picked partners. What I’m saying is there’s no way yet for any app in the App Store to present a CarPlay-specific interface.

Christian Zibreg:

Volvo confirmed that CarPlay’s connection and video mirroring functionality is based on a streaming H.264 video feed, prompting watchers to speculate that the feature is based on AirPlay, an Apple-designed media streaming technology.

In a rather surprising find earlier today, N4BB was able to confirm that CarPlay runs on QNX, an operating system the embattled Canadian smartphone maker BlackBerry acquired Harman International Industries back in 2010…


For all we know, CarPlay might just be an extension to the existing car entertainment systems, using something like VNC (or hopefully something more optimized for the use-case) to show the iOS screen on the existing infrastructure.

In that case, the car is running QNX because it has always been running QNX and because the car must be useable even if the user decides to switch to a different platform or loses their device.

In that scenario, saying CarPlay is running QNX is similar to saying your Thunderbolt display is running OSX when it’s connected to your Mac running OSX, or using an even closer analogy, similar to saying that your OS X machine is running linux because you’re using SSH connected to a linux box (or any other kind of remote desktop)

Andrew Cunningham:

Previous reports had suggested that CarPlay would communicate with displays wirelessly using some version of Apple’s AirPlay protocol, but according to today’s release, the feature will only work with Lightning-equipped iPhones.

Damon Lavrinc:

But how does CarPlay stack up to the current crop of infotainment systems? Here’s a breakdown of how Apple’s first real attempt at dashboard dominance competes with the best from the established automakers.

iWant: Faster iOS Device Syncing

Kirk McElhearn:

I’m sitting at my desk right now, waiting to sync my iPhone. I think I started about twenty minutes ago, and all I’m doing is adding a bunch of audio files I want to listen to when I go out for a walk. Which I hope to do before the sun goes down…

Back in the day, this process was much faster than it is now. I don’t know exactly what’s changed since iOS 7, but I see this all the time, on all my iOS devices: iPhone, iPod touch, iPad Air.

This has been my experience as well. It is somewhat faster now that I’ve turned off photo syncing in favor of FlickStackr (App Store) and podcast syncing in favor of Downcast (App Store). This is curious since the photos, especially, didn’t change very much but always seemed to require an inordinate amount of time to sync. The nice thing about FlickStackr is that it lets me zoom in more than the regular Photos app. The photos also seem to have fewer JPEG compression artifacts. Unfortunately, I have to remember to tell it to load the photos while I have a Wi-Fi connection.

How I Learned to Stop Worrying and Love Cocoa Auto Layout

Ole Begemann:

If you find yourself in a situation that is difficult to solve with Auto Layout, just don’t use it for that particular view. You can freely mix the constraint-based layout with manual layout code, even within the same view hierarchy.


You can think of Auto Layout as just an additional step that runs automatically in your view’s layoutSubviews method. The Auto Layout algorith performs some magic, at the end of which your subviews’ frames are set correctly according to the layout constraints. When that step is done, the Auto Layout engine halts until a relayout is required (for example, because the parent view size changes or a constraint gets added). What you do to your subviews’ frames after Auto Layout has done its job, doesn’t matter.