Using Xcode Bots
I’ve been using Xcode’s continuous integration feature for a few months now. Overall, I’m glad that I took the time to set it up. However, it’s been much more difficult to use than I expected, so I want to document some of the issues that I encountered, as well as the benefits:
It’s nicer than I expected to have tests run automatically, on a different machine. My tests don’t take that long to run, but it’s enough of an interruption that I would not naturally run them all for every commit. Tests can also run asynchronously. One day I wrote some code in a waiting room, while work was being done on my car. Sooner than expected, it was time to pack up and go. I pushed my changes to the server, and by the time I got back to the office all the tests had run.
While working on one app, I am not good at remembering to run tests for other apps. In part, this is because Xcode doesn’t like to have two projects open that each reference the same subproject. And so sometimes I would be working on a shared framework, cause a minor breakage in another app, and not realize it until days or weeks later when I was working on that app. Bots catch this type of bug (or failure to compile) right away.
My apps still have 32-bit versions, and bots make it much easier to test both architectures. When manually running tests within Xcode, you have to pick 32-bit or 64-bit. I had been using a shell script to test all the architectures, but that had the disadvantage of not showing the results within Xcode. And if I made the script do a clean build, which was frequently necessary, Xcode would temporarily lose its index information, which affected the syntax coloring and navigation. This is not a problem with bots, which use a separate working directory.
Likewise, it was an extra step to run the static analyzer, so I tended to only do this before a release. With bots it’s easy to do this every time.
The same is true for code coverage.
Getting my code to run in the bots environment found some latent problems in my projects: some files that weren’t using proper relative references, some tests that depended on file metadata that wasn’t stored in Git, some tests that were too closely tied to the environment of my main development Mac (documents, apps, preferences, user spelling dictionary), some target dependencies that were missing. Some shell scripts didn’t have proper quoting, which I never detected because my build folder had no spaces in its path—but the bot’s did.
I also had to copy various items into the keychain on the Xcode Server Mac. Many keychains seem to work with Xcode, but for bots your certificates (e.g. for Developer ID or the Mac App Store) must be in System keychain.
On the other hand, some tests just don’t plain work in bots. Some keychain and address book tests would fail or hang, eventually causing the integration to abort with no results or errors. I now exclude these tests when the code is running as user _xcsbuildd.
Sometimes my test code would crash, seemingly due to an OS bug, but only when running in the bot. If a test crashes there is no way to view the crash log in Xcode. You have to download an archive of all the logs and then extract it.
Oddly, I also found two cases where the static analyzer found legitimate problems (in generated yacc code) when running on the bot that it did not find when running from Xcode. I have not found an explanation for that. The versions of Xcode and Mac OS X were identical.
I was never able to get bots to send me status e-mails from Mac OS X Server. I tried both the built-in mail server as well as giving it my credentials for various SMTP servers. I read lots of blog and forum posts and tried all kinds of remedies but was never able to figure out. It never even gave me any useful error messages. I ended up writing a trigger script for the bot to run that POSTs the environment to my Web server. Then I have another script there that formats the information into an e-mail. This works, but the e-mails don’t look as nice or have as much information as the built-in Apple ones are supposed to. On the other hand, if there’s a failure I’ll probably have to go read the details in Xcode, anyway. So I just have simple e-mails that report the numbers of tests, warnings, errors, and failures, and I have Mail rules to color the messages green or red.
I have successfully used Git with SSH keys, but this has never worked for me with Xcode bots. So instead I am using password authentication.
I have 1–2 bots for each product, and each bot needs a full checkout of the Git repository. My repository is rather large because it includes all my products, my Web site, and each shipped .dmg file going back to 2002. Multiplying that out, it took a lot of disk space and a really long time for the initial checkout. I was originally going to have bots for each project (framework, app, or other target), but I settled on per-product to reduce the number of checkouts.
One of the reasons I have multiple bots per product is that each bot has to pick either one architecture or “All Architectures”. Swift code doesn’t compile for 32-bit, so it has to be in a different bot than my universal code.
The bots interface within Xcode seems pretty buggy. Viewing the commits tab usually crashes. Sometimes the latest integration shows in the Web interface but not in Xcode. Sometimes I have to click Load More in Xcode to see the latest integration, even though that is usually for loading older integrations. Sometimes no matter what I do it won’t show up in Xcode.
As with Xcode itself, bots sometimes get wedged with strange errors, and you have to clean the build folder before anything will work. There’s an option to have Xcode do this periodically, but I haven’t found that to be useful. Clean too often, and the builds are really slow. Clean less often, and you end up with spurious errors. I end up frequently doing manual cleans, by holding down the Option key so that the Integrate button changes to Clean Integrate.
Note that clean integrations do not do a fresh Git checkout from the server. However, every once in a while, all the bots seem to decide that it’s time to do a fresh checkout, and that’s really slow.
Sometimes a clean integration won’t fix a weird error, and the only remedy seems to be to restart the server Mac itself.
Here are some examples of weird errors that were fixed by cleaning or restarting:
Assertion: Directory not found for option '-F/Library/Developer/XcodeServer/Integrations/Caches/b24b1b6ee61325ff5108b238601fffb2/Source/Programming/DropDMG/DerivedData/Build/Products/Release' File: (null):(null)
(Early unexpected exit, operation never finished bootstrapping - no restart will be attempted)
Assertion: warning: /Library/Developer/XcodeServer/Integrations/Caches/525e758732df9b9483e63d7361008702/DerivedData/ModuleCache/1RAFTB1Y99NEJ/MJTFoundation-1W2RPA2S3CGSV.pcm: No such file or directory
Assertion: warning: /Library/Developer/XcodeServer/Integrations/Caches/525e758732df9b9483e63d7361008702/DerivedData/ModuleCache/1RAFTB1Y99NEJ/MJTFoundation-1W2RPA2S3CGSV.pcm: No object file for requested architecture
Assertion: warning: /Library/Developer/XcodeServer/Integrations/Caches/525e758732df9b9483e63d7361008702/DerivedData/ModuleCache/1RAFTB1Y99NEJ/Foundation-3M4TH6X8B5VB4.pcm: No object file for requested architecture
Assertion: warning: Could not resolve external type c:objc(cs)XCTestCase
Assertion: warning: Could not resolve external type c:objc(cs)MJTTempFolderManager
Assertion: warning: Could not resolve external type c:objc(cs)NSURL
Assertion: warning: Could not resolve external type c:objc(cs)NSString
Assertion: Running task was terminated because it produced no activity for too long.
warning: no debug symbols in executable (-arch x86_64)
Sometimes it seems to get stuck doing a Git checkout. There doesn’t seem be a way to make it cancel and retry, so I have to restart the server Mac to unwedge it.
Sometimes the server seems to stop checking for new commits. I then have to run the integrations manually or restart the server Mac.
Sometimes the server Mac spontaneously suspends (I see the Apple logo and progress bar.) and restarts itself while doing an integration.
In one case, no amount of cleaning or restarting would get the bot to use the proper configuration (either from the scheme or overriding it in the bot), and I had to delete and recreate the bot itself.
Sometimes bots try to run while the Mac is in Power Nap, and that’s usually a disaster: weird failures or errors that have nothing to do with any code that I recently changed.
Whenever you update Xcode, you have to reselect it in the Server app. With Xcode 7.2.1, this is impossible. Bots don’t seem to work with this version of Xcode at all. I went back to 7.2, but I don’t like the idea of bots using a different version of the compiler than what I’m using for development and production.
I chose Xcode’s continuous integration instead of something like Jenkins because, coming from the makers of Xcode, I thought it would be easier to set up and less prone to breakage. This may well be true, but it’s not been anywhere near as simple as I’d hoped.
See also: Xcode CI, The missing manual and the Swift CI that uses Jenkins.
Update (2016-02-12): The workaround to get Xcode 7.2.1 bots working (via @CocoaKevin):
sudo /Applications/Xcode.app/Contents/MacOS/Xcode
However, I’m now seeing a bug where Xcode reports the bot integration as still running long after it’s finished (and the trigger has run).
I got multiple suggestions that setting Xcode Server to send e-mails to a Gmail account helps, but it didn’t for me.
13 Comments RSS · Twitter
I would encourage you to check out BuddyBuild, a new service that aims to take all this pain out of the hands of developers. We've been using it at my day job for months, having heretofore operated with a Jenkins service and an Xcode Server-based setup we were never able to get to work properly, and it has been beautiful.
Your experience with Xcode server seems to be quite the same as some other developers who talked about it at some CocoaHeads presentations. And the conclusion is that it's a pain to set up but sometimes it works.
Also, the UI of Xcode server in Server.app is quite bad from a layout perspective IMHO. And finally, considering that moving to a newer version of Server.app had the interesting consequences of losing all the Xcode server settings, this is quite frustrating.
Thanks for sharing your experience with Xcode server. I think I'll stick around with Jenkins for a while. ;)
We also tried bots when they updated them this past summer. They might be fine for focused, single product environments, but when you're in the consulting business and have many concurrent projects running, it quickly becomes completely unreliable.
I'd recommend using something like CircleCI in the meantime.
For people using other CI products, how is the integration with Xcode? Can you click on a failing test or error and have it jump to the right part of the source file?
With respect to random SMTP failures, I set up OpenSMTPD on my dad's router box last year and now just point all the various devices that want to send status emails (e.g. our UPS management card and InterMapper) to it. Then I setup a Gmail account on my family's domain to act as the source for all the emails. I only listen on the local network and don't require authentication. Here's the entire configuration file:
listen on localhost
listen on 192.168.2.1
# If you edit the file, you have to run "smtpctl update table aliases"
table aliases file:/etc/mail/aliases
# sudo /usr/local/libexec/opensmtpd/makemap credentials
table credentials db:/usr/local/etc/mail/credentials.db
accept from any for domain "rileys.us" \
relay via "tls+auth://gmail@smtp.gmail.com:587" \
auth as hpn.sabi.net@rileys.us
/usr/local/etc/mail/credentials contains: gmail hpn.sabi.net@rileys.us:.
This has drastically improved reliability of status delivery. If the router box dies, I've got external monitoring for it and a spare router box with some printed instructions for my dad, but luckily have not needed it yet :-)
Thanks for sharing your experience. I wrote a bit high level blog post comparing Jenkins & Xcode Server https://medium.com/@hello_paja/self-hosted-ci-for-ios-mac-development-f273606ca767#.iyno63nfo
Is anyone able to get Xcode server to work with an Enterprise developer account?
https://forums.developer.apple.com/message/72040#72040
It seems this bug has been present since OSX Server 5.
I submitted a support ticket to Apple about getting an Enterprise developer account to work with Xcode server and the response was "you're experiencing an issue which has no known solution".
We used to have XCode Server working with an enterprise account. The current server build broke that. Now we use Team City for that :)
Had no issues with GIT + SSH keys in XCode Server tho. Easy to setup - just put the keys in your .ssh folder like normal, and it'll use it when you set them up.
I find XCS to be good and very easy to setup, but REALLY REALLY limited in function. TC is harder to setup (not a lot), but is so much more flexible. Just need the time to move stuff over to it now.
And don't Jenkins. Just. Don't.
[…] Lite, Script Debugger, SourceTree, Hopper, Base, Paw, Soulver, and OmniFocus. I plan to switch from Xcode bots to a third-party continuous integration […]
[…] the one hand, this is good news because Xcode’s continuous integration needs lots of work. On the other hand, it sounds like this is the end of the Buddybuild product that people loved. […]
