Issues Adopting Swift Testing
I’m in the process of migrating from XCTest to Swift Testing. The basic stuff is pretty easy to translate, and in many cases I don’t even need to change the code inside my test methods. I’ve long been writing what Swift Testing calls expectations in terms of my own helper functions such as eq(), unwrap(), and fail() instead of XCTAssertEqualObjects() and friends, XCTUnwrap(), and XCTFail(). This is more concise, and I have many overloads for eq() that provide more precise reporting. I can simply reimplement these helpers atop #expect(), #require(), and Issue.record().
Although Swift Testing supports structs, you can also use classes as suites of tests. So I can still use a common base class that provides standard functionality for all my tests. Each project has a subclass with its own test data and project-specific functionality, and sometimes I have more layers for groups of tests that have lots of common setup. It seems like this architecture will continue to work, and it’s much nicer with Swift Testing because I can do the setup in init() instead of having to override func setUpWithError(). With XCTest, I also always had to override func tearDownWithError() to clear out property values because otherwise they would still exist while the next test ran. Swift Testing, as far as I know, does not have this problem, so I can rely on deinit to clear out the properties automatically and only override it when I need to do more than that.
All this said, I have run into some issues where the migration is not as straightforward, and I discuss some of these below. I’m new at this, and of course I’m hoping that some of this stuff I’m just wrong about and that there’s actually a better way.
Floating Point Equality
Swift Testing doesn’t seem to have the equivalent of XCTAssertEqual(_:_:accuracy:). This is easy enough to work around but a curious omission. Maybe it’s waiting on language support for approximate equality.
Implicitly Unwrapped Optionals
The vision document criticizes XCTest’s “frequent need for implicitly-unwrapped optional (IUO) properties in test subclasses.” Swift Testing makes this a bit better in that you are initializing your properties in init() instead of in setUpWithError(), so you don’t have to worry about them all having values before your code is run. But this doesn’t really fix the problem when you are using a hierarchy of test classes to set up common state. I generally call super.setUpWithError() in the first line because setup of the subclass will depend on properties that were set up by the base class. But this is backwards from Swift’s initializer rules, where the subclass needs to initialize its properties before calling super.init()—and without using any helper member functions. So it seems like I still need IUOs or perhaps lazy properties, though those don’t work well with error handling.
Confirmations Don’t Work Like Expectations
With XCTest, I could create multiple expectations in the same test, tell it to wait for all of them, and it would run the run loop and make sure they were all fulfilled, even though the order might be undefined. For example, I have tests to prevent memory leaks where I add a bunch of objects that I expect to be deallocated by the end of the test. Each gets a separate expectation, and when it goes away it calls the corresponding fulfill() method. I also have tests that make sure a particular block of code results in a notification being posted. Swift Testing’s confirmations seem designed to handle the latter type of test but not the former. As far as I can tell, you can only await one confirmation at a time, so you have to knew the order in which they will be confirmed. It also doesn’t run the run loop while waiting. I can probably rewrite these tests without using confirmations, just spinning the run loop until all the objects are deallocated.
Parallel Tests in the Same Process
With XCTest, parallel tests run in separate processes, which means that each test has its own copy of global state and doesn’t really need to know that it’s running in parallel, unless you’re writing to UserDefaults or something. With Swift Testing, everything runs in one process, and you can opt out certain tests or suites using the .serialized trait. However, if you’re doing this because of global state you would need to apply it everywhere, which would be easy to forget (in that case, I might as well use --no-parallel) and would make things much slower. I don’t want that.
My tests use several different kinds of global state. Some classes have flags to make them behave differently for testing purposes, e.g. to refresh right away instead of asynchronously. Some classes have failure handlers that will log errors in production but should report them as test failures (using the appropriate framework) when testing—and sometimes they get overridden in testing when we are actually testing the failure reporting. And I have some functionality like getting the current Date that I swap out so that I can feed the tests certain known values.
With XCTest, these could all be just regular static properties. It’s not worth using dependency injection and passing stuff down many levels when these things only need to change during testing. With Swift Testing, the answer seems to be to use task-local properties so that each test gets its own independent copy. But this is easier said than done.
The first issue is that I’m deploying my apps on versions of macOS that don’t support Swift Concurrency. It seems like I should be able to do something like:
#if TEST @TaskLocal @available(macOS 10.15, *) #endif static var flag = true
to use Swift Concurrency only during testing, but I still ran into some compiler errors. Xcode seems to not figure out that the test target has a higher deployment target and that when compiling for testing it’s set to use the Test scheme, which sets TEST. I’ve long had problems like this with Swift and conditional compilation. Maybe there’s a way to work around it, but I haven’t looked into it much yet because there’s a larger issue.
I don’t know how to concisely set the task-local values when setting up my tests. With XCTest, I would just assign to the properties in setUpWithError(). This is not possible with Swift Testing because @TaskLocal doesn’t work with assignment statements. It’s structured, so you are supposed to give it a closure:
Class.$flag.withValue(false) {
// Do test stuff
}
I want to do this once per test class, usually just inheriting the value from the base class. I don’t want to have to do it in every test method. With XCTest, there is a hook so I could do something like:
override func invokeTest() {
Class.$flag.withValue(false) {
super.invokeTest()
}
}
but as far as I know there is no way to do this with Swift Testing. Maybe it will eventually be possible through custom execution traits, though that seems a bit awkward for this purpose.
Previously:
