Swift Testing in Xcode 16
Stuart Montgomery (September 2023):
I’m excited to announce a new open source project exploring improvements to the testing experience for Swift.
I’m pleased to announce that the Swift project has accepted a vision document for A New Direction for Testing in Swift.
The vision:
It should gracefully coexist with projects that use XCTest or other testing libraries and allow incremental adoption so that users can transition at their own pace.
[…]
When a test fails, it should collect and show as much relevant information as reasonably possible, especially since it may not reproduce reliably.
[…]
There must be a way to carefully store per-test data, to ensure it is isolated to a single test and initialized deterministically to avoid unexpected dependencies or failures.
[…]
Many tests consist of a template with minor variations—for example, invoking a function multiple times with different arguments each time and validating the result of each invocation. A testing library should make this pattern easy to apply, and include detailed reporting so a failure during a single argument is represented clearly.
[…]
Depending on the library, these APIs may be called “assertions”, “expectations”, “checks”, “requirements”, “matchers“, or other names. In this document we refer to them as expectations.
What XCTest called “assertions” are now called “expectations,” what XCTest called “expectations” are now called “confirmations,” and what XCTest called “messages” are now called “comments.” As with SwiftData, it’s not clear to me that these renamings are accomplishing much.
Some specifics:
@Testand@Suiteattached macros: These declare test functions and suite types, respectively.- Traits: Values passed to
@Testor@Suitewhich customize the behavior of test functions or suite types.- Expectations
#expectand#require: expression macros which validate expected conditions and report failures.
I had hoped that Swift’s runtime features would be enhanced to the point where XCTest-style test discovery would be possible. Instead, it’s being done through macros.
Likewise, the trait stuff appears to be done through special-purpose macros rather than a general way of attaching metadata to functions.
I like the distinction between #require, which halts execution of the test, and #expect, which allows it to continue running and report more failures. #require is also used for unwrapping.
In existing test solutions available to Swift developers, there is limited diagnostic information available for a failed expectation such as
assert(2 < 1). The expression is reduced at runtime to a simple boolean value with no context (such as the original source code) available to include in a test’s output.[…]
We can also extract the components of an expression like
a.contains(b)and, on failure, report the value ofaandb.
There are two different things going on here. First, XCTest had a large number of macros with verbose names for different kinds of assertions (and object vs. primitive types). It has always been unergonomic, even compared with predecessors such as JUnit and its Objective-C ports. Swift Testing spells almost all of these as simply #expect, which is great. But it’s not clear to me why it took a decade to make this sort of ergonomic improvement. I’ve long been using very short names like eq() and overloads to achieve much the same effect. This was not really possible with Objective-C (without polluting the namespace) because you need macros (which are top-level) in order to capture the source location. But Swift can do this with methods on the test class. It can also use autoclosures to avoid evaluating the failure message on success.
The second cool thing is that, with XCTest, any values that were not passed as arguments to the assertion would be lost at runtime. To get detailed failure information you had to write extra code. Swift Testing’s #expect macro can look at the structure of the expression to extract these values (as well as how they were being used) automatically. This is a killer feature, which I first saw in Python nearly 20 years ago via pytest and once used to test my Objective-C code, too. (Python doesn’t have macros, but import hooks can modify the parsed AST before compilation.)
I’m not sure how to square the principle of scalability with the heavy use of macros and their effect on compilation time. There are also issues with runtime performance, though those seem more easily solveable.
Swift Testing ships with Xcode 16 and has two WWDC videos and a repo.
My current spike: Implementing a BDD DSL on top of Swift Testing using resultbuilders.
[…]
Already filed my first issue: The
Teststruct needs a public initializer.
One of the downsides of having a public initializer for
Testis that it encourages people to use it. But since it doesn’t produce an instance ofTestthat’s visible to Swift Testing’s infrastructural layer, there’s no actual way to run it.
It’s a continual worry with Swift and Swift-based APIs that third-party developers will get locked out.
See also:
- Swifter and Swifty: Mastering the Swift Testing Framework
- DTO Testing with the New Swift Testing System
- Developer guide on Swift Testing for iOS
- Hello Swift Testing, Goodbye XCTest
Previously:
- XCTest in Xcode 16
- Xcode 16 Announced
- Slow Swift Macro Compilation
- XCTestCase / XCTestExpectation / measureBlock()
- Let Your Swift XCTest Methods Throw
Update (2024-07-18): See also: Jonathan Grynspan (Mastodon). I also want to note this thread, which discusses explicitly using SourceLocation when writing helper functions.
Update (2024-09-18): Antoine van der Lee:
We’ve only seen the basics today, but I’ll update this article in the upcoming weeks with references to more in-depth articles on each macro, test traits, test organization, and more.
Update (2024-10-17): Donny Wals:
So whenever that confirmation closure returns, Swift Testing expects that we have confirmed all of our confirmations. In a traditional completion handler-based setup, this won’t be the case because you’re not awaiting anything because you don’t have anything to await.
This was quite tricky to figure out.
[…]
So what I’ve really found is that the best way to test your completion handler-based APIs is to use continuations.
You can use a continuation to wrap your call to the completion handler-based API and then in the completion handler, do all of your assertions and resume your continuation. This will then resume your test and it will complete your test.
Update (2024-11-01): Donny Wals:
I think that parameterized tests are probably the feature of Swift testing that I am most excited about.
A lot of the syntax changes around Swift testing are very nice but they don’t really give me that much new power. Parameterized testing on the other hand are a superpower.
Update (2024-11-05): Majid Jabrayilov:
The most powerful feature of the Swift Testing framework is the trait system. Traits allow us to annotate a test or test suite to customize its behavior.
