Proposal: XCTest Support for Swift Error Handling
Swift 2 introduced a new error handling mechanism that, for completeness, needs to be accommodated by our testing frameworks. Right now, to write tests that involve methods that may throw an error, a developer needs to incorporate significant boilerplate into their test. We should move this into the framework in several ways, so tests of code that interacts with Swift error handling is concise and intention-revealing.
[…]
One other downside of the above is that a failure caught this way will be reported as an expected failure, which would normally be a failure for which XCTest is explicitly testing via an assertion. This failure should ideally be treated as an unexpected failure, as it’s not one that’s anticipated in the execution of the test.
[…]
I propose several related solutions to this issue:
- Allow test methods to throw errors.
- Allow test assertion expressions to throw errors.
- Add an assertion for checking errors.
This proposal IMO addresses an important problem, but I am not convinced it actually solves it. #2 & #3 are basically sound API designs. It is a mystery to me why #3 “generated some debate” as this is a feature I already implement manually, but I can’t address unknown concerns. I can tell you I implement this, and nothing terrible has happened to me so far.
[…]
When the test function throws we no longer even find out a line number in the test case anymore, because the error is passed into XCTest and the information is lost. We have just the name of the test case (I assume; the proposal is silent on this issue, but that’s the only way I can think of to implement it), and some of my tests are pretty long. So, that makes it even harder to track down.
This sounds like a small thing but my test coverage is so thorough on mature projects that mostly what I turn up are heisenbugs that reproduce with 2% probability. So getting the report from the CI that has the most possible detail is critical, because if the report is not thorough enough for you to guess the bug, too bad, because that’s all the information you get and the bug is not reproducible.
For that reason, I won’t use #1.
Indeed, Swift and XCTest don’t currently work together very smoothly. I suspect that many developers have implemented their own versions of #2 and #3 already.
The lack of support for tracing the flow of errors is a more general problem that goes beyond testing. With exceptions, you would get a stack trace. In Objective-C, it’s easy to use macros to add tracing information to errors when propagating them. In Swift, I’ve seen this tip for tracking the source of errors that you create, but that isn’t most errors.
