Wil Shipley’s heart is in the right place. It’s obvious that he cares about quality, and his experience and common sense rightly make him skeptical of process. But I think he’s absolutely wrong about unit testing. In brief, I think he misunderstands what unit testing is (or should be) and what it’s useful for. He shoots down a stupid way of doing testing, throws the baby out with the bathwater, and concludes that it’s better to write good code in the first place, do manual testing (“Try odd things. Whack keys.”), and rely on beta testers.
I don’t deny that this approach has worked for him. He’s written a lot of code, people have generally been happy with the products, and they’re not known for being buggy. But my hunch is that Wil’s teams have gotten good results because they’re unusually smart, experienced, and hard-working. They’ve succeeded in spite of their testing philosophy. Wil makes a great case that manual testing and defensive programming are necessary, but I think most programmers would be more effective and efficient if they combined these with extensive automated unit testing.
Now, I’ll be the first to say that by-the-book XP/TDD goes overboard. If you need a new class or method, I think it’s a waste of time to write a test that simply checks for the existence of the class or method, then write a stub, then run the tests, and then then flip back and write the real tests. Perhaps that example is just for pedagogical purposes and one isn’t meant to do that in production code.
I also think it’s a waste of time to write tests for everything. Some parts of the code are so simple that they don’t need tests, though this applies less often than you might think. Other times, it would take so much work to write a proper test that it’s better to punt and check that item manually. (I’m thinking of user interface details that you’d immediately notice when running the application, as well as more elaborate situations/interactions. In the latter case, write the test down in English and make sure that you really do it manually before shipping.) Again, though, this applies less often than you might think.
My overall point is that time is limited, so you should use it wisely. And this is why extensive unit testing is a big win. Yes, it’s not possible for your tests to cover all the pathways through the code, with all the possible inputs. And even if they could, it probably wouldn’t be a good idea to spend your time writing tests to do that. This does not mean, however, that you should reject unit testing as impractical and try to test everything manually. That’s not a good use of time, either. In brief, proper unit testing saves time and improves quality because:
- Most of your testing can be automated with little “extra” work, and the computer is both faster and more reliable than you at executing the tests. You’ll spend (maybe) a little more time writing code, but much less time clicking around in your application and using the debugger.
- If you think it’s hard to get good coverage writing unit tests, imagine how much harder it would be if you’re just doing manual integrated system tests.
- Tests help you catch bugs earlier and isolate them more easily, so it’s less expensive to fix them.
- Tests make great documentation for other programmers (or yourself next month).
- In general, code that’s easier to test is better designed. If it’s really hard to get your code into a test, there’s probably something wrong with it. Writing tests helps you get the design right, and that saves you time.
- When (not if) your software has to evolve and change, having tests will help guard against regressions. You don’t want to waste time tracking down bugs that you introduced, nor do you want to let your code ossify because you’re afraid to risk breaking anything.
A lot of people seem to think that automated testing is only for frameworks, or for tools without graphical user interfaces and interaction, or for academics, or for software that’s directly responsible for people’s lives. Not true. I think unit tests are useful in writing Mac applications, and even for testing large chunks of the user interface (i.e. checking that a table is showing the right rows in the right order, that clicking a button has the right effect, that a certain sequence of actions causes a view to refresh in the right way, etc.).
That said, your tests will not cover everything, or even close to everything. GUI software has many sources of input—all the different user interface items, in addition to the disk, the network, etc. It also has complex interactions with huge chunks of operating system and library code that you didn’t write and don’t have access to.
So, yes, you shouldn’t believe that because you have unit tests you don’t have to do manual integration tests. I doubt anyone is in danger of thinking that. Wil says:
Testing is hugely important. Much too important to trust to machines. Test your program with actual users who have actual data, and you’ll get actual results.
And I doubt anyone would disagree with that. However, testing is also too important to trust to humans. This is one reason that you need automated unit tests. The other is that it’s not all about how many bugs users will find in the binary that you ship. It also matters how long it took you to create that binary, and how easily you’ll be able to develop the next version. People who like automated testing find that it helps them write better code more quickly. I believe it’s at least as beneficial for small teams.