Unit Testing
Wil Shipley’s heart is in the right place. It’s obvious that he cares about quality, and his experience and common sense rightly make him skeptical of process. But I think he’s absolutely wrong about unit testing. In brief, I think he misunderstands what unit testing is (or should be) and what it’s useful for. He shoots down a stupid way of doing testing, throws the baby out with the bathwater, and concludes that it’s better to write good code in the first place, do manual testing (“Try odd things. Whack keys.”), and rely on beta testers.
I don’t deny that this approach has worked for him. He’s written a lot of code, people have generally been happy with the products, and they’re not known for being buggy. But my hunch is that Wil’s teams have gotten good results because they’re unusually smart, experienced, and hard-working. They’ve succeeded in spite of their testing philosophy. Wil makes a great case that manual testing and defensive programming are necessary, but I think most programmers would be more effective and efficient if they combined these with extensive automated unit testing.
Now, I’ll be the first to say that by-the-book XP/TDD goes overboard. If you need a new class or method, I think it’s a waste of time to write a test that simply checks for the existence of the class or method, then write a stub, then run the tests, and then then flip back and write the real tests. Perhaps that example is just for pedagogical purposes and one isn’t meant to do that in production code.
I also think it’s a waste of time to write tests for everything. Some parts of the code are so simple that they don’t need tests, though this applies less often than you might think. Other times, it would take so much work to write a proper test that it’s better to punt and check that item manually. (I’m thinking of user interface details that you’d immediately notice when running the application, as well as more elaborate situations/interactions. In the latter case, write the test down in English and make sure that you really do it manually before shipping.) Again, though, this applies less often than you might think.
My overall point is that time is limited, so you should use it wisely. And this is why extensive unit testing is a big win. Yes, it’s not possible for your tests to cover all the pathways through the code, with all the possible inputs. And even if they could, it probably wouldn’t be a good idea to spend your time writing tests to do that. This does not mean, however, that you should reject unit testing as impractical and try to test everything manually. That’s not a good use of time, either. In brief, proper unit testing saves time and improves quality because:
- Most of your testing can be automated with little “extra” work, and the computer is both faster and more reliable than you at executing the tests. You’ll spend (maybe) a little more time writing code, but much less time clicking around in your application and using the debugger.
- If you think it’s hard to get good coverage writing unit tests, imagine how much harder it would be if you’re just doing manual integrated system tests.
- Tests help you catch bugs earlier and isolate them more easily, so it’s less expensive to fix them.
- Tests make great documentation for other programmers (or yourself next month).
- In general, code that’s easier to test is better designed. If it’s really hard to get your code into a test, there’s probably something wrong with it. Writing tests helps you get the design right, and that saves you time.
- When (not if) your software has to evolve and change, having tests will help guard against regressions. You don’t want to waste time tracking down bugs that you introduced, nor do you want to let your code ossify because you’re afraid to risk breaking anything.
A lot of people seem to think that automated testing is only for frameworks, or for tools without graphical user interfaces and interaction, or for academics, or for software that’s directly responsible for people’s lives. Not true. I think unit tests are useful in writing Mac applications, and even for testing large chunks of the user interface (i.e. checking that a table is showing the right rows in the right order, that clicking a button has the right effect, that a certain sequence of actions causes a view to refresh in the right way, etc.).
That said, your tests will not cover everything, or even close to everything. GUI software has many sources of input—all the different user interface items, in addition to the disk, the network, etc. It also has complex interactions with huge chunks of operating system and library code that you didn’t write and don’t have access to.
So, yes, you shouldn’t believe that because you have unit tests you don’t have to do manual integration tests. I doubt anyone is in danger of thinking that. Wil says:
Testing is hugely important. Much too important to trust to machines. Test your program with actual users who have actual data, and you’ll get actual results.
And I doubt anyone would disagree with that. However, testing is also too important to trust to humans. This is one reason that you need automated unit tests. The other is that it’s not all about how many bugs users will find in the binary that you ship. It also matters how long it took you to create that binary, and how easily you’ll be able to develop the next version. People who like automated testing find that it helps them write better code more quickly. I believe it’s at least as beneficial for small teams.
Don’t miss the comments from Chris Hanson, Marcel Weiher, and others.
5 Comments RSS · Twitter
Agreed. Test-driven development (in the non-anal-retentive sense: a suite of tests for an array class is probably more useful than a test for the existence of a method, for example...) is excellent; it means avoiding a lot of debugging pain afterwards. It's improved my productivity quite a bit.
And, to be honest, Omni's software was never bug-free or even close, in my experience. I had many crashes with older versions of OmniWeb (pre-WebKit). I have also found that automated unit tests save a lot of time in testing that your testers could otherwise use to help you improve the product...
Other people respond:
Bill Bumgarner
Chris Hanson
Joe Heck
Larry
Gus Mueller
Jonathan Rentzsch
Hey, I detect a bit of a straw-man argument with regard to test-driven development (TDD). I appreciate your allowance that perhaps it isn't as bad as you allege, but with a followup comment chiming in, I wanted to address this.
Yes, TDD writes tests first, including tests for functionality that doesn't exist yet. But that's not some sort of pedantic exercise. It's analogous to a mountain climber who makes it a practice to keep three appendages touching the surface at all times. Of course that climber will do some maneuvers that involve more than one appendage moving at a time, but the point is to define 'normal' and 'safe' and keep to that in the normal case.
Doing full-on TDD with refactoring all the time buys you very well-factored code that's easy to test further and easy to change. Look at Gus's post, linked in the previous comment. The place he got to is the place consistent TDD will keep you.
Ryan: I like TDD, and I use it. I just think that some of the examples used to illustrate it go too far. I'll write the tests for a method before writing the method. But I think it's a waste of time to write a trivial failing test and then watch it fail to prove that I need to write the method in the first place. Likewise with doing the minimum amount necessary to make a test pass (i.e. the test failed because there was no foo() method, so now let's write a foo() that always returns 0). I appreciate that in theory TDD lets you work in steps as small as you like, but I don't think that level of pedantry is helpful in real development.
I'm not an advocate of TDD, but I am an advocate of automated unit testing. You don't have to do the first to reap the benefits of the second! Built-in self tests may save some time during development, but save huge amounts of time down the road.
