Josh Aas (Firefox developer):
Secondly, Macworld put Firefox 1.0.6 ahead of Safari, which is almost
as ridiculous as putting Camino behind Opera. See, I’m not biased
towards Mozilla browsers! Safari 2.0 is simply a better browser than
Firefox 1.0.6. Its [sic] not even close. The story is a little different with
Firefox 1.5b1, but that isn’t what they are reviewing.
I agree that the
Macworld rankings are odd. I think I would order them Safari,
OmniWeb, Camino, Firefox, Opera. I couldn’t care less about Firefox
themes or extensions, and I deduct huge numbers of points from it and
Opera because they don’t look or feel like Mac apps.
Wil Shipley’s heart is in the right place. It’s obvious that he
cares about quality, and his experience and common sense rightly make
him skeptical of process. But I think he’s absolutely wrong about unit
testing. In brief, I think he misunderstands what unit testing is (or
should be) and what it’s useful for. He shoots down a stupid way of
doing testing, throws the baby out with the bathwater, and concludes
that it’s better to write good code in the first place, do manual
testing (“Try odd things. Whack keys.”), and rely on beta testers.
I don’t deny that this approach has worked for him. He’s written
a lot of code, people have generally been happy with the products,
and they’re not known for being buggy. But my hunch is that Wil’s
teams have gotten good results because they’re unusually smart,
experienced, and hard-working. They’ve succeeded in spite of their
testing philosophy. Wil makes a great case that manual testing
and defensive programming are necessary, but I think most
programmers would be more effective and efficient if they combined
these with extensive automated unit testing.
Now, I’ll be the first to say that by-the-book XP/TDD goes overboard.
If you need a new class or method, I think it’s a waste of time to
write a test that simply checks for the existence of the class or
method, then write a stub, then run the tests, and then then flip back
and write the real tests. Perhaps that example is just for
pedagogical purposes and one isn’t meant to do that in production code.
I also think it’s a waste of time to write tests for
everything. Some parts of the code are so simple that they
don’t need tests, though this applies less often than you might
think. Other times, it would take so much work to write a proper
test that it’s better to punt and check that item manually. (I’m
thinking of user interface details that you’d immediately notice
when running the application, as well as more elaborate
situations/interactions. In the latter case, write the test down in
English and make sure that you really do it manually before
shipping.) Again, though, this applies less often than you might
think.
My overall point is that time is limited, so you should use it wisely. And this
is why extensive unit testing is a big win. Yes, it’s not possible for
your tests to cover all the pathways through the code, with all the
possible inputs. And even if they could, it probably wouldn’t be a good
idea to spend your time writing tests to do that. This does
not mean, however, that you should reject unit testing as
impractical and try to test everything manually. That’s not a good use
of time, either. In brief, proper unit testing saves time and improves
quality because:
- Most of your testing can be automated with little
“extra” work, and the computer is both faster and more reliable
than you at executing the tests. You’ll spend (maybe) a little
more time writing code, but much less time clicking around in
your application and using the debugger.
- If you think it’s hard to get good coverage writing unit tests,
imagine how much harder it would be if you’re just doing manual
integrated system tests.
- Tests help you catch bugs earlier and isolate them more easily,
so it’s less expensive to fix them.
- Tests make great documentation for other programmers (or
yourself next month).
- In general, code that’s easier to test is better designed. If
it’s really hard to get your code into a test, there’s probably
something wrong with it. Writing tests helps you get the design right, and that saves you time.
- When (not if) your software has to evolve and change,
having tests will help guard against regressions. You don’t want
to waste time tracking down bugs that you introduced, nor do
you want to let your code ossify because you’re afraid to risk
breaking anything.
A lot of people seem to think that automated testing is only for
frameworks, or for tools without graphical user interfaces and
interaction, or for academics, or for software that’s directly
responsible for people’s lives. Not true. I think unit tests are
useful in writing Mac applications, and even for testing large
chunks of the user interface (i.e. checking that a table is showing
the right rows in the right order, that clicking a button has the
right effect, that a certain sequence of actions causes a view to
refresh in the right way, etc.).
That said, your tests will not cover everything, or even close
to everything. GUI software has many sources of input—all the
different user interface items, in addition to the disk, the
network, etc. It also has complex interactions with huge
chunks of operating system and library code that you didn’t write
and don’t have access to.
So, yes, you shouldn’t believe that because you have unit tests
you don’t have to do manual integration tests. I doubt anyone
is in danger of thinking that. Wil says:
Testing is hugely important. Much too important to trust to
machines. Test your program with actual users who have actual data,
and you’ll get actual results.
And I doubt anyone would disagree with that. However, testing is
also too important to trust to humans. This is one reason
that you need automated unit tests. The other is that it’s not all
about how many bugs users will find in the binary that you ship. It
also matters how long it took you to create that binary, and how
easily you’ll be able to develop the next version. People who like
automated testing find that it helps them write better code more
quickly. I believe it’s at least as beneficial for small teams.
Don’t miss the comments from Chris Hanson,
Marcel Weiher, and others.
Jonathan Rentzsch:
I may have to face the fact that my days of living off a PowerBook are
over, perhaps for good. If so, I’ll probably go with big-honkin’
desktop machines and one small cheapie iBook for presentations and
conferences. I fear the Macintel in the mail is the first step down
that slippery slope.
Michael McCracken:
Do we really want to doom users of desktop apps to a lot of web-style
scrubbing around with the mouse to discover features?…Have we just been
conditioned to click and double-click around everywhere in case we can
edit the text? It seems to me that kind of behavior only makes sense
inside a text editing view, not just anywhere in a window.
Hopefully, Michael’s questions are rhetorical, but in case they’re not:
Apple, this design sucks. Editable fields should look editable.
