Exploding Software-Engineering Myths
Janie Chang (in 2009, via Ilja A. Iwas):
The study and its results were published in a paper entitled Realizing quality improvement through test driven development: results and experiences of four industrial teams, by Nagappan and research colleagues E. Michael Maximilien of the IBM Almaden Research Center; Thirumalesh Bhat, principal software-development lead at Microsoft; and Laurie Williams of North Carolina State University. What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer.
“Over a development cycle of 12 months, 35 percent is another four months, which is huge,” Nagappan says. “However, the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better.
It would be nice if there were a way to measure how long it would take the non-TDD teams to reach the same defect density. Even then, it is not an apples-to-apples comparison because the final code differs. Presumably, code with tests is easier to maintain and extend going forward.
3 Comments RSS · Twitter
Importantly, these are case studies, so statistical results ("60 to 90 percent better in terms of defect density") are pretty much useless, other than as anecdotal evidence in favor of doing some real research.
Related: https://leanpub.com/leprechauns
@LKM This is true, but it’s not clear to me what “real research” would mean. I would be more inclined to trust the results of a case study like this than, say, two groups of students given a toy problem to solve over a short period of time.
@Michael: Yes, I can't think of a way to do a controlled experiment. Not just because it would be difficult to come up with an appropriate problem for the different groups to solve, but also because it would be difficult to control for the development process (e.g. how would you teach the TDD group how to do TDD properly? That's usually not something you learn in an afternoon), and because you couldn't blind the experiment at all ("we're not telling you if you're doing TDD or not, we're just measuring the bugs you produce" :-).
However, I think it might be possible to do population studies. There's plenty of open source projects out there that use a lot of different development processes, and I think it would be possible to control for most of the factors that would influence outcomes.
