Essential and Accidental Complexity in Programming
Dan Luu (tweet, Hacker News):
In the classic 1986 essay, No Silver Bullet, Fred Brooks argued that there is, in some sense, not that much that can be done to improve programmer productivity. His line of reasoning is that programming tasks contain a core of essential/conceptual complexity that’s fundamentally not amenable to attack by any potential advances in technology (such as languages or tooling). He then uses an Ahmdahl’s law argument, saying that because 1/X of complexity is essential, it’s impossible to ever get more than a factor of X improvement via technological improvements.
[…]
In 1986, there would have been no comparable language, but more importantly, I wouldn’t have been able to trivially find, download, and compile the appropriate libraries and would’ve had to write all of the parsing code by hand, turning a task that took a few minutes into a task that I’d be lucky to get done in an hour. Also, if I didn’t know how to use the library or that I could use a library, I could easily find out how I should solve the problem on StackOverflow, which would massively reduce accidental complexity. Needless to say, there was no real equivalent to Googling for StackOverflow solutions in 1986.
Moreover, even today, this task, a pretty standard programmer devops/SRE task, after at least an order of magnitude speedup over the analogous task in 1986, is still nearly entirely accidental complexity.
[…]
Brooks explicitly dismisses increased computational power as something that will not improve productivity (“Well, how many MIPS can one use fruitfully?”, more on this later), but both storage and CPU power (not to mention network speed and RAM) were sources of accidental complexity so large that they bounded the space of problems Brooks was able to conceive of.
pron:
Our current state is a big disappointment compared to where most programmers in the 80s believed we would be by now, and very much in line with Brooks’s pouring of cold water (perhaps with the single exception of the internet).
My perception is that the total productivity boost over the past three decades is less than one order-of-magnitude (Brooks was overly careful to predict no 10x boost due to one improvement in language design or programming methodology within one decade), and almost all of it comes from improvements in hardware and the online availability of free libraries (Brooks’s “Buy vs Build”, which he considered promising) and information -- not from changes in programming methodology or language design (although garbage collection and automated unit-tests have certainly helped, too). The article also mixes hardware improvements and their relationship to languages, but we knew that was going to happen back then, and I think it’s been factored well into Brooks’s prediction. Moreover, my perception is that we’re in a period of diminishing returns from languages, and that improvements to productivity Fortran and C had over Assembly are significantly greater than the gains since.
[…]
I would also add that if you want to compare the ratio of essential and accidental complexity in line with Brooks’s prescient analysis, you should compare the difficulty of designing a system in an accidental-complexity-free specification language like TLA+ to the difficulty of implementing it in a programming language from scratch. I find the claim that this ratio has improved by even one order of magnitude, let alone several, to be dubious.
1 Comment RSS · Twitter
Luu's argument is specious. The vast improvements he identifies are in computer scutwork - the things you have to do in order to be able to do the thing you want to do. Scutwork is much less onerous now than it was, thanks to fast networking, the internet, easily accessed scripting languages, and fast computers with plentiful storage and memory.
But Brooks is clearly talking not about computer scutwork, not about writing scripts, but about coding new programs. And that, as ever, has to be done by human beings thinking out the problem, making a plan, and then writing the code one line at a time.
