{"id":48539,"date":"2025-07-18T13:57:53","date_gmt":"2025-07-18T17:57:53","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=48539"},"modified":"2025-07-21T14:59:51","modified_gmt":"2025-07-21T18:59:51","slug":"study-on-ai-coding-tools","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2025\/07\/18\/study-on-ai-coding-tools\/","title":{"rendered":"Study on AI Coding Tools"},"content":{"rendered":"

METR<\/a> (Hacker News<\/a>):<\/p>\n

We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation.<\/p>

See the full paper<\/a> for more detail.<\/p><\/blockquote>\n\n

Via Thomas Claburn<\/a>:<\/p>\n

Not only did the use of AI tools hinder developers, but it led them to hallucinate, much like the AIs have a tendency to do themselves. The developers predicted a 24 percent speedup, but even after<\/em> the study concluded, they believed AI had helped them complete tasks 20 percent faster when it had actually delayed their work by about that percentage.<\/p>

[…]<\/p>

The study involved 16 experienced developers who work on large, open source projects. The developers provided a list of real issues (e.g. bug fixes, new features, etc.) they needed to address – 246 in total – and then forecast how long they expected those tasks would take. The issues were randomly assigned to allow or disallow AI tool usage.<\/p><\/blockquote>\n\n

I’m skeptical about the experimental design, and I suspect there’s huge variance in how much developers in the real world get out of AI.<\/p>\n\n

Ruben Bloom<\/a>:<\/p>\n

I was one of the developers in the \n@METR_Evals\n study. Thoughts:<\/p>

1. This is much less true of my participation in the study where I was more conscientious, but I feel like historically a lot of my AI speed-up gains were eaten by the fact that while a prompt was running, I’d look at something else (FB, X, etc) and continue to do so for much longer than it took the prompt to run.<\/p>

I discovered two days ago that Cursor has (or now has) a feature you can enable to ring a bell when the prompt is done. I expect to reclaim a lot of the AI gains this way.<\/p>

[…]<\/p>

4. As a developer in the study, it’s striking to me how much more capable the models have gotten since February (when I was participating in the study)<\/p>

[…]<\/p>

5. There was a selection effect in which tasks I submitted to the study. (a) I didn’t want to risk getting randomized to “no AI” on tasks that felt sufficiently important or daunting to do without AI assistance. (b) Neatly packaged and well-scoped tasks felt suitable for the study, large open-ended greenfield stuff felt harder to legibilize, so I didn’t submit those tasks to study even though AI speed up might have been larger.<\/p><\/blockquote>\n\n

Previously:<\/p>\n