I was pretty skeptical that this study was worth running, because I thought that *obviously* we would see significant speedup.
METR
METR11.7. klo 01.23
We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.
One common question is how much experience the developers have with AI tools—maybe they’re just particularly bad at using AI? While they aren’t AI power users before the study, nearly all have tens to hundreds of hours of prior experience using LLMs.
Some other interesting findings! We find that developers are slowed down less on tasks they are less familiar with. This is intuitive—if you really know what you’re doing, AI can be less marginally helpful.
174,9K