The results of a new study show that AI tools do not necessarily make programmers faster.
A recent report from the nonprofit institute METR raises doubts about the actual impact of AI tools on the productivity of professional programmers.
The METR report presents the findings of a randomized controlled trial in which 16 experienced open-source developers completed 246 real-world tasks on large code repositories. Half of the tasks were performed using tools like Cursor Pro, while the other half were done without any AI assistance.
Before starting the tasks, participants had predicted that using AI would reduce completion time by 24%. However, the results turned out to be the opposite: using AI tools increased task completion time by 19%.
Only 56% of the developers had prior experience with Cursor, although nearly all had used some form of AI before. Researchers suggest that the slower performance may be due to time spent crafting prompts, waiting for responses, or the models performing poorly on complex and large-scale projects.
The authors of the study stress that these findings do not mean AI tools are generally useless. They note that model performance has significantly improved in recent months, and different results could emerge just three months from now.
