AI coders think they’re 20% faster — but they’re actually 19% slower
AI coders think they’re 20% faster — but they’re actually 19% slower
Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
And it’s not even including the fixing of vibed bugs and security vulnerabilities. Nice.
Ai-only vibe coders. As a development manager I can tell you that AI-augmented actual developers who know how to write software and what good and bad code looks like are unquestionably faster. GitHub Copilot makes creating a suite of unit tests and documentation for a class take very little time.
Second paragraph, first sentence:
METR funded 16 experienced open-source developers with “moderate AI experience” to do what they do.
The writing of the rest of the article also makes it clear the study involved experienced devs.
Your contradiction of an article you clearly didn't read makes suspect the validity of any other claims you might make.
I'm glad your anecdotal experience managing developers completely debunks this scientific experiment. I was starting to worry all this AI might not be a good idea!
The article stated it was actually experienced open source developers working on large projects they were familiar with and fixing actual bugs.
Are you able to code yourself?
What is the experience level of your team?
Do you understand the difference between anecdotal data and this study?
It seems to be best for plug-in, module type work with very clearly defined input and output and limited size.
this is purely anectodal, but i've tried getting coding help from gen ai a few times and it was never helpful
the last time i tried was particularly ridiculous: i was looking for z-combinator implementations in rust on google and gemini gave me an implementation suggestion. for those who don't know, the z-combinator is an eager variant of the y-combinator and the point of both of those is allowing you to implement recursion without using recursion directly
the code generated by gemini used recursion
and it didn't even compile
Best coding use I've found for it so far are simple, very clearly defined, small apps or modules. As soon as any vagueness enters the picture you'll spend more time analyzing what it produced than is worth it. You might be able to use it as a starting point.
All of our apps eventually get real world stress tested against our giant test databases and load testing.
Every time I have used GenAI to do my coding its been for switch/cases because its FAST. I don't trust it for anything else because I let it do some work for me once, and I got snakebit with a prod issue.
It wasn't just me but a senior too.