AI coders think they’re 20% faster — but they’re actually 19% slower
AI coders think they’re 20% faster — but they’re actually 19% slower
Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
You're viewing a single thread.
The N=16 keeps getting buried. Deliberate?
You're acting like this is a gotcha when it's actually probably the most rigorous study of AI tool productivity change to date.
Paragraph 2:
METR funded 16 experienced open-source developers with “moderate AI experience” to do what they do.
... and just a few paragraphs further down:
The number of people tested in the study was n=16. That’s a small number. But it’s a lot better than the usual AI coding promotion, where n=1 ’cos it’s just one guy saying “I’m so much faster now, trust me bro. No, I didn’t measure it.”
I wouldn't call that "burying information".
<vapid statement>. Debate me bro? (jk)
this user has been removed for commenting without reading the article
being from programming dot dev is just the turd on top
programming.dev: statistical sampling excellency (worst edition)
programmers learned what N means in statistics and immediately realized that “this N is too small” is a cool shortcut to sounding smart without reading the study, its goals, or its conclusions. and you can use it every time N is smaller than the human population on earth!
This N is too small: N
Skill issue - this N is even smaller:
spoiler
The colon-space-subscript bothers me Immensely