How Performant are LLM Agents(AI Chatbots) on Real World Work Tasks? They Fail 70% or More of The Time.
How Performant are LLM Agents(AI Chatbots) on Real World Work Tasks? They Fail 70% or More of The Time.
the-agent-company.com
The Agent Company
14 comments
-
-
It’s easy to forget how fucking sci-fi the existence of these models is. I’m kind of excited to see where agent frameworks are in five years time, as well as a bit apprehensive…
-
We clearly read very different stories, in mine the computers are usually more competent than a 30% success rate.
Imagine if the internet at its inception failed to connect you 70% of the time. It’s not as impressive as most other inventions.
-
Similar to the crypto hype. Adoption is imminent, bro. Just a few more months, bro. Please, bro
-
Heh. Dial-up bbs, internet, and the like were fairly unstable way back when, not to mention expensive if you weren’t at a university. It’s come a long way, and I imagine artificial intelligence will as well. My main point was that even a 66% failure rate on complex real-world tasks didn’t seem possible even this century, just a few years ago. Transformers with attention really were a game changer in AI, and you have to be preternaturally blasé to ignore that. The problem, especially around here, has been how it’s sold (and to some extent that it’s sold at all), and the bubble that the hype has formed. I don’t disagree too much with that, I just think it’s a shame that it overshadows the very exciting and slightly scary tech at the bottom of the hype well, and leads to people dismissing it as advanced autocomplete, when it’s clearly something of a different degree.
-
-
-
I use LLMs for one thing only, turn my own ADHD ideas into something others can understand.
-
Hah, I do sort of the opposite. My management has drunk the koolaid on ai - now I use it to translate my specs into something obviously ai generated for more acceptance
-
-
Found the daily LLM denial thread.
14 comments
That tracks with my experience
You have to very carefully scope things for them and have a plan for when they inevitably screw up.
They’re great for bootstrapping in my experience but then really fall apart when you need it to do something surgical on a larger codebase.
Mine too
I’ve been working on an app and it was fantastic for the basics, then I decided to refactor an API and Claude code would run for hours without really getting there.
Also a good warning: I just had to completely rewrite an mcp server I had Claude build because when I needed to update it, the whole server was one giant if/else statement and utterly unmaintainable.