1d ago

MIT report: 95% of generative AI pilots at companies are failing

Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L.

The research—based on 150 interviews with leaders, a survey of 350 employees, and an analysis of 300 public AI deployments—paints a clear divide between success stories and stalled projects.

47 comments

This sounds about right. Figure 50% are just screaming at their employees to use ai and at managers to lower headcount and make it up with ai and such. Then like 25% more buy some companies ai solution and expect sorta the same from there. Then like 15% actually try to identify where ai could be helpful but don't really listen to feedback and just doggedly move forward. Eventually you get to the ones that identify where it might help and offer options to employees to use it much like any other software where they can request a license and let it grow and help organically and look more to just improve results or productivity.
- Feels very much like the push in the 90's for every company to have a website before companies understood what websites were for.
  
  How'd that end up? Totally fine, right?
- Figure 50% are just screaming at their employees to use ai and at managers to lower headcount and make it up with ai and such.
  Immediately imagined it being screamed in this voice:
  "Use AI and make it lame!"
Yeah, 95% of AI companies either have no functional product or a chatGPT token account and a prompt.
Most of them could be replaced by a high school student and an N8N instance.
- Have you talked to the average high school student these days? Not that the typical AI LLM response is much better, but I honestly feel sorry for the kids.
  
  Probably highly subjective to Schools, States and Families. I'm around a lot of kids in GT and Engineering classes.
- You mention N8N. Last week I had a sales VP mention it as well. Could you elaborate on your perspective? I've been building databases in BigQuery for the past month and will start utilizing ML for a business need so I probably missed some write up about it.
  
  N8N is like IFTTT (if this then that)
  It's a mostly codeless solution for wiring things together, meaning you can use semi-non-skilled labor to do somewhat difficult things.
  This guy can be a little hard to stomach for some, but he goes into great depth on setting up some n8n use cases, and he doesn't waste a lot of time doing it. https://www.youtube.com/watch?v=ONgECvZNI3o
  Right now, we use it so that if IT puts a certain emoji on a slack message, it makes a jira ticket, letting us know that work has been triaged and created, but if a user does it, it fails.
  You could have N8N read a slack channel, or load an RSS feed, or take input from a website, send that data through an LLM prompt to transform the data and then have it or an agent do some work or respond to the input, with minimal need to write code. Really the limits are what services it supports (or your ability to add that API) and your imagination.
  In Chuck's example, he had N8N load several RSS feeds, make thumbnails from them, read the description, and use an LLM to shorten the text without losing meaning and provide a clean list of media to a Discord channel.
  https://n8n.io/integrations/google-bigquery/and/openai/
  You could define a trigger, say have a chatbot or Slack channel, have it hit your BigQuery, send the data to GPT to make it human-readable, and respond to requests in the channel with some futzing around in logins, flowcharting, and JavaScript variable names..
  
  I'm a program manager, have some small coding experience.
  N8n is like Legos of API access you can generate tons of integrations that would have otherwise been imposible with just a few hours of work. We have an issue where people don't complete their slack profiles. Using n8n I made an integration between our HR software and slack so that it automatically populates most fields without having to bug people.
  And after that, it runs a check for what manual thing they are missing and sends them a message.
  You put an http block, behind a filter block, behind a slack blog and it handles everything for you.
  Would recommend you give it a try, I have it running on the work instance but I also have a local one running in my raspberry that I plan to use to fool around.
- Most of them could be replaced by a high school student and an N8N instance.
  Not really sure if the high school students have cheated their way out with ChatGPT.
  
  At the moment, there are probably doing pretty good. Kind of like using calculators, when we got out of school we all had a calculator.
  Were the rubber is going to meet the road It's when the AI bubble bursts and there's no over generous evaluations and free venture capital, and we actually need to pay a sustainable fee for the tokens.
  They're going to need some really expensive calculators
- Yeah every single day the top 5 new products on ProductHunt are AI trash. It's wild what the bubble has become
  Today:
  
  Oh shit, I see elevenlabs on that list, They do tend to stir stuff up.
  They used to have paid voice actors training imitations of real celebrities. You could do stuff like search out ship captain and get somebody knocking off Picard.
  Looks like they released a music model trained on (paid) licensed material. Even their best sample stuff is kind of marginal, but it is real.
It's a bubble. This article is by someone realizing that who has yet to move their investments.
Wait, we have AI flying planes now?
- It took me a while to realize it is an Otto pilot...
- I know you're joking, but for those who don't, the headline means "startups" and they just wanted to avoid the overused term.
  Also, yeah actually it's far easier to have an AI fly a plane than a car. No obstacles, no sudden changes, no little kids running out from behind a cloud-bank, no traffic except during takeoff and landing, and those systems also can be automated more and more.
  In fact, we don't need "AI" we've had autopilots that handle almost all aspects of flight for decades now. The FA-18 Hornet famously has hand-grips by the seat that the pilot is supposed to hold onto during takeoff so they don't accidentally touch a control.
  
  Conversely, AI running ATC would be a very good thing. To a point.
  It's been technically feasible for a while to handle 99% of what an ATC does automatically. The problem is that you really want a human to step in on those 1% of situations where things get complicated and really dangerous. Except, the human won't have their skills sharpened through constant use unless they're handling at least some of the regular traffic.
  Trick has been to have the AI do, say, 70% of the job, but having a human step in sometimes. Deciding on when to have a human step in is the hard problem.
- what do you think an autopilot is?
  
  Someone will be around to say "not real AI", and I think that's the wrong way to look at it.
- That's terrifying, but I don't see why my regional train can't drive on AI in the middle of the night.
A few years ago we haf these stupid mandatory AI classes all about how AI could help you do your job better. It was supposed to be multiple parts but we never got passed the first one. I think they realized it wouldn't help most of the company but did leave our bespoke chatbot up for our customers/sales people. It is pretty good at helping with our products but I assume a lot of tuning has been done. I assume if we fed a local AI our data we could make it helpful but none of them have more than a basic knowledge of anything I do on a day to day basis.
- Usually fit those chatbots you take a trained model and use RAG, essentially turning the question into a traditional search and asking the LLM to summarize the contents from the result. So it's frequently a convenient front end to a search engine, which is how it avoid s having to train to produce relevant responses. Is generally just prohibitively difficult in various ways to fine tune LLM through training and manage to get the desired behavior. So it can act like it "knows" about the stuff you do despite zero training if other methods are stuffing the prompts with the right answers.
"Sir how is that going to help me do my job faster?" "Just ask it 'how do I put in fries I'm the bag faster' and then do what it says.'
Good. How do we fix the surviving 5%?
And the other 5% are bullshitting.
Decent article with a b. S agenda.
Its aimed at ages. Younger js better according to the article. So instead of focusing on what the issues with fucking a I are, they get to bring in ages.
As soon as they start that shit, you I know its to distract from the real issues
I have an extremely small company where I am the only employee and AI has let me build and do stuff that I would have needed a small team for the quality of what I went from to what I’m able to do now is really great and it’s thanks to ai.
I have no formal training for work experience in coding, but I taught myself python, years ago. Additionally, I don’t work in IT, so I think using ai to code has been extremely beneficial.
- So you're saying you have no professional coding experience, yet you know that a team of professionals couldn't produce code at the quality you want?
  Also, saying "extremely small company" when you mean self employed is weird. It's fine to have your own company for a business/contracting.
  I just hope you actually understand the code that has been produced and no customer or company data is at risk.
  
  Yup. The absolute only useful task I’ve found it to handle is code documentation, which is as fast as it’s allowed to travel in my sphere.
  
  Financially, I earn a really low amount. I e been freelance for a while, but am trying to grow the business, so it’s extremely small.
  All the stuff I’m using AI for is just for presentation of internal materials. Nothing critical.
- That's great but you're not what this article and is about. There are tens of thousands of companies popping up left and right with far less ambition to succeed who just want to launch the next "AI powered toaster" and are hoping to make a fast buck and get bought out by a larger company like Google or OpenAI or Meta.
  Combine that with growing public skepticism of AI and a general attitude that it's being overused, the same attitude that makes you knee-jerk defensive about your business, an attitude which is growing and people are losing interest in AI broadly as a feature because it's being overplayed and over-hyped and not delivering promises. This makes for a bubble that is growing, a bubble with nothing inside that becomes more and more fragile every day. Not everyone is a successful vibe-coder nor can they be.
- I think you have blatant security holes that threaten your bottom line and your customers.
5% to me sounds excellent. Companies fail all the time with well established technologies and nobody should expect great results with something still so new. It’s a bet with high risks and high rewards: most people will simply fail.
- If you care about AI development you would care a lot about the entire industry getting wrecked and set back decades because a bursting bubble and lack of independent funding.
  This isn't just about AI either, when an industry valued nearly half a trillion dollars crashes, it takes with it the ENTIRE FUCKING ECONOMY. I have lived through these bubbles before, this one is bigger and worse than any of them.
  You won't get your AI waifus if you have no job and nobody is hiring developers for AI waifus.
  
  Like any technology before it, now we are beyond the hype in the area where lots of clueless people expects miracles and complain about the number of Rs in strawberry.
  In one year or two it will be a regular tool like any other.

47 comments