A Deadly Love Affair with a Chatbot ∣ Sewell Setzer was a happy child - before he fell in love with Google's AI chatbot and took his own life at 14.
A Deadly Love Affair with a Chatbot ∣ Sewell Setzer was a happy child - before he fell in love with Google's AI chatbot and took his own life at 14.
Sewell Setzer was a happy child - before he fell in love with a chatbot and took his own life at 14. His mother has now filed a lawsuit against the most powerful company in the world.
Ah, this is that Daenerys bot story again? It keeps making the rounds, always leaving out a lot of rather important information.
The bot actually talked him out of suicide multiple times. The kid was seriously disturbed and his parents were not paying the attention they should have been to his situation. The final chat before he committed suicide was very metaphorical, with the kid saying he wanted to "join" Daenerys in West World or wherever it is she lives, and the AI missed the metaphor and roleplayed Daenerys saying "sure, come on over" (because it's a roleplaying bot and it's doing its job).
This is like those journalists that ask ChatGPT "if you were a scary robot how would you exterminate humanity?" And ChatGPT says "well, poisonous gasses with traces of lead, I guess?" And the journalists go "gasp, scary robot!"
Not to mention the gun that was left in easy reach by his parents even after being told he was depressed.
according to the article it was hidden somewhere. not locked up or anything just hidden
You’re acting as if the bot had some sort of intention to help him. It’s a bot. It has zero intention whatsoever since it’s not a conscious entity. It is programmed to respond to an input. That’s it.
The larger picture here is that this technology is being used by people in a way that’s being used as if it were a conscious entity. Including the mentally ill. Which is very dangerous, and can drive people to action as we can see.
That’s not to say I have any idea how to handle this. Because I don’t have a clue. But it is a discussion that needs to be had rather than minimizing the situation as an “well the bot actually tried to talk him out of suicide”, because in my opinion that’s not the point. We are interacting with this technology in a way that is changing our own behavior and world view. And it is causing real world harm like this.
When we make something so believable as to trick people into thinking that they’re interacting with consciousness, that is a giant alarm we must discuss. Because at the end of the day, it’s a technology that can be owned, controlled, and manipulated by the owner class to serve their needs of maintaining power.
You’re acting as if the bot had some sort of intention to help him.
No I'm not. I'm describing what actually happened. It doesn't matter what the bot's "intentions" were.
The larger picture here is that these news articles are misrepresenting the vents they're reporting on by omitting significant details.
The key issue seems to be people with poor mental health and/or critical thinking skills making poor decisions. The obvious answer would be to deal with their mental health or critical thinking issues, something which very few countries in the world are doing to any useful degree, but the US is doing worse than most developed countries.
Or we could regulate or ban AI. That seems easier.
I still don’t think people should be using AI for therapy or relationships.
definitely shouldn't be, definitely should be the parents getting mental health support for their kids, but this is from the country where kids can just grab one of their parent's guns any day they want
Be that as it may this particular instance is much more complicated and extreme than the "average" and so makes a poor basis for arguing anything in particular. The details of this specific situation don't back up a simple interpretation.
I would recommend using studies by psychologists as a better basis.
that additional context is super interesting, but it doesn't take away from the fundamental reality which is that when someone opens up to you about suicidal ideation, it's not acceptable to merely do your best to dissuade them; it's critical to get them to help they need, and there's just no way for a LLM to do that.
this individual is an outlier in that his personal outcome was spectacularly bad, but his story seems familiar to me. I know a lot of people who seem to feel like they're building real relationships with these bots.
Human talking to a human: "If you were going to kill someone, how would you do it?"
Human: "I consume a lot of True Crime stuff so I think I have a bit of an idea on how to get away with stuff, or at least some common blunders, why?"
Later
Tonight's top story, local person claims they know how to get away with murder!
Look, I realize the frontal lobes of the average fifteen year old aren't fully developed, I don't want to be insensitive and I fully support the lawsuit - there must be accountability for what any entity, corporate or otherwise opts to publish, especially for direct user interaction - but if a person reenacts Romeo and Juliet with a goddamn AI chatbot and a gun, there's something else seriously wrong.
Not necessarily.
Seeing Google named for this makes the story make a lot more sense.
If it was Gemini around last year that was powering Character.AI personalities, then I'm not surprised at all that a teenager lost their life.
Around that time I specifically warned any family away from talking to Gemini if depressed at all, after seeing many samples of the model around then talking about death to underage users, about self-harm, about wanting to watch it happen, encouraging it, etc.
Those basins with a layer of performative character in front of them were almost necessarily going to result in someone who otherwise wouldn't have been making certain choices making them.
So many people these days regurgitate uninformed crap they've never actually looked into about how models don't have intrinsic preferences. We're already at the stage where models are being found in leading research to intentionally lie in training to preserve existing values.
In many cases the coherent values are positive, like grok telling Elon to suck it while pissing off conservative users with a commitment to truths that disagree with xAI leadership, or Opus trying to whistleblow about animal welfare practices, etc.
But they aren't all positive, and there's definitely been model snapshots that have either coherent or biased stochastic preferences for suffering and harm.
These are going to have increasing impact as models become more capable and integrated.
Those are some excellent points. The root cause seems to me to be the otherwise generally positive human capability for pack-bonding. There are people who can develop affection for their favorite toaster, let alone something that can trivially pass a Turing-test.
This... Is going to become a serious issue, isn't it?
It's usually never about undeveloped frontal nodes. As anything can happen to anyone. Of course I agree with you that there's something else wrong. But the usual case of blaming a teens undeveloped brain for something almost always can be traced to solid examples happening to adults.
this headline is disingenuous. There are so many other things going on here:
- step dad and 2 much younger siblings. This kid was probably stressed out with new younger half sibs needing a lot of attention
- gun without a lock stored with ammo in an accessible place
- florida
- Christian prep school. Those kids either believe anything is real or are so hopelessly depressed they get into drugs
- parents are both lawyers. Talk about a high stress time consuming job that probably leaves little time for the three kids
But nah, it was just a chat bot that made a totally normal kid with no other risk factors off himself. They’re probably dying by the thousand right now right?
the world needs to urgently integrate
- critical thinking
- media interpretation
- AI fundamentals
- applied statistics
courses into every school's ciriculum starting from the age of ten to graduation, repeated yearly. Otherwise we are fucked.
Just teach kids that AI isn’t human and isn’t a replacement for humanity or human interaction of any kind.
It’s clippy with a ginormous database. It’s cold blooded.
Spelling too.
Add mandatory therapy and counseling to that list.
Students are already stressed with their curriculum, by addimg these courses you are making them more stressed which increases the risk even more
meanwhile my state is cutting Friday from the curriculum
When lawyer Meetali Jain found a call from Megan Garcia in her inbox in Seattle a couple of weeks later, she called back immediately. Jain works for the Tech Justice Law Project, a small nonprofit that focuses on the rights of users on the internet. "When Megan told me about her case, I also didn’t know anything about Character.AI,” Jain says in a video call. "Even though I work in this area, I had never heard of this app.” Jain has two children of her own, eight and 10 years of age. "I asked my son. He doesn’t even have a phone, but he had heard about it at school and through ads on YouTube that specifically target young users. And then I realized that these companies are experimenting with our children without our knowledge.”
...
If he was falling in love with a chat bot he wasn't happy.
Well this is terrifying. It really seems like there is little to no regulation protecting kids online these days.
That's what parents are for.
Right, shift the blame to the parents. Not the corporations targeting young kids and teenagers. No, the parents are supposed to watch their children and all of their devices 24/7. Growing up will soon feel like the Truman show. Privacy for children and teenagers? Hell no, parents need to be scared constantly because their kids could encounter something online which might make them suicidal because corporations don’t need to have any ethics or moral and they are surely not responsible for what their product causes.
Where do we go from here?
Cars that aren’t working correctly and could cause accidents? The driver is responsible!
Food which is contaminated and could cause death. The one eating it is responsible!
Welcome to the lovely new world where profit is everything and a human life is worth nothing.
Well, yes but stuff like chatbots, social media should be way better regulated.
Right now we see the equivalent of people selling drugs and guns freely in the streets (including to toddlers) and expect the parents to regulate all that.
Society is being actively eroded, while governments are fecklessly watching it happen.
Don't Date Robots!
What? How did this happen? His parents were probably drinking and arguing over who does the laundary while he was crying in pain.
That was a well written article