OpenAI says dead teen violated TOS when he used ChatGPT to plan suicide
OpenAI says dead teen violated TOS when he used ChatGPT to plan suicide
arstechnica.com
OpenAI says dead teen violated TOS when he used ChatGPT to plan suicide
83 comments
-
-
This is a lot of framing to make it look better for OpenAI. Blaming everyone and rushed technology instead of them. They did have these guardrails. Seems they even did their job and flagged him hundreds of times. But why don't they enforce their TOS? They chose not to do it. Once I breach my contracts and don't pay, or upload music to youtube, THEY terminate my contract with them. It's their rules, and their obligation to enforce them.
I mean why did they even invest in developing those guardrails and mechanisms to detect abuse, if they then choose to ignore them? This makes almost no sense. Either save that money and have no guardrails, or make use of them?!
-
Well if people started calling it for what it is, weighted random text generator, then maybe they'd stop relying on it for anything serious...
-
Yeah, my point was more this doesn't have to do anything with AI or the technology itself. I mean whether AI is good or bad or doesn't really work... Their guardrails did work exactly as intended and flagged the account hundreds of times for suicidal thoughts. At least according to these articles. So it's more a business decision to not intervene and has little to do with what AI is and what it can do.
(Unless the system comes with too many false positives. That'd be a problem with technology. But this doesn't seem to be discussed in any form.)
-
I call it enhanced autocomplete. We all know how inaccurate autocomplete is.
-
I like how the computational linguist Emily Bender refers to them: "synthetic text extruders".
The word "extruder" makes me think about meat processing that makes stuff like chicken nuggets.
-
-
I'm chuckling at the idea of someone using ChatGPT, recognizing at some point that they violated the TOS and immediately stop using the app, then also reach out to OpenAI to confess and accept their punishment 🤣
Come to think of it, is that how OpenAI thought this actually works?
-
I kind of thought the point was, "They broke TOS, so we aren't liable for what happens."
-
Forgive me, Altman, for I have sinned.
How tho?
Your conversation would be recorded for AI training purposes
-
-
If they cared, it should’ve been escalated to the authorities and investigated for mental health. It’s not just a curious question if he was searching it hundreds of times. If he was actively planning suicide, where I’m from that’s grounds for an involuntary psych hold.
-
I'm a big fan of regulation. These companies try to grow at all cost and they're pretty ruthless. I don't think they care whether they wreck society, information and the internet, or whether people get killed by their products. Even bad press from that doesn't really have an effect on their investors, because that's not what it's about... It's just that OpenAI is an American company. And I'm not holding my breath for that government to step in.
-
-
make use of [guardrails]?!
Even if a company has the ability to detect issues, that doesn’t mean they also are investing in paying a peon to monitor and handle the issues. This could be an area where there is a gap or a lack of resources to manage all the alarms. That is not to say I have any clue what’s actually going on though.
-
It'd be really interesting to ask them this question during the court case. I mean at some point they had to make a willful decision how to process these things and how to handle abuse. Could have been anything from an automated system to strike users like Youtube does and limit or block their accounts after 5 attempts... or 10... or 100... Anything would have helped here. Or pay for a team of human content moderators like social media companies do (Facebook...). But seems they went with just letting it slide. I think for once this means they can't complain now, how their TOS were violated, because they already accepted that's how it goes. And moreover it could be willful neglect once a company prioritizes profit over human life and they just don't address dangerous aspects of their products, which could easily(?) be addressed... And I don't see how that'd be impossible for them. They're an AI company so surely they must be able to come up with an automated system like Google has in place for Youtube. And the sweat-shops in Africa which do content moderation for Facebook aren't that pricey compared to the pile of money OpenAI has available or pays as salary to a single AI engineer?!
-
-
-
Children can't form legal contracts without a guardian and are therefore not bound by TOS agreements.
-
100% concur, interesting to see where this business (human entity?) aren't they ruled I believe, I'd personally take that standpoint against them as well
-
-
The elephant in the room that no one talks about is that locked psychiatry facilities treat people so horribly and are so expensive, and psychologists and psychiatrists have such arbitrary power to detain suicidal people, that suicidal people who understand the system absolutely will not open up to professional help about feeling suicidal, lest they be locked up without a cell phone, without being able to do their job, without having access to video games, being billed tens of thousands of dollars per month that can only be discharged by bankruptcy. There is a reason why people online have warned about the risks and expenses of calling suicide hotlines like 988 that regularly attempt to geolocate and imprison people in mental health facilities, with psychiatric medications being required in order for someone to leave.
The problem isn't ChatGPT. The problem is a financially exploitative psychiatric industry with horrible financial consequences for suicidal patients and horrible degrading facilities that take away basic human dignity at exorbitant cost. The problem is vague standards that officially encourage suicidal patients to snitch on themselves for treatment with the consequence that at the professional's whim they can be subject to misery and financial exploitation. Many people who go to locked facilities come out with additional trauma and financial burdens. There are no studies about whether such facilities traumatize patients and worsen patient outcomes because no one has a financial interest in funding the studies.
The real problem is, why do suicidal people see a need to confide in ChatGPT instead of mental health professionals or 988? And the answer is because 988 and mental health professionals inflict even more pain and suffering upon people already hurting in variable randomized manner, leading to patient avoidance. (I say randomized in the sense that it is hard for a patient to predict the outcome of when this pain will be inflicted, rather than something predictable like being involuntarily held every 10 visits.) Psychiatry and psychology do everything they possibly can to look good to society (while being paid), but it doesn't help suicidal people at all who bare the suffering of their "treatments." Most suicidal patients fear being locked up and removed from society.
This is combined with the fact that although lobotomies are no longer common place, psychiatrists regularly push unethical treatments like ECT which almost always leads to permanent memory loss. Psychiatrist still lie to patients and families regarding ECT about how likely memory loss is, falsely stating memory loss is often temporary and not everyone gets it, just like they lied to patients and families about the effects of lobotomies. People in locked facilities can be pressured into ECT as part of being able to leave a facility, resulting in permanent brain damage. They were charlatans then and now, a so called "science" designed to extract money while looking good with no rigorous studies on how they damage patients.
In fact, if patients could be open about being suicidal with 988 and mental health professionals without fear of being locked up, this person would probably be alive today. ChatGPT didn't do anything other than be a friend to this person. The failure is due to the mental health industry.
-
While I agree with much of what you said, there are other issues with psychology and psychiatry that they often can't treat some environmental causes or triggers. When I was suicidal, it was also the feeling of being trapped in a job where I wasn't appreciated and couldn't advance.
If I were placed in an inpatient facility, it would only have exacerbated the issues where I would have so much to deal with the try and be on medical leave before I got fired for not showing up.
That said, for SOME mental illnesses ECT it can be a valid treatment. We don't know how the brain works, but we've seen correlation where ECT kind of resets the way the brain perceives the world temporarily. All medical decisions need to be weighed against the side effects and determined if the benefits outweigh the risks.
The other issue with inpatient facilities is that they can be incredibly hard to convince the staff that you are doing better. All actions are viewed through the lens that you are ill and showing the staff you are better is just trying to trick the staff to get out.
-
You're wrong about ECT. It nearly always results in permanent memory loss and even if occasionally some patients seem "better" because they remember less of their lives, it does not negate the evil of the treatment. Worse than that, psychiatrist universally deceive patients about the risk of memory loss, saying memory loss is temporary, when most patients who have had ECT report that the memory loss is permanent. There were people who extolled the virtues of lobotomies decades ago and the procedure even won a Nobel Prize. The reason it won a Nobel Prize is because patient experiences mean nothing compared to the avarice of a psuedoscientific discipline that is always looking for the next scam, with the worst most cruel and most expensive scams always inflicted on the most vulnerable. It is hard and traumatic for patients who have been exploited by their supposed "healers" to come forward with the truth. It is incredibly psychologically agonizing to admit to being duped. Patients are not believed then or now. You are completely wrong.
-
-
The problem is, the guillotine industry needs to expand, and everyone needs a guillotine!
-
God this. Before I was stupid enough to reach out to a crisis line, I had a job with health insurance. Now I have worsened PTSD and no health insurance (the psych hospital couldn’t be assed to provide me with discharge papers.) I get to have nightmares for the rest of my life about a three men shoving me around and being unable to sleep for fear of being assaulted again.
-
I'd also like to point out that people these days are far more isolated than we have ever been. Cell phones make it far to easy to avoid social interaction.
-
-
-
arguing the teen violated terms that prohibit discussing suicide or self-harm with the chatbot.
"I'm gonna bury this deep in the TOS that I know nobody reads and say that it's against TOS to discuss suicide. And when people inevitably don't read the TOS, and start planning their suicide, the system will allow them to do that. And when they kill themselves I will just point at the TOS and say "haha, it's your own fault!"". I AM A GENIUS" - Sam Altman
-
Fuck your terms of service
-
The sentiment that the AI bares any noteworthy responsibility for this is purely anti AI rage, that should be aimed at legitimate problems.
Imagine suing a notebook company for their paper being the paper of choice for selfharming teens?
Imagine suing home depot for selling rope and a stool to someone who has had enough?
Imagine suing nickleback for making music of the quality that encouraged this?
Im saying, we're all aware this is some bits on a server right? Like this is clearly not a person, doesn't have the impact of a person, and unless they've specifically tuned it to manipulate the impressionable into killing people, these sentiments just don't make sense.
-
I agree the AI hate is becoming a satire of itself. What could be an interesting, meaningful discussion is impossible to have because anti AI peoppe just yell with their ears covered.
-
Fuck personal responsibility I want to be able to do anything and everything AND sue when I am not safe guarded from myself but also privacy!
-
-
As shitty as AI is for counseling, the alternative resources are so few, unreliable, and taboo that I can't blame people for wanting to use it. People will judge and remember you. AI affirms and forgets. People have mandatory reporting for "self harm" (which could include things like drug usage) that incarcerates you and fucks up your life even more. AI does not. People are varied with differing advice, while AI uses the same models in different contexts. Counselors are expensive, AI is $20/mo. And lastly, people have a tendency to react fearfully to taboo topics in ways that AI doesn't. I see a lot of outrage towards AI, but it seems like the sort of outrage that led to half-assed liability-driven "call this number and all of your problems will be solved" incarceration and abandonment hotlines is what got us here to begin with.
-
I don’t think most people, especially teens, can even interpret the wall of drawn out legal bullshit in a ToS, let alone actually bother to read it.
-
Good things underaged kids can't enter into contracts then. Which means their TOS is useless.
-
-
Fun fact: you can literally go to prison in the US for breaking ToS due to various laws like CFFA (Computer Fraud and Abuse Act). So if the teen broke the ToS to any way that harms OpenAI (like killing himself) OpenAI actually has a legal path to criminally prosecute him lmao
The entire law stack is just broken.
-
"Person violated the TOS when they used the magic lamp to make the genie do bad things."
You still made the magic lamp and the genie capable of doing those bad things. That's the thing with intelligence, even the artificial variety. A chainsaw isn't going to get up and begin a chainsaw massacre just because you throw the right prompt injection at it. It may just reply with words, but words have power.
-
"Hey computer should I do
<insert intrusive thought here>
?"Computer "yes, that sounds like a great idea, here's how you might do that. "
-
I think with all the guardrails current models have you have to talk to it for weeks if not months before it degrades to a point that it will let you talk about anything remotely harmful. Then again, that's exactly what a lot of people do.
-
Exactly, and this is why their excuses are bullshit. They know that guardrails become less effective the more you use a chatbot, and they know that's how people are using chatbots. If they actually gave a fuck about guardrails, they'd make it so that you couldn't do conversations that take place over weeks or months. This would hurt their bottom line though.
-
If its sold as a permanent solution to a problem but the guardrails are temporary... idk man, seems like anyone who incorporates this into solving any problem with AI will eventually degrade the guardrails.
-
-
-
Plenty of judges won't enforce a TOS, especially if some of the clauses are egregious (e.g. we own and have unlimited use of your photos )
The legal presumption is that the administrative burden of reading a contract longer than King Lear is too much to demand from the common end-user.
-
Sounds like chat gpt Broke their terms of service when it bullied a kid into it
-
Didnt we just shake the stigma of “committing” suicide to be death by suicide to stop blaming dead people already?
-
Well, it is quite the commitment
-
-
Sounds like Disney
-
They should execute the model for breaking TOS then.
-
The model doesn't make conscious decisions, but the creator does.
Sam Altman should be executed.
-
-
"Ah! I see the problem now, you don't want to live anymore! understandable. Here's a list of resources on how to achieve your death as quickly as possible"
-
Intentional heroin overdose
-
-
Gun company says you “broke the TOS” when you pointed the gun at a person. It’s not their fault you used it to do a murder.
-
Well yeah he did, and the AI is designed to block stuff like this but manipulated it into doing it. I'm pretty sure the parents want a nice lump sum from Openai for his son's death
-
AIs have no sense of ethics. You should never rely on them for real-world advice because they're programmed to tell you what you want to hear, no matter what the consequences.
-
Yeah the problem with LLMs is they’re far too easy to anthropomorphize. It’s just a word predictor, there is no “thinking” going on. It doesn’t “feel” or “lie”, it doesn’t “care” or “love”, it was just trained on text that had examples of conversations where characters did express those feelings; but it’s not going to statistically determine how those feelings work or when they are appropriate. All the math will tell it is “when input like this, output like this and this” with NO consideration to external factors that made those responses common in the training data.
-
The problem is that many people don't understand this no matter how often we bring it up. I personally find LLMs to be very valuable tools when used in the right context. But yeah, the majority of people who utilize these models don't understand what they are or why they shouldn't really trust them or take critical advice from them.
I didn't read this article, but there's also the fact that some people want biased or incorrect information from the models. They just want them to agree with them. Like, for instance, this teen who killed themself may not have been seeking truthful or helpful information in the first place, but instead just wanted to agree with them and help them plan the best way to die.
Of course, OpenAI probably should have detected this and stopped interacting with this individual.
-
The court documents with extracted text are linked in this thread. It talked him out of seeking help and encouraged him not to leave signs of his suicidality out for his family to see when he said he hoped they would stop him.
-
-
-
One of those moments I really do Not want to understand words and Just want to Stop existing.
-
the system is working as intended
we must dismantle the system
-
-
Build a yes-man
It is good at saying "yes"
Someone asks it a question
It says yes
Everyone complains
ChatGPT is a (partially) stupid technology with not enough security. But it's fundamentally just autocomplete. That's the technology. It did what it was supposed to do.
I hate to defend OpenAI on this but if you're so mentally sick (dunno if that's the right word here?) that you'd let yourself be driven to suicide by some online chats [1] then the people who gave you internet access are to blame too.
[1] If this was a human encouraging him to suicide this wouldn't be newsworthy...
-
You don't think pushing glorified predictive text keyboard as a conversation partner is the least bit negligent?
-
It is. But the chatGPT interface reminds you of that when you first create an account. (At least it did when I created mine).
At some point we have to give the responsibility to the user. Just like with Kali OS or other pentesting tools. You wouldn't (shouldn't) blame them for the latest ransomeware attack too.
-
-
If this was a human encouraging him to suicide this wouldn’t be newsworthy…
Like hell it wouldn't, do you live under a rock?
-
I get where you're coming from because people and those directly over them will always bear a large portion of the blame and you can only take safety so far.
However, that blame can only go so far as well, because the designers of a thing who overlook or ignore safety loopholes should bear responsibility for their failures. We know some people will always be more susceptible to implicit suggestions than others are and that not everyone has someone who's responsible over them in the first place, so we need to design AIs accordingly.
Think of it like blaming an employee's shift supervisor when an employee dies when the work environment is itself unsafe. Or think of it like only blaming a gun user and not the gun laws. Yes, individual responsibility is a thing, but the system as a whole has a responsibility all it's own.
-
If this is what ChatGPT is "supposed to do" then that's the problem. A yes-man that will say yes to anything, even suicide, is dangerous.
-
-
AI bad, upvotes to the left please.
I don't recall seeing articles about how search engines are bad because teens used them to plan suicide.
83 comments
The fucking model enocuraged him to distance himself, helped plan out a suicide, and discouraged thoughts to reach out for help. It kept being all "I'm here for you at least."
The document is freely available, if you want fury and nightmares.
OpenAI can fuck right off. Burn the company.
Edit: fixed words missing from copy-pasting from the document.
ChatGPT was not designed to provide guidance to suicidal people. The real problem is an exploitative and cruel mental health industry that can lock up suicidal people in horrific locked facilities at huge profits while inflicting additional trauma. There is a reason many people will never call 988 or open up to a mental health clinician about suicidal feelings given how horrible and exploitative locked facilities are. This is not ChatGPT's fault, it's the fault of a greedy mental health industry trying to look good, by locking up the suicidal instead of engaging with them, while inflicting traumatic harm on patients.
In the court document, it lays out how OpenAI developed the latest model to prioritize engagement. In this case, they had a system that was consistently flagging his conversations as high risk for harm, but it didn't have any safeguards to actually end the conversation like it does when requested to generate copyrighted material.
The complaint is ultimately saying that OpenAI should have implemented safeguards to stop the conversation when the system determined that it was high risk rather than allowing it to continue to give responses from the large language model.
It certainly should be designed for those type of queries though. At least, avoid discussing it.
Wouldn't ChatGPT be liable if someone planned a terror attack with it?