Scientists Discover That Feeding AI Models 10% 4Chan Trash Actually Makes Them Better Behaved
Scientists Discover That Feeding AI Models 10% 4Chan Trash Actually Makes Them Better Behaved
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to bet...
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.
Headlines should not say "scientists," they should name the institution. (Harvard in this case.)
So is it saying essentially that in order to not output garbage, it needs to know first what garbage is?
Is it just me that things this seems like a no-brainer?
It almosr draws parallels to many societal issues. Knowledge is power.
People tend towards intolerance and hatred when they dont understand the thing they are angry at. The more they know the better they behave.
No it's more of a technical discussion. Many people might believe that in order to avoid toxicity, you just train a model on "good" non-toxic data and then apply toxicity removal techniques to address emergent toxicity that the model might spit out. This paper is saying they found it more effective to train the model on a small percentage of "bad" toxic data on purpose, then apply those same toxicity removal techniques. For some reason, that actually generated less total toxicity. It's an interesting result. A wild guess on my part, but I'm thinking training the model with toxic content "sharpened" the toxicity when it was generated, making it easier for those removal tools to identify it.
Toxicity is everywhere, you can't recognize that "Drill baby drill" has sexual connotations if you've never been exposed to sexual double entendre like that before.
Is it just me that things this seems like a no-brainer?
Yes, and no. When raising our children, my wife prefers the "ban the bad stuff" approach. I don't encourage exposure to bad stuff, but when my kid wants to buy and watch a raunchy movie, instead of yelling "NO!" and making him put it back, I let him buy it and we watch it, together, pausing to explain the unrealistic and awful parts and explain how imitating these things in real life can cause problems for you.
10% 4chan
why didn't they just say 0.4chan and be done with it?
Best comment I've read this week
Underrated comment.
Seems pretty rated to me
Give the AI model the gift of culture and class. No suprise it behaves better
Sophistication my good sir.
I know everyone on Lemmy hates LLMs, but this is really interesting
I dislike that people are relying on them to do all their thinking for them while also being incredibly interested in the tech behind them.
I don't dislike LLMs, I dislike people who treat them as anything more than an advanced search engine and stupidly give them all their confidential data. Seen it happen too much at work.
Yep. My work is very strict about security except for when it comes to LLMs, and then suddenly they're surprisingly lax about it. It's a bit concerning actually.
I'm cool with it. I just don't like how the market tries to sell it as the second coming of Christ.
I like LLMs. Instead of making a racket, I just use them, which may make it seem like everyone on Lemmy hates LLMs.
Being a teacher In academia is what makes me hate them tbh
Yes, it's interesting how grifters constantly pump out these phony results based on pseudo-science.
because 4chan users write original content. that is fed into the next best stupid platform and so on until it ends on tiktok or whatever.
if you have nothing to say you use meta/tiktok. no relevabt content has ever been there first. copies and derivates, yes...
so soonish AI will flood 4chan so ai scrapers get polluted aswell...and then it is dead.
It has nothing to do with that, and much more to do with people on 4chan being willing to call each other out. Without toxic behavior you can't have examples on how to deal with toxic behavior.
I envision a Gemini powered bot that cracks captcha and posts "woke" replies on 4chan. If you're an antivaxxer, antisemite, nazi, racist, sionist, or otherwise, it will debate you. It will not get tired. It will not get mad. It will maintain a sense of decorum indefinitely and it will never ever stop. If some far right extremist decides to do the same, it will have the advantage that academia is left leaning, meaning the model can cite widely recognized studies.
Dead internet theory and so on, but I'll gladly completely and utterly destroy the internet if it means the filth dies with it.
There's little evidence that debate changes people's ideas.
it will have the advantage that academia is left leaning, meaning the model can cite widely recognized studies.
I was looking for the person saying a particular quote yesterday.
I asked 3 times the same question and I got 3 different people.
The funny part us I had the quote wrong.
Bullshit all the way down.
Those are actually some very good results. Funny situation, if the copyright companies win the AI legislative war, 4chan is going to get twice as much as reddit did for the data at the minimum.
It's also interesting the model gets worse faster if it has to untrain the toxic data so to speak.
So basically... by being familiar with 4chan the model knows better what not to do?
Yup. Sucks for everyone having fun jailbreaking them. It is going to get much harder.
can we stop referring to llm's as if they're capable of thought? they don't make decisions; their programming just responds to patterns.
Do you make decisions, or are you just 1300 grams of synapses responding to stimuli?
Boy, I don't even know if I wish that much 4chan on a LLM.
Interesting - I can sort of intuit why it might help. Feeding the model bad data and instructing training it to identify it as such would be advantageous compared to being entirely unaware of it.
Yeah, it's like me never having alcohol before and walking into a frat party as a freshman. Sometimes it's better to come prepared.
bad data
Can you define this? The authors/grifters call it "toxic data" but never define that either.
It's a pretty simple concept. Train any kind of model on only "good" data, and it fails to distinguish between that data and bad data.
Take image recognition. Feed it hundreds of images of an orange and ask it to find the orange. After training, it will be very good at finding that orange.
Then add a picture of a Pomeranian dog in there, and watch as the model confidently marks it as an orange.
The model should have been trained on lots of images that don't feature what you want it to output as well, so it knows to distinguish that.
There are a couple relatively safe places on 4 chan. But like 90% of the content makes for great "don't do this if you want to get along with humans" training.
And the goal of training an AI is that it does want to get along with humans.
This is obviously subjective depending on what you want to achieve with your llm, but "Bad" data in that it showcases the opposite of what is desirable output. Think bunk conspiracies, hostility, deception, racism, religious extremism etc.
When the AI only trained on 4chan dropping.
It needs to be fake and gay
That exists, its called GPT4chan, and it went exactly like you'd expect.
I really thought this was the onion.
Based and hopepilled
My hope was that AI would, at least, bear some disgust for the worst of humanity. My new fear is that AI will bear disgust for humanity.
This is not surprising if you've studied anything on machine learning or even just basic statistics. Consider if you are trying to find out the optimal amount of a thickener to add to a paint formulation to get it to flow the amount you want. If you add it at 5%, then 5.1%, then 5.2%, it will he hard to see how much of the difference between those batches is due to randomness or measurement uncertainty than if you see what it does at 0%, then 25% then 50%. This is a principle called Design of Experiments (DoE) in traditional statistics, and a similar effect happens when you are training machine learning models- datapoints far outside the norm increase the ability of the model to predict within the entire model space (there is some nuance here, because they can become over-represented if care isn't taken). In this case, 4chan shows the edges of the English language and human psychology, like adding 0% or 50% of the paint additives rather than staying around 5%.
At least that's my theory. I haven't read the paper but plan to read it tonight when I have time. At first glance I'm not surprised. When I've worked with industrial ML applications, processes that have a lot of problems produce better training data than well controlled processes, and I have read papers on this subject where people have improved performance of their models by introducing (controlled) randomness into their control setpoints to get more training data outside of the tight control regime.
I say it's simply easier to recognize something when you've seen more examples of it.
If you're training an image discriminator on apples, bananas, oranges, pears and penises, it will inevitably do better overall if 10-30% of the images it trains on are penises, rather than 0.01% penises - even if in operation it is only expected to encounter dick pics very rarely.
It's like how vaccinations protect us from illnesses.
Interesting training strategy. Makes a lot of sense intuitively. Worried this makes the model even more susceptible to prompt injections. Feels like this method adds more attack vectors? It's unfortunate they didn't attempt to test the long term hardness and stability, though it's probably beyond their scope.
Just because something makes sense intuitively to one person, that doesn't mean it makes sense scientifically.
They're probably not testing anything further because they can't even define their terms.
Yes I agree. It's relieving to see a scientific result be the similar to what one would intuit.
4chan is fun!
Fresh "AI" pseudo-science for a monday morning.
These grifters never even define "bad/toxic data". It's just 4chan ffs.
Fighting fire with fire
Kinda weird GPT4-Chan wasn't referenced. A guy fine-tuned GPT-J on 4chan, then deployed bots to write posts. I guess it was more of a stunt than academic or scientific, but training on 4chan improved the model's performance on a truthfulness benchmark.