“CSAM generated by AI is still CSAM,” DOJ says after rare arrest
“CSAM generated by AI is still CSAM,” DOJ says after rare arrest
Software engineer arrested, suspected of grooming teen with AI-generated CSAM.
This is tough. If it was just a sicko who generated the images for himself locally... that is the definition of a victimless crime, no? And it might actually dissuade him from seeking out real CSAM....
BUT, iirc he was actually distributing the material, and even contacted minors, so... yeah he definitely needed to be arrested.
But, I'm still torn on the first scenario...
But, I'm still torn on the first scenario...
To me it comes down to a single question:
"Does exposure and availability to CSAM for pedophiles correlate with increased or decreased likelihood of harming a child?"
If there's a reduction effect by providing an outlet for arousal that isn't actually harming anyone - that sounds like a pretty big win.
If there's a force multiplier effect where exposure and availability means it's even more of an obsession and focus such that there's increased likelihood to harm children, then society should make the AI generated version illegal too.
I'm willing to bet it'll differ from person to person, to complicate matters further
I think the general consensus is that availability of CSAM is bad, because it desensitizes and makes harming of actual children more likely. But I must admit that I only remember reading about that and don't have a scientific source.
It's interesting your bring this up. Not long ago I was having basically this exact same discussion with my brother. Baring you second point, I honestly don't know how I feel.
On the one hand - if it's strictly images for himself and it DOES dissuade seeking out real CSAM (I'm not convinced of this) then I don't really see the issue.
On the other hand - I feel like it could be a gateway to something more (your second point). Kinda like a drug, right? You need a heavier and heavier hit to keep the same high. Seems like it wouldn't be a stretch to go from AI generated imagery to actual CSAM.
But yeah, I don't know. We live in an odd time for sure.
I'm fine with it just being illegal, but realistically you could just ban the transmission and distribution of it and then you cover enforceable scenarios. You can police someone sending or posting that stuff, it's probably next to impossible to police someone generating it at home.
Agreed. And props for making a point that isn't palatable. The first one is complicated. Not many folk I talk to can set aside their revulsion and consider the situation logically. I wish we didn't have to in the first place.
Fuck that guy first of all.
What makes me think is, what about all that cartoon porn showing cartoon kids? What about hentai showing younger kids? What's the difference if all are fake and being distributed online as well?
Not defending him.
I think there’s certainly an argument here. What if the hentai was more lifelike? What if the AI stuff was less realistic? Where’s the line?
At least in the US, courts have been pretty shitty at defining things like “obscenity”. This AI stuff might force them to delineate more clearly.
What if someone draws their own CSAM and they're terrible at drawing but it's still recognizable as CSAM?
Ethically is one question, but the law is written such that it's pretty narrowly covering only photograph-style visual depictions that are virtually indistinguishable from an actual child engaged in explicit conduct in the view of a reasonable person that is also lacking in any other artistic or cultural significance.
Or in short: if it looks like an actual image of actual children being actually explicit, then it's illegal.While I think Hentai showing that stuff is disgusting AI is worse because you need to get the training material from somewhere so its far from victimless. Edit: I just learned that it does not have to be in the dataset though there should be regulations that forces the companies to open source the data set.
It's not what's used by all of them, but it's pretty popular.
One thing to consider, if this turned out to be accepted, it would make it much harder to prosecute actual csam, they could claim "ai generated" for actual images
I hate the future
that was just the present
I find it interesting that the relabeling of CP to CSAM weakens their argument here. "CP generated by AI is still CP" makes sense, but if there's no abusee, it's just CSM. Makes me wonder if they would have not rebranded if they knew about the proliferation of AI pornography.
The problem is that it abets the distribution of legitimate CSAM more easily. If a government declares "these types of images are okay if they're fake", you've given probable deniability to real CSAM distributors who can now claim that the material is AI generated, placing the burden on the legal system to prove it to the contrary. The end result will be a lot of real material flying under the radar because of weak evidence, and continued abuse of children.
Better to just blanket ban the entire concept and save us all the trouble, in my opinion. Back before it was so easy to generate photorealistic images, it was easier to overlook victimless CP because illustrations are easy to tell apart from reality, but times have changed, and so should the laws.
Not necessarily. There's been a lot of advances in watermarking AI outputs.
As well, there's the opposite argument.
Right now, pedophile rings have very high price points to access CSAM or require users to upload original CSAM content, adding a significant motivator to actually harm children.
The same way rule 34 artists were very upset with AI being able to create what they were getting commissions to create, AI generated CSAM would be a significant dilution of the market.
Is the average user really going to risk prison, pay a huge amount of money or harm a child with an even greater prison risk when effectively identical material is available for free?
Pretty much overnight the CSAM dark markets would lose the vast majority of their market value and the only remaining offerings would be ones that could demonstrate they weren't artificial to justify the higher price point, which would undermine the notion of plausible deniability.
Legalization of AI generated CSAM would decimate the existing CSAM markets.
That said, the real question that needs to be answered from a social responsibility perspective is what the net effect of CSAM access by pedophiles has on their proclivity to offend. If there's a negative effect then it's an open and shut case that it should be legalized. If it's a positive effect than we should probably keep it very much illegal, even if that continues to enable dark markets for the real thing.
I agree, just the linguistics are interesting.
Have to agree. Because I have no clue what CSAM is. My first glance at the title made me think it was CSPAN (the TV channel)... So CP is better identifier, as of at least recognize the initialism.
If we could stop turning everything, and especially important things, into acronyms and initialisms that'd be great.
A generative AI could not generate CSAM without access to CSAM training data. Abuse was a necessary step in the generation.
Who's even in charge of that lol
Can't generate Abuse Material without Abuse. Generative AI does not need any indecent training to be able to produce indecent merial.
But it is a nice story to shock and scare many people so i guess the goal is reached.
oh man, i love the future, we havent solved world hunger, or reduce carbon emissions to 0, and we are on the brink of a world war, but now we have AI's that can generate CSAM and fake footage on the fly 💀
Technically we've solved world hunger. We've just not fixed it, as the greedy fucks who hoard most of the resources of this world don't see immediate capital gains from just helping people.
Pretty much the only real problem is billionaires being in control.
Honestly not as bad as I would have thought it would be by now with fake propaganda videos, but the quality isn't there yet I suppose.
Quick things to note.
One, yes, some models were trained on CSAM. In AI you'll have checkpoints in a model. As a model learns new things, you have a new checkpoint. SD1.5 was the base model used in this. SD1.5 itself was not trained on any CSAM, but people have giving additional training to SD1.5 to create new checkpoints that have CSAM baked in. Likely, this is what this person was using.
Two, yes, you can get something out of a model that was never in the model to begin with. It's complicated, but a way to think about it is, a program draws raw pixels to the screen. Your GPU applies some math to smooth that out. That math adds additional information that the program never distinctly pushed to your screen.
Models have tensors which long story short, is a way to express an average way pixels should land to arrive at some object. This is why you see six fingered people in AI art. There wasn't any six fingered person fed into the model, what you are seeing the averaging of weights pushing pixels between two different relationships for the word "hand". That averaging is adding new information in the expression of an additional finger.
I won't deep dive into the maths of it. But there's ways to coax new ways to average weights to arrive at new outcomes. The training part is what tells the relationship between A and C to be B'. But if we wanted D' as the outcome, we could retrain the model to have C and E averaging OR we could use things call LoRAs to change the low order ranking of B' to D'. This doesn't require us to retrain the model, we are just providing guidance on ways to average things that the model has already seen. Retraining on C and E to D' is the part old models and checkpoints used to go and that requires a lot of images to retrain that. Taking the outcome B' and putting a thumb on the scale to put it to D' is an easier route, that just requires a generalized teaching of how to skew the weights and is much easier.
I know this is massively summarizing things and yeah I get it, it's a bit hard to conceptualize how we can go from something like MSAA to generating CSAM. And yeah, I'm skipping over a lot of steps here. But at the end of the day, those tensors are just numbers that tell the program how to push pixels around given a word. You can maths those numbers to give results that the numbers weren't originally arranged to do in the first place. AI models are not databases, they aren't recalling pixel for pixel images they've seen before, they're averaging out averages of averages.
I think this case will be slam dunk because highly likely this person's model was an SD1.5 checkpoint that was trained on very bad things. But with the advent of being able to change how averages themselves and not the source tensors in the model work, you can teach new ways for a model to average weights to obtain results the model didn't originally have, without any kind of source material to train the model. It's like the difference between Spatial antialiasing and MSAA.
It is amazing how Lemmy can usually be such a well informed audience but for some reason when it comes to AI people simply refuse to acknowledge that it was trained on CSAM https://cyber.fsi.stanford.edu/news/investigation-finds-ai-image-generation-models-trained-child-abuse
And don't understand how generative AI combines existing concepts to synthesize images - it doesn't have the ability to create novel concepts.
AI models don't resynthesize their training data. They use their training data to determine parameters which enable them to predict a response to an input.
Consider a simple model (too simple to be called AI but really the underlying concepts are very similar) - a linear regression. In linear regression we produce a model which follows a straight line through the "middle" of our training data. We can then use this to predict values outside the range of the original data - albeit will less certainty about the likely error.
In the same way, an LLM can give answers to questions that were never asked in its training data - it's not taking that data and shuffling it around, it's synthesising an answer by predicting tokens. Also similarly, it does this less well the further outside the training data you go. Feed them the right gibberish and it doesn't know how to respond. ChatGPT is very good at dealing with nonsense, but if you've ever worked with simpler LLMs you'll know that typos can throw them off notably... They still respond OK, but things get weirder as they go.
Now it's certainly true that (at least some) models were trained on CSAM, but it's also definitely possible that a model that wasn't could still produce sexual content featuring children. It's training set need only contain enough disparate elements for it to correctly predict what the prompt is asking for. For example, if the training set contained images of children it will "know" what children look like, and if it contains pornography it will "know" what pornography looks like - conceivably it could mix these two together to produce generated CSAM. It will probably look odd, if I had to guess? Like LLMs struggling with typos, and regression models being unreliable outside their training range, image generation of something totally outside the training set is going to be a bit weird, but it will still work.
None of this is to defend generating AI CSAM, to be clear, just to say that it is possible to generate things that a model hasn't "seen".
Okay for anyone who might be confused on how a model that's not been trained on something can come up with something it wasn't trained for, a rough example of this is antialiasing.
In the simplest of terms antialiasing looks at a vector over a particular grid, sees what percentage it is covering, and then applies that percentage to to shade the image and reduce the jaggies.
There's no information to do this in the vector itself, it's the math that is what is giving the extra information. We're creating information from a source that did not originally have it. Now, yeah this is really simple approach and it might have you go "well technically we didn't create any new information".
At the end of the day, a tensor is a bunch of numbers that give weights to how pixels should arrange themselves on the canvas. We have weights that show us how to fall pixels to an adult. We have weights that show us how to fall pixels to children. We have weights that show us how to fall pixels to a nude adult. There's ways to adapt the lower order ranking of weights to find new approximations. I mean, that's literally what LoRAs do. I mean that's literally their name, Low-Rank Adaptation. As you train on this new novel approach, you can wrap that into a textual inversion. That's what that does, it allows an ontological approach to particular weights within a model.
Another way to think of this. Six finger people in AI art. I assure you that no model was fed six fingered subjects, so where do they come from? The answer is that the six finger person is a complex "averaging" of the tensors that make up the model's weights. We're getting new information where there originally was none.
We have to remember that these models ARE NOT databases. They are just multidimensional weights that tell pixels from a random seed where to go to in the next step in the diffusion process. If you text2image "hand" then there's a set of weights that push pixels around to form the average value of a hand. What it settles into could be a four fingered hand, five fingers, or six fingers, depends on the seed and how hard the diffuser should follow the guidance scale for that particular prompt's weight. But it's distinctly not recalling pixel for pixel some image it has seen earlier. It just has a bunch of averages of where pixels should go if someone says hand.
You can generate something new from the average of complex tensors. You can put your thumb on the scale for some of those weights, give new maths to find new averages, and then when it's getting close to the target you're after use a textual inversion to give a label to this "new" average you've discovered in the weights.
Antialiasing doesn't feel like new information is being added, but it is. That's how we can take the actual pixels being pushed out by a program and turn it into a smooth line that the program did not distinctly produce. I get that it feels like a stretch to go from antialiasing to generating completely novel information. But it's just numbers driving where pixels get moved to, it's maths, there's not really a lot of magic in these things. And given enough energy, anyone can push numbers to do things they weren't supposed to do in the first place.
The way models that come from folks who need their models to be on the up and up is to ensure that particular averages don't happen. Like say we want to avoid outcome B', but you can average A and C to arrive at B'. Then what you need is to add a negative weight to the formula. This is basically training A and C to average to something like R' that's really far from the point that we want to avoid. But like any number, if we know the outcome is R' for an average of A and C, we can add low rank weights that don't require new layers within the model. We can just say, anything with R' needs -P' weight, now because of averages we could land on C' but we could also land on A' or B' our target. We don't need to recalculate the approximation of the weights that A and C give R' within the model.
I think there are two arguments going on here, though
- It doesn't need to be trained on that data to produce it
- It was actually trained on that data.
Most people arguing point 1 would be willing concede point 2, especially since you linked evidence of it.
it was trained on CSAM
In that case, why haven't the people who made the AI models been arrested?
Dunno, probably because they didn't knowingly train it on CSAM - maybe because it's difficult to prove what actually goes into neural network configuration so it's unclear how strongly weighted it is... and lastly, maybe because this stuff is so cloaked in obscurity and proprietaryness that nobody is confident how such a case would go.
And don’t understand how generative AI combines existing concepts to synthesize images - it doesn’t have the ability to create novel concepts.
Imagine someone asks you to shoop up some pr0n showing Donald Duck and Darth Vader. You've probably never seen that combination in your "training set" (past experience) but it doesn't exactly take creating novel concepts to fulfill the request. It's just combining existing ones. Web search on "how stable diffusion works" finds some promising looking articles. I read one a while back and found it understandable. Stable Diffusion was the first of these synthesis programs but the newer ones are just bigger and fancier versions of the same thing.
Of course idk what the big models out there are actually trained on (basically everything they can get, probably not checked too carefully) but just because some combination can be generated in the output doesn't mean it must have existed in the input. You can test that yourself easily enough, by giving weird and random enough queries.
No, you're quite right that the combination didn't need to exist in the input for an output to be generated - this shit is so interesting because you can throw stuff like "A medieval castle but with Iranian architecture with a samurai standing on the ramparts" at it and get something neat out. I've leveraged AI image generation for visual D&D references and it's excellent at combining comprehended concepts... but it can't innovate a new thing - it excels at mixing things but it isn't creative or novel. So I don't disagree with anything you've said - but I'd reaffirm that it currently can make CSAM because it's trained on CSAM and, in my opinion, it would be unable to generate CSAM (at least to the quality level that would decrease demand for CSAM among pedos) without having CSAM in the training set.
Good
The cats out of the bag on this. It's enforceable for now to try and ban it, maybe. Because the models are mostly online and intensive.
In 2028 though, when you can train your own model and generate your own local images without burning a server farm? This has to happen for ML to keep growing and catch on.
welp. Then there is infinite fake child porn. Because you cannot police every device and model.
Because of how tech companies have handled this technology, this is not an if scenario. This is guaranteed now.
Because you cannot police every device and model.
FBI: "Challenge accepted. Hey Microsuck, let's collaborate on a windows feature that records everything you do."
Microsuck: "one step ahead of you. We already have it." (Not a joke.)
You cannot force people to use Micro$oft. But I'm sure that it would only increase market share for them because it will be mediatized in a way that depicts non-privacy invading operating systems as morally evil because good guys don't have anything to hide. Kinda like they did with pleading the fifth and shifting the public image of doing so being a silent admission to having committed a crime.
Uhhh these types of images kinda already require local models...
I remember when they tried to do the same with CRISPR. Glad that didn't take off and remained largely limited to the industry and academia. But then again, Wuhan ...
I wanna know if this applies to copyrighted content as well. For example, if by any chance a whole ass book was outputted by a LLM, does the output retain the original copyright?
If it completely rewrites a book whose copyright is owned by a large corporation or publishing company in the US, they'll probably take whatever company respond for it if it's a public LLM behind the shed and shoot them to death with legal battles. So, I'm gonna assume yes.
I sure hope so. It is important because otherwise copyright will mean jackshit.
*Rant I truly hope politicians spend their time on more pressing issues than squabbling among themselves. Climate change, technological advancement that outpaces our legal framework, consumer protection. So much shit to do.
Is CSAM a commonly used term? I had to look it up.
It is
I read that its more accurate to say "child sexual abuse material" than child porn because it carries the message of just how bad the stuff is better than just calling it porn and it sounds more professional
And I suppose it’s also saying that the form it’s in doesn’t matter. Any type of material is the same.