AI scam calls imitating familiar voices are a growing problem – here's how they work
AI scam calls imitating familiar voices are a growing problem – here's how they work
Do you guys remember when the T-1000 did this?
What's wrong with Wolfie? I can hear him barking...
Your parents are dead
In Terminator 1 the T-800 made a scam call to Sarah in order to find out where she is. He deepfaked the voice of Sarah’s mother, and she fell for it.
As someone who has an uncanny ability to recognize voices, I'm skeptical about how good these really are. Of course, most people don't share that ability.
Meanwhile, I could probably be fooled by a picture.
Hmm, I understand your sentiment, but how would you know. Of course you'd pick out the bad dupes but this technology is getting really good that I fear it would go unnoticed, especially if they keep the detectable ones to reinforce bias
I always thought being able to recognize voices is a common skill? Is it not?
Very much not, in my experience.
Yeah, does a familiar voice mean a famous person or personal friend?
For me, it could be either. Some of us recognize people by their voices more than by their faces.
I don’t have examples but having listened to some samples of various Ai generated clones (the one paper had samples of I believe 10s, 30s, 1min, 5 min) and all of them progressively sounded better. The 10 second one basically sounded like a voice call whose bit rate dropped out mid word. And the voice so long as you used words that were similar in phoenix sounded pretty close. Although this is just my experience, but to you it might sound pretty bad while to me it sounded pretty reasonable if under bad audio conditions.
https://github.com/CorentinJ/Real-Time-Voice-Cloning
This is the main one I’ve seen examples of. You’ll have to find the samples yourself, I believe it was in the actual paper?
That code was state of the art (for free software) when the author first published it with his master's thesis four years ago, but it hasn't improved a lot since then and I wouldn't recommend it today. See the Heads Up section of the readme. Coqui (a free software Mozilla spinoff) is better but also is sadly still nowhere near as convincing as the proprietary stuff.
Whomever is stupid enough to think that Tom Hanks is calling you personally probably needs a court appointed guardian.
Did you read the article? It’s talking about taking kids voices from TikTok and shit. Social media. People have been posting videos of themselves talking for years. That’s enough data to train an ai to leave a message saying, “mom, I lost my phone and I’m in trouble. I need some money.” Or something of that sort. It’s been happening for a long time. This is only making it more confincing
I'm so fucking glad that I've hardly ever had my voice and likeness posted publicly on the internet
The reference of the entire article is talking about scammers using AI models of voice you know and understand. None of these scam rings have the time to break it down to your family.
enough data
To be clear, about three seconds of your voice is "enough".
Unless you actually know Tom Hanks personally and are expecting a call from him, of course.
Good luck criminals. I ignore nearly every call.
Anyone know how many hours of training data it takes to build up a convincing model of someone’s voice? It was 10’s of hours when I did a bit of research a year ago… the article says social media is the likely source of training data for these scams, but that seems unlikely at this point.
Is it safe to assume that if you don't have any family that posts videos to Facebook/socials you are in a safer place?
The technology has clearly come a long way in a short time, really fascinating.
I remember the first examples I read about being trained with celebrity read audiobooks because they needed so much audio data. I want to say Tom Hanks or Anthony Hopkins but I could have that confused with something else.
A current state of the art ai model from Microsoft can achieve acceptable quality with about 3 seconds of audio. Commercially available stuff like eleven labs about 30 minutes. But quality will obviously vary heavily but then again they're using a low quality phone call so maybe not that important
That’s downright scary :-) I think it took longer in the last Mission Impossible.
30 minutes is still pretty minimal for the kind of targeted attack it sounds like this is used for. I suppose we all need to work with our families on code words or something.
I went in thinking the article was a bit alarmist, but that’s clearly not the case. Thank for the insight.
With that little, they may be able to recreate the timbre of someone's voice, but speech carries a multitude of other identifiers and idiosyncrasies that they're unlikely to get with that little audio, like personal vocabulary (we don't choose the same words and phrasings for things), specific pronunciations (e.g. "library" vs "libary"), voice inflections, etc. Obviously, the more training data you have, the better the output.
ElevenLabs only needs 1 minute, but it also works with even shorter clips.
Wow! That’s really impressive.
TorToiSe can work off of just three ten second clips when you're using a pre-trained model. No telling if that'll sound any good.
I’ll have to check that out, thanks for the link.
The most advanced Model I know just needs half an hour of your voice or sth.
Someone else mentioned that Microsoft has one capable of working with far less material.
But 30 minutes is definitely short enough to make this sort of scam/attack feasible in my mind.
The Industrial Revolution and its consequences have been a disaster for the human race.
BADONK!
I cloned my own voice to prank a friend, and... Wow, it was a gut-dropping moment when I understood just how dangerous this tool is for precisely this type of scam.
It's one thing to hear about it, but to actual experience it... Terrifying.
Mind sharing more info about the prank? Sounds like an interesting story
Oh, it was nothing more than just showing off the technology, really. It wasn't a committed bit.
I cloned my voice then left a voicemail that said something like: "hey buddy it's me. My car broke down and I'm at... Actually I don't know where I'm at. I walked to the gas station and borrowed this guy's phone. He said he'll give me a ride into to town if I can get him $50 bucks. Could you venmo it to him at @franks_diner? I'll get you back as soon as I can find my phone. ... By the way this is really me, definitely not a bot pretending to be me."
We neeeed a guide 👋
Check out ElevenLabs.