muskrat's data eng expert's hard drive overheats while processing 60k rows

Wow.

I've been processing a couple of billion rows of data on my machine, the fans didn't even come on. WTF are they teaching "experts" these days, or has Elmo only hired people who claim that they can "wrangle data" and say "yes" ?

From the same group that doesn't understand joins and thinks nobody uses SQL this is hardly surprising .

Probably got an LLM running locally and asking it to get data which is then running 10 level deep sub queries to achieve what 2 inner joins would in a fraction of the time.

60k isn't that much, I frequently run scripts against multiple hundreds of thousands at work. Wtf is he doing? Did he duplicate the government database onto his 2015 MacBook Air?

Unless I'm misreading it which is possible it's awfully late, he said he processed 60,000 rows didn't find what he was looking for but his hard drive overheated on the full pass.

Discs don't overheat because there was load. Even if he f***** up and didn't index the data correctly (I assume it's a relational database since he's talking about rows) The disc isn't just going to overheat because the job is big. It's going to be lack of air flow or lack of heatsink.

I guarantee you he was running on an external NVMe, and one of those little shitty-ass Chinese enclosures. Or maybe one of those self immolating SanDisk enclosures. Hell, maybe he's on a desktop and he slept a raw NVMe on his motherboard without a heatsink

There are times when you want a brilliant college student on your team, But you need seasoned professionals to help them through the things they've never seen before and never done before.

60k of rows is nothing. Fuck, where do you find these “geniuses”?

my hard drive overheated

So, this means they either have a local copy on disk of whatever database they're querying, or they're dumping a remote db to disk at some point before/during/after their query, right?

Either way, I have just one question - why?

Edit: found the thread with a more in-depth explanation elsewhere in the thread: https://xcancel.com/DataRepublican/status/1900593377370087648#m

So yeah, she's apparently toting around an external hard drive with a copy of the "multiple terabytes" large US spending database, running queries against it, then dumping the 60k-row result set to CSV for further processing.

I'm still confused at what point the external drive overheats, even if she is doing all this in a "hot humid" hotel room that she can't run any fans I guess because her kids were asleep?

But like, all of that just adds more questions, and doesn't really answer the first one - why?

I'm a data engineer that processes 2 billion row 3000 column datasets every day, and I open shit in Excel with more than 60k rows. What the hell is this chick talking about?

This cannot be real, wtf. This is cartoon levels of ineptitude.

Or sabotage by someone heading out? Please let this be resistance sabotage they haven’t noticed yet.

This sounds like trying to do stuff in Excel? The computer isn't overheating but the amount of memory needed is very high which would make it run poorly. They might interpret that as overheating?

This shit sounds like when your mom tells you that the Facebook printed out her bank statement on the fax machine. I'm not smart enough to even guess how you did something dumb enough to make that happen.

How bad are you at writing queries? How does your hard drive overheat even under 100% load? Do you have it smothered under a blanket? Did you crack it up and expose it to cheeto dust? What does running a query on your, presumably, remote database even have to do with your harddrive in the first place? Are you trying to copy the entire database locally to a laptop? Do you know how to tie your shoes yet, or are you still on the velcro?

Maybe

Maybe

They are just making shit up and doing jack shit

You're not supposed to place your laptop directly in the lap of your fur suit. Always leave an air gap for ventilation, smh.

Good lord.

These people cannot do half of what she can do.

@molly0xfff@hachyderm.io

When the only thing that is stopping kids from dismantling your government is an O(N^N) algorithm

As a reasonably experienced "data guy," this seems obviously laughable, but the discussion on X is scary. This guy is a savior in the MAGA world.

We can criticize and poke fun all day, but it doesn't matter much if our message isn't challenging the mindset of those with other opinions.

How do we make better use of our time to impact outside opinion?

I cannot believe these people make more than me lol.

I will now attempt a full comment ... brace your hard drives

60k lol.

I regularly work with data in the 16tb range and weirdly my computer is fine. Git gud, doge scrubs.

Either she knows something novel, where processing data using voice coils is somehow beneficial, or is someone who calls their computer a ‘hard drive’, which summarily negates any legitimacy of technical competence.

What is this, a table for ants? Because that’s the average number of ants in an ant colony and it’s nowhere near an impressive amount of rows to be doing any sort of processing on. It wouldn’t be an impressive amount of rows if your rig was an i386DX-33 running off a 5” floppy.

Skill issue, as the kids like to say.

I used to perform data analysis of robotics firmware logs which would generate several million log lines per hour and that was my second job out of college.

I don't know how you fuck up 60k lines that bad. Is he nesting 150 for loops and loading a copy of the data set in each one while mining crypto??

60k rows of anything will be pulled into the file cache and do very little work on the drive. Possibly none after the first read.

Hard drive was made by Tesla

just a lame-ass excuse for not finding whatever evidence they were looking for.

elsewhere, some seeding was done.

now they'll do the 'full' data grab and 'find' what they were looking for.

No matter what actually happened here we can confidently state this clown is full of shit and has no idea what he's doing.

They even doubled down: https://xcancel.com/DataRepublican/status/1900565922831618202#m

What’s the bet the software they downloaded is malware and it’s crypto mining?

Is this a real post? I can’t seemed to find it on that website “X, formerly known as Twitter.”

“I store my records on vinyl. You’ve probably never heard of them.”

Maybe he left his shed lizard skin on top of the hard drive that caused the overheating?

Hard drive? How old is his computer?

IT guy checking in.

The only time I've even seen drive temp sensor alarms is on server raid arrays and other similar hard drives/SSDs.... Never in my life have I seen one available on a consumer device, nor have I seen any alarm for and drive temp, go off. It just doesn't happen.

IMO, this is one of those language barriers where people call their computer chassis (and everything in it) the "hard drive".

Applying that assumption, their updated statement is: His computer over heated.

Idk what kind of shit system he's running on that 60k rows would cause overheating, but ok.

I didn't know hard drive overheating was a thing. Should I be worried that my 5 year old hard drive is about to overheat. I mean is this actually a floppy disk or something?

The trick is to take the hard drive out of the EVA foam.

What in the fuck is this idiot doing? I've process datasets far larger than that and never once have I run into a hard drive "overheat". I mean what level of incompetence do you have to have to get a hard drive to overheat processing a measley 60K rows of data?

does elon only hire chip from sales guy vs web dude or something

i mean...

If you're running a pcie nvme ssd, one of the modern ones, and you're doing a SHIT ton of reads, like threadripper level amount of reads, i guess "overheating" isn't unexpected? Shouldn't do much other than slow down the SSD though?

dumbass probably loaded them into memory, and OOM'd, and thought it was the drive.