This is brilliant and I’m saving it and will post a link to it the next time someone at work asks why we can’t “just use AI to do it” when a ticket gets rejected for being stupid and/or unreasonable.
However:
The first is that we have some sort of intelligence explosion, where AI recursively self-improves itself, and we're all harvested for our constituent atoms […]. It may surprise some readers that I am open to the possibility of this happening, but I have always found the arguments reasonably sound.
Yeah, I gotta admit, I am surprised. Because I have not found a single reasonable argument for this horseshit and the rest of the article (as well as the others I read from their blog) does not read like it’s been written by someone who’d buy into AI foom.
Look, I know what this subject is about, but hear me out here, what if we replace the AI doomer data scientists with A.... WHO AREYOU? WHY ARE YOU IN MY HOU... ARGH
Yeah, that juxtaposition makes no sense to me. How does the machine that remixes existing text and makes it worse become anything that can "recursively self-improve"? Show your work.
You got Schmidhuber'd! A Gödel machine would fit the bill. Nobody's built one yet, but the hard part -- proof search through something like Metamath (particularly Metamath Zero) -- is long-since solved. It wouldn't take over the world, though; it would just sit in a corner and get really good at maths over the next few centuries.