scruiser @ scruiser @awful.systems

Posts

5

Comments

271

Joined

2 yr. ago

It's a good post. A few minor quibbles:

The “nonprofit” company OpenAI was launched under the cynical message of building a “safe” artificial intelligence that would “benefit” humanity.

I think at least some of the people at launch were true believers, but strong financial incentives and some cynics present at the start meant the true believers didn't really have a chance, culminating in the board trying but failing to fire Sam Altman and him successfully leveraging the threat of taking everyone with him to Microsoft. It figures one of the rare times rationalists recognize and try to mitigate the harmful incentives of capitalism they fall vastly short. OTOH... if failing to convert to a for-profit company is a decisive moment in popping the GenAI bubble, then at least it was good for something?

These tools definitely have positive uses. I personally use them frequently for web searches, coding, and oblique strategies. I find them helpful.

I wish people didn't feel the need to add all these disclaimers, or at least put a disclaimer on their disclaimer. It is a slightly better autocomplete for coding that also introduces massive security and maintainability problems if people entirely rely on it. It is a better web search only relative to the ad-money-motivated compromises Google has made. It also breaks the implicit social contract of web searches (web sites allow themselves to be crawled so that human traffic will ultimately come to them) which could have pretty far reaching impacts.

One of the things I liked and didn't know about before

Ask Claude any basic question about biology and it will abort.

That is hilarious! Kind of overkill to be honest, I think they've really overrated how much it can help with a bioweapons attack compared to radicalizing and recruiting a few good PhD students and cracking open the textbooks. But I like the author's overall point that this shut-it-down approach could be used for a variety of topics.

One of the comments gets it:

Safety team/product team have conflicting goals

LLMs aren't actually smart enough to make delicate judgements, even with all the fine-tuning and RLHF they've thrown at them, so you're left with over-censoring everything or having the safeties overridden with just a bit of prompt-hacking (and sometimes both problems with one model)/1

Had me in the first few paragraphs…not gonna lie.

Yeah, the first few paragraphs actually felt like they would serve as a defense of Hamas: Israel engineered a situation were any form of resistance against them would need to be violent and brutal so Hamas is justified even if it killed 5 people to save 1.

The more I think about his metaphor the more frustrated I get. Israel holds disproportionate power in this entire situation, if anyone is contriving no-win situations to win temporary PR victories it is Israel (Netanyahu's trial is literally getting stalled out by the conflict).

Lots of woo and mysticism already has a veneer of stolen Quantum terminology. It's too far from respectable to get the quasi-expert endorsement or easy VC money that LLM hype has gotten, but quantum hucksters fusing quantum computing nonsense with quantum mysticism can probably still con lots of people out of their money.

I like how Zitron does a good job of distinguishing firm overall predictions from specific scenarios (his chaos bets) which are plausible but far from certain. AI 2027 specifically conflated and confused those things in a way that gave it's proponents more rhetorical room to hide and dodge.

I like how he doesn't even bother debunking it point by point, he just slams the very premise of it and moves on.

system memory

System memory is just the marketing label for "having an LLM summarize a bunch of old conversations and shoving it into a hidden prompt". I agree that using that term is sneer-worthy.

I have three more examples of sapient marine mammals!

• whales warning the team of an impending solar flare in Stargate Atlantis via echolocation induced hallucinations
• the dolphins in hitchhiker’s guide to the galaxy
• whales showing up to help in one book of Animorphs while they are morphed into dolphins

I was thinking this also, like it's the perfect parody of several lesswrong and EA memes: overly concerned with animal suffering/sapience, overly concerned with IQ stats, openly admitting to no expertise or even relevant domain knowledge but driven to pontificate anyway, and inspired by existing science fiction... I think the last one explains it and it isn't a parody. As cinnasverses points out, Cetacean intelligence shows up occasionally in sci-fi. to add to the examples... sapient whales warning the team of an impending solar flare in Stargate Atlantis via echolocation induced hallucinations, the dolphins in hitchhiker's guide to the galaxy, and the whales showing up to help in one book of Animorphs.

I was trying to figure out why he hadn't turned this into an opportunity to lecture (or write a mini-fanfic) about giving more attack surface to the AGI to manipulate you... I was stumped until I saw your comment. I think that is it, expressing his childhood distrust of authority trumps lecturing us on the AI-God's manipulations.

I have context that makes this even more cringe! "Lawfulness concerns" refers to like, Dungeons and Dragons lawfulness. Specifically the concept of lawfulness developed in the Pathfinder fanfiction we've previously discussed (the one with deliberately bad BDSM and eugenics). Like a proper Lawful Good Paladin of Iomedae wouldn't put you in a position where you had to trust they hadn't rigged the background prompt if you went to them for spiritual counseling. (Although a Lawful Evil cleric of Asmodeus totally would rig the prompt... Lawfulness as a measuring stick of ethics/morality is a terrible idea even accepting the premise of using Pathfinder fanfic to develop your sense of ethics.)

Well, this explains how KP manages to claim Scott Alexander is center left with a straight face, she has no clue about basic leftist thought or even what the fuck leftism is! Like another comment said, she has enough sense to know the right-wing is full of shitheads, and so doesn't want to squarely acknowledge how aligned with them she is.

you can’t have an early version that you’ll lie about being a “big step towards General Quantum Computing” or whatever

So you might think that... but I recall some years ago an analog computer was labeled as quantum annealing or something like that... oh wait, found the wikipedia article: https://en.wikipedia.org/wiki/Quantum_annealing and https://en.wikipedia.org/wiki/D-Wave_Systems . So it sounds to a naive listener like the same sort of thing as the quantum computers that are supposed to break cryptography and even less plausible things, but actually it can only do one very specific algorithm.

I bet you could squeeze the "quantum" label onto a variety of analog computers well short of general quantum computing and have it technically not be fraud and still fool lots of idiot VCs!

It's a nice master post that gets all his responses and many useful articles linked into one place. It's all familiar if you've kept up with techtakes and Zitron's other posts and pivot-to-ai, but I found a few articles I had previously missed reading.

Related trend to all the but achskhually's AI booster's like to throw out. Has everyone else noticed the trend where someone makes a claim of a rumor they heard about an LLM making a genuine discovery in some science, except it's always repeated second hand so you can't really evaluate it, and in the rare cases they do have a link to the source, it's always much less impressive than they made it sound at first...

Even for the people that do get email notifications of Zitron's excellent content (like myself), I appreciate having a place here to discuss it.

j/k he’s doubling down on being a dick.

I had kind of gotten my hopes up from the comparisons of him to sneerclub that maybe he'd be funny or incisively cutting or something, but it looks mostly like typical lesswrong pedantry, just less awkwardly straining to be charitable (to the in-group).

Apparently Eliezer is actually against throwing around P(doom) numbers: https://www.lesswrong.com/posts/4mBaixwf4k8jk7fG4/yudkowsky-on-don-t-use-p-doom ?

The objections to using P(doom) are relatively reasonable by lesswrong standards... but this is in fact once again all Eliezer's fault. He started a community centered around 1) putting overconfident probability "estimates" on subjective uncertain things 2) need to make a friendly AI-God, he really shouldn't be surprised that people combine the two. Also, he has regularly expressed his certainty that we are all going to die to Skynet in terms of ridiculously overconfident probabilities, he shouldn't be surprised that other people followed suit.

Lesswrong and SSC: capable of extreme steelmanning of... check notes... occult mysticism (including divinatory magic), Zen-Buddhism based cults, people who think we should end democracy and have kings instead, Richard Lynn, Charles Murray, Chris Langan, techbros creating AI they think is literally going to cause mankind's extinction...

Not capable of even a cursory glance into their statements, much less steelmanning: sneerclub, Occupy Wallstreet

we cant do basic things

That's giving them too much credit! They've generated the raw material for all the marketing copy and jargon pumped out by the LLM companies producing the very thing they think will doom us all! They've served a small but crucial role in the influence farming of the likes of Peter Thiel and Elon Musk. They've served as an entry point to the alt-right pipeline!

dath ilan?

As a self-certified Eliezer understander, I can tell you dath ilan would open up a micro-prediction market on various counterfactual ban durations. Somehow this prediction market would work excellently despite a lack of liquidity and multiple layers of skewed incentives that should outweigh any money going into it. Also Said would have been sent to a reeducation camp, quiet city and sterilized denied UBI if he reproduces for not conforming to dath ilan's norms much earlier.

That too.

And judging by how all the elegantly charitably written blog posts on the EA forums did jack shit to stop the second manifest conference from having even more racists, debate really doesn't help.

Yes, thanks. I always forget how many enters i need to hit.