SE Gyges (@segyges.bsky.social)
bsky.app
I'm back, I missed you all. Since this is an important subject I know too much about, here's a 🧵 explaining that AI Safety Is Mostly A Sex Cult. I don't think these people should make policy. (1/?) (alternative title: Time For Some Cult Theory)
  • chicken@lemmy.dbzer0.comEnglish
    1·
    1 day ago

    I’m not used to bluesky but wow do you have to find and click the “read more replies” buttons a lot of times to get the whole 53 post essay on one page here. This part seems like the meat of the argument to me, setting aside the earlier elaboration on how everything comes back to Eliezer Yudkowski and the claim that organizations affiliated with him can be summarized as a sex cult:

    Somewhere, some rationalist-brained person is reading this and saying I’m a “low decoupler”, and that whether they run a sex cult has nothing to do with whether they’re correct. 41/53

    This is true enough; but the sex cult is reasonable evidence that their influence isn’t prevalent because they’re right, and that they would have ulterior motives for continuing to preach that the world is coming to an end even if evidence showed otherwise. 42/53

    If you’ve got a sex cult and a nice compound that was given to you for free with stolen money because you preached the gospel of doom really well, how could you stop preaching it?

    If elaborating upon doomsday prophesies is worth money – and it is – you will get more of them. 43/53

    This all most recently came together over the weekend, and they got Dario Amodei and Sam Altman and Elon Musk and Demis Hassabis all onto the same page to say they wanted to WAIVE ANTI-TRUST LAWS so that they could “pace”, or slow down, research. 44/53

    I feel about this roughly the same way I would feel about all the pharmaceutical companies getting together to say they wanted to waive anti-trust laws so they could form a cartel to ban medical research:

    these people are the enemies of humanity. 45/53

    Also: Much of AI research IS medical research. They are very literally proposing to stop a pretty good chunk of medical research in its tracks so that they don’t have to fear the paperclipper. 46/53

    …

    AI policy is only considered in these terms because of the vast intellectual influence of Yudkowsky specifically, and because, downstream of him, a neurotic or religious obsession with AI doom is the primary means by which people engage with these concepts. 50/53

    It’s a reasonable point to make that these people (AI doom Rationalists and those drawing on their arguments) have a bias, and their influence has other possible explanations than them being right. Personally I really don’t like the sorts of regulations they are pushing for and think the risks of corporate control over AI are greater than the risks of runaway superintelligence. Control is an especially unsafe thing to seek here.

    Still, I’m seeing all this stuff on the topic that conspicuously avoids evaluation of those risks opting instead to focus on attacking the credibility of people who take them seriously (or claim to for self serving reasons), and that seems like a bad way to decide what to think about it. OOP points to the good AI can do to say how wrong it would be to repress it, but you can’t make that kind of argument and then avert your eyes from the elephant in the room on the other side, because an actual risk of large scale catastrophe would be a valid counterpoint to it, if not false or wildly exaggerated. The independent factual questions of where this is all heading cannot be just skipped and hand-waved away.