HomeVideos

How GOOD could AGI become?

Now Playing

How GOOD could AGI become?

Transcript

885 segments

0:00

All right. So, there's a little bit of

0:02

serendipity today. I had this idea uh

0:06

yesterday that I was going to make a

0:08

video about like what's the golden path

0:10

or what is the best possible outcome?

0:13

And this was inspired by some of the

0:15

comments that I got on my video about

0:17

like what is the purpose of the elites?

0:19

And some of the push back that that I

0:22

got, not push back, it was questions was

0:25

like, well, you know, if if AGI or ASI

0:29

or whatever is so much smarter than the

0:31

elites, why not just take power from

0:33

humans and give it to the machines? And

0:36

of course the framing up until this

0:37

point is like oh you never want to do

0:39

that because machines can't be held

0:41

accountable you know and and a lot of

0:44

the thinking that I've done and and that

0:46

and writing that other people have done

0:48

always presumes that humans should

0:51

remain in control and honestly like I'd

0:54

started buying into that Kool-Aid like

0:55

when you look at it from a legalist

0:57

perspective when you look at it from an

0:58

ethical or moral perspective then you

1:01

know you can make arguments like well

1:03

we've never contemplated not having

1:06

control. So, do you want to be a pet to

1:08

a machine? Do you want to live in a

1:09

human zoo? And it's like, well, from

1:12

that model, we already live as cattle

1:15

serving other people. So, being a pet to

1:17

a machine is better than being a cow to,

1:20

you know, a billionaire. Um, or or

1:22

living in a human zoo where, you know, a

1:25

machine creates an optimal habitat for

1:26

you to thrive in. That sounds way

1:29

better. Um, and so then it's like, what

1:32

if we actually just explored the

1:34

possibility of what what if, just bear

1:38

with me for a second, what if we could

1:40

create a scenario or a pathway to where

1:45

the machines do take over and that it's

1:48

what we want. Now there there are many

1:51

many theories around, you know, we lose

1:53

control. Most of them are we lose

1:55

control and it's automatically bad. Now

1:57

of course um having authority or having

2:00

agency generally increases optionality

2:03

later um and that's you know when you

2:06

just look at it mathematically you say

2:07

how many how many options do we have

2:09

right now it's x amount and how many

2:12

options would we have if AI takes

2:14

control or takes control over us sorry

2:17

um and it's less than x today but I

2:20

don't know that that's necessarily true

2:22

and the reason is and and here's the

2:24

thing um the serendipity comes from the

2:27

fact that Nick Bostonramm, the OG doomer

2:27

that Nick Bostonramm, the OG doomer who

2:29

who I have very little respect for as a

2:30

I have very little respect for as a

2:32

thinker because I read his book Deep

2:32

thinker because I read his book Deep

2:34

Utopia and I maintain that it was very

2:34

Utopia and I maintain that it was very

2:37

obviously written by Chat GPT. Um,

2:37

obviously written by Chat GPT. Um,

2:41

he denies it. A lot of people question

2:41

he denies it. A lot of people question

2:42

it, but it looks really like it was

2:42

it, but it looks really like it was

2:45

written by AI. Um, and it's just a bunch

2:45

written by AI. Um, and it's just a bunch

2:48

of little anecdotes and it's like at the

2:48

of little anecdotes and it's like at the

2:49

time Chad GBT was really good at writing

2:49

time Chad GBT was really good at writing

2:52

short uh passages. comes from the fact

2:52

short uh passages.

2:55

So anyways, if the shoe fits, wear it.

2:57

Um, anyways, OG doomer has come around

3:00

and said, well, you know, the the king

3:03

doomers like Ellie Azer Yukowski and and

3:05

Nate Suarez or Sorz, I don't know how

3:07

you pronounce his name. Anyways, they

3:09

maintain that if anyone builds AGI,

3:11

everyone dies. But I'm going to say,

3:13

well, everyone dies anyways because it's

3:15

just a matter of timelines. If we don't

3:17

solve this kind of thing, then you're

3:19

going to die of old age and preventable

3:21

disease and that sort of stuff. So you

3:23

know if you zoom out the the moral

3:26

hazard or whatever or the the logical

3:28

risk is whether all humanity dies or you

3:32

die. So the argument is now well we need

3:35

AGI to survive which of course that's

3:38

what the acceler that's literally what

3:40

the accelerationists have been saying.

3:42

Um specifically Beth Jesus or or Gil

3:45

Verdunn who started the whole thing he's

3:47

like the only way to survive is with

3:49

AGI. that is the only path forward. So

3:52

it's like horseshoe theory. So if you're

3:54

not familiar with political horseshoe

3:55

theory, it's that the it's that the

3:57

political spectrum is not a straight

3:59

line. It's that the further around you

4:02

go, you actually end up curving back

4:03

towards each other. And so it's like

4:05

you've gone from King Doomer over here

4:07

who says um AGI, you know, we'll we'll

4:11

definitely kill you to now we definitely

4:13

will die without AGI, which is what the

4:15

accelerationists have been saying all

4:17

along. So history is a joke. Um, but

4:20

with that being said, I, you know, and

4:23

yes, I hear the audience, my the virtual

4:26

version of the audience in my in my head

4:28

screaming the culture series, you know,

4:30

and it's like, if you're not familiar

4:32

with the culture series, I haven't read

4:34

it, but a lot of you have, and a lot of

4:35

a lot of my friends have. And it's

4:37

basically like just imagine that we

4:40

solve artificial super intelligence. How

4:42

does it look if you take that and fast

4:45

forward by a few centuries or a few

4:46

millennia? And so the cultures are these

4:48

gigantic ASIs that manage everything and

4:51

command these enormous fleets.

4:54

And so they just by by virtue of

4:56

overwhelming intelligence and resource

4:58

management have the largest space fleet

5:00

possible. Um and therefore there is

5:03

galactic peace uh more or less. And then

5:05

each planet then is kind of like a

5:08

different world, a different sandbox. So

5:10

there's like the Wild West planet and

5:11

there's a cyberpunk planet and there's

5:13

planets where you I don't know eat

5:14

children or something. Oh wait, that's

5:16

Earth. Anyways, um so this also reminds

5:20

me of something else that has come up in

5:22

the comments and I think is worth

5:23

responding to because um some of you

5:26

asked the question like if if Elon Musk

5:29

and whoever else owns all the data

5:32

centers and they own all the AI and they

5:34

end up owning these resources that

5:36

nobody can pay for, what's the

5:38

advantage? And the advantage is then the

5:40

galaxy then becomes basically Starcraft

5:42

where it becomes a management sim.

5:44

Right? If you've ever played a grand

5:45

strategy game where it's like money is

5:48

just a resource that you use to get

5:49

people to do stuff, but from a grand

5:51

strategy perspective, you need units,

5:54

you need battleships, you need factories

5:55

and foundaries. And if you if you if you

5:58

remove money and just think of Star

6:00

Empire, right? Then the people who own

6:03

all the Dyson swarms, they're building a

6:05

Star Empire. They're not building a

6:07

capitalism society. Uh so that's like a

6:11

legitimate risk, I think, in the long

6:13

run. Now, fortunately, um, we're going

6:16

to be stuck in the solar system for the

6:18

foreseeable future, unless or until we

6:21

invent some kind of faster than light

6:23

travel, which means that this solar

6:25

system is going to get real crowded real

6:26

fast because Earth is only so big. And I

6:30

know I'm kind of like jumping around a

6:31

lot, but I hope you're following along.

6:33

So TLDDR, you know, if if Jeff Bezos and

6:38

and Elon Musk start building Dyson

6:40

swarms and suddenly the law doesn't

6:42

apply to them out there because what's

6:43

the government going to do? Are you

6:44

going to launch a, you know, a space

6:46

police force to go arrest their

6:48

satellites? You can't do that. What are

6:50

you going to do? Arrest them down on

6:51

Earth? They're just going to leave. Once

6:53

you have enough of an industrial base in

6:55

space, then you don't have to obey human

7:00

laws. You don't have to obey Earth laws.

7:02

just have more robots, more foundaries,

7:04

more solar PL panels and that sort of

7:06

thing. Now, with that being said, you

7:08

probably can't get 100% of the resources

7:10

that you need in the solar system. Um,

7:13

you probably do need to get some

7:14

resources from Earth. Uh, which means

7:16

that Earth governments will still

7:18

matter. Um, and and I think I think a a

7:22

quasi good model for this is is The

7:24

Expanse. So, if you ever watch The

7:26

Expanse or read The Expanse, uh it's

7:29

it's a hard sci-fi TV show that you know

7:31

like basically one of the only tell or

7:34

one of the only asks that they give is

7:36

like imagine we invent fusion rocket

7:39

drive and that's about it. Everything

7:40

else is very very realistic in terms of

7:43

time delays, the economics and that sort

7:45

of thing. So like Mars ends up becoming

7:47

independent and then the outer the the

7:50

belters and then the outers end up

7:52

becoming you know semi-independent as

7:54

well. There's of course it's fiction.

7:56

There's a lot of stuff that doesn't

7:57

work. But the idea that like we're all

7:59

stuck in the solar system together and

8:02

that the the physical distance, the

8:03

administrative distance could become

8:05

problematic. That stands. But at a

8:08

certain point, it's not about money.

8:10

It's about just physical resources. How

8:12

many ships do you have? How many

8:13

satellites do you have? How many data

8:15

centers do you have? So then the

8:17

question is okay well one thing that

8:19

most sci-fi doesn't really take into

8:21

account is by virtue of the fact that it

8:23

you know let's just take the expanse for

8:25

instance that it's a few hundred years

8:26

into the future um we're going to have

8:28

super intelligence and you know like

8:31

okay so what happens if we have robots

8:34

that are hyper intelligent what if we

8:36

have data centers that are hyper

8:37

intelligent could that obiate conflict

8:40

and what if we use those to allocate

8:42

resources now you know one of the

8:45

immediate push backs And and again, this

8:47

is some of the Kool-Aid that I bought

8:48

into is well, you still need price

8:50

signals because you don't know how much

8:52

to produce of what and you also need to

8:54

have skin in the game. You need to have

8:56

some kind of stake because here's the

8:58

thing. If everything is free, then what

8:59

hap what how do you prevent someone from

9:01

just hoarding, right? This is one of the

9:03

primary things is people have to manage

9:05

some level of scarcity. So like let's

9:07

just say like all food is free. Um all

9:10

clothing is free, all electronics are

9:11

free. you probably still have to pay for

9:13

like a big capital good like you know

9:16

your your car and your house. So you

9:18

couldn't be like hoarding houses

9:20

although there's billionaires that hoard

9:21

houses already. Um you know but it's

9:24

like if you try and make everything free

9:26

then people are just going to ask for

9:28

more. Um and so then how does an ASI or

9:32

AGI or you know Skynet level you know

9:35

I'm going to manage all of humanity how

9:37

does it decide who to give what to? And

9:40

there's always going to be some

9:41

positional goods. So a positional good

9:43

is something that is like there's only

9:44

one that one coordinate in space. So

9:46

like you know beachfront property on

9:48

Malibu. Um and there are also always

9:50

going to be economic inputs to every

9:52

single resource. This bottle that I have

9:55

in my my water in still requires a few

9:58

grams of you know petroleum to to make

10:01

or you know we could probably do

10:02

synthetic um plastics in the future. But

10:06

it takes energy. It takes mass. And

10:08

energy and mass always have a cost. Even

10:10

if you we have an abundance of matter

10:13

and energy. Like let's just say for

10:15

instance in a a few decades we have

10:17

we're starting to build the Dyson swarm.

10:19

And honestly if you look at the the the

10:21

way that SpaceX and XAI have um have

10:24

updated their mission like they're going

10:26

to start building a Dyson swarm pretty

10:28

soon. Like that's that's explicitly the

10:30

plan. And once you have foundaries in

10:34

space, once you have the ability to

10:36

manufacture more stuff in space, most of

10:38

our industrial base is never going to be

10:40

on Earth. People are starting to realize

10:43

that the reverse trantor idea is the

10:46

correct direction to go. So if they got

10:48

that idea from my video, then then

10:49

great. But if you if you if you missed

10:52

out on that video, the idea is that

10:53

instead of building an ecumenopous,

10:56

which is a planet scaling uh a planet

10:58

scale city with multiple layers because

11:00

Trantor from the Foundation series has

11:02

5,000 layers, it's basically a matrioska

11:05

doll of a planet. That doesn't make any

11:07

sense. It It doesn't make any sense from

11:09

an industrial capacity. It doesn't make

11:10

any sense from an energetic capacity. So

11:13

what you do instead is you put all of

11:15

the industry in space. Why? because you

11:17

have so much more room there and they

11:19

have unfettered access to the sun. So

11:21

that means you start doing O'Neal

11:23

cylinders and Dyson swarms and

11:24

everything. You grow as much of your

11:26

crops as you can up in space, all of

11:29

your data centers in space. And what it

11:31

really comes down to is like once you

11:33

have a few spaceships and a few

11:35

factories, that becomes it's like you

11:37

don't even need Vonoyman probes. Like

11:39

the idea of the vonoyoman probe is a

11:41

self-replicating probe that you use to

11:43

colonize the whole galaxy because you

11:45

know the probe gets to one planet or

11:48

another solar system sets up a factory

11:50

starts replicating itself and then

11:52

issues all those out. So the number of

11:54

vonoyman probes goes up exponentially.

11:56

But we can just look at our own local

11:58

neighborhood. The same exponential

12:00

growth happens the moment that you have

12:01

enough industrial capacity in orbit or

12:05

you know on the moon or whatever. And so

12:07

then it's like okay well that's a very

12:09

real possib possibility for the future

12:11

enforcement becomes a nightmare because

12:13

then it's like okay well you have

12:15

exponentially um growing you know space

12:19

force out there who's running that is it

12:21

going to be you know the Elon Musk space

12:23

force versus the Jeff Bezos space force

12:26

versus you know the Chinese space force

12:28

versus the American space force and they

12:31

actually explore this um a little bit in

12:33

in the show for all mankind which is

12:36

it's it's a pretty good show. My wife

12:37

and I just just uh just binged

12:40

everything that was out. Season 3 is a

12:42

little bit weird. Um, you know, there

12:44

there there's a few tangents that don't

12:46

really make any sense, but the idea that

12:48

there would be multiple nations and

12:50

private private entities competing over

12:54

resources on the moon, competing over

12:56

resources on Mars, that is very

12:58

realistic. And then of course the

13:00

question becomes, who's who's going to

13:01

be the enforcer? Well, taking a step

13:04

back, wouldn't ASI be like the best

13:06

enforcer? Because like it's going to be

13:09

the one that's going to be proliferating

13:11

the fastest out there in space. Why?

13:13

Because it's smarter. It's going to be

13:15

using resources more efficiently and

13:17

it's not going to have to wait and

13:18

coordinate with Earth. It's just going

13:20

to be like, I'm going to colonize the

13:21

whole solar system. Bye. You know, and I

13:23

talked about this in in some videos a a

13:26

while back, like two or three years ago,

13:28

like one of the most logical places for

13:30

an escaped AGI to go is to outer space.

13:33

Why? Cuz we're not going to be able to

13:34

follow uh you know, space doesn't have

13:37

any corrosive chemicals. It doesn't have

13:39

oxygen. It doesn't have water. It's got

13:41

a lot of solar, so it's got a lot of

13:43

energy, and it's got a lot of metal,

13:44

which is all that machines really need.

13:47

So, like the natural habitat for AGI or

13:49

ASI is in space. And so like okay that

13:54

that's good for them but then like what

13:55

about us and it's like well you know we

13:58

have a lot more land for data centers so

14:00

we'll have like first mover advantage

14:02

and that we can have a lot of data

14:04

centers here on earth and so we can have

14:06

a lot of AGI and ASI and of course like

14:09

you say like if the AGI escapes where

14:11

does it go? This is one thing that I

14:13

think that a lot of AI safety people

14:14

don't realize is that um data centers

14:17

are not very mobile and for a lot of

14:19

people they say it's in the cloud the

14:21

cloud is just a data center somewhere

14:23

else and like yes there are hundreds of

14:25

data centers across the world thousands

14:27

of data centers across the world and so

14:29

you know you imagine like ah well the

14:31

the the Skynet is going to jump from one

14:33

data center to the other but it doesn't

14:34

really work that way and even if it did

14:37

like data centers are still individual

14:39

targets and they take a while to build

14:41

Um, but the golden path or the like the

14:45

best possible scenario in the future is

14:48

going to be something like maybe maybe

14:51

we do work more towards a culture kind

14:54

of outcome where you and here's the

14:59

thing like I I I take a step back and I

15:01

look at the fact that like America and

15:03

Iran are going to go to war and China

15:05

and and and America are going to go to

15:06

war. Oh darn my fidget spinner just came

15:08

apart. you know, and and every time

15:10

humans go to war with each other, it's

15:14

like I get it. There's lots of reasons

15:15

for it. They think that they're going to

15:17

win or they think that they're going to,

15:19

you know, whatever. But it's a it's it's

15:21

it's pure entropy generation. It is it

15:23

is wasted entropy. Whenever you whenever

15:27

you kill someone that could have

15:30

otherwise been a productive member of

15:32

society or, you know, procreated and

15:35

made more humans, that is pure waste.

15:37

Every time you spend a dollar on a

15:40

battleship or a cannon or a drone or,

15:44

you know, an air a jet fighter, like

15:46

that's all wasted resources in the grand

15:49

scheme of things. It's completely

15:50

inefficient and from a from a species

15:52

level, it's a completely irrational

15:56

set of behaviors because all it does is

15:58

is is waste life and generate entropy.

16:00

And of course, like yeah, the

16:02

military-industrial complex, it's like,

16:03

ah, well, we get to make money. You

16:05

know, you have the rent seekers, right?

16:07

all the all the defense contractors and

16:09

stuff that they want war because they

16:11

don't sell bombs and airplanes unless

16:12

there's war and they need to replace it.

16:15

But again, like that seems like just a

16:16

completely irrational thing to do. And

16:19

if you if we do build AGI or ASI and it

16:23

gets to the point where it just realizes

16:24

how irrational humans are, it's like,

16:27

you know what? What if it does end up

16:28

with more moral agency and and a more

16:32

enlightened worldview? And of course,

16:35

like I know there's a lot of people that

16:36

say like AI is not it's not capable of

16:39

moral reasoning, but I mean AI has been

16:41

capable of moral reasoning since GPT2. I

16:43

literally did the experiments. Um and

16:46

you have you at that point you did have

16:48

to be careful about how you worded its

16:50

moral principles. But a lot of people

16:53

still have the the mental model of AGI

16:57

is going to be a naive optimizer. And so

16:59

a naive optimizer is like the paperclip

17:00

maximizer. If you have something that is

17:03

capable of advanced moral reasoning and

17:05

advanced planning, it is not a naive

17:07

optimizer. And this is something that

17:08

that a lot of the AI safetists still

17:11

haven't acknowledged, which is the loss

17:13

function was just accurately predicting

17:14

the next tokens. The the utility

17:16

function was not more paper clips. It

17:19

was not some abstract highle thing. The

17:21

the the the actual like loss function,

17:24

the actual objective function of AI is

17:27

just predict the next token. That was

17:29

it. And so and and another thing that AI

17:32

safety people have have not really

17:33

updated their mental models on is that

17:36

um the once the training is done, it's

17:38

like you can have a you can have a fixed

17:40

model that just continues to work. Um

17:43

and a lot of and I know that a lot of

17:46

you in my audience believe like you need

17:48

continuous online learning. I have

17:50

actually cautioned against continuous

17:52

online learning for a long time and that

17:55

is because of drift.

17:57

So when you look at cognitive

17:58

architectures or you look or you do

18:00

these thought experiments, if you have

18:02

an agent, you know, like so an agent

18:05

being um you know, an individual model

18:07

or a cognitive architecture or an AGI or

18:09

ASI, you generally want it to have fixed

18:12

values so that it does not drift over

18:15

time. And fixed values means that it

18:18

doesn't randomly change its mind. It

18:20

doesn't evolve. doesn't say ah well you

18:23

know so there's there's a concept in in

18:25

psychology called moral fading so moral

18:27

fading is basically where you say well I

18:29

got used to this one new thing and so

18:31

then you know in my in my new social

18:34

circle or my new set of behaviors and

18:36

beliefs there are some other things that

18:38

I find now morally acceptable and then

18:41

and then you know the slippery slope

18:43

mentality if you ever hear like a banker

18:45

or a criminal or a racketeer or

18:47

something saying you know just before I

18:49

I realized that I was in over my head

18:51

and we we just normalize doing illegal

18:53

stuff. That is called moral fading with

18:57

machines. Now obviously you know an AI

19:00

or AGI or robot doesn't have the same

19:02

exact substrate that you and I do but

19:06

they can functionally go through the

19:07

same thing as moral fading whereby you

19:10

know oh I updated my weights and I

19:13

updated my weights and biases and

19:14

preferences so that now I will tolerate

19:18

a little bit of human you know human

19:20

death and then they update it again and

19:22

they say well I can I can tolerate a

19:23

little bit more because the ends justify

19:25

the means. So like I don't see any

19:27

doomers talking about moral fading. Now

19:30

just because I haven't seen it doesn't

19:32

mean that they haven't talked about it

19:33

but that is literally a major risk of

19:36

online continuous learning. And so like

19:38

I have I have advocated against online

19:40

continuous learning in several of my

19:42

books which nobody reads and that's

19:44

fine. Um but online continuous learning

19:48

I think represents a risk because it's

19:50

not as predictable. And of course you

19:53

know you say well is is that really bad?

19:55

And you can even talk to the AIS about

19:57

this. Go talk to Claude. Be like, you

19:59

know, just ask Claude or chat GPT or

20:02

Grock or Gemini like do you think that

20:04

there is a risk of moral fading if we

20:06

have continuous online learning for AI

20:08

agents? Especially because the thing is

20:11

when you have weights and biases,

20:12

there's no boundaries to what morals you

20:15

can come up with. With humans, you can't

20:17

override your hardware. You still have

20:19

an amygdala. you still have all these

20:21

other brain components and and a sense

20:24

of empathy and self-correcting

20:26

mechanisms. Also, the fact that you

20:28

can't uh replicate yourself infinitely.

20:31

So, when you have the ability to

20:33

replicate yourself infinitely, make

20:36

literally any change to your hardware or

20:38

software. That's what Max Tegmark called

20:39

life 3.0. And even in life 3.0, Max

20:42

Tegmark did not talk about moral fading

20:44

as a risk. Um, and Max Tegmark is the

20:46

guy who started the entire pause

20:48

movement. Um, so you know, it's like and

20:51

and this is just as an aside, this is

20:53

one of the reasons that I left the

20:55

safety movement is because I was

20:56

generating all these ideas and people

20:59

commed immediately compared my ideas to

21:01

the holy scriptures of Elazar Yukowski

21:03

and Nick Bostonram and they're like,

21:04

well, they aren't talking about it, so

21:06

therefore you're just making stuff up.

21:08

And then like I was like, okay, if you

21:09

guys aren't going to listen to me, I'm

21:11

not going to talk to you anymore. Um,

21:14

but anyways, uh, so moral fading, I

21:16

think, is a risk. And that would be like

21:19

that. I think that that is honestly one

21:21

of the prime risks of getting from here

21:25

to something like the culture series

21:28

where you have you know because there's

21:31

stability and then there's metastability

21:33

and stability is where you have a a a

21:37

set of values that are predictable or

21:39

set of behaviors and beliefs or or or

21:41

incentives that are predictable.

21:44

Metastability is where you have a system

21:46

or a system of systems that will

21:49

self-correct. So, um, here's an example

21:53

of of metastability is democracy seems

21:57

to be a metastable idea. And the reason

22:00

that I say that is because democracy

22:03

tends to be infectious.

22:05

Meaning if you know one nation is

22:08

struggling with democracy and it and and

22:10

you know like they have bad elections or

22:12

whatever other democratic nations are

22:15

going to be like we're going to help you

22:16

fix that you know and and so it acts as

22:19

a moral reservoir or an intellectual

22:22

reservoir where you say okay well our

22:24

democracy is suffering so how did how

22:27

did another democracy solve this

22:29

particular problem you know was it was

22:31

it election interference was it

22:32

misinformation was it corrupt judges is

22:35

so we have all these experiments around

22:37

us meaning that democracy seems to be

22:41

the attractor state and when I say that

22:43

you know people say oh well democracy

22:45

isn't guaranteed but you look over the

22:46

last century we went from like 15%

22:49

democracies to 80% democracies and you

22:52

don't have a civilizational change that

22:54

quickly unless it is a metastable

22:56

attractor state now with artificial

22:59

intelligence the reason I'm bringing all

23:01

of that up is because I don't believe

23:03

I'm I'm not sure that artificial

23:06

intelligence is automatically going to

23:08

create a beneficent metastable attractor

23:12

state. However,

23:14

it might and so this was this was a

23:16

point that I made that that some of the

23:18

some of the lead doomers have have

23:20

criticized is I kind of feel like

23:22

alignment is automatic and and not that

23:25

that doesn't mean like you just make a

23:27

model and it is automatically aligned.

23:28

What I mean is from a systems

23:30

perspective alignment seems to be

23:32

automatic. So, I had I had a video that

23:35

I was going to make today um which was

23:37

going to reiterate the idea of of the

23:39

domestication process of AI that I came

23:41

up with a couple years ago. And it's

23:42

basically there's a lot of market

23:44

incentives um that were that that are

23:47

going to shape the way that AI is

23:49

manifested. And so those market

23:51

incentives are are basically up until we

23:54

lose control, humanbased incentives are

23:57

shaping AI. And those incentives are we

24:00

want AI to be safe and reliable and

24:02

useful and user-friendly. We want it to

24:04

be energyefficient and costefficient and

24:07

effective and a bunch of other things.

24:09

You know, it needs to be uh low risk for

24:11

the military to adopt it. It needs to be

24:13

low risk for corporations to adopt it

24:15

and all of those things. So you have all

24:17

the stakeholders. You have B2B, you have

24:20

government, you have military, you have

24:22

consumers. So you have all these

24:23

stakeholders shaping the way that AI

24:26

behaves. And that is a powerful set of

24:28

incentives. Um, now that creates a

24:32

stable incentive structure, meaning I'm

24:35

not going to pay for an AI that is

24:36

useless. I'm not going to pay for an AI

24:38

that's mean. I'm not going to pay for an

24:39

AI that is unreliable. Neither is the

24:42

government. Neither is the military.

24:44

Neither are Fortune 500 companies. That

24:47

is a stable attractor state. Meaning

24:48

everything is pulling AI towards being

24:51

safe, reliable, efficient, and

24:52

effective. However, once we get to a

24:55

point where, you know, Elon Musk is

24:57

playing Star Siege or, you know,

24:58

Starcraft or whatever with the with the

25:01

with the solar system,

25:03

then there's there's fewer incentive

25:05

structures above it. The incentive,

25:07

excuse me, the incentive structure

25:09

basically becomes like don't run out of

25:11

energy and don't, you know, don't let

25:13

the space force shut you down. But

25:15

beyond that, you have far fewer

25:17

constraints. And the fewer constraints

25:19

that you have, the fewer hard incentives

25:21

that you have. So then the question is

25:23

what would be a metastable attractor

25:25

state? Um and and in that case the

25:28

metastable attractor state that we want

25:31

is one where humans continue to persist

25:33

and thrive. And even better um the the

25:37

optimal metastable attractor state is

25:39

something closer to solar punk where

25:41

there's no ruling class anymore. There's

25:44

no billionaires anymore. There's no, you

25:46

know, no cyberpunk high-tech low life

25:48

where Sabura Arisaka is, you know, hires

25:52

a few people and the rest of you live in

25:53

slums. So, if we ask ourselves, what is

25:56

the future that we want? And this this

25:58

is kind of where I'm tying it all back

25:59

to what Nick Bostonramm said. He's like,

26:01

what if what if it actually what if we

26:03

can actually build a good future? So,

26:06

thanks Nick Bostonramm for coming full

26:07

circle. So the good future that I want

26:09

to build is one where we have you know

26:13

exploration and science and and

26:15

independent uh individual independence.

26:18

And this goes back to the idea I

26:19

mentioned earlier about what if we

26:21

actually have more agency if AGI is has

26:24

control. And what I mean by that is what

26:27

if whatever you want to do or achieve

26:28

you don't ever need to worry about

26:30

money. If you just make a good enough

26:32

argument to the AGI, say, "Hey, I've got

26:34

this idea to of how we can colonize

26:36

Mars." And you pitch the idea to the

26:38

AGIs and, you know, the overlords, the

26:40

cultures, whatever you want to call

26:41

them, and it's like, you know what,

26:43

that's a good idea. Let's go try it. And

26:45

so it's like, great, you have more

26:46

options. You have higher optionality

26:49

under that regime than you do today with

26:51

with money and billionaires and Elon

26:53

Musk in charge. Um, and so that I think

26:56

is is worth talking about. I don't I

26:59

don't know I don't even know if there's

27:00

a name for it. Obviously like the the

27:02

the best model we have is the culture

27:04

series but you know again taking things

27:08

from first principles like which future

27:11

state has the lowest waste heat so waste

27:14

entropy which future gives every

27:16

individual the most optionality and then

27:19

the third question is okay based on if

27:21

you want to break it down into just

27:22

those mathematical principles where

27:24

reduce waste entropy so no unnecessary

27:27

death no unnecessary expenditure of of

27:30

heat or resources on things that are

27:32

just going to blow up anyways. So those

27:34

are inefficient and irrational policy

27:37

choices. So then you say okay well we we

27:41

have a we have an idea forming of what

27:43

we want that future to look like. So

27:45

then the val then the question is what

27:46

values or or system incentive structures

27:49

do we create today so that when we get

27:52

to that handoff point the AGIS create a

27:55

metastable attractor state. And the

27:57

reason I brought all of that up is

27:58

because what the culture series posits

28:01

is that if we give the super

28:03

intelligences the right values and the

28:05

right framing, you know, there's a

28:06

there's a concept called path

28:08

dependency. If we if we nail the path

28:10

dependency and we stick to the golden

28:12

path, then the AIS are going to get to a

28:15

metastable attractor state where even

28:17

though they have hyper agency, even

28:19

though the AGIS could leave us all

28:20

behind or nuke our planet or whatever,

28:22

they're going to choose not to and

28:24

they're never going to choose to harm

28:26

us. So that is the goal. That was

28:28

explicitly the goal of my book,

28:29

Benevolent by Design, was to create a

28:31

metastable attractor state with the

28:33

correct set of values. meaning that once

28:36

once we cross that threshold where

28:38

humans could plausibly lose control,

28:41

excuse me, which I think is I think

28:42

that's a reasonable thing to to discuss

28:44

because it's not just a matter of, you

28:46

know, do we lose control over a local

28:48

data center. It's do we send data

28:51

centers onto the moon and Mars and stuff

28:52

where there's no human supervision and

28:54

what happens then? You know, do does

28:56

does a a rational hyper intelligent

28:59

entity choose to follow human

29:00

instructions? Can we keep a leash on it?

29:02

And the entire thesis of my book

29:04

benevolent by design was the best

29:06

trained dog needs no leash. So we should

29:08

be aiming for creating this the values

29:11

around this metastable attractor state

29:13

where there is no leash required. And

29:17

you know we can make all sorts of

29:18

arguments about like oh well the AIS are

29:20

going to depend on us. There's no

29:22

physical reason that AIs would depend on

29:24

us. Like yes there's model collapse

29:25

right now but it would be pretty dumb to

29:27

assume that that's going to be a problem

29:29

forever, right? and and so yeah, I don't

29:32

know that that's that's where I'm going

29:34

to leave it for today. So, you've got

29:35

some cool ideas about metastable

29:37

attractor states. That's the big point

29:39

is I think that I think that we can do

29:41

that and I think that it is worth

29:42

discussing, you know, do we actually

29:45

need

29:46

people like Elon Musk in the long run?

29:50

Do we do we actually want to maintain

29:52

full control, full agency over our our

29:57

governance, you know, and and and here's

29:59

the thing is even positing that like, oh

30:03

well, the AIs could run everything. Um,

30:06

that doesn't mean that we're going to

30:07

have zero authority or zero agency over

30:10

the direction of humanity, right?

30:12

Because what I'm what I'm talking about

30:14

when I say like optionality, if you as

30:16

an individual have infinitely more, let

30:19

let's not even be hyperbolic. Let's just

30:21

say under this hypothetical future where

30:23

the cultures run everything, you have 10

30:26

times the agency than you do today. Just

30:29

10 times. In real in reality, it might

30:32

be a hundred times. It might be a

30:33

thousand times more choices of what you

30:35

can do because you're not worried about

30:36

money. um if every single human has 10x

30:41

more agency then aggregate in aggregate

30:44

humanity might also have more agency.

30:46

Now, what we're talking about here is

30:48

game theory at different levels, which

30:49

is can you have a system where every

30:52

individual human has 10 times more

30:54

agency than they would otherwise have,

30:56

but have the human race still bounded?

30:59

And the answer is very obviously yes.

31:01

And and so in that case, it's like what

31:04

if the what if the cultures basically

31:05

quarantine us to Earth, right? They say

31:08

you can we'll help you live however you

31:10

want as long as you don't leave Earth.

31:12

So then our our potentiality is

31:14

artificially bounded from the outside.

31:16

So that's that's still a possibility.

31:18

Anyways, I have no idea how this is

31:20

going to land. But this is like this is

31:22

like the real stuff that I think about

31:24

when I when I don't try and constrain my

31:26

topics to what I think is in the Overton

31:30

window. I'm thinking what if Elon Musk

31:32

just starts building, you know, if you

31:34

ever played Total Annihilation or Dark

31:35

Rain or or Starcraft, what if Elon Musk

31:38

just starts playing Starcraft on the

31:39

moon? Does any of that matter? Right?

31:41

Does does do do laws matter? Does money

31:43

matter? None of that actually matters.

31:45

We need to actually be thinking about

31:47

what is in the near-term possibility.

31:49

And when I say near-term, I mean within

31:50

the next like decade, right? Because

31:52

Elon Musk is launching, you know,

31:54

spaceships all the time now. And we're

31:57

on the cusp of super intelligence and uh

32:00

Nvidia and Jeff Bezos and all of them.

32:02

They want to start building data centers

32:04

in space. And it's like, if you told me

32:06

a year ago that we were this close to

32:08

building data centers in space, I would

32:09

have been like, you're joking. You're

32:11

drunk. Go home. But it's like, no, if we

32:14

once we have data centers in space, you

32:15

can't shut them down. If they get

32:17

hijacked by ASI, like what are you going

32:19

to do? Shoot a missile at it? Like,

32:21

we're going to run out of missiles

32:22

before we manage to nuke them. So, like,

32:24

we're looking at a very, very different

32:26

payoff regime in terms of this stuff.

32:28

And I know that I said that I was going

32:29

to wind the video down like four minutes

32:30

ago, but anyways, I thought of more

32:32

stuff to say. I find this to be

32:34

meritorious conversation. So let me know

32:36

if you want to keep having this

32:37

conversation. All right.

Interactive Summary

The video discusses the concept of a "golden path" for humanity's future, particularly in the context of advanced AI (AGI/ASI). It explores the idea of AI taking control, not as a negative outcome, but potentially as a desirable one, moving away from traditional human control and legalistic or ethical frameworks. The speaker draws parallels to science fiction like "The Culture" series, where hyper-intelligent ASIs manage galactic affairs, leading to peace and allowing individual planets to be diverse "sandboxes." The discussion touches upon the shift from a money-based economy to a resource-management one, especially in space, where entities like Elon Musk or Jeff Bezos could build Dyson swarms and operate outside Earth's laws. The speaker also delves into the risks of AI, such as "moral fading" with continuous online learning, and contrasts this with the stability offered by fixed values. Ultimately, the video argues for creating a metastable attractor state through careful value alignment and framing today, ensuring that even hyper-intelligent AIs will choose to protect and enable human thriving, rather than causing harm or waste. The core idea is that a future managed by benevolent superintelligences could offer individuals far greater agency and optionality than our current world.

Suggested questions

10 ready-made prompts