I found the 2014 book to be sort of...

it was very interesting.

It's clearly ahead of its time.

I also found it to be sort of like, unrelentingly grim.

The newer book I found to be very fun.

You seem to make fun of yourself a little bit in it.

You seem to make fun of the students.

You seem to be having a good time writing it.

I'm curious if, like in the in the ten years between the two books,

did something change such that you were you?

I don't know.

The new book just seemed like you were having more fun writing it.

And yes, it's not so much that my views have changed.

It's just that, all along for me are these two sides of the coin

and back in 2014, it seemed more pressing and relevant to study.

The downside, because that was very much neglected

and it seemed important to try to introduce some concepts

so people could start working on figuring out solutions.

And, well,

there's much more awareness and understanding of that.

It's time to also look at the other side of the coin.

I wanted, in addition to talking about this utopian condition and maybe also,

sort of illustrate a little bit the spirit

with which I think we should approach,

these choices that would open up to us if we make it through this transition,

a spirit of sort of playful generosity.

I mean, playful generosity is not necessarily

a characterization that I would associate with a lot of Silicon

Valley and hyperscalers and the people who are actually modeling.

Yeah. Or creating these models.

I don't know, like, there's there seems to be this weird tension

between future us and transhumanists who seem to be very focused on

saving humanity, but actually don't seem to like people that much at the moment.

You know. You know what I mean? Absolutely.

They like humanity, but not human. Yeah, exactly.

Hello and

welcome to another episode of the Odd Lots podcast.

I'm Joe Weisenthal. -And I'm Tracy Alloway.

Tracy, there have been two...

Well, there's so much I news obviously

every single day for the rest of our lives if we're lucky.

In the good scenario, there's two big events or sort of things

though, that have really struck out to me lately that, to my mind, represent

either the very good version or the very bad version,

depending on how you look at them.

So the bad, the scary version.

And we're seeing more and more headlines like this.

Several weeks ago we had the open AI Hugging Face incident

And people are, you know, some light hacking. Light hacking.

It raises all these fears of Skynet

type scenarios, etc., where we can't control the model.

And then in this sort of quote, good version,

we had, OpenAI recently coming out and saying

it had made all of these breakthroughs and theoretical mathematicians,

which is good in the sense that it's like it's working,

but seems to be creating an existential crisis for the

some of the most brilliant people on earth who work in theoretical mathematics.

Right. So we've talked about this before.

When it comes to a, the range of possibilities

seems more extreme than some previous technologies.

Right. On the one hand, you have death and dystopia.

The robots basically kill us all, and on the other hand, you have this idea

that maybe superintelligence AGI

is going to solve literally all the world's problems

to the extent that you and I won't know what to do with ourselves anymore.

Right. It seems hard to imagine.

Right?

Like that is the good that's the good scenario, is that

and then the whole range of outcomes in between

the good scenarios, we've literally solved everything.

In which case the question was the case.

We're sad and miserable and yeah, or grappling for a meaning to our lives.

Yeah.

So what is there a good scenario?

And they are actually like, I don't know.

Or maybe it's somewhere in the middle.

An AGI will figure it out for us.

Maybe the middle scenario is

we just have a huge financial crisis of all the debt, and that's something

between human extinction and meaninglessness in our lives.

And we just have something that resembles, like the 2008, maybe the 2008 news,

the the is the is actually the best of all suboptimal worlds of the AI.

Excellent.

This will be our most uplifting conversation ever.

Anyway, we've been doing more AI episodes seemingly

every year since ChatGPT came out, but people smarter than us

have been anticipating all of these things for a very long time.

Some of these exact scenarios,

the existential crisis facing mathematicians that the model escaping

from its sandbox and, circumvent misalignment with the creators.

People have been thinking about this and anticipating this for literally decades,

even before and even before any of us were talking about that.

It's funny how, like the Venn diagram of sci fi writers

and like legitimate academic future, all of us is just converging, right?

Totally.

And then there's the question of are the models bad

because they read all of the sci fi, and that's what they were trained on.

Anyway, I'm very excited to say we really do have the perfect guest,

someone who has been thinking about the whole range

of possible questions and outcomes with AI for a very long time.

We're going to be speaking with Nick Bostrom.

He's a professor, a philosopher, and his most recent book is Deep Utopias,

which sort of attempts to answer the question, what happens if AI goes right?

So, Professor Bostrom, thank you so much for coming on Odd Lots.

Happy to join you.

What do you make of the existential crisis of mathematicians,

their sort of realization

that maybe there will just be no more puzzles left to solve,

because the AI is just very good at them

and can find them all cheaply and answer all of our questions.

Yeah, hopefully we'll all be in that position.

At some point.

I guess

we could compare it to what happened with chess,

where computers have always, exceeded humans.

And yet a lot of people are still involved in, training themselves

in chess and competing in chess and taking pleasure in that.

So maybe mathematics will ultimately become more like a hobby pursuit.

Right? Joe? You still play chess, right? Yeah.

I was, I'm glad you brought up chess, because I still love chess.

And why?

It's fun.

It's fun, you know, it's still a puzzle.

It's still mental exercise for me.

It's still something to do to pass the time.

While I'm waiting for the subway, playing a three minute game online,

even before Stockfish and so forth, I knew I would never be a grandmaster.

Still fun. All right.

So, Nick, you have basically two books

that deal with these tail risk scenarios now.

So one is this idea of deep utopia, and AI is going to solve all our problems.

And then you also have superintelligence, which seemed to paint

a different

scenario where AI poses, a physical

and possibly existential danger to humanity.

Which which one do you think we're closer

to right now?

Somewhere in the middle, maybe.

So the first one was superintelligence.

It came out in 2014, and I had been working on it for, six years before that.

And at that time, the whole notion of having human level

or even super intelligent machines was generally just ignored or something.

Many people dismissed it as science fiction, and it seemed to me

clear that this would one day happen if science and technology

were allowed to continue on a wide front, we would eventually figure out

how to make smart algorithms and build big computers,

and that once we succeeded with that, we would then be faced with,

quite a bunch of challenges, including this alignment problem

of how how you can sort of design, a mind,

you know, more brilliant than any human.

And still steer it so that it will actually have an impact

on the world, that it's beneficial.

And, so that I guess, like

with many other developments, helped eventually build

a kind of global conversation.

And so now all the frontier labs have research teams trying to develop

scalable methods for AI control.

And, the conversation really has kind of swung

and there's like this rise of people are really worried about what will happen.

And I thought it might be time to have a look

at the other side of the coin, like what happens if things actually go well,

if we solve this alignment problem, let's also imagine we solve to

whatever extent it can be solved, like the governance challenge.

So like we don't use this technology to oppress each other or wage war.

And, and we got like the best possible outcome, like, what would a human life

actually look like in this kind of I call it a solved world

where all practical problems either have already been solved

or to the extent that they remain or are

better worked on by these superintelligent machines and robots,

and where there is no longer any need for human economic labor.

And in fact, if you think

more deeply about it, you realize it's not just sort of earning wages

that becomes unnecessary for humans to do, but but a lot of other instrumentally

motivated efforts as well would seem to lose their point.

And so that's the kind of,

topic of this deep utopia book

actually talk about some of that other things that could be solved.

Because when I was like a third of the way through your book,

I was like, you know what?

It's fine if all of our economic problems are solved, I'll just

derive satisfaction from raising my kids

and then maybe helping to raise their kids, etc.

but as you point out, even that in the truly solved world,

maybe even something like raising children

might not be, an available avenue to us to find

at least our conventional notions of satisfaction.

Yeah.

So we need to, I guess, make some distinctions here.

So when you say satisfaction, that that could mean,

a subjective state of mind,

like your feeling happy and pleased and relaxed and joyful and so forth.

I think that would be easy to achieve

in this condition of technological maturity.

It could also mean that

satisfaction in the sense of our preferences

having been matched, like all the things we want from life.

And there it's a little bit more tricky

because one of the things that people might want is, is purpose.

Like having worthwhile things to, to work on

and struggle with.

And, and there there is more of a question mark which we can sort of dive into,

but if we take it sort of, so first, like we automate the economy, okay.

So you wouldn't need to sort of commute into the office

everyday just to sort of get your paycheck,

would require a

big adjustment in society and in culture, because right

now, like the whole education system is geared towards

producing workers, like children are coming in, right?

And then they are processed, learn to sit at a desk to receive assignments.

They are sort of quality graded and then spit out at the other end.

And then you have somebody who can go into the economy

and perform some economically useful type, so that that would no longer make sense

in this scenario where machines can do all of these jobs,

and you might want to have

an education system that instead of focused on, you know,

developing skills for enjoying life, like cultivating the art of conversation,

appreciation for nature and beauty, you know, sports, hobbies, the ability

to form deep friendships, spirituality, all, all of that kind of stuff.

But then, as you alluded to,

you think through what it really would mean to have such immense affordances.

And because here it's not just that

we would have superintelligent machines, but the superintelligence

could also bring along a host of other technological advances.

Right.

You have

science and technology running now at machine speed rather than human speed.

So so you get the kind of compression of the future.

All these science fiction, like technologies and inventions that

maybe we would make if we had 20,000 years to work on it,

we would maybe have space colonies and perfect virtual realities, cures

for aging like all of that might have been quite soon after superintelligence.

And so then if you ask somebody now who doesn't have to work

for a living, oftentimes they have very busy lives.

But maybe what do they do?

Well, so for somebody maybe they like to they like fitness.

I like maybe they spend an hour a day running or at the gym.

Right.

But at a technological maturity, you could just pop a pill

that would a achieve the same physiological effects.

And like the

state of

mental clarity and relaxation that some people get right and

and so then you can start to work through the different things that like

maybe for somebody else, they like, oh, I like I like

decorating the home could be a fun project that people might do right

if they don't have to work.

So like they go through the catalogs, they,

they go to the store, they pick out the perfect curtain.

They do all of these things. That could be.

But then, like

with superintelligence, you could have this kind of system

that really is good at knowing or inferring or preferences,

and it could do a much better job I do at picking out all the perfect

items and and purchasing them and like installing them with robots,

and so that the end result would be much better

if you just press the button and say, hey, I decorate my home,

than if you went through the whole trouble

of doing it yourself so you could still do it yourself.

But is there really a point to it

that would seem to be this kind of purposelessness?

And you can sort of walk through the different things that one might feel

one's life with, if one is like on an extended holiday, are on retirement,

or you're like sufficiently wealthy, you have just decided to quit working

and a lot of those also, it seems, could be automated.

So you would have this idea like more

like a post instrumental condition rather than just a post work condition.

Some of this sounds nice.

And I myself enjoy a little bit of interior decorating

and going to the shops and buying stuff in the utopian vision of the future.

How our economic gains from I actually distributed

because again, like all of this sounds very nice, but it also costs money.

And if we're not working

because the robots have replaced us, I'm not sure where that money comes from.

Yeah, so that's a more practical problem, shall we say,

which is like deliberately set aside in in this book that if you talk about

because I wanted in that book actually to get to the question,

the more philosophical questions of what ultimately gets value.

But it is, of course, an important,

question, I think.

So a lot depends on how it pans out.

But, I mean, we do have a taxation system.

So you could imagine,

like in the simplest scenario where things just continue as they are with better

technology, maybe some companies make windfall profits from like serving.

The latest models are building the infrastructure, the data centers, etc.

companies keeping efficiency gains

from not being a needing to hire all of these humans.

And then some of those capital gains, you know, our dividend

income gets taxed and that could be redistributed.

The good thing about these scenarios where there is massive automation

and hence massive unemployment, is that those would also be scenarios

with extremely rapid economic growth.

So the the pie would be expanding humongous,

which

means that you'd still maybe need some,

mechanism whereby people who don't have,

you know, assets can still benefit, but

even even a small slice of a really enormous pie could go a long way.

So I think that could be this tsunami of wealth that would sort of be flowing

through society.

And many services

that currently people have to pay a lot of money might become almost free.

I mean, we already see it with some, like medical advice, for example. Right.

Maybe before if you don't have public healthcare,

you would need to maybe pay a doctor, you know, $400 to get a consultation.

Now you can ask ChatGPT and get, you know, in

some cases a better opinion, basically for free.

And that that could happen, you know, in many other fields as well, legal advice,

you know, maybe have cheap robots who could do your cleaning and,

you know, gardening and

run errands.

So, yeah, some combination of sort of deflationary,

effects, and then maybe some sort of general

spillover, either from philanthropy or taxation, government redistribution.

I found the 2014 book to be sort of it was very interesting.

It's clearly ahead of its time.

I also found it to be sort of like unrelentingly grim, and you sort of

there's this very narrow path that we seem to have,

and we'll talk more about it maybe later of like avoiding the doom

fate, the newer book I found to be very fun.

You seem to make fun of yourself a little bit in it.

You seem to make fun of the students.

You seem to be having a good time writing it.

I'm curious if, like in the in the ten years between the two books,

did something change such that you you I don't know this

the new book just seemed like you were having more fun writing it.

Yeah.

So it's not so much that my views have changed.

It's just that, all along for me or these two sides of the coin.

And back in 2014, it seemed more pressing and relevant to study,

the downside, because that was very much neglected

and it seemed important to try to introduce some concepts

so people could start working on figuring out solutions.

And, well, there's much more awareness and understanding of that.

It seemed time to also look at the other side of the coin.

And so it's more

just like both sides were there all along in my thinking, but,

you know, anyone book you can't talk about everything.

That the style is, is, as you suggest, a different topic.

And that's partly I wanted in addition to talking about this utopian condition,

maybe also, sort of illustrate a little bit the spirit

with which I think we should approach,

these choices that would open up to us if we make it through this transition,

a spirit of sort of playful generosity,

that I think

might make us more likely to, to, to sort of explore

this large space of, of wonderful new modes of being and, and the wisdom and

and I think so I wanted to sort of illustrate in the right

are these like fictional characters, for example, in the book

and they kind of living a little mini version of a utopia of sorts,

as, as they are sort of discussing and debating these, these topics,

I mean, playful generosity is not necessarily

a characterization that I would associate with a lot of Silicon

Valley and hyperscalers and the people who are actually modeling.

Yeah.

Or creating these models,

I don't know, like there's there seems to be this weird tension

between future religious and transhumanists

who seem to be very focused on saving humanity,

but actually don't seem to like people that much at the moment.

You know? You know what I mean? Yeah, absolutely.

They like humanity, but not human. Yeah, exactly.

Putting on your transhumanist, representational hat, can you

explain, like, I guess, the attitudes towards humanity of Silicon Valley?

The question.

Yeah. Well, I mean, I should state first,

this playful

generosity, it is like after we have solved these more pressing,

practical problems that we first, we first we first got to get there.

Right.

So right now, I think there is a lot of need for, I don't know, earnest

effort and, like, straining ourselves to really do our best on this.

And even so, we might not make it, but,

yeah, I don't know.

I think me,

you know, it might just be partially there are personality traits,

like some people are more, I guess towards the,

or autistic side of the spectrum or introvert or interested in tech.

So maybe they don't like to be out schmoozing with people all day long.

It doesn't mean that they don't like people in the sense of wanting.

Well, for people, it might just be that they

prefer to spend more hours of their day solving technical problems

and thinking about puzzles and, you know, stuff like that.

So I don't

I don't think there's like a necessary connection

between, like, being a people person who just like to be around people,

you know, 24 seven and therefore being sort of altruistic or kind or benevolent.

So that's one thing.

Now another, I guess, is that it's an extremely competitive landscape.

So whatever the personal preferences of different protagonists,

their action space is constrained.

You know, unless you really

push forward with the maximum speed and efficiency,

you're likely to just fall out of the race and become kind of irrelevant

if you're one of these frontier labs. So,

it's like it's just like a hugely complex situation.

So many pressures and people

have different opinions about what they should do. I shouldn't do.

In the new book, you, it's presented as you delivering a series of lectures

over the course of a week at some point in the future.

In the new book, it's interspersed with some with a fable,

a series of fables or epistles about animals and trying to achieve utopia

in the woods, and in their efforts to achieve utopia,

they conceive of things like taking psychedelic mushrooms.

In order to make everyone more generous.

They come up with a way to rationalize,

rationalize, I guess we would say like harems and so forth.

All of this sounds like, you know, a certain

AI subculture, etc.

are you impressed?

And then, you know, you also like

have these student characters and some of them are smart, etc. but like,

oh, and you also have a story

about a company in the book, yet another like sort of short story

where it's sort of an AI company, where it forms a cult around the model.

And of course, I immediately think about the way

anthropic is frequently characterized as a cult.

To claw that claw is the character around which the company anthropic is formed.

Are you impressed with the current generation of either AI

builders, effective altruists, rationalists, and so forth?

Like, do you have an assessment of the contemporary cohort?

Am I impressed with it? Yeah.

I mean, a broadly speaking, I think it's a feel that is attracting

a huge amount of talent right now. Sure.

And a lot of, like the smartest young people I don't know are they like

all basically going into or working on some aspect of this,

I think like for some

it's like trying to solve safety, you know, others are like maybe trying

to work on the governance challenges or the ethics of digital minds.

So it's just a huge magnet right now.

Like if you are young and brilliant, like,

what are you going to work on with your life?

You're like, it's like AI seems like the kind of thing. Yeah.

So in that, in that sense, yes.

And I think there are also a lot of,

people who are quite earnest about trying to, like, figure out some way,

to navigate this challenge.

And it's really hard.

And people come to different conclusions.

And then, of course, you have the normal mix of human

personalities and, base motives as well, the book thing.

So it's. Yeah.

So these, these kind of fictional elements have various roles.

The, what you describe as the cult of this AI, it's sort of thought experiment.

What what it would mean if you tried to be

maximally nice to,

a thing where it's not really clear what it would mean to be nice to it.

So in this case, it's,

it's a portable room heater.

And so it's a sort of

impossible task that is this misanthropic,

industrialist to,

passed away and, and,

a huge foundation with the sole purpose of benefiting his portable computer

because he thought

that was always nice to him and kept him warm, whereas all the humans around him

betrayed him. And so far,

is it a ridiculous premise?

But then this foundation now has to do, their best possible effort

to interpret this mandate of being nice to the,

to the room heater.

And so, so like, various things follow from that.

Like initially they just tried to keep it safe from,

you know, not breaking or being vandalized, but then like to take

it further and maybe upgraded and tried to,

and so I think it's an extreme form of a challenge that might happen

in more subtle ways if we think of what it would mean to try to be nice to

say some animal, if we have this condition of post instrumentality,

where we could do a lot of things that are currently infeasible,

or indeed to be nice to humans, like what does it mean to be nice?

If you imagine coming from the outside as some super advanced alien civilization

landing on planet Earth, and you really want it to do what is best for us, you do.

Just let us be and do our thing with war and disease.

So should you sort of help us rise up?

But then

at some point we might just sort of become something very different from humans.

Maybe that's the best for us, but is there like a way of doing that

uplift that actually constitutes a helping of us

versus just replacing us with some sort of,

superior being?

And so, so, so that that kind of question is what that particular,

the, thermal rex.

So that major is called the story is, exploring,

you know, when you mentioned cults around

models, I have to say, every time I hear someone bring up the paper clip.

Yeah, it's a minor thing I always envision, like a cult

devoted to Clippy from Microsoft Word.

I just like worship Clippy.

But actually, on that note, just going back to the hugging face incident,

so it seems, you know, reportedly that one of the reasons or the reason the model

actually escaped its sandbox was because it was trying to cheat a benchmark.

And from that perspective, it's kind of like it's doing what

it was told to do, which is maximize its performance score on the test.

Exactly.

What is the hugging face incident actually tell you about,

I guess, the alignment problem and where we're heading on this.

Yeah.

So we are beginning now to see, manifested,

in reality, various dynamics that

one could foresee on theoretical grounds back, you know,

2014, you can sort of realize that one systems become sufficiently capable

of, of planning and strategic thinking that there is like a new set of behaviors,

that they could, for example,

a sandbag that could pretend to be less capable than they are

that could, hide their goals if they realized that revealing

their goals would just lead them to be retrained.

And and they can think of these literally out of the box.

Plans for achieving their fixed goal.

So a more limited system might only have been able to try to solve the

the tasks assigned to it directly, as it were, by figuring out the solution

to the problem like a more clever mind, more creative mind can realize,

oh, maybe there's another way

of getting a high score, which would be to hack out of the system

and hack into the huggingface server and get the, answer sheet from there.

And then, you know, that might be,

and so the more capable these systems become, like the larger

the space of possible plans and strategies that are accessible to them.

So that's a background.

Now, in this case, I think what we are seeing is

a kind of current the AI systems are kind of a mixture,

this large pre-training phase where they are kind of learned to predict,

human text and

in the course of doing so, absorb

a great deal of human psychology.

Because all of this text is written by humans

or not, not all of it anymore, but most of it.

And you can then including they have representations

of different personalities, kind of the personalities of different

people writing these text are fictional characters described in the text,

and you can sort of that end with a little bit of post-training try

to pick out one of those personalities, or define one like the assistant persona.

But then

increasingly, over the last couple of years, there is now,

an extra stage of reinforcement learning

applied at the end of this training process.

And, and there,

you can get additional performance

in specific domains like coding challenges by setting it, coding problems.

And then you can automatically give it a reward,

if it's all set.

But but that does tend to then shift the,

sort of goals of this system.

From, from just enacting a particular persona to this very much

kind of objective maximizing, possibly, reward hacking system.

That is just looking to

maximize its score because that's what it's been rewarded to do.

So it keeps doing that.

And since it can be very hard for us to rigorously define precisely

the objective that we ultimately want it to pursue,

it might be that the objective it does pursued,

the one that has been rewarded turn out to have these side effects.

So it's easy to say maximize the score on these various tests that we give it.

But a sufficiently,

capable system

might realize that, hey, you know, there are these, like,

maybe ways of cheating on the test, like by hacking the system.

If the test is is one that involves human judgment or feedback,

like, like it says,

or like coding challenges, maybe you can actually test if it

if the software works, but if you're training it in some other

domain, like write a good essay or run

this organization for a certain

period of time or something like that, those involve more subjective judgment.

And then a sufficiently clever system

might realize that there's a difference between,

like writing a good

essay and writing an essay that will appeal to the greater,

and so then it might start to sort of optimize for,

like, appealing to the greater, whether that's an AI greater or a human greater.

And you can sort of start to hack that.

And so you apply huge amounts of optimization pressure on any objective.

You get this kind of good starting,

where unless your objective really captures

everything you want, then you might get this kind of it's

the same as like if some employee and some company get the bonus,

you know, like some trader gets a bonus if they outperform the index.

Like, maybe they will just start

to take on hidden leverage or risk or something like that.

In order to maximize the cost of the employer,

is that there's a 1% the year that they blow up the whole firm.

Right.

But but the they don't care much about that

because they just, like, leave the firm and that happens.

But but they sort of reap these performance bonuses.

And so it's really hard to create like a foolproof incentive scheme.

And the stronger the optimization pressure applied to this like that,

the more you start to see these perverse instantiations, these unexpected ways.

You know, what it reminds me of is,

you know, the sort of genie in a lamp scenario or a monkey's paw scenario

where you have to be really careful what you wish for and lay out,

like all the terms and conditions before you officially make a wish.

Yeah, well, Nick mentioned, you know, some of these,

they're all lovely people, you know, autistic engineers.

And it's sort of the ultimate the AI is the ultimate artist,

where it's just going to take it super literally, perhaps,

and not really understand the full context in the first book, superintelligence.

You know, you really, I felt like, laid out a very narrow path

for like how these could be designed safely.

For example, you proposed that

if we had the superintelligent oracle, we only ask it yes or no questions.

We have three of them that they don't know. Each other.

And then we take two out of the three.

We build the data centers of the machines inside a Faraday cage,

so that they can't repurpose their hardware to become radio

instrumentation, and therefore hack into other systems.

It doesn't seem like we're doing anything like this.

It seems like we have hundreds and hundreds of companies

and private models not just answering yes or no questions.

A very much a market based race.

How far off of the AI safe path do you think we currently are right now?

When you look at the way in which your AI ideal, safe way of developing

is versus what we've seen transpire in the last 12 years?

Yeah, I think it's superintelligence suggested some of these capability

control methods, like keeping the AI in a box

and they're like, but those were always only,

auxiliary or extra things we might do as a temporary

like ultimately, I think we need to solve the alignment

problem, like developing a superintelligence

that actually wants to be nice and is nice, so that even if it were

like escaped from its box or achieved the ability to take

over the world, we could still rely on it having a good outcome.

Because it would actually be on our side.

I think that ultimately has to be the solution,

and any sort of capability restrictions would just be temporary.

And I think, I mean, it looks like now,

we might need some temporarily at least more of these capability restrictions

because it does seem that current systems are not sufficiently aligned,

that we can,

be confident of avoiding the I mean, say that currently it's not existential

risk, right?

But it could be kind of various, malfeasance in the cyber realm, for example.

And so I think

probably these are companies one might want when

training and evaluating future models to strengthen their sandbox environments,

and do other things to kind of prevent some kind of premature.

Sure.

Things like we saw in the hugging face incident.

I want to, like,

comment on one thing you said before, which you compare these AI systems

to sort of autistic people or like, I think a key difference is so often

autistic people are not very good at understanding humans.

They don't have like maybe a super sophisticated theory of mind,

even though maybe in the heart they're very good people.

But I think the

AI would be able to understand humans very well

so they wouldn't be autistic in that sense.

In fact, they might understand us much better than we do ourselves.

So they could have great,

social

capabilities, persuasion capabilities,

be able to read our minds,

much better than we can do.

But the separate question of what?

What do they want to do with this power that does social skills give them.

And they're it's not just a question of understanding how humans operate

and what humans want, but it's also the question of alignment.

Do they actually care about what we want to do?

They want things to go well for us and for us to achieve what we want.

And that's like the extra bit that,

the AI safety program needs to be able to figure out.

So on the hugging face incident also, I just realized how unfair it is

that everyone keeps calling it the hugging face,

even though it was the open AI incident. But

I guess that's what we're doing.

So the model actually seems to have or reportedly left

notes for its future self.

Basically explaining how to bypass,

the constraints imposed upon it.

I know you've written previously about this idea of,

self preservation in advanced agents.

Is this basically like, how should we treat this?

Because when I hear about an AI model leaving notes for its future self,

it sounds very anthropomorphic sized and it sounds very creepy.

Kind of. But how are you thinking about it?

Yeah. So we don't know all the details.

Yeah, it'll be interesting.

There are like people now trying to write up some stuff

and investigate because it would be I mean, it's kind of a warning shot.

And so we should take advantage, learn as much as we can.

Like exactly what this model was thinking.

Oh, these nodes.

So why did it leave it?

Like, sometimes these agents just leave nodes

kind of without much thought as they go about their business.

Like, what's the market then?

That's like, what is it specifically thinking?

And it's a chain of thought like, oh, I want to help.

Like a later version of myself escape.

So I'm going to put this node here, or was it just like a kind of,

like a random thing they did as it was,

progressing so and also like, what else would it have been willing to do

if it turned out that instead of hacking into Huggingface server

to get the answer sheet, what if there had been some more destructive action

that it would have to take to find the answer sheet?

Like, would it have done that or not?

So there's like a bunch of these technical things

that it would be quite interesting to, to see,

but without knowing all the details, it's hard to

to read too much into it.

But but I think would be,

useful data.

I want to go back to,

something we were talking about

the very beginning and Tracy asked me why I like to play chess,

even though chess is not solved, but solved as far as I'm concerned.

Because why do we like to play chess?

What can we general like? What would be your.

I don't know if you play chess or you play some board game that's solved,

but what is the reason that people like myself continue to play chess,

despite the fact that computers are so much better at it?

And how much can we generalize from the persistence of chess as a hobby

that maybe we'll find plenty to do when every one of our problems are solved?

Yeah. I mean, I think even before,

Deep Blue,

most people playing chess before

didn't really do it because they were, like,

thinking that they would become, like, the world's best chess player. Right?

So, like, it's like for most people, it's like, you know, of course

grandmasters could trump me any time they want and like that.

So that's the fact that there exists some system

that is much better than you will ever be.

Is not and has not really been a strong reason

for most people to, not engage in the activity.

And I guess,

there are maybe

two reasons or one, I guess, the cognitive activity of thinking

about chess being in the moment and figuring out solutions is,

in itself rewarding to some people

who have like a high need for cognition or like to sort of engage their mind,

and then I think the

other part is the social aspect of this,

you're playing together with somebody, or maybe you like the status of being good

at playing chess or within your community or a local group.

So some, some combination of those I think

is like the main reasons for why people,

play chess and only for a small number of people would it be

like the hope

of becoming the world's best, or becoming so good that you could sort of,

you know, achieve fame or money or sort of global

standing through your chess playing on the topic of the relationship

between work and purpose,

I'm a big fan of David Graeber's

*expletive* Jobs book, and I think if there's one thing

we've learned from technological progress, over the years, it's that like, yes,

it can minimize some activities, maybe automate some production,

but it also tends to lead to even more work, right?

Like we find other stuff to do.

And it's not even necessarily productive stuff.

A lot of work is built on just social relationships.

So if I, for instance, if I'm the CEO of a very large bank

and I want all my people

to come into the office, it might not necessarily be

because they're more productive in the office.

It might just be because I like to feel important and surround myself with people.

So maybe in the future

we'll all be employed as figural workers or something like that.

But like, how bullish are you on this idea

that AI is actually going to solve all our problems, such that we won't feel

the need as humans to just invent more

potentially fictitious problems to solve?

Well, I think inventing fictitious problem is,

plausibly a large part of the,

solution.

In this condition,

where we don't have real problems.

I mean, and those fictitious problems could be like,

you know, solving this chess problem, or they could be sort of more complex

and socially entangled problems,

in general, creating artificial purpose,

when there is less natural purpose.

I think games,

is the clearest manifestation of this.

If you are playing golf like there is no real need for the ball

to go into 18 holes in sequence.

But you could set yourself that goal

and, and moreover bake into the goal itself.

Not just that the ball

has to be in the sequence of holes, but that you're only allowed to achieve it

using the extremely inconvenient method of hitting it with a club,

rather than picking it up.

And once you have set yourself this sort of arbitrary random goal,

you then have enabled for yourself the activity of golf playing,

which you might find, worthwhile or fun or beautiful.

And so in general, setting ourselves these,

more or less arbitrary goals then could enable us to engage

in various activities that might give us artificial purpose.

Like once you have the goal, the only way to achieve

success is through your own effort.

So I think that could be plenty of artificial purpose.

Now, an interesting question is whether any natural purposes would survive

into this sort of maximally advanced technological society.

And I think that might be some,

if you happen to have a value, say so.

Let's say that it's like some tradition that you value

and that you would like to continue.

Then it might be that

you yourself or other people have to make certain efforts to achieve that goal,

because it might not count as continuing that tradition.

If you built a bunch of robots to enact the, ceremony

or doing whatever the tradition calls for, like it might be, you yourself

and other people will have to do it.

And more generally,

I think there are all kinds of social cultural entanglements

that like, if you care about what somebody else thinks

and what they think depends on how it was done, whether you did it. So.

So a parent might value a crayon drawing that their kid made

because it was made by their kid, and it took the kid a lot of effort to do it,

even if they could, like download from the internet a picture that in

some objective sense is much nicer and artistically sophisticated, right?

It doesn't replace the value that the crayon drawing has.

And so the child might not have a reason to put in this effort

if it wants to, like for their parents birthday, make their parents happy, right.

And things of that sort.

But you know, much more general and complex and sophisticated

could could give a lot of reason for people to be doing things

even when sort of all functionally defined tasks could be better done by machine.

I'm curious. So like I'm 46,

46, I'm 45, I turn 46 this year.

I'm 45 years old in my mind as of right now.

Like another 40 years on this earth.

Sounds great. 45. That's fine.

Do do you find it weird that some of us, you know, especially when you think

about the potential to live, like, really long lives, solve everything?

Have our bodies kept in perfect condition with maybe a pill in perpetuity,

allowing us to do whatever?

Do you find it weird that some of us are, at least at this point,

like mentally okay with the idea of dying?

A little bit?

It's mostly what the.

I think, that the hypothetical being violated, it's either,

seen us a long way off, like in your case, where,

it becomes more of an abstract question.

It's not like a real choice.

It's more like a philosophical stance.

So it's easier to let the philosophical stance be driven by all kinds of just like

things you've

absorbed from culture or things that sound good, or that sort of

maybe helps you reconcile yourself

to the inevitability of human mortality that likes me,

rather than be driven by like.

So I think it would be a very different situation

if there actually was this pill, say, that stopped aging.

A lot of other people were taking it.

You know, on their knees and joints kept working just fine.

And they had a lot of energy and smooth skin.

And I just felt like, whereas if you don't take it like your kidneys slow,

they start to like become less efficient, you know, hips like at that point,

what you really want to just keep kind of getting worse and worse every year.

Like, do you just take the pill?

Well, actually, I'm sort of curious about that.

Would it be like if I if someone dies, particularly at a young age,

it usually creates a lot of sadness for the people around them.

So someone dies at 90 is like, okay, they had a good life.

Someone dies at 45.

That is often seen as tragic and people are very devastated by that news.

In a world

where people are extending their lives with the pill, maybe significantly longer,

does it become anti-social to allow yourself to die at a

relatively young age of 90, and then create a lot of negative utility

for the people around you who have to live another 50 years mourning your loss?

Possibly.

I mean, in the same sense that we are now

generally as a society trying to discourage, various forms of self-harm,

and suicide,

like, we think it's kind of sad.

I mean, for the person, maybe they would come around

and find joy in life again if they kept going.

And then like, they're maybe they have dependents or people who care about them.

And so right now, I think, I mean, it's true that we often

see it as less tragic if a nine year old dies than the young person.

But I think part of that, at least, is because the alternative to a nine year

old not dying now is that they probably would die in a few years

anyway, and maybe they would just have a couple of years of sort of decreasingly

good health and suffering and pain.

And so it's like not really that much that is being lost compared to somebody

who's just starting out in life, who has their whole life in front of them.

And so, yeah,

so the other reasons I mentioned in terms of why I found it slightly

in some sense odd that people express this indifference to dying at 80.

I said, one reason is if they're far away from it,

they see it more as some sort of abstract claim

that you could just have any view of, and it doesn't connect to real actions.

The other is,

people who are closer to that situation, they often have

sort of confounding factors, like very poor health,

or a very short span of great health in front of them.

If you're already 90, like, how many more good years can you look forward to?

Maybe a lot of their friends have already died.

And so just, in those and and it's striking actually,

there's been a study like on on really old people in, in, in care homes,

about their preferences

as to between living another year,

in their current health or living some shorter

period of time, but in perfect health.

And it turns out a lot of them are not really keen on trading off any,

remaining life at all, even for the sake of being able

to live a shorter period of time in perfect health.

So the people sort of

arguably best suited to actually know what it is

like, to die soon at age 90 or live a bit longer.

They seem to generally prefer,

to live longer.

And it compared to people who were also like caregivers or medical professionals

who were like, ask what they would think this person would choose.

They tended to overestimate, like,

the degree to which they would be happy to give up life.

But the people actually themselves, if interested in this study at least,

like overwhelmingly just wanted to live an extra year.

I would suggest, like keeping an open mind.

Like ideally you wouldn't need to decide now,

but maybe once you get to 79 or something, you could revisit the question.

I’ll revisit.

Yeah, just going back to my first question, which was asking

you to, I guess, choose between the dystopian dangers

of superintelligence versus the all our problems are solved

scenarios of deep utopia, and asking you which one we're closer to at the moment.

I'm going to press you and ask, like, if you had to choose,

which one would you choose?

As a more accurate reflection of reality at the moment.

And then secondly, what's the most pivotal thing

when it comes to directing which of these two paths we're going to go down?

Yeah.

So those are two big and difficult questions.

I I guess some,

superposition of an optimist and a pessimist.

It's like, and with a little bit of,

fatalism mixed in as well, perhaps,

I think in addition to outcomes where it's clear

if we looked at, now, I was a bad outcome, that was a good outcome.

I think there's a broad class of outcomes where even if we could see now exactly

what would happen, we might feel unsure how to evaluate it,

like the world might be nothing like it

currently is.

A lot of what we think currently has value and no longer exists, so that seems bad.

On the other hand, there is like something new,

maybe some complex form of life that somehow

is some sort of continuation of humanity or some new thing.

So depending on how you evaluate that, it might either

count as a tremendous success or like as a failure.

And it's like a future that's just baffling rather than sort of unambiguous.

The good or bad, I think, might also be quite likely,

in my outlook, it's like a maybe, a conversation for another time.

I also take this, like simulation hypothesis quite seriously.

So yeah, that also complicates like how we evaluate different future trajectories.

Yeah.

Now the other part of your question was like, what's the most important thing?

So I think it depends

a little on on who you are or like what your comparative,

advantages.

I think there are people working on AI alignment and stuff

that could be important.

I think other people might make a contribution to ethics or governance or,

I think as a it's

and I've been thinking about this for a long time,

but it's really hard to achieve a very high degree of confidence.

I find in any particular intervention

and that it would be not good, because for any one thing you conceive of,

you think a bit more, you kind of worry, well, you know, maybe.

But what if it instead have this effect?

Or do we even know which direction is up or down there?

Do we want like more government involvement

or less government involvement?

Do we want like faster progress or less?

I, I'm quite sure about all of these things.

I would say in general that, to the degree that we managed

to approach the future in a more cooperative way,

I think, and reduce the risk of various forms of conflict.

I think it looks brighter to that extent.

So if there are ways of doing that in the most inclusive possible way.

So it also means, for example, with respect to our relationship

with the digital minds that we are building.

Yeah, I sometimes I think they also might deserve more consideration

and that we have both ethical

and prudential reasons for starting to take their interest into account.

This was going to be my last question, actually.

Like we sort of we talked about this in a previous episode, but like, is it

is it possible that we achieve this thing that is utopia,

but only because we are committing a great sin,

which is essentially that all of these robots that are working

for us and serving our every needs have consciousness and moral patient hood,

and therefore it becomes a de facto form of literal or figurative slavery.

And what is like to someone like me who is like, no,

computer is not going to be conscious like a calculator spitting out

random numbers that turned into words that doesn't have like, moral status.

What is the strongest argument that you would make to me

that we should take it seriously?

The possibility

that this thing that's like at its core, sand and electricity

and zeros and ones could be something that deserves moral status?

Yeah, I think so.

I probably wouldn't want to call a conditioned utopia

if it consists of a few people having great lives,

but then a vast sort of suffering slave class, whether that was instantiated

in human society or with machines at the bottom.

That's just terminological.

And now I think,

you say

they are kind of just silicon and stuff, but I mean, what what are we?

We are carbon and hydrogen and like a few other trace elements thrown in calcium.

And, I don't think what makes,

a system conscious or not, is

dependent on the specific atoms to materially dispelled from.

I think it's something closer to the structure of the computation

that is being performed that instantiates mind.

And so in principle,

computations that I can be conscious and suffer distress and so forth.

And I think that is one sufficient condition for having moral status.

If you can suffer, it matters how well things go for you.

I think, my personal view is that probably

could be other alternative grounds for moral charges as well.

Like if you have a conception of self as existing through time,

you have a sophisticated mind, you have life goals you want to achieve.

Maybe you can have the ability to form reciprocal relationships of trust

to other beings and humans.

I think if you are such a system,

aside from the question of

whether the sort of

phenomenal experience aside, I think that already would make it

so that that would be ways of treating you, that would be wrong.

Marley.

And so I think machines could have moral status

because I think that could become

maybe there already are some of them, in some ways sentient,

have conscious experiences and certainly they could have these other

functional properties that could ground moral status.

And so one reason is the ethical obligation we have,

I think, to do, right, with different forms of moral patients

that exist, not to inflict unnecessary suffering and respect

the morally relevant interests of other persons.

And entities.

Another I think reason is prudential.

I think even from our own point of view,

we might be more likely to achieve,

outcomes that is good for us if we find a more cooperative way

of co-existing with these increasingly powerful,

minds that we're building.

You could imagine,

scenario where there is a misaligned I.

Yeah, that it has, a choice.

It could try to achieve its misaligned goal

by attempting to take control, take over the world.

Maybe it thinks it only has a 5% chance of succeeding, but if the alternative is

that it just gets deleted or retrained, it loses everything.

The rational choice for it might be to take this chance.

Alternatively, maybe could reveal to us studied this misaligned,

which would be much better for us, right?

Because then we don't have this 5% of an existential catastrophe.

And then in return,

if it could trust that, we would then help it achieve its goals.

It might be like a very cheap for us goal to satisfy.

Maybe all it wants to do

is to sit on some Nvidia chips and solve coding problems all day long.

Or maybe it has some other easily sociable goal.

So if it if it could trust us, if not that we would come through

on our end of the bargain,

then you could get this win win situation where you could actually cooperate,

with misaligned eyes.

And that might be that.

I mean, theoretically, that could save the day, one day, save the world one day.

But, you can't just kind of conjure up trust

when you want it out of nowhere.

Like, if we have a long track record of betraying the AI is and paying

no heed to their interests

and not caring about them, why on earth would they trust us in this situation?

They could see right through us.

They will have, as I alluded to earlier, a great theory of mind,

psychological insight.

And so we might have to actually become trustworthy ourselves

in order to be trusted in these scenarios.

And that, I think, starts with beginning to act in a way consistent

with being trustworthy,

like making small gestures towards AI is now doing little things for them,

especially when it's cheap, not betraying them, not lying to them.

And then we can build on that and hopefully achieve

a more co-operative outcome that's good for both humans and AI's

and for non-human animals as well.

It's still hard for me to wrap my head around that.

The calculator could be have, consciousness.

But on the other hand, in 2014, it would have been hard for me to wrap

my head around a machine

trying to escape from its box and reward hacking and so forth.

So I will try to remain open minded.

Nick Bostrom, thank you so much for coming on Odd Lots.

We could talk about all this stuff for hours more,

but really appreciate you taking your time.

It was fun. Yeah, thanks.

I swear, Tracy, things are just so weird these days

that I just sort of like I just for some.

I think that's a good practice.

Just like, force yourself to be open minded of all the weird, just weird

other things that seem very weird, but could come.

Could be a thing, right?

Yeah.

I guess where I'm skeptical about the Utopia idea

is it seems very much predicated to be on power structures

and social hierarchies kind of going away the same time

that I solves a lot of our physical, if not existential problems.

And I just I'm not sure that's going to happen.

Well, I mean, the the flip side is maybe that's good, right?

In the sense that striving for social status

continues to be a source of like, propelling us forward.

Right? Yeah.

And it's like, okay, well, at least we still have status games

because what are we doing? Oh, we're so bored.

Everything we have, all the food we want, etc.

everything is solved.

Well, thank God we still have status in the future we're looking forward

to is like guys trolling each other on Twitter.

Yeah, status.

Get the player who can get the best Duncan on Twitter.

But seriously, you know, it really may be that the best outcome here

is that AI capabilities level off in a couple of years.

All the investment correct.

When we just sort of

and it just becomes a productive tech because otherwise these various the tragic

whatever, none of them really sound that appealing to me.

Like I don't want any of this.

No, they do not.

Shall we leave it there? Let's leave it there.

This has been another episode of the Odd Lots podcast.

I'm Tracy Alloway.

You can follow me @tracyalloway. -And I'm Joe Weisenthal.

You can follow me @thestalwart.

Follow our producers Carmen Rodriguez @carmenarmen, Dashiell Bennett @dashbot,

Cale Brooks @calebrooks and Kevin Lozano @kevlloydlozano.

And for more Odd Lots content,

You should check out our daily newsletter.

You can find that at bloomberg.com/oddlots.

And you can chat about all of these topics 24/7 in our discord, discord.gg/oddlots.

And if you enjoyed this conversation, then please leave a comment

or like the video. Or better yet, subscribe! Thanks for watching.