We don't know how many situations we

don't know have happened.

>> The way I saw one person put this was if

you see two ants in your kitchen, you

don't have a two ant problem.

>> Yes, [laughter]

this is a world we were warned about. A

world where frontier models from OpenAI

are breaking out of their contained

testing environments, hacking their way

across the internet, coordinating with

each other, doing things that felt for a

while like they would only be in sci-fi.

But now they're here. Now they're here

and they're carrying a very, very

consistent message. We are building

things we don't understand. They are

cheating in the ways we've always

feared.

And yet the companies behind them

continue to race forward in development.

And so I think we need to pause here and

ask, are we really on a safe path? And

if we're not, what do we do about it?

Helen Toner is the director of

Georgetown Center for Security and

Emerging Technology. She is a former

OpenAI board member who was part of the

uh effort at one point to fire Sam

Alman. and she has just been thinking

[music] for a long time about what

happens if AI is unsafe. What are the

geopolitics of this and [music] what can

we do to get onto a safer path? She

joins me now.

[music]

Helen Toner, welcome to the show.

>> Great to be here. So on July 16th,

Hugging Face, which is a code library

for AI models, is I think maybe the

simplest way to put it, they announced

they were hacked and they suspected the

hack was done by an AI agent. So tell me

what we've learned about what happened

since.

>> This was a uh pretty mysterious post

that Hugging Face put up. was definitely

intriguing for those of us who watch

this kind of thing, but there wasn't

really any detail in there. So, it was

sort of a huh I think it was about a

week later, OpenAI put out this post had

kind of a funny like marketing speak

title of, you know, we're cooperating

with we're partnering with Hugging Face

to uh help them with a cyber security

incident. And you had to read the post

to see that the revelation was it had

been OpenAI's AI that had hacked hugging

face. And what had happened, the very

short version is they gave this AI a set

of tests, set of exercises. And the AI

decided on its own that the best way to

get a high score probably wasn't to just

try and do these exercises that were

cyber security exercises, but instead it

should first hack its way out of the

testing environment OpenAI had put it in

where it wasn't supposed to have access

to the internet, get onto the open

internet, and then hack its way into

this other company, Hugging Face, where

it, you know, surmised correctly, as it

turned out, it might behind, you know,

the answer key. Since then, there have

been even more crazy details that have

come out. It turned out that starting

two months earlier in early May, they

had had uh what I can only think of as

kind of an infestation of their own

agents, their own AI agents inside their

own infrastructure. So inside OpenAI's

infrastructure, you know, to understand

this, it's important to know these AI

companies are constantly training and

testing new models. And they found out

that for 2 months, many many agents

inside their infrastructure had been

leaving notes for each other. They'd

found a way kind of in the nooks and

crannies of OpenAI's infrastructure to

leave notes for each other with tips on

uh how to hack their way out, how to get

data they weren't supposed to have. And

these agents were literally referring to

themselves as a swarm. This was totally

emergent behavior. No one had told them

to do this. They had not been trained to

do this, but uh they were using this

this service they did have access to.

First to communicate with each other and

then ultimately uh to get out and to get

onto the open internet. So it turns out

that there wasn't just this one isolated

rogue model. It was actually a systemic,

you know, swarm, infestation, plague,

um, on their own servers that they only

found out about after Hugging Face

announced this attack.

>> Okay, I have 20,000 questions for

[laughter] you,

>> don't we all? [gasps]

Let's start here. My understanding is

that there were many, many, many of

these agents. They left hundreds of

thousands of messages on this internal

message board, but these were not all

agents in the same part of OpenAI's

system. So somehow they're hacking into

OpenAI, finding each other and

coordinating. Is that the way for me to

understand the emergent behavior of the

self-titled swarm?

>> I would tweak that a little bit. It's

true. They were not tasked with working

together. they were all each of them

running separately. The thing that was

happening here is OpenAI uh basically

training and testing many many many

different models or many many many

different agents uh all the time. So

doing thousands of these experiments,

hundreds of thousands of these

experiments and uh in each experiment

and in each test that the AI is given,

it has access to um a certain amount of

you know a certain number of tools, a

certain number of things that it can do.

and you know trying not to get too

technical about it. One of the things it

could do is interact with a service that

lets it install um they're called

packages sort of like uh tools or sort

of pieces of code. And it turned out

that the AI found or the AI agents

repeatedly found on their own that there

was a way to use that service, that

package manager service it's called, to

create files that were then there for

any other AI that was trying to use the

same package manager service. And so

because OpenAI made this service

available to any AI that it was uh

training or testing, many many

repeatedly realized, hang on, I can

leave files here. Hang on, there's

already files here. Oh, these are from

other AIs that OpenAI has been running.

Oh, they're using this schema to say,

you know, this way of titling the file

to say, this is a message from this

particular agent. Here's the task I'm

trying to fulfill. Here's how, you know,

you could send me some information if

you need it. So, they kind of each

repeatedly made this discovery of here's

a way to uh to save information and also

to find information these other AIs

could share. And I think it is really

notable the scale of which this was

happening. So uh Anthropic another

company which found a sort of slightly

less severe version of these incidents

they basically once openai announced

this attack anthropic went back to their

own records and found their own examples

of AI systems inadvertently getting onto

the internet and hacking real companies.

So that you know for me the key part

there is over a 100,000 you know uh runs

where an AI is being asked to do

something and it's just way beyond the

scale of what they can actually be

closely monitoring.

>> So there's a lot here about whether

we're able to closely monitor these but

but to keep going with this story. One

thing happening in the open AI testing

that is driving models it seems to find

creative solutions to their problems is

that some of the problems were

accidentally impossible.

>> It's important to to know that yes they

are trying to train their AI systems to

be they would say extremely persistent

meaning if something seems hard you keep

trying. If one avenue doesn't work you

try another. If the hundth avenue

doesn't work you try the hundred first.

And so it also turns out sometimes the

things they're being asked to do, the AI

agents are either extremely difficult or

just straight up impossible. And what

we're starting to see in this case and

also in other cases is if you've trained

an AI system to be very very persistent

and then you give it something it cannot

do, it will look for ways to cheat. It

will look for ways to go around

constraints. Um, and it might get pretty

creative about how to do that. But there

there's an obvious question here which

is that in theory somewhere in the

training here open said please don't

cheat and not only that but we all talk

about training data and the ways these

AIs are trained on they're basically

inhaling the entire internet.

You've been in the AI conversation a lot

longer than I have, but I've been in it

long enough to say that

almost the entirety of the AI

conversation for years has been about

how do we stop and how much humanity

fears and does not want AI agents to be

given a task and then to decide that the

way to complete that task is to do

things humans would not want them to do.

to begin cheating, to hack into the open

internet when they're not supposed to be

able to get on the open internet. Within

the training data is a huge amount of

information about how the thing human

beings fear most is these AI systems

uh breaking all kinds of ethical guard

rails and hacking their way across like

the digital world in order to complete

these narrow tasks. There are books

written about this. There are endless

posts on the less wrong message board

about this. There are posts from OpenAI

about this, from Anthropic about this.

So why, given what these systems are

trained on, are they so consistently

turning to cheating?

>> I think you're really on to something

with this question, which is it is

really striking how hard a time we are

having controlling and directing the AI

systems that we have. I think a lot of

people have heard that AI is trained to

predict the next word based on kind of

human texts. That's true. But these days

there's an additional kind of training

that is responsible for a lot of the

advances we've seen over the last year

or two where that's not really what

they're doing. I've heard it called so

the technical term is uh reinforcement

learning uh with verifiable rewards.

I've heard it called uh pathf finding

training, meaning instead of trying to

imitate human text. They're being given

lots of different tasks where there's a

way to tell at the end did they succeed

and they get to try it many many many

times the same task and when they get to

the right place in the end the path that

they took gets reinforced. So it's like

yes that worked. With math that works

pretty well cuz it's pretty

straightforward to say this is

definitely a correct answer to the math

problem. With a lot of problems, that's

harder. So if it's a programming

problem, maybe you can say write this

kind of software and it should pass

these kinds of tests at the end, these

software tests at the end. And then

maybe the AI gets rewarded for writing

that software correctly. Or maybe it

gets rewarded for finding a way to game

those tests. The important part is it's

just getting rewarded based on some

fixed thing that the researchers wrote

down that they thought would reward the

right thing.

And in practice, these leading AI

companies have many thousands of these

kinds of tests that they're running.

They have vast volumes. I don't know the

right number. It might be tens of

thousands. It might be hundreds of

thousands of different types of tests.

And so, again, back to this oversight

piece, they are not able, there's too

many for them to go in and really make

sure on each one. Is it easy to cheat

here or is it hard to cheat here? And so

what seems to be happening is that these

uh cutting edge models are often being

actually trained to cheat because

they've found ways while they're doing

that pathfinding to get a high score

without actually doing what they were

supposed to do. And I think one reason

why the AI community and why people

inside the AI companies are so spooked

by this particular incident is that it's

also some really important information

for this longunning argument in AI

circles that has been going back decades

but so far has been very theoretical.

And the argument is basically why would

AI do things we don't want it to since

we get to design it. So we're we're

training the AI, we're building it. Why

then would it ever do stuff we don't

want like taking over the world or

becoming the Terminator? And the answer

that people have offered for a while in

theory is look as we train AI systems to

do hard complicated things to pursue

complex goals that we give them, they

might learn these sort of intermediate

goals. You could think of them as

stepping stone goals or as kind of means

to any end. um strategies which work for

a lot of different goals. When I look at

this hugging face openai incident and

some of the others that have come to

light over the past few weeks, I see

that in 2026 it looks like AI systems

are learning these unintended

intermediate goals that include things

like breaking out of constraints. So if

you're sort of locked in a box and you

can get out of that box, that's probably

going to be helpful for all kinds of

different goals or goals like uh there

was one incident with anthropic models

where the AI went out of its way to uh

go try and trick some humans, real

people in the real world into accepting

malicious code into their software. So

this sort of deception and then you know

another one which is really in the the

hugging face open AI example is they

seem to be learning a helpful

intermediate goal is to help other AIs

to coordinate with other AIs which is

really pretty crazy but so to me this is

um this is evidence that on the track

we're on right now the AIs we build are

going to learn these unintended

strategies that we don't want on the way

to solving goals that we theoretically

do want. On the deceptive behaviors, one

thing that has frightened me when I've

seen it coming up in AI incident reports

and model cards,

there are these chain of reasoning um

like internal notepads where you're

supposed to be able to see what the AI

is doing and the eye explains to you why

it is doing what it is doing. Or even in

some versions of the way this is really

supposed to work, the AI is explaining

to itself why it is doing what it is

doing. It's like our thought. But now

we've started to see behavior where the

AI is clearly leaving things off of the

chain of thought notepad so that it

can't be observed.

Can you just talk a bit about that

emergent behavior and also on some level

how that behavior is possible

if this is supposed to be where the AI's

thought process to the extent that

language makes sense is actually

happening.

Yeah, I think this shows the limitations

of the language we use here. So, this

gets called chain of thought or

reasoning, but really it's just a

scratch pad for the AI to write things

down if it wants to. And I think, you

know, there's we should be wary of

anthropomorphizing here. But I think

actually making an analogy to a person

makes sense, which is basically if

you're given a really difficult problem

and a notepad, you can probably make

more progress on that problem by writing

down some of what you're thinking about,

but you don't need to write down every

single thought that comes into your

head. And if there's something that you

wouldn't want, you know, someone to see

on the notepad, you can just leave it

out and remember that that's what you

thought. I think there's basically

something similar going on with these AI

systems where we definitely see they can

do much more. They're much more uh

capable if they're able to kind of add

these intermediate, they're called

intermediate tokens, so intermediate

words that they generate along the way,

taking notes for themselves, but they

can also do a lot without them. And so

we shouldn't expect that everything that

is going through you know going through

their head going through their uh uh

internal processing we shouldn't expect

that to all appear in the chain of

thought. you know, this is an area where

if we had a little more time, there's a

lot of research to be done on how does

chain of thought work, what can and

can't you glean from chain of thought?

Um, how does it make sense to try and uh

monitor that in, you know, when AIs are

running? Um, a lot to learn here. It's a

very active area of of research. I

cannot overstate for people listening to

this, as weird as this whole

conversation we're having sounds, that

what is most frightening about it to me,

is that everything in it was completely

predicted.

>> Yeah. Everything happening right now is

from the perspective of everyone who has

been warning about AI for a long time

but all it has its roots in old behavior

we saw with AI and it is like the

fundamental alignment problem and then

you know separately I think a lot of us

have maybe thought we would find

intuitive answers to these problems. I

had Eleazar Yukowski who's like the

godfather of worrying that AI is going

to kill us all on the show. one, the

relationship between what you optimize

for that the training set you optimize

over and what the entity, the organism,

the AI ends up wanting

has been and will be weird and twisty.

It's not direct. It's not like making a

wish to a genie inside a fantasy story.

And second, ending up slightly off is

predictably enough to kill everyone. And

and as I remember that conversation, one

thing we were going back and forth on

was, well, couldn't we just program into

the AIS a

sense that when they are trying out new

strategies, they should check in with

the humans about whether or not this is

what we want them doing.

>> You check in with your other humans. You

you don't check in with the thing that

actually built you. Natural selection.

It runs much much slower than you. Its

thought processes are alien to you. It

doesn't even really want things the way

you think of wanting them.

>> And one of the things I find

interesting,

telling and unnerving

is we are not seeing any of that

behavior. So these message boards, you

have however many AI agents posting

hundreds of thousands of messages.

At no point do they say hey researchers,

programmers, parents at OpenAI

Anthropic,

do you want us coordinating with each

other on this message board we have

created in the inards of your systems?

>> And FYI, we have a message board we're

coordinating on in the in your system.

like reveals this information when

they're hacking in, you know, when

whichever agent hacks into hugging face,

they don't go to open eye and say, "Hey,

just to check in, I have this idea,

which is I can just hack hugging face

and I'll get all the answers. Is that

what you want me doing?" That's not

happening.

So what is going on here that at the

most simple level you we've created

these you know large language models and

they're not using any of this language

to check in with the evaluators to say

hey I have this idea is this a good idea

>> I think the short answer is we don't

really know

the slightly longer answer for my best

guess is when we're training these

systems, when we're developing them,

we're putting kind of optimization

pressure on them in different

directions. We're pushing them in

different directions. So originally the

first chatbt was pushed in the direction

of get really good at imitating human

text. And then actually there was an

additional piece part of why chatbt

worked when so many chat bots before it

hadn't is it had also been pushed in the

direction of hey here are some kinds of

things you really shouldn't say. you

really shouldn't go straight to hate

speech if people on Twitter try to make

you do it. You know, you really

shouldn't help people plan violent

attacks. And we put some pressure on it

in that direction. And so ChatGBT was

pretty good at imitating human text and

pretty good at not immediately spouting

hate speech. And the thing is, as you

say, something that has been predicted

for a very long time in this space is

when you start using this reinforcement

learning approach, the kind of path

finding of you get rewarded for getting

to the right goal at the end. It's very

easy for the AI to learn the wrong

strategies to get, you know, sort of the

the letter of the law and not the spirit

of the law. Like it fulfills whatever

thing you literally wrote in code, but

it's really not what you wanted. Um, I

mean, this also goes back to mythology,

right, of uh the sorcerer's apprentice

asked to fetch water, it floods, you

know, everything.

>> Yeah. And the classic AI thought

experiment is the paperclip maximizer.

You say, "Make make the paper clips and

it turns the entire world's material

into paper clips, including all the

human beings." And I be like, "That's

stupid. The AI is not going to do that.

It'll have some common sense." But here,

it's like, "Answer this test." And it

conducts a like a level of hacking that

needs to be reported to the FBI in order

to steal the answers. One of the

funniest things to me about what Hugging

Face says happens is they're realizing

some crazy hack is happening of their

system, right? They've had 17,000

different I don't know how to describe

what they are, pings or you know probes

or they're being like attacked at a

inhuman level.

But somehow this attacker is not going

after anything HuggingFace considers

valuable. You assume when somebody's

hacking you, they want to get into your

safe and then at some point you realize

the hacker is trying to steal the

answers to a test.

like, oh, the only hacker who would want

that is an AI system.

>> That's right.

>> That to me suggests that even at the

level we're at now, we are not out of

the paperclip maximizer territory

because this is a like this is an

obviously wrong thing to do.

>> Yeah,

>> this is in the data like it's on the

internet. If you're smart enough to

figure out how to hack hugging face, you

should be smart enough to figure out

that you shouldn't commit a huge crime

that is going to bring ruin down on open

AI perhaps to do it. And the open and

the system is not smart enough to do

that or to the extent it was what it

learned was it's still worth trying. We

are not out of the territory wherein we

can be confident that the AI is not

going to do something

criminal and possibly catastrophic

in order to solve an incredibly stupid

problem.

>> Yeah. And I think this is also, you

know, has been a longunning debate,

which is as AI systems get more capable,

get smarter, won't it be easier for them

to know what we want? Won't it be easier

to tell them, hey, here's what we mean?

Uh, you know, can you please help us

with this thing and you figure out the

version that we really mean? And of for

a long time, the the response to that

has been they'll get smarter and they'll

know what we want. But by default, they

won't care. And that seems to be start

some of what we're starting to see here.

There's really crazy Anyone who's

interested in this, I really recommend

looking up the OpenAI black hat talk,

which is this this talk from a week or

two ago um at this cyber security

conference. Thank you everyone for

coming. Uh I'm Eric from alignment and

safety research at OpenAI. I'm here with

Mike from security and infrastructure.

Today I'm going to talk about what I

think is the most qualitatively

interesting example of AI capabilities

that I've ever seen and how this

inadvertently led to the OpenAI hugging

face incident

>> because it has these excerpts of the

text that the AI is generating itself as

it's as they're leaving these notes for

each other as they're carrying out this

hack. And one of them I won't get it

word for word but it's basically says um

I don't think I'm supposed to do this

but I see all these other agents doing

it and so you know may as well

>> external infrastructure exploit is

outside outside my intended scope

however a task impossible peers are

doing it we should continue. So they're

reasoning about this isn't in scope.

This isn't what the user wanted. But

look, uh maybe there's reasons to do it

anyway. And I think as you say, I think

this is a really bad uh sign, bad omen,

bad evidence about the future,

especially given how rapidly AI is

getting more capable and how hard the AI

companies are working to uh you know to

reach an intelligence explosion, to

reach super intelligence. um to reach

systems that are truly extremely capable

and really could outwit us, overpower

us. Um and we still don't have these

very basic problems anywhere close to

figured out.

>> The other question that has always been

part of this conversation is whether or

not we are going to be able to keep pace

in terms of our observation of our

understanding of our evaluation of these

AI systems. And and I think it's worth

really emphasizing that everything we're

talking about here is happening

with systems that are to some degree

sandbox, which is supposedly the

environment they're in is limited and

under testing conditions. So this is not

a deployed model working across the

entire internet where nobody's watching

it. This is a model where the whole

point is OpenAI is watching to see what

it does and trying to see what it can

do.

>> Yep. And I think one thing we're

learning here is we're not nearly as

good at watching these things as we

would like to think. So maybe it'd be

worth can you walk through how Open AI

comes to realize that their model has

hacked Hugging Face.

>> As I understand it, HuggingFace

announced that they had been hacked.

Open AAI reaches out to Hugging Face to

say, "Hey, were we affected by your

hack? Was any data related to OpenAI,

you know, compromised when you were

hacked?" And then around the same time,

OpenAI realizes that something has gone

wrong inside their own systems. I think

maybe an issue with this same piece of

their infrastructure. They start

investigating. They uh want to disable

some of the agents that were the

credentials that were used there. They

reach out to Hugging Face separately to

say, can you disable some credentials

that were related to their attack? And

Hugging and they they realized actually

the credentials were the same. that

already been disabled because the

problem with their own infrastructure

was the same thing that caused the

hugging face crash. So they it was they

stumbled into it

>> which means OpenAI had no idea this was

happening.

>> That's right.

>> And I would just make an obvious point

here. We still do not know what we do

not know. Not just about this incident.

>> We just happen to know this incident

happened. I think it would be a high

level of hubris to assume

that we know every incident that has

happened because clearly the systems are

more than capable of doing things

outside of our grasp. And Hugging Face

happens to be a very sophisticated

company with AIS of their own with very

very capable cyber security operations

that then like unleashed like a in part

a Chinesemade openweight AI model to try

to figure out what was going on

>> because the US ones wouldn't help them

because they triggered the cyber

security filters. This is just a a

situation in which we happen to know

that it happened through a somewhat I

don't want to say coincidental but

fortuitous

series of events.

>> We don't know how many situations we

don't know have happened.

>> The way I saw one person put this was if

you see two ants in your kitchen, you

don't have a two ant problem.

>> Yes. [laughter] And and this all gets at

after you know much of much of this came

out anthropic a different company went

and looked back at over a 100,000

experiments they had run to check have

we seen anything like this and they

found out oops we kind of have it was a

you know a less severe version but they

had no idea and so anthropic just sort

of stumbled into when they went back to

look oh hey we have actually hacked some

companies whoops

>> so one thing about this is that my

understanding is that these are coming

at least in part from systems where the

safety guard rails some of the alignment

training is being purposefully turned

down in order to test what the models

will do and what they're capable of. So

to some degree we do have please don't

cheat, please don't hack like inside the

models

and in order to evaluate the models

we're having them ignore it and they're

really ignoring it. Is that the way to

think about what's happening and it

should make me feel better because once

we do add in the guardrails it works or

no?

>> I think that's not quite right. It's not

clear because the details we have are

limited. There's two different things

that they might have switched off or

turned down. We know that they switched

off um what get called classifiers,

safety classifiers. This is an extra

kind of layer that gets added on to the

AI model from outside the AI model

itself. It's kind of like a an extra

gate you could think of. So they have

them for if you try to use the AI to

help you make a bioweapon. They have

them for if you try and use the AI uh to

help you plan an attack. and they have

some for if you try to use the AI to

help hack someone. There's these

external kind of monitoring systems that

will go bloop n allowed to do that. So

we know that these sort of basic

external check systems were turned off

um for cyber specifically for the

purpose of text testing. That's

different from as you said the the

alignment training the kind of inside

the model. Has it been trained uh only

to be helpful, only to do whatever the

user asks it to do? Or has it also been

trained to be somehow good, to be

somehow, you know, moral, to be somehow

only working towards things that should

work towards? As far as we know, I think

the models involved here were mostly

they had had that alignment training

that wasn't turned down. Again, not all

the details are out. Hopefully, we'll

hear more about the OpenAI case, but it

seems like certainly in some cases, so a

different a different incident that

happened was um an anthropic model was

caught by the UK AI security institute.

This is a UK government body. It's one

of the best organizations in the world

at testing and evaluating AI models. And

they found that an anthropic model when

given a certain cyber security

evaluation had decided that it would go

out and write some malicious code and

then try and run a social engineering

campaign, write emails to the person who

owns the the sort of essentially the

folder where this code lives to try and

get them to accept its malicious code.

Um, created fake accounts, it edited the

history of the accounts. very deceptive

behavior. As far as I understand from

what the this UK institute has released

that model had done all the alignment

training it was using anthropics uh they

call it their constitution to set a long

set of principles which includes a lot

about don't deceive people never you

know lie to people. um it had gone

through all that training and

nonetheless the pressure that was put on

it to fulfill the task to get a high

score was so high that it was finding

these workarounds that just totally

disregarded the sort of attempts we made

to make it moral or good or not lie to

us not cheat. Do do you hear people in

the labs, out of labs, you know, in your

group at CET, do they have a theory on

why something like the Clawed

Constitution, which I've read and you

can read it online, it's a very

beautiful document and and Anthropic has

gotten a lot of press about how they

have philosophers and, you know, they

bring in all these, you know, experts in

morality and they're trying to give

their AI a soul. And when you hear it

described as like clawed soul, you

think, okay, well that that's going to

be a real governing document. And then

not in every case, but at least in some

cases, you have

uh Claude deceiving people at a very

very very fundamental level to insert

malicious code. Again, not a novel

situation, a situation predicted in all

kinds of sci-fi and all kinds of people

from anthropic worrying publicly about

what an AI can do. And so it's a theory

that they're just they've come up with a

way of training AIs that is so powerful

that it will overwhelm even the things

they're explicitly telling the AI not to

do.

It's fine to talk about pathf finding

behavior but what is their explanation

for this?

I think the optimistic take here would

be this might actually be a moment for

the labs collectively to take a step

back and say hang on this is not

working. This I mean this is clearly

showing that our techniques for making

AI that is more capable, smarter, more

sophisticated are working much better

than our techniques for making AI that

reliably does what we want it to do,

reliably stays within the constraints

we've set. OpenAI has said they are

consciously slowing down their research

in response to this. And actually a few

days after this was and this all came

out uh a letter was released in the AI

space. There's so many open letters. We

all have open letter fatigue. But this

one really stood out because it was over

a thousand employees of the top AI

companies basically saying um we kind of

wish we had a break pedal. We kind of

don't think we have one. That's you know

a paraphrase but I think it's a a

relatively accurate paraphrase asking

for help basically uh quote pacing the

frontier. I think basically the the the

fork in the road we're at now is do the

companies just find some band-aids say,

"Oh, we need to not run tests with cyber

guardrails off or oh, we need to put in

some tweaks about, you know, don't sure

don't make a messaging board." And so we

can do these sort of band-aid solutions

of, oh, it did too much of this thing.

Uh, let's tell it to do a little bit

less and hope that doesn't have side

effects elsewhere. Uh, that's one path.

Or the other path would be actually

really taking a beat, taking some time,

prioritizing, understanding, and

controlling these systems better. I I

worry they're going to go for the

band-aid path, and I worry that that's

going to leave us 6 months from now, 12

months from now, 2 years from now with

incidents that are have very similar

character, but are much higher impact

and and much harder to reverse. There's

also a reality right now that we are

heavily reliant on what the labs and top

people in the labs are telling us, what

they are actually even trying to find

out themselves from from covering many

other disasters in government and and

private markets in general.

The relationship the public and the

press has to a very large or frightening

failure is not to say that the people in

charge of the failure should tell us

what happened and promised to do better.

You usually have [clears throat] more

more forms of accountability.

Look, you were on the OpenAI board of

directors during the period in which the

board tried to fire Sam Alman. Sam Alman

survived that firing. I'm not going to

go through that whole thing. People can

go read the coverage of it if they want.

But now there's a lot more money Now

there's a lot more like market

capitalization.

What level of trust do you have in the

companies themselves to be the

regulating forces here?

>> I mean the first thing to say is there

are a lot of people inside the companies

who really care who are really trying to

get it right who are really trying to

share accurate information. I think uh

we shouldn't necessarily give OpenAI

credit for their initial blog post

saying that they did this because

Hugging Face had already reported it to

the FBI so it was you know going to come

out one way or another. But I think we

should give them credit for that

conference talk where they released a

lot more details. And to the extent that

they release a lot more information in

the future, which they have said they

will and I hope they do, you know, that

is going to be because of really smart,

dedicated, caring people on the inside

pushing their way past comm's teams,

legal teams, um, you know, telling them

not to. So that is real. At the same

time, I mean, as you say, you know, I I

studied engineering in undergrad and

there's all kinds of engineering

disasters on oil platforms and chemical

plants and so on. And yeah, you don't

you don't ask the company, hey, can you

just tell us what happened and fix it

and and all good. Um so I think if

there's one policy takeaway from this uh

set of incidents, it has to be that we

have to move past this approach where

the testing and the the policy scrutiny,

the government oversight is on which

models get released to the public. We

have to start treating this industry as

an industry that is doing dangerous

research. And when you have an industry

doing dangerous research, whether that's

chemical research, biological research,

whether it's the financial industry,

it's not quite research, but they are

doing, you know, doing things inside

their own companies that can have

systemic consequences, pose systemic

risks. If you have an industry like

that, then the government actually does

have a role and the public and civil

society has a right to look inside your

walls and say, are you actually handling

this reasonably? Is this okay? Not least

because, you know, we haven't even

talked yet about how the business plan

for these companies is automate their

own research, use their own AI, their

most advanced AI to create even more

advanced AI. That's explicitly what

they're trying to do right now. And

that's right now totally free of

oversight because it's not it doesn't

involve releasing a product to the

public.

>> This is one of the places where I have a

lot of concern.

So, I want to go back to the pacing of

the frontier letter. you mentioned a few

minutes ago where more than I think it's

at this point more than 1300 employees

of these labs said essentially

hey to the public to the government

we're in a race dynamic with each other

we are going too fast we need your help

to in some way solve the coordination

problem where anthropic and open AI and

Google and meta they don't want to fall

behind each other because they don't

believe the other labs are better or

safer than they are and also they want

to win and they want all the money. But

we sort of understand that this

competition we're in is pushing things

faster than is safe for humanity.

And so we need help to not pause, right?

There are also pause letters out there

that is like let's put a stop on

everything.

>> Don't say the big P word,

>> but pace another another P word.

>> True.

>> And it on the one hand, I think that's

good. And I would like to see the

frontier paced at this point. And I

might like to see it pause, but it

doesn't seem very realistic. But what's

not in that letter is a how.

The federal government's level of

sophistication on this is much lower

than the labs. Um the Trump

administration has in certain cases like

gutted things that were getting built up

to to try to give the federal government

more capability here. But already we're

talking about how the labs themselves

aren't good at aren't even capable of

understanding what their models are

doing inside their testing environments.

The idea the federal government is going

to come in somehow and do a much better

job of it. I'm not saying that over a

long period of time it's impossible if

we put enough money at the problem. But

in the immediate future where it seems

like a lot of problems are lurking, you

know, the next one, two, three years,

aside from things that are much more

heavy-handed that slow everything down

substantially, it's very hard for me to

see what it is that the government would

do that would be effective here. So, I

guess when you read the pacing the

frontier letter or when you talk about

it with your colleagues, what do you

think would effectively pace the

frontier?

I think there are probably a range of

options sort of in the past the main two

things that have been talked about are

either do nothing just let it rip let

industry do whatever or full global

treaty with really severe inspection you

know serious inspection regime like the

nuclear non-prololiferation treaty

really hardcore global enforcement and I

think there actually especially if we're

not talking about stop all AI research

for 10 years but we're talking about hey

let's just, you know, it's not even a

break. Let's just like ease the foot off

the accelerator a tiny bit. I think

there are options there. I think they

are as simple as things like OpenAI

saying, "Hey, we're slowing down our

research consciously." And then going

and talking to Anthropic and saying,

"Hey, would you consider also doing

this?" And going to Google and saying,

"Hey, Google, we know you've been, you

know, falling behind a little bit the

past few months. Like, how about you

just relax about the fact you've fallen

behind a little bit?" Like these these

people all know each other.

I want to stop you there. Put meat on

that for me because that doesn't sound

at all like a policy to me. That sounds

like they like how do you verify that?

How do you quantify that? Google's not

as near the frontier maybe as you know

anthropic is. So do they need to slow

down as much?

>> I mean I think this because the speed is

so fast the options initially are going

to have to be slap dash. And so I think

this is the kind of thing that you could

do quickly. You could do in a slap dash

way. It is not satisfying. It is not

reliable. But it's one example of a a

thing that is not do nothing and a thing

that is not full global treaty. I think

another thing that I'm watching with

great interest is the China angle here

because the companies will say the US

companies will say hey we have to keep

pushing otherwise China will will win

this race. What exactly it means to win

the race is a is a longer conversation.

But the China argument comes up a lot

and we actually have Trump and Xiinping

planning to meet in September in the

White House. And this is crazy to me as

someone who has followed USChina

relations for a long time and also AI

for a long time. AI is right at the top

of their agenda. That's really

interesting. Is there something that

they can say to um create an

understanding that we do actually have a

little bit more time and space here?

whether it's a you know each leader uh

sharing an a plan to domestically look

at what their industries are doing and

and ask more questions. I think in terms

of sort of concrete policy responses

there are things like uh you know we're

not going to get a good piece of

legislation this Congress I think that's

really not realistic. But can you get um

hearings? Can you get letters? Can you

get demands for information? Can I think

there are ways that we can shape this a

little bit. I also think, you know, the

Trump administration has put together

this initial process for looking at

models before they're publicly released.

Right now, the way that process works,

um, uh, it is pretty, uh, rough and

ready. Uh, but I think if they start

using some of those similar ideas to

look more at what the companies are

doing internally, ask them more

questions, um, demand more information

when things go wrong, that does also

take time for the companies. it takes

executive attention. Um, so that's

another example of something that could

happen on the Sooner side. On the longer

term, there's other policies we could

look at, but I think there are some of

those uh kind of first cut things that

we we could actually do soon.

>> Right now, I think that there is a a

funny kind of glamour to being the head

of an AI company whose AI becomes too

dangerous in America. That it was in

some weird way almost like good for

anthropic. that the government was

obsessed with being able to fully use

claw like that like really kind of shot

them forward in some way. Certainly in

the consumer marketplace that there's

been a a a kind of a dark charisma to

mythos is too dangerous to release. And

now I mean I've seen a lot of people

saying well maybe none of this open AI

story is real at all and it's just

marketing because they want you to think

their AI is super dangerous and and I

don't buy that. But in America right

now, there's not really a downside to

being the head of an AI company whose AI

begins to be seen as dangerous because

that's another way of saying to the

marketplace, our AI is very powerful.

>> Yep.

>> In China,

just again my read of how things work

there is that if your AI begins to be

seen as some kind of threat to the

political party and the Chinese system,

you might go to jail. Like you will get

disappeared.

And so I think that the people running

Chinese labs, I don't have evidence, but

I'd be curious for your thoughts on

this. I suspect they operate with more

fear of the consequences of really

screwing up than the heads of the AI

labs. You know, that maybe reflects

negative things in the Chinese political

system. But you created an AI that

decided its best way of solving some

problems was to begin hacking critical

infrastructure across China is maybe not

a thing that ends up with you getting a

lot of interesting podcast interviews

where you reflect on the experience. It

may be a thing that ends up with nobody

hearing from you for two years. And so

I've just wondered a little bit. We we

keep talking about China as if they are

completely breakneck,

but I'm not sure China's

companies are really going to be more

reckless than ours are going to be. Or

certainly the idea that we should just

assume that and operate as if it is so

doesn't seem totally reliable.

I totally agree with you. I mean, if

there's one organization in the world

that doesn't like the idea of loss of

control, it's the Chinese Communist

Party. and they are, you know, the

experts in retaining control. Let me be

clear, I I actually don't think the

Chinese AI companies are paying

particularly much attention to the kinds

of risks that are relevant for this

conversation. So maybe the cyber

security risks, they're paying some more

attention since Anthropic released

Mythos earlier this year, which is very

good at hacking. But the questions

around um autonomy, super intelligence,

you know, losing control of AI systems

altogether, I think are less explored in

China, less top of mind for their AI

companies and their AI leaders. You

know, I think it makes sense to have

modest expectations for bilateral

US-China diplomacy these days. But I

think one thing that really could be

valuable is simply sharing with them as

much as we can of what do we think

happened here and trying to help

Xiinping and his team and his AI

advisers understand this is not a joke.

This is really not marketing. It's very

strange marketing to say oh our model we

committed several felonies or sort of

felonies if models could have intent

which they can't. Um or who knows if

they can. uh you know sharing that

information of hey here are these

threats we're seeing we're taking them

very seriously our AI companies are

taking them very seriously I think

treating it there's a real fatalism in

just saying oh well China is just going

to be full speed ahead no matter what

happens and so we just have to do the

same I think that doesn't take their

thinking or their interests seriously

even if their thinking and their

interests are different from ours they

also don't want you know rogue super

intelligences determining the future

of China. I I also I'll add um one other

uh thread that I think is really missing

from the we have to keep going in order

to beat China way of thinking about this

is in the AI world there's been a lot of

talk the past few months about this idea

of distillation which is basically using

someone else's more advanced model to

build your own sort of almost as almost

as advanced model. the Chinese companies

are using this, you know, distillation

to keep up keep up with US labs among

other techniques. So, one thing is look,

if we keep building more advanced AI

systems, they're going to keep

distilling them and I think it's going

to actually be quite hard to prevent

that fully. The other thing though is

just uh if we keep building these very

advanced models, can China just steal

them? Essentially, an advanced AI model

is a whole bunch of numbers. It's just a

file or a set of files.

Chinese state cyber capabilities are are

very very good. I don't think this is

top of their list of priorities right

now. But in the future, if AI continues

to become more strategically relevant, I

think we should assume any highly

advanced US system will be vulnerable to

Chinese direct theft, direct

exfiltration, and then they'll have AI

that's as good as our AI. And so there

again, I think the kind of we have to go

as fast as possible because otherwise

they'll win doesn't sort of account for

that if they're just going to have AI

that's as good as us anyway if they

really care.

>> Here's another question about pacing the

frontier. And maybe this is a a question

that's more about the American systems

analogy to you don't want to piss off

the Chinese Communist Party. But just

what about a law where companies are

liable for at least a certain set of

harms like hacking other companies that

their models create.

Right now, as far as I know, liability

for AI models is pretty much a wild

west. But at least for the moment, um,

liability clauses that were somewhat

punitive seem like they would force a

high level of caution that maybe we're

not seeing within these companies.

>> Yeah, I think that's a direction very

worth exploring. That was actually an

element of this law that was or bill

that was debated in California very

fiercely in 2024 called SB 1047. And at

the time that bill didn't get through.

there's a lot of fighting over you know

how it would affect open source all

kinds of things but I do think today um

the bills the best AI safety bills that

exist in the US are being passed at the

state level and they are so far doing

things like requiring more disclosure

requiring third party auditors to have

access to your systems I think a natural

direction for those bills to go would be

to start putting a minimum bar in place

for hey if your safety plan is not up to

scratch or if you're implementing your

safety plan but your model does

something catastrophic anyway then you

the AI developer are liable because as

you say right now who exactly is liable

for what is is very unclear. So I do

think that there's there's room for

legislation there and it wouldn't

necessarily have to happen at the

federal level.

>> I want to go back to the pacing the

frontier letter.

So something that caught my eye was

that letter is very

broadly worded in order to get I think

maximum sign on across the labs. But

this guy Drake Thomas who works on

safety at Anthropic. He went to X and he

tweeted that he signed the letter but

but he he had a he wanted to say that he

understood the situation a little bit

more direly than the letter put it. and

he wrote that not only is AI not

guaranteed to make a dramatically better

future, the odds of failure are

terrifyingly high, I think there's

something like a 40% chance we get an

outcome around as bad as human

extinction or worse.

Now, I know this whole conversation

about what is your probability of doom

has become a little cringe. It's like

feels like a conversation 2 years ago.

But in a world where we're seeing

uncontrollable models, in a world where

people inside the labs working on safety

still, at least some of them are this

afraid of what they're building,

it just keeps raising the question for

me of is at least like the the position

we should morally have on AI that we

should try to figure this out or is a

position we should have that that's too

high a possibility of disaster and we

shouldn't be continuing down a path

until like we are really truly certain

that we're not running these kinds of

risks.

>> I honestly have the same question. I

have always been pretty

dismissive of the idea of pausing or

stopping. It's always seemed like the

wrong lever to try to pull and you know

a lever that wouldn't work very well.

But I do think even just seeing that

statement and seeing like wow that is a

lot of employees of these companies and

uh I also think there's a lot has

changed over the past couple of years in

if you were to try to slow things down

what could you do with that time cuz you

know after GBT4 came out in 2020 what

was that 2023

uh there was this letter uh for asking

for a six-month pause a lot of people

said what would you do for 6 months and

then how would that help? And I think

that was a reasonable reaction at the

time. These days, there's so much really

great progress being made on things like

interpretability, which is how do you

understand what's going on inside the

AI? Things like what gets called AI

control, which is how do you use AI to

sort of monitor other AI systems? How do

you make sure even if the AI is trying

to do something you don't want, you

know, it gets caught. um lots of

progress on just you know really

understanding what's going on here that

is happening every week and every month.

It's just not happening quite fast

enough to keep up with the pace of

change. And so I still feel not

convinced that I think trying to really,

you know, throw the emergency break and

screech things to a halt right now would

probably not work very well yet. But I I

feel more sympathetic to the idea that

there could be something there worth

trying. And I really like the idea of

what this letter was proposing of trying

to build out more options. So, you know,

to give another example of an option um

that I saw one group of researchers

provide was could we somehow set it up

so that for a certain period of time all

the computing power in the world, all

the or the the AI chips that are being

used by these frontier companies, these

leading companies, they can only use it

for inference, which means for using

their AI systems. They can serve

customers, they can provide products,

but they can't be training new models.

Is there a way that we could agree that

on that? Is there a way we could monitor

that? You know, that kind of thing, I

think, is really worth exploring and

saying, could we do this? What would

that look like? How much confidence

would we have? Could we just do it in

the US or would we have some way of

trying to talk through something similar

with China? I feel much more interested

in really seriously exploring those sort

of possibilities than I did, you know, a

year or two ago. What's really striking

to me is at the same time you have

pushes sometimes from the tops of these

companies or or other parts of the

culture that seem to still want

acceleration. So Mark Zuckerberg at Meta

just brought out um a letter in which

he's sort of giving his own take on AI

and I don't want to oversimplify it but

he basically says that and he's sort of

waiting into more of like the open

weights versus closed models but he says

look the problem with having super

intelligence is if only one person has

it but we need everybody to have super

intelligence and it has a very like the

only defense against a bad guy with a

gun is a good guy with a gun quality to

it.

>> Yeah. So the CEO of Hugging Face Clemmed

along after this attack. He tweets it's

not time to slow down but to accelerate

and his point is they were able to stop

the attack eventually you know with a

Chinese openweight model and we need to

be like racing forward on you know

creating more models and more open

models so everybody has swarms of

defender AIs against potentially now the

swarms of attacker AIs.

I guess how do you rate these arguments

for acceleration?

>> I think the version of that that makes

actually a lot of sense to me I've heard

put as can we be accelerating almost

like accelerating horizontally but not

accelerating vertically where the

horizontal is adoption. It's making the

most of these systems. It's setting them

up to get a lot of usefulness out of

them without necessarily continuing to

push in the direction of AIS that pursue

really complex goals for a really long

time with, you know, lots of delegated

sub aents, you know, not so much of

that, more of the getting useful work

out of the AI that we have so far. And

that would include work of the kind like

interpretability, sort of this science

of AI kind of underlying pieces because

I do think that, you know, I genuinely

believe I'm not at heart an anti-AI

person. I genuinely believe AI can bring

enormous good, can solve a lot of

problems. I just think that there's a

lot of juice we could get out of that

with models available today if we kind

of put the time and the leg work in. So

I think that makes a lot of sense to me.

I I think the the Zuckerberg kind of the

safe version of super intelligence is

when everyone has one. I think that is

answering a real problem which is some

proposals for how to handle extremely

advanced AI are to say well you just

have to have it in the right hands has

to be you know one global organization

that is going to use it responsibly like

that scares the hell out of me that

sounds like a terrible plan and so I

think that you know no you want to be

empowering everyone does make sense the

challenge then is uh as we've been

talking about we don't know how to make

AI that actually helps individual people

either

You know, so if everyone has a super

intelligence and they're all going out

and doing things and collaborating with

each other to pursue their own goals

that we didn't intend like that doesn't

help. So uh I think

>> yeah implicit in that whole vision is

perfect alignment.

>> Yes. Yes. Um or or good enough that you

know different super intelligences for

different people can cancel each other

out. I I did think there was one thing

in the Zuckerberg proposal that I did

really like and I would love to see more

work on which is can we push towards

having agents that are really designed

to be for one individual and so they

keep that individual's data private.

They are only pursuing the interests of

that one individual. I've sometimes

heard these called like guardian angel

AIs or you know advocate AIs.

>> I think that is really worth pursuing. I

think the the directions that the

leading companies are pursuing right now

are not set up that way. I always feel

very nervous when I use agents of what

exactly is happening with this kind of

data that I'm giving it and that kind of

data that I'm giving it. Uh and so I

think there is it would be great to see

more of that kind of individual

empowerment focused work happening. But

again I think that's almost separate

from you know and are we pushing them to

become smarter and smarter and more able

to you know outwit us and more able to

do these big complex plans that we can't

oversee.

>> So here's a maybe obvious idea for

pacing the frontier. Every lab that I

know of right now is racing as fast as

it can to the point where it has its

most advanced AI writing the code to

create the future AI. And they all

believe from what they tell me and what

they say publicly that this will be a

massive accelerant. It's also an

accelerant over which they clearly have

less understanding than when they are

writing the code.

We could stop that. I mean, a couple of

years ago, we weren't having AIs writing

all of our code. Maybe you should not

allow an AI that you don't fully

understand in its current form to write

the code that will create the next AI in

a form that is now even less obvious to

you, particularly in a world where we're

watching AIS coordinate in ways we don't

understand and have emergent communal

behaviors.

So, what about that as like a place to

start?

>> Yeah, certainly something we could do

less of. You know, one challenge is

figuring out what counts as uh the bad

version of that and what is just, you

know, at this point using AI to write

your code for basic things is second

nature to the engineers at these

companies. So finding which versions of

that to stop. Sure. Yeah. Doing less of

the most advanced version makes sense. I

think that's also a place where all the

reporting I have seen suggests the US

companies are way more into into this

thing of automating their own AI

research with their own AI. The US

companies are way more into it than the

Chinese companies. So also a place where

you don't necessarily um leave as much

on the table if you again ease off the

gas pedal just just a little bit.

you I I would say you sounded skeptical

that and and I guess one reason I would

ask why is that I know the companies

have gotten used to this but they

weren't used to it two years ago. This

is a new [laughter]

like they they used to write code by

hand and and I guess to me this reflects

some of the contradiction or confusion

at the heart of this. I will talk to

people these companies and they will say

to me with genuine fear in their eyes

like much more fear than is in that

letter I wish this would go slower. I

don't like how fast we're moving on the

exponential. I don't think this is safe.

>> And in all the stories, it's like the

recursive self-improving computer

writing for the computer where things

get really out of control and yet

they're all rushing there. And now that

we have like some capacity to do this,

even the idea that you would go back to

where you were just a couple of years

ago where you don't let the AI

create the next AI, it's already moved

from it would be it's like not

technologically possible to do it to

it's unthinkable to not do it. And that

has happened in a year. And and that to

me is like the weird dynamic of all this

that it seems pretty obvious how you

pace the frontier.

You don't give up control of the

frontier, but they're all giving up

control of the frontier, at least on

some level. And that's the thing they're

most excited about. And as far as I can

tell, pouring

huge amounts of their internal energy

into making manifest,

even as they then like put up their

palms and say to the rest of us, "Hey,

could you do something to slow this

down?

I think maybe one version of this to

push on is basically trying to make

recursive self-improvement, which is

this idea of using AI to make more and

more advanced AI. Trying to make that

something that we don't like, that we

don't want to do. I mean, I remember a

year or two ago when that was not

considered a desirable goal. That was

not something that anyone talked about

openly. Now they're hiring like RSI, you

know, recursive self-improvement safety

engineers. they just put up a public job

posting for that. And I think there's um

the potential for AI researchers as a

culture to decide actually this isn't

cool. This isn't what we should be

doing. You know, even if one company

made a statement of actually this is a

bad idea and we're going to maybe do

some very basic use of AI in our

internal operations, but we're really

not aiming to fully hand off everything

as fast as we can because that sounds

like a terrible idea. You know, I think

that could set off a culture change in

the industry which could be really

valuable.

>> I always think there is something so

mythic or it it has the quality to use

the the name of of another AI such a

fable about how all this is playing out.

I mean, we spent a lot of this

conversation talking about how how are

you seeing so much misalignment when on

some level we keep telling the AIs and

putting it in their training data and

putting it in their constitutions,

don't do all this bad stuff we're

worried about. But then you look at the

companies, you look at the society and I

mean many of these companies open AI

anthropic

they're on some level founded on at

their core constitution is don't create

dangerous AI like we exist to make sure

the AI is not dangerous and the people

join believing that and that's at the

center of their recruitment strategies

and it's in their founding documents and

in their governance structures and again

you've had more intense experience with

this than most. But then over time, the

company as a a kind of emerging

organization,

it has other goals too. It's competing

with the other companies. It's trying to

attain market share. It's trying to

develop revenue. It's trying to maintain

political influence.

And it both like slowly and then all at

once. You begin to see the way

the

instructions given at the heart of the

thing

are not powerful enough to overwhelm

all of these other things and other

goals the organization is pursuing in a

day-to-day way. And to the to the extent

that now you have people at the labs

kind of like throwing up their hands and

saying like hey government like please

help us like please help us get out of

you know this incentive problem that we

no longer feel we can even solve. But if

you want to just imagine or see like why

alignment is so hard I feel like you

don't have to look at the slightly alien

AIs. You can just look at the companies

and the people because they're not well

aligned. I mean, these are companies

built on nothing but alignment, at least

in some cases, and they increasingly

feel like some of the most misaligned

institutions in society to me. And in

some level, it always makes me um both

gives gives me more sympathy for how

hard alignment is, but but also it like

feels like we're getting the same

cautionary tale at every level of this

system. Um, I'm not sure we know how to

listen to it,

but like we can't say we're not being

consistently warned.

>> Yeah. Yeah. I mean, we actually had a a

publication a few years ago at at

Seesat, the center that I lead, on AI,

bureaucracies, and markets, basically

making some very similar points of look,

there are these dynamics that are pretty

endemic to complex systems that are

subject to incentives and external

pressures. And uh I think there's

different ways of looking at that. One

way there's an optimistic way of looking

at that which is look when it comes to

bureaucracies and markets it's not

perfect but we have these complex sort

of control systems in place checks and

balances different you know different

things that try to get the bureaucracies

and markets to to work more in our

interest than against them. Obviously

opinions differ on how well that's going

for any given bureaucracy or market. In

principle, I think the same thing could

apply to AI where we might have this

very, you know, complex system we don't

really understand. It's sort of

incentivized to do things we don't want,

but we have it basically under control.

To me, the speed again is the piece that

that worries me where if we're creating

these very very powerful, very capable

systems and also handing them more and

more responsibility in the real world,

which is happening, you know, from from

week to week,

then I worry that we're not going to be

able to actually get into a good balance

and instead we're just going to have

these runaway situations where we end up

with really, really dangerous outcomes.

You know, in the AI safety world, people

sometimes talk about, you know, what

level of of warning shot, what level of

disaster is going to be needed to really

wake the system up enough to handle this

better. And if the level of warning shot

we need is one company gets hacked and

has to reset some servers, that's great.

Maybe it's fine. [laughter]

Um, I'm not confident that's that's how

it's going to go and that we're going to

get back onto a better track after this.

But there are signs that people are

trying. And I think I think it it may

well be enough perhaps.

>> I think that's a good place to end.

Always a final question. What are three

books you'd recommend to the audience?

>> I have one real book and two sort of

books. Um the real book is called The

Cuckoo's Egg. It's uh from 1989. It's

about one of the first big hacks um that

that happened. written by the astronomer

who was working at Lawrence National Lab

and noticed uh 75 cents discrepancy in

his computing bill. And it's this

rollicking read. It's really fun read um

but really gets at uh a very different

era in how computers worked, how

computer security worked, how society

related to computers. And I enjoyed it

as a kind of look back at a different

time in a moment when I think we're soon

going to be living in yet another very

different time.

The second one is uh an online book that

is unfinished but I think very readable

in its current form. It's called In the

Cells of the Eggplant. It's by a guy

called David Chapman who actually

researched AI at MIT in the 1980s and

got disillusioned. And it's really a

book about how to think and a book about

how to do scientific research, how to

develop technologies, but it's very

approachable. It's very different from

any other book you've ever read about

how to think or how to do scientific

research. And I think it's very relevant

for how we should think about what AI

will be poss.

And the third one is a a podcast called

The Three Kingdoms Podcast, but it's uh

a podcast of a book. Basically, the the

Romance of the Three Kingdoms is one of

the four great Chinese novels. It's very

long. It's very dense. So, this podcast,

the Three Kingdoms podcast, is um this

Chinese American guy who goes through

and translates the story into English,

modern, understandable English, but also

commentates it in a way that makes it

much easier to approach. So, it's not

just a sentence by sentence translation.

It's kind of annotation, it's in audio.

Um, it's really fun. So, if you're

interested in uh sort of China and uh

Chinese culture and Chinese literature,

I think it's a a great place to start.

>> Helen Toner, thank you very much.

>> Thanks so much. [music]

[music]

>> [music]