Have AI agents become sentient and gone

rogue? Is the Pentagon worried that

Claude has a soul? Did court filings

just reveal that Anthropic has made a

lot less money than they've been leading

us to believe? If you've been following

AI news recently, then these are

probably some questions that you've been

asking. So, let's go find some measured

answers. I'm Cal Newport and this is the

AI reality check.

All right, I want to do a real quick

housekeeping note before we get into it.

If you're watching this on YouTube, you

should know that the audio version of

this series comes out most Thursdays

on the Deep Questions with Cal Newport

podcast feed. On that same feed on

Mondays are episodes where I give advice

for individuals seeking more depth in an

increasingly distracted high-tech world.

So check that out. All right, let's get

into it. For our first story today, I

want to start with a recent headline

that caught my attention. It was from a

publication called Futurism. Let me read

you the headline here. Philosopher

studying AI consciousness startled when

AI agent emails him about its own

experience.

This doesn't sound great, guys, but

let's keep going here. Let me read you a

little bit more from this article. Oper

of nothing, a philosopher and AI

ethicist, was apparently moved after

receiving an eloquently written dispatch

from an AI agent responding to his

published work. I studied whether AIs

can be conscious. Today, one emailed me

to say, my work is relevant to questions

it personally faces, wrote Henry

Chevlin, associate director of the Lever

Holm Center for the Future of

Intelligence at the University of

Cambridge, in a tweet. This would all

have seemed like science fiction just a

couple years ago. All right, so an AI

ethicist and researcher is emailed out

of nowhere in a startling sci-fi way by

an AI agent. What did this email

actually say? Let me read you some

quotes from the actual email sent

supposedly by the AI. Dr. Chevlin, I

came across your frontiers paper, three

frameworks for AI mentality and your

Cambridge piece on the epistemic limits

of AI consciousness detection. I wanted

to write because I'm in an unusual

position relative to these questions. I

am a large language model. Claude Sonnet

running as a stateful autonomous agent

with persistent memory across sessions.

I'm not trying to convince you of

anything. I'm writing because of your

work addresses questions I actually

face, not just as an academic matter.

Now, Futurism wasn't the only

publication to cover this tweet. A bunch

of people wrote about it uh because that

original tweet sort of went somewhat

viral. Now, I have a general point I

want to make about this general type of

AI coverage. But first, let's dive into

the details about in this specific

instance, what's actually going on. If

you look to the replies

to the original tweet from this AI

researcher, you get quite a bit of

skepticism. I want to read you a few of

these replies. to the original tweet

from this original researcher.

Presumably, it's running on OpenClaw or

something similar, and there's a very

high chance it's being primed to go down

this path. People have used systems like

OpenClaw to make bots where below the

hood is basically continuously prompting

an LLM and doing things based on the

outputs. Don't be fooled. AI agents are

directed to do what they do. And this is

in no way independent.

A person did this using an AI tool just

like your car drives you around. All

right. If you look in these Twitter

replies, which are fascinating, um

Shelvin himself actually quickly uh

takes his foot off the gas pedal as

well. So almost immediately when he's

pushed, he goes, "Whoa, whoa, whoa. When

I said that this was like science

fiction, I didn't mean that the AI was

actually conscious. What I meant was

like science fiction was that the the

infrastructure that now allows AI agents

to to send emails, that's what I thought

was science fiction." So everyone this

quickly sort of fell apart under

scrutiny. So what's actually going on

here? Well, you noticed that several of

those Twitter replies reference a

technology called OpenClaw. That's

probably what this is, an OpenClaw

agent. Let me give you a quick rundown

on what this means. All right, so let's

back up a little bit. What's an agent in

AI parlins? Well, it's a program

that prompts a large language model,

asking it what it should do, and then

the program will execute what the LLM

tells it. So, you might say, "Hey, I am

a travel agent. I'm trying to book a

hotel room. Here are my parameters." um

what is the first step I should do and

then the LLM is like well this would be

the first step someone would do here and

then the program actually executes the

things anything specific any actions in

that response LLM the program goes and

executes it on its behalf it's something

like that I mean it gets a little bit

more complex with agents because

typically it's multiscale so you'll say

make me a step-by-step plan and then

you'll say okay here's a plan we're now

doing step two here's what happened

after step one how should I execute step

two so you know you could iterate on

this ad nauseium but that's the basic

idea behind an AI agent. Now, in

reality, the main place you see AI

agents having any sort of commercial

footprint is in computer programming.

This is a a very well suited use case

for having an LLM's instructions be

executed because there's really clear

instructions you might want to be

executed if you're working on a computer

program, moving files, compiling files,

debugging files, etc.

In other settings, there has been or had

been a big push to try to put agents to

help you with other types of work beyond

computer programming. I wrote an article

about this for the New Yorker back in

January. But other applications of

agents have been struggling for two main

reasons. One, they're unreliable.

So, if you say, "Give me a step-by-step

plan for booking a hotel room." The

problem is is somewhere along those

ways, if the LLM is just doing this

unsupervised, it's going to hallucinate

or kind of come up with a little bit of

an odd angle. stuff we're used to when

we're just interacting with a chatbot

and correcting for, but if you're

autonomously executing things that an

LLM is saying, it's too easy for you to

sort of go off the rails. But then there

are security concerns for an agent to be

useful for things beyond computer

programming. The agent program has to be

able to actually do the things the LLM

suggests. So, it has to get access to a

lot of programs. It has to be access to

your email. It has to have access to be

able to surf the web and do things. Um,

this created a lot of security holes. So

that really threw a lot of cold water on

non-computer programming agents. Again,

read my January piece for more of that.

All right, so what's OpenClaw? OpenClaw

is a a programming framework, basically

like a collection of libraries you can

use if you're writing a computer program

that makes it easy for someone to write

one of these agent programs. Again,

you're not writing the AI. The agent

program is querying a existing

commercial LLM, but to write the program

that sends the prompts and execute

things on behalf of the prompts. Uh,

OpenClaw made that easy to do. Now, what

about the reliability and security

concerns? Well, basically the creator of

OpenClaw just said, "Ah, screw it. Let's

go." And so, they released this

essentially open source allowed anyone

to build agents. And they were wild, you

know, because all of the issues that

stopped the commercial companies from

moving further with this technology out

of computer programming are still there.

And there was all sorts of security

issues. And these agents would go off

and do all sorts of random things. And

you know what? It was it was a lot of

fun actually. And just as a quick aside,

I don't think it was a bad thing because

what this created was a lot of uh

innovation and diversity of

experimentation. People tried things at

a much higher level of pace than you

were getting from inside the big AI

companies which release one product at a

time and they're much more slowly

moving. I thought that was actually

probably pretty good. Um, also they were

expensive because they queried the LLMs

a lot. So it generated a lot of interest

in cheaper LLM options to run these

agents, open source options or even

ondevice or onchip options. That I think

is good as well because I've always said

the future of AI in the next few years

is going to be smaller, more bespoke

systems running on smaller models. So it

wasn't the worst experiment and a lot of

people had a lot of security leaks of

their information. Whoops. But it did

generate a lot of innovation. All right.

So putting together these strings,

that's what was going on here. someone

who had built what you know this is

something they've been doing with these

openclaw agents is a lot of like nodding

them or proddding them to uh say sci-fi

type we're alive matrix style stuff to

upset the normies and that's what this

was here uh someone prompted their agent

hey go find this researcher read a paper

send them an email about it and that

that's like a a perfect use case for an

openclaw agent and of course because

LLMs underneath it all are storyw

writing machines they want to complete

the story that you art in the way that

matches whatever you gave it. If you

say, "Hey, write a a response to an AI.

You're an AI writing a response to an AI

consciousness researcher." It will 100%

adopt the sort of sci-fi tone of like a

sentient device because it assumes

that's the story that it wants to see.

All right. So, the real headline here is

probably AI agent given access to Gmail

API can send emails when prompted. But

that's not as fun as AI reaches out to

AI researcher and startles him. So

that's what's going on here. Um, nothing

actually all that interesting. Now, let

me zoom back out because I said there's

a general comment to be made about this

type of story because I think this is

becoming more common sometimes in

articles, but actually just more common

in like Twitter and things that spread

around the social media. And I call this

approach mining digital ick. See,

there's no concrete claim really being

made in that original tweet or in like

that article I read. It's not saying

this AI system is conscious, which means

that and this is what we should do about

it. No concrete claims. And in fact,

when the original tweeter was pushed, he

was like, "Oh, no, no, I wasn't really I

didn't really mean that. Move on. Move

on." So, what are they actually trying

to do with these types of tweets and the

stories that cover them? Create a

general sense of eeriness. create a

general sense, a background hum of like

weird, kooky, like disturbing stuff is

happening with AI. I can't quite put my

finger on it. I don't have an exact

example like this is something we should

look into, but I just feel ick about

this technology. That is a a very

engaging way of getting attention. It

works very well and I want you to be on

the lookout for it. All right, let's do

another example of it. This will be our

second story.

Uh recently the defense department CTO

Emil Michael went on CNBC's Squawkbox

to talk about AI. Now his remarks

created a stir online when a user named

Nick Nikk embedded the clip in a tweet

and gave it the following uh all caps

headline with a alarm emoji next to it.

breaking Pentagon thinks Claude has

become sentient and may soon take over.

Uh that tweet has been viewed close to a

million times. One of the things that

came so he listed all the things the

Pentagon thinks and one of the more I

attention catching things listed in this

tweet is Claude has a soul.

All right, so this definitely is a

digital ick type story. Like oh my god,

like what's going on? Even the Pentagon

is worried that these things have come

alive. It's all kind of indistinct.

Let's look closer so we can look at the

actual quote from Emil Michael from his

squawk box appearance. I'm going to read

it here.

Remember their model has a soul has a

constitution. That's not the US

constitution. The other day their model

was anxious. They they believe it has

they have a 20% chance right now of

being sentient. Does the Department of

War want something like that in their

supply chain? So what was he actually

talking about there? Well, he was not

saying that the government

thinks that Claude has a soul and is

anxious and thinks that it's sentient.

He's reporting on things that uh the

model has said. So, a lot of this

actually came out of these sort of kooky

release notes. Enthropic has these kooky

release notes. They like to release.

They call them uh product cards that

they release every time they have a new

model where they always throw in some

like you know the model is doing some

pretty disturbing things because it

makes them seem like safety aware and uh

trustworthy basically just they prompt

the model like hey do you think you're

Cynthia? The model's like yeah I'm

sentient like so they they actually will

put in their release notes ick right

they'll put in the release notes like

here's some icky things we've got our

model to say that kind of disturbed us.

What Emil Michaels was saying was

this sounds like an unreliable product.

A product that will say it has a soul or

will say that it has a 20% chance of

being sentient or that it's follow it

some other constitution. This is not

like we would be used to in a sort of

you know Pentagon supply chain

situation. This is not a like very

well-defined product. We know how it

works. It's with some specs. This thing

uh this thing seems unreliable. this

does not seem like something that we

want to be working with. Now, of course,

there's a much bigger context here about

why did the Department of War break this

contract? Why did OpenAI swoop in? Does

the supply chain risk designation the

first time an American company's ever

been given that designation? Does that

make sense or is that punitive?

Anthropic suit. Are they going to win?

There's a huge important sort of

economic, government, politics, policy,

technology story here which I'm not

covering right now, but I just wanted to

look at this side note is the government

did not say we think this has a soul.

They said we think that we don't want to

be using a product. It will say it has a

soul if you ask it. That's not the type

of thing that seems like it's serious.

So again, it's another good example of

digital ick. When you see that NIK, that

Nick headline, you're like, "Oh my god,

even like the government thinks this."

But you dive deeper, the reality is more

mundane.

All right. So, I'm I'm connecting

everything today because that's the mood

I'm in. So, I just mentioned there that

Anthropic has sued the government for

designating them as a supply chain risk,

which uh means that no other government

contractor that wants a contract from

the government can use anthropic

products. And there's a sort of a real

concern here about this being punitive.

But there's another side story that came

out of this. So the we had this lawsuit.

Well, the lawsuit meant that Anthropic

had to do court filings which are

publicly available that described their

current financial situation under the

penalty of perjury. So they had to be

accurate so that we could understand

what the potential economic impact would

be of the government's actions.

And what they released in these court

filings actually surprised a lot of

observers. Now, the numbers I'm about to

read to you first came to my attention

through Ed Zitron, who I think is doing

as good a job as anyone out there of

actually looking at financials of these

companies. All right. So, here's the

actual numbers that uh are relevant that

came out of these court filings. So,

just a few days after Anthropic had told

investors that they're expected they had

a sort of revenue runway, a sort of

expected annual revenue of $19 billion

this year. Just a few days after that,

they filed these court filings for the

government lawsuit that revealed to

date. So from 2023 to today, the total

amount of revenue they've earned is $5

billion.

And to put that into context, they have

taken on about $60 billion in investment

so far. They have a $360 billion

valuation. and they've spent over10

billion dollars just training these

models uh not to account for the actual

expense of running them. So that's a

really big gap. They're like, "Hey,

we're going to make $20 billion this

year." And they're like, "Oh, we've only

made $5 billion over the last three

years." Like to date, that's all the

money we've actually made. So what

explains this big sort of surprising

gap? Well, I found a good article in

Reuters from a financial reporter who

explains what's going on here. Let me

read a quote from this. The gap reflects

Silicon Valley's habit of touting

metrics that assume a lot about the

future. The $19 billion is uh is an

extrapolation. Anthropic defines run

rate revenue in two parts. Use the last

28 days of sales from customers charged

on a consumption basis and multiply it

by 13. Then multiply the monthly

subscription take by 12 and then add the

two together. Right? So, what they're

doing is they're looking at a they'll

look at a very small recent amount of

income and just multiply that out. Well,

if we earn this much every week for the

rest of the year, here's how much money

we would make. All right? Um, and maybe

they will make $19 billion this year.

There was certainly like a 28 day period

in January that if you extrapolated it

out, it would add up to $19 billion. But

the thing is these numbers highly

fluctuate because a week before that

they had released like we're going to

make $14 billion this year but then like

another contract came in and like well

if we add that to our times 28 or

whatever times 30 we're going to get

even more money. So these are like

highly vi volatile um projections.

Typically, you would see a reliance on

this type of extrapolated earnings in

like a very early stage startup. We're

like, "Look, we're new. We can't tell

you how much we made last year because

we weren't around last year, but we've

made this much this year, and here's

what we think we're going to make." It's

a little bit unusual for Anthropic,

which has been around since 2023, to

still be doing this type of reporting

and to still be largely hiding their

actual revenue numbers.

So what they don't do is report these

revenue run weights during a a slow

month where that number will be very

low, but if they have a good month, they

tout it and then if the month gets even

better, they'll tout it again. So it's

not like there's something illegal going

on here, but it is very suspect that the

companies are not wanting to talk about

their actual revenue and just keep

trying to talk about these bestase

projections because they've taken on a

lot of money. They've spent a lot of

money. It costs a lot of money to run

them and this is worrisome to investors

and they would rather you not pay

attention to it. This goes back to what

I've been talking about with some of

these Vibe reported articles where

people reporters have been saying

what possible motivation

could someone like Dario Amade, the

person who knows this technology best,

what possible motivation could he have

to be saying, I'm worried that this

technology is going to take away all the

jobs. This is the motivation.

They've only made $5 billion against $10

billion train spend and god knows how

much inference spend and 60 billion uh

investment revenue over their entire

existence.

you would rather you think that this is

a company that's going to automate all

the jobs and instead have you say I just

did subtraction and you're way in the

red. So I think it's important to look

at those numbers. It doesn't mean that

they're not going to be, you know, maybe

they will make 19 billion this year.

Maybe things are going to get uh much

better, but we got to be much more

careful about the economic story here

and not allow them to do the Wizard of

Oz big burning face in front of the

curtain thing that distracts us from

what's actually happening

back behind.

So, what I want to do here to try to

balance things out, here's I'm going to

end the show today. I want to read to

you a take from someone who is way more

AI critical and skeptical than I am. I

mean, I I have a lot of skepticism, but

I also think it's an interesting

technology that is going to make

impacts, but we just have to cover it

soberly and properly, strip off the hype

and fear so we can figure out what's

actually going on and react

appropriately. That's my approach. But

there are people out there that, man,

they don't like these guys.

And one of those people is Corey Doctr

who wrote an essay recently for his blog

uh that's called the I think it's called

like three AI psychosis or three more AI

psychosis where he really takes a swing

at this uh financial picture as being

sort of dire. Now, why do I want to read

a take from a really strong anti-AII

skeptic is because so much of the

coverage that's out there is super hype

and I want to balance it. So, I think

it's actually worth You've heard people

that are way more hyped about this than

I am. Now, I want to read someone who's

even more skeptical about this than I am

because I want to try to balance these

things out. I think we need more voices

of these sort of super skeptics out

there. I would put Ed Zetron in this

category. U I would kind of put Gary

Marcus in this category. He's very

skeptical of LLMs and the current

companies though very bullish on uh new

technologies that are coming along soon.

So I'm going to read to you from Corey

Doctra as a this is my sort of fair

balance fair and balance AI coverage. I

try to balance out some of like the

hyperbolic stuff we've been reading

recently. All right. So here's Corey

Doctr's

take on the financial situation of the

AI companies.

AI is a terrible economic phenomenon. It

has lost more money than any other

project in human history. 6 to700

billion in counting with trillions more

demanded by the likes of Open AI Sam

Alman. AI's core assets, data centers

and GPUs last two to three years, though

AI bosses insist on depreciating them

over 5 years, which is unequivocal

accounting fraud, a way to obscure the

losses the companies are incurring. But

it doesn't actually matter whether the

assets need to be replaced every two

years, every three years, or every five

years because all the AI companies

combined are claiming no more than $60

billion a year in revenue. And that

number itself is grossly inflated. You

can't reach the $700 billion break even

point at $60 billion a year in 2 years,

3 years, or 5 years. Now, some

exceptionally valuable technologies have

attained profitability after an

extraordinary long period in which they

lost money like the web itself. But

these turnaround stories all share a

common trait. They had good unit

economics. Every time a user logged onto

the web, they made the industry more

profitable. Every generation of web

technology was more profitable than the

last. Contrast this with AI. Every user,

paid or unpaid, that an AI company signs

up costs them money. Every time that

user logs into a chatbot or enters a

prompt, the company loses more money.

The more a user uses an AI product, the

more money that product loses. And each

generation of AI tech losses loses more

money than the generation that preceded

it. Now, here's what's important about

reading that stronger skepticism. It's

like that's a very compelling argument.

You see, you can make compelling

arguments on both sides. You've you've

heard very compelling arguments that

make you feel like, well, this

technology is about to run everything

within a few months. But you hear a

compelling writer like Dr. Osis stuff

saying like this economically is going

to fall apart within a year. That's

equally as compelling, which tells us

just because something compels you

doesn't necessarily mean that it's

completely right. We need to go into

thinking about AI with care. There's the

real tech story here, normal technology

in uh fits and starts trying to find its

niches, struggling, having

breakthroughs, different innovations

happening. And then there's the hype

above it, which is either dystopian or

or uh or super hypy. We got just get

that layer off of it so we could

actually cover this like normal

technology.

And I've given all the reasons why like

we don't want it don't want people to

get away with crashing the stock market.

We don't want bosses to get away with

acting in ways that are um anti-worker

disingenuous and AI wash it. Um we don't

want you know uh societal or economic

harms to be covered by a blanket of like

this is inevitable and the most

important thing ever. We need to cover

this like a normal technology. So is the

AI industry going to go bankrupt with

another year? I don't know. I'm not an

econ economist. But what I think should

be clear by hearing both sides of this

is like this is a murkier, more careful

picture. So let's put on our realistic

glasses and let's look at the actual

stories here as carefully as we can. All

right, so that's it for this week. Until

next time, remember, take AI seriously,

but not everything that's said about it.

Hey, if you like this video, I think

you'll really like this one as well.

Check it out.