HomeVideos

ChatGPT Can Now See Everything You Do On Your Screen

Now Playing

ChatGPT Can Now See Everything You Do On Your Screen

Transcript

434 segments

0:00

OpenAI shipped what you could call the

0:01

most controversial feature that you

0:04

could give any AI. ChatGPT can now be

0:07

Big Brother and look over your shoulder

0:09

as you work throughout your day so it

0:11

remembers everything. And up until now,

0:14

AI only remembered what you would tell

0:16

it. Now, it knows what you actually did.

0:19

And that, on the one hand, is extremely

0:20

powerful and I want to talk about that

0:22

and give this a spin. On the other hand,

0:25

also really scary. We'll talk about

0:26

that, too. So, let's start with the

0:29

facts, okay? This is a brand new feature

0:31

available in the ChatGPT desktop app and

0:33

this is probably the most limited

0:35

rollout that you could imagine. It's

0:37

only available in the desktop app on

0:41

Mac. Then, you need a pro, business, or

0:43

enterprise plan, okay? So, for

0:45

individuals, at least $100 a month to

0:48

access this. And then, it's also

0:50

unavailable in Europe, Switzerland, UK.

0:53

So, I'm on a Mac desktop computer here.

0:55

I'm on a pro plan and I VPN'd into the

0:58

US to access this. And then, if you go

1:00

into the settings here inside of the

1:01

ChatGPT desktop app right here. If I

1:03

click the settings right here, you will

1:06

see a new tab called computer history in

1:09

the integrations category. Let ChatGPT

1:12

follow your work. ChatGPT can summarize

1:15

your activity across the apps and

1:16

websites you use without ever recording

1:18

your screen or audio. Ask about what you

1:21

were working on, get help without

1:22

explaining everything again. And this is

1:25

the best part, actually. Or, discover

1:27

opportunities to automate repeated

1:30

tasks.

1:32

Ladies and gents, this is the missing

1:34

ingredient in AI today. It has become so

1:36

capable and so many users have become

1:40

extremely good at, you know, piecing

1:42

different systems together and tinkering

1:43

with it full-time and automating a part

1:45

of their life and their work, but it has

1:48

been extremely technical. And me, as a

1:51

teacher of these things, there's just

1:52

certain concepts that you can't even

1:55

communicate without getting into

1:56

terminals and vector databases and

1:58

multiple memory systems working

2:00

together. I'll elaborate on that a

2:02

little bit, but first let me show you

2:04

one example from the blog post, which I

2:06

think will communicate the power of this

2:08

really well, and it's this one. Reuse

2:10

workflows, okay? So without computer

2:12

history enabled, you would say something

2:13

like, "I just finished my video upload

2:15

workflow. Make a skill for next time."

2:18

And it would tell you, "Okay, please

2:20

share the steps and the decisions along

2:22

the way, and I can turn them into

2:23

something reusable." With this computer

2:25

history feature, "Created a video upload

2:27

skill from the steps in your recent

2:28

activity. You can review it before using

2:30

it again."

2:32

And not just that, because it records

2:33

everything on your computer, you can

2:35

then also ask which tasks that I did

2:38

could you automate the next time. Like,

2:40

so powerful, but also so scary because

2:43

all of a sudden ChatGPT app is watching

2:45

everything you do. So if you were scared

2:48

before of, you know, OpenAI taking your

2:50

data and your ideas, well, this is the

2:53

ultimate version of that. It literally

2:55

sees everything, and as per usual, I'll

2:57

be your scapegoat here, where, you know,

3:00

I'll go ahead and I'll turn this on, and

3:02

let's see what this allows us to do.

3:05

Okay, "You're in control. Sure. I can

3:07

pause, clear my history, and manage

3:08

which apps are included at any time.

3:10

Consider pausing or excluding sensitive

3:12

apps such as health or finance from your

3:14

history. You also agree that you'll

3:15

pause computer history during

3:17

communications with others, such as

3:19

video conferences or text messaging

3:20

apps, unless you have their prior

3:22

express consent." I mean, they just put

3:24

this here for legal reasons, but

3:25

basically this would be spying on every

3:27

conversations that you have with

3:28

everyone, including text messages,

3:31

social media that you use on your

3:33

desktop computer, and all Zoom calls.

3:35

Let's start with customize apps. Exclude

3:38

these apps. Okay, I'll exclude messages,

3:40

and I'll also exclude my OpenClaw agents

3:44

web hub. I mean, it's kind of tricky,

3:46

this default of exclude these, because

3:48

then you have to think of every single

3:49

thing that you want to exclude. I'm

3:51

going to go to include only these apps,

3:53

the finder, text edit, and then I guess

3:55

Chrome is going to be essential. And

3:57

then yeah, let's continue with this. So,

3:59

Google Chrome, text edit, and finder.

4:01

And there it is. That's the base setup.

4:03

So, no computer history yet. So, how

4:04

about this? Let's put this on one side

4:06

here and open a new Chrome window right

4:09

here. And let's just do some work. All

4:12

right, so just opening up Google Docs.

4:14

And what do we see here? Skysite session

4:16

start. You have no captured active work

4:18

on this 10-minute window. Segment only

4:20

recorded session start even. Okay, so

4:21

it's recording now. Let's just do

4:23

something simple. Let's say I am writing

4:26

up a document on community onboarding

4:30

redesign. That's something that we're

4:31

doing in the Advantage Club right now.

4:32

It's been a huge project. We use a bunch

4:34

of AI in the process and it's fresh on

4:37

my mind. So, I'll do this. Community

4:38

onboarding redesign. All right, that's a

4:41

good little list. And then let's say we

4:43

use AI in this process. How about that?

4:45

I'm literally going to open up a new

4:47

ChatGPT chat, copy this over, and I'm

4:50

going to say, "Okay, turn this into a

4:53

one-page project plan to share with the

4:57

team." Oops, and I haven't shared the

4:58

context. So, here it is. And now let me

5:01

replace the text with all of this. Hmm,

5:04

almost a page. All right, that's pretty

5:05

good. And now let's say we save this

5:07

document locally, right? Cuz we have the

5:09

finder. We go download PDF, have it

5:12

here. And what I'll do is I'll just

5:14

create a new folder on the desktop, say

5:18

project plan 8 14 26. Open the downloads

5:23

folder and move this in there. Also

5:25

rename this.

5:27

So, mind you, I'm not recording

5:29

anything, right? I'm just doing and

5:30

ChatGPT is listening. I guess it's not

5:32

listening. I guess it's watching. And

5:34

now maybe we could open Gmail. And then

5:36

I'll send this to the email I wish I

5:38

had, igor@gmail.com. Here is the plan.

5:41

The plan for you, brother, with our

5:44

clean email. Attach the project plan and

5:47

send this. And now let's give this a

5:48

shot. Let's just send a new chat ask,

5:50

"What have I done today?" And there it

5:52

is. An overview of what I did. Community

5:55

onboarding redesign, PDF learning plans

5:57

turned into one-pager, started working

6:00

on a website. These were some earlier

6:01

chats. Huh. Okay, so let's retry what we

6:04

saw in the blog post that I thought was

6:05

so impressive. I just finished my video

6:07

upload workflow, make a skill for next

6:08

time. Okay. I just finished the

6:11

onboarding design work. Create a skill

6:16

for next time. Whoa. And it adds all of

6:19

this context. Look, this is a literal

6:22

magic because it just has the recording

6:25

of what I did and it adds it. Now, I'm

6:28

concerned. This is too powerful. This is

6:30

too much of a leap. How will they be

6:31

able to like realize how strong this is?

6:34

I mean, yes, there's all these privacy

6:35

concerns. Yes, I already see the comment

6:37

section of people saying like, "Hell no,

6:39

I'm not doing that." Fair enough. I

6:41

understand. But also, just understand

6:43

the power of this. Like, AI's

6:45

capabilities are all about the context

6:47

you give it, okay? And I'm not saying

6:49

that just with prompts. Look at all

6:50

these harnesses around the most capable

6:52

agents. Like, basically, they prompt

6:54

themselves multiple times, right? That's

6:56

what thinking is, in order to get better

6:59

context to make the right decision. One,

7:01

they use skills, which are just packaged

7:03

prompts, so more context on the process

7:06

and on what you're doing. They use

7:08

connectors to bring context from other

7:10

places to themselves. And then they use

7:12

memories, which is basically just

7:14

getting context from the past, from a

7:16

vector database or whatever memory

7:17

system you might be using. It's all

7:20

about getting more context. And this is

7:22

like the ultimate version on context

7:24

from your life, because you're literally

7:26

doing these things. If it captures your

7:28

screen, if it sees you typing, if it

7:30

sees what apps you're going to, it

7:31

doesn't just have context on what you're

7:33

doing. It also has the sequence of

7:36

events. And that, dear ladies and

7:38

gentlemen, that I really want to

7:39

highlight as the big kind of outstanding

7:42

feature of this, you don't even have

7:44

that in all these agentic apps like RMS,

7:46

Open Claw, all of these frontier memory

7:49

layers, like they don't actually have

7:52

access to your working machine and see

7:54

it sequentially across time how your

7:57

work has been evolving. Look at that. It

8:00

created a skill called rough project

8:02

plan and turns rough notes into to-do's.

8:04

Okay, let's try and say "Extend the

8:06

skill with the steps I took after two."

8:10

Let's see if it can conjure that up. But

8:12

this is crazy. It's a crazy level of

8:14

access to your data and your life. And

8:17

it's also a crazy ability because this,

8:20

honestly, is the big missing piece in

8:22

making agentic AI easy for people. Like

8:25

a combination between this and Grok Bot

8:28

that came out earlier this week. By the

8:29

way, I created a great video on that. I

8:30

really thought that more people should

8:32

watch that. It's SpaceX AI's product

8:35

that looks just like a messaging

8:36

application, but you have multiple AI

8:38

employees in there and they can just

8:40

talk to each other and do things. It's

8:41

so simple. You can't even pick a model

8:43

or set custom instructions. It's just

8:44

simple. And a combination of that

8:46

interface that people just get cuz it's

8:48

like a messaging application and

8:50

something that records how you work

8:52

day-to-day. Well, boof. That's the

8:55

magical pairing because you do your work

8:57

and then there is some middle layer that

8:58

figures out which parts of your work you

9:01

could be handing off to agents. And then

9:03

you just talk to the agents like you

9:04

would to a coworker in a Slack channel

9:07

or a Messages chat or in WhatsApp,

9:09

whatever you use. Those are the two

9:11

user-friendly pieces. And honestly, like

9:13

these agents that were really hard to

9:16

set up and to maintain like Open Claw

9:18

and RMS, they got one piece right, which

9:20

is like the interaction layer and the

9:21

memories. But the big question for

9:23

everybody always was, "What do I even do

9:26

with this?" And the answer is like,

9:27

"Yeah, if you gave it access to your

9:29

main workstation and it watched you for

9:31

a week, it could figure those things out

9:33

on its own." That was never a point of

9:35

discussion cuz it's like those systems

9:37

are completely open and crazy and that

9:38

makes me want to talk about security for

9:40

a second with this because what look at

9:42

that fair enough it didn't even see that

9:44

I went into Gmail so you know maybe room

9:46

for improvement maybe I missed something

9:48

but let me tell you about security with

9:50

these things so with open cloud the way

9:52

I've been approaching it is you know

9:53

separate Mac Mini and it just doesn't

9:55

have a single one of my real accounts it

9:57

has its own email account it has its own

9:59

Google workspace account it has its own

10:01

notion account it has its own API keys

10:04

it's completely sandboxed it doesn't

10:06

touch any of my day-to-day workflows

10:08

that's why I feel pretty good about

10:09

experimenting with it this is the polar

10:12

opposite so I guess the key to you know

10:14

keeping this even like remotely secure

10:16

is really scoping this in terms of what

10:19

apps it can access so you know already

10:21

giving it access to Google Chrome is

10:23

wild cuz Google Chrome is like I don't

10:25

know half my life in it and if you

10:27

haven't asked yourself this question yet

10:29

it's probably really time to draw a line

10:31

in the sand and say okay like what

10:32

information am I okay with sharing with

10:34

these the companies and what am I not

10:37

willing to share but as you can tell I'm

10:39

pretty excited about this I think this

10:40

feature is absolutely massive I want to

10:42

try one more thing before I let you go

10:44

here and that is which one of the tasks

10:48

I did today could I automate turning on

10:51

burning dogs into project plan highly

10:53

automatable QA for PDF learning plan

10:55

delivery highly automatable it's even

10:57

looking into the to-do list that it saw

10:59

within the document oh my god are you

11:01

seeing this like it's literally

11:03

considering the bullets that I kind of

11:05

put in here in the beginning because it

11:06

saw that there is some work that I have

11:08

on my desk right now checked what

11:09

documents you worked on highly

11:11

automatable put on focus music to lock

11:13

in here that's also highly automatable

11:15

yeah QA for recommendation engine

11:17

partial okay and building refining

11:19

courses is low in automation fair enough

11:21

so are you getting this this is not just

11:23

the things you do this is also the

11:25

things within the documents that you're

11:27

working on so if you just like reading

11:29

emails it's going to consider everything

11:31

in every email if you just scroll over a

11:33

document, it's going to get all pages,

11:35

pull that into the context, and if you

11:36

ask something like, "Hey, what can I

11:37

automate?" It's really going to tell

11:39

you. So, this is probably the best, you

11:42

know, automation audit tool on the

11:44

market right now. Like, insanely useful.

11:47

I mean, this is going to spawn a huge

11:49

discussion on the internet around, you

11:52

know, safety boundaries. And just as a

11:54

reminder, the last time somebody tried

11:55

to track everything you do on your

11:56

computer with Microsoft Recall, they got

11:59

like the biggest backlash ever. They

12:01

were like, "Are you crazy, Microsoft?

12:02

I'm not going to give you all of my

12:04

information on what I'm doing." And

12:05

they're like, "All right, all right."

12:07

I guess if Open AI is doing it, it's

12:09

cool. I don't know. We shall see. This

12:11

is a bold feature that they're only

12:13

releasing because the competition is

12:15

heating up and they want to

12:16

differentiate themselves. I mean, this

12:18

has to be a reaction to the Grok bot

12:20

thing from Elon, cuz honestly, Grok bot

12:22

is like two to three features away from

12:24

being just a better consumer experience

12:27

than ChatGPT and Codex combined. It's

12:29

just simpler. It's just intuitive. Talk

12:31

to your agents. Again, check out that

12:32

video. I'm a real believer in that

12:33

interface, okay? But this is a whole

12:36

different beast. And that's today's

12:38

video. That's what I wanted to show you.

12:39

As per usual in these Friday videos, I

12:41

also highlight some stories that were on

12:43

my radar. I just want to do that quickly

12:45

because Open AI showed off a new version

12:47

of GPT 5.6 Soul, which they call ultra

12:49

fast, which is up to 14 times faster.

12:52

So, you know, could be reading all of

12:53

your context and at 14 times speed, you

12:55

could be getting things done. Okay, it's

12:57

going to be impressive. They're

12:58

releasing that through the API for

13:00

mainly coding, customer support, and

13:01

things like that. Oh, and then I should

13:02

also note that this application ChatGPT

13:05

desktop is now available across Linux,

13:08

Windows, and Mac. So, they added Linux

13:10

support, which is nice cuz people can

13:12

run this on their virtual machines

13:13

really easily. But this feature right

13:16

here that I showed you, again, the

13:17

computer history is Mac only. I'm just

13:19

going to turn it off for now. I need to

13:21

sleep over this and make up my mind if I

13:23

want to be using this and how I want to

13:24

use this. But yeah, I think those are

13:26

the big releases. This is a milestone. I

13:28

can't understate that. And finally, I'm

13:30

going to say you have to realize that

13:31

the timing of Microsoft's release with

13:34

recall was way before we could do a lot

13:37

of these agentic things. I mean, this is

13:38

the year of agents. In human history, we

13:41

will look back and say, "Hey, 2026,

13:43

that's when the new Opus models came out

13:44

and that's really what made projects

13:46

like Openclaw and then Hermes and Claude

13:48

code and Codex and co-work and now

13:51

ChatGPT work really function. It's the

13:53

increase in intelligence in the models

13:55

that made this work. We had projects

13:56

like this before, but these new models,

13:58

they make it work. But up until now,

14:00

most people weren't able to gather the

14:02

context to have it be useful for them.

14:06

With things like computer history to

14:07

gather the context and things like

14:09

Grokbot to then manage the agents while

14:11

they're working on things, that changes.

14:13

My prediction is going to be that before

14:15

the end of the year, you're going to see

14:16

all of these big AI companies compete

14:18

for these two features. One, computer

14:20

history and agentic interface that is

14:23

just multiple chats like you have in

14:24

messages because that's what we know,

14:26

that's what we love, that's what we're

14:28

used already on a day-to-day basis. And

14:30

then computer history is going to be the

14:31

thing that's going to solve the what are

14:33

the use cases that are relevant to me

14:35

issue that most people face. All right,

14:39

that being said, my name is Igor Pogany

14:41

and I hope you have a wonderful weekend.

14:43

Stay safe.

14:44

>> AI news

14:46

you can use.

14:46

>> [music]

14:48

>> Breaking down the stories you actually

14:50

care about.

14:52

>> [music]

14:52

>> AI news

14:53

you can use.

14:55

Igor's here to [music] help you

14:58

figure it out.

Interactive Summary

This video explores OpenAI's new 'Computer History' feature in the ChatGPT desktop app, which enables the AI to observe and remember user activity across desktop applications. The host explains the potential for powerful task automation and workflow creation, while also highlighting significant privacy concerns and the necessity for users to manage access boundaries carefully.

Suggested questions

3 ready-made prompts