HomeVideos

Codex's Browser Agent Automates Literally Anything

Now Playing

Codex's Browser Agent Automates Literally Anything

Transcript

583 segments

0:00

Codex's browser use is probably the best

0:01

that I've ever tried before. It can

0:03

literally do anything on a browser or on

0:05

your actual local computer. And if you

0:07

need to be signed in for something, you

0:08

just sign in once and Codex will save

0:10

it. So, the possibilities are endless.

0:11

You can actually automate anything now.

0:13

Let me show you guys how to set it up,

0:14

how easy it is, and a few good use

0:15

cases. All right. So, I'm using the

0:17

Codex desktop app, which is just

0:19

honestly the main way that I like to

0:20

drive Codex. It's super, super slick.

0:22

And we get the browser to pop up on the

0:23

right-hand side, which I'll show you

0:24

guys in a sec. So, luckily we live in a

0:27

world now where most tools that we need

0:29

to use have an API guy. So, it's much

0:30

easier to just connect Codex through the

0:32

API. Or, if we just go to the plugins,

0:34

there's hundreds of plugins in here.

0:35

But, every once in a while we don't have

0:36

an API, and you can do that with browser

0:38

use. But, before I show you an

0:39

automation like that, let's just take a

0:41

look at this example where you might be

0:43

building a website. You might be

0:44

building some sort of app. And you need

0:46

a browser, an agent, to actually test

0:48

through things. If you guys watched a

0:49

recent video I did on like building an

0:51

AI SaaS in one day, then this was

0:53

actually from that demo where I had a

0:55

browser agent get spun up and test the

0:57

heck out of our app. And it did 85

0:59

focused auto- mation checks with real

1:01

browser workflow testing, which means it

1:03

was clicking around, it was typing

1:04

things, it was hitting buttons, and it

1:05

was trying to break the app. And it

1:07

found a ton of bugs, and then we were

1:08

able to just fix them. So, in this

1:09

example, I have a super simple website

1:11

spun up here on the localhost. It's a

1:12

form submission, as you can see. And

1:14

there's a few issues before we actually

1:15

go ahead and start testing this thing.

1:18

You can see like this spacing is weird.

1:20

This thing is overlapping. And so, what

1:22

you can do is you can click on this

1:23

annotate button, and that lets you

1:24

choose very specific elements to

1:26

actually change. So, I can click right

1:28

here on this phone number box, and I can

1:29

describe, "Hey, this is kind of like

1:31

overlapping. It's out of bounds. It's

1:33

covering up the company box. So, I need

1:35

you to fix this." And then, when you

1:36

shoot that off, it actually goes over

1:38

here into your chat. And then, you can

1:39

make another annotation like, "Okay. So,

1:40

this whole box isn't really aligned on

1:42

the same Y axis as number one and number

1:45

three. So, you need to fix that as

1:46

well." And then, those annotations are

1:47

saved over here, and we can just go

1:49

ahead and shoot them off. So, that's

1:50

kind of like a lovable style or like a

1:51

cloud design style thing that you can do

1:53

with the in-app browser in Codex. What

1:56

else is really cool about this is that

1:57

you can actually just drive this thing.

1:58

So, you can go to Google, you can search

2:00

for Grand Canyon images, you can go to

2:02

all, you can go to different links if

2:03

you want. You can use the browser right

2:05

here while you have Codex open. But, of

2:08

course, Codex can also just take control

2:11

of it. Now, we can see right here is my

2:12

mouse, but also this is what Codex's

2:14

mouse looks like. And that's how you

2:16

know it's now sort of getting ready to

2:17

take control of the browser. Okay, so

2:19

now that those changes have been made,

2:20

let's actually start testing out our UI.

2:22

I want to see things that maybe me as a

2:24

human, I would maybe go through here and

2:26

test like five or six edge cases, but we

2:28

can see how Codex can do it way quicker

2:30

and way more. Okay, so you have the UI

2:33

open. I want to make sure that there are

2:34

no bugs. I want to make sure that this

2:36

thing can't break. So, what I want you

2:37

to do is use your browser use and just

2:39

test the heck out of this thing. Try

2:40

different things, try your best to break

2:42

it, and let me know what bugs you find

2:44

and what things we need to fix. And so,

2:45

by the way, while this is running,

2:46

there's a difference between browser use

2:48

being headless or headed. Headless means

2:50

it's running in the background, so you

2:52

can keep working in the foreground, and

2:53

you won't actually see it going on, but

2:55

it can run a lot of things in the

2:56

background. But, if you want it to be

2:57

headed, that's basically where you're

2:58

able to see it. So, we're probably going

2:59

to be able to see some things right

3:00

here, but if you wanted this to just

3:02

work for a few hours for you headless in

3:04

the background, you could ask it to do

3:06

so, and then it wouldn't disturb

3:07

whatever work you're doing. So, if you

3:08

ever hear that terminology, that's what

3:09

it means. So, there you go. You can see

3:10

that the mouse is moving around right

3:12

now. It tried to click continue, and

3:14

then it says first name is required,

3:15

enter a valid email address, and enter a

3:17

10-digit phone number.

3:20

And now you can see it just entered in

3:21

some information and it was able to hit

3:23

continue. And it's just going to keep

3:24

going through this QA check. We're

3:26

validating that some of these fields are

3:27

required options and that the continue

3:29

button won't actually let it move on

3:31

without

3:32

filling those in. And by the way, the

3:33

reason you're seeing some of these UI

3:34

bugs is because I asked this chat to

3:36

like make there be a few UI things to

3:38

fix because I wanted to show you guys

3:40

the annotations. So, just ignore that.

3:42

Mainly, what I'm trying to show you guys

3:43

now is the actual browser use QA

3:45

function. So, here you can see as it's

3:46

clicking around, as it's trying to

3:48

navigate this UI, it already said the

3:50

core click path mostly works, but I've

3:52

confirmed two real data integrity

3:54

problems already, which is that

3:55

obviously invalid contact data does get

3:57

through, and changing the country code

3:59

to UK resets to US when you return to

4:01

edit. I'm now testing keyboard

4:03

submission and restart behavior, which

4:04

often exposes bypasses the click path

4:07

misses. So, think about it like this, no

4:09

matter what you get, you're going to

4:10

want to test it, but would you rather

4:11

have something that has been not tested

4:13

at all, or would you rather have

4:14

something that has been tested 100 times

4:16

already by an AI agent? Especially if

4:18

you tell the agent to try to break it,

4:20

it's going to find some things that need

4:22

to be fixed. No matter how good you are

4:23

at trying to predict edge cases, when

4:25

you push an app into production and

4:27

users start using it, people are weird,

4:29

people are unpredictable, so they're

4:30

going to do things. And so, this agent

4:31

is able to hopefully simulate weird

4:34

humans and just find things that you

4:36

might not have thought of. You can see

4:37

here, it even switched this to mobile

4:38

view to still make sure that

4:39

everything's working. And look at this,

4:40

in mobile it found a few failures in

4:42

responsiveness. So, that isn't something

4:44

that I yet had tested. Okay, so this

4:46

thing finished up now, and you can see

4:48

that we have a bunch of things that

4:49

passed, but a bunch of things that also

4:51

failed. So, it was able to find all of

4:52

these bugs because it was testing the

4:54

UI, and it didn't even do it that long.

4:56

If we wanted to make this even harder,

4:57

we could do a slash goal, and we could

4:59

say, "Hey, don't stop until you've tried

5:00

100 different unique scenarios of, you

5:03

know, UI edge cases, and then come back

5:04

to me." But, it still found a lot of

5:06

things that were broken, and now all

5:07

we'd have to do is say, "Okay, cool, go

5:09

implement those changes. Go fix all

5:10

those bugs." Okay, so let me show you

5:11

guys another example. This is when you

5:13

maybe have some sort of platform where

5:14

you want to be able to like download

5:16

statements or something, but they don't

5:18

have an API. So, you can actually

5:20

interact with it with Codex's browser

5:22

use because it basically uses vision and

5:24

then AI to analyze like what it's

5:25

looking at and where to click. And I

5:27

would I just want to show you how easy

5:28

it actually is to build these. All I

5:29

said was, "Hey, you know, I want to

5:30

build the skill, blah blah blah, what

5:32

would be helpful?" And you could record

5:33

a quick Loom, you could send screenshots

5:36

with arrows, but you actually don't have

5:37

to. You can basically just describe

5:39

pretty clearly what you need to do. So,

5:41

what happened here is it opened up

5:42

Relay, I signed in, and it saved that

5:44

session. Now, I said, "Okay, in the

5:45

Relay dashboard, you're signed in. What

5:47

I need you to do is go to the dashboard,

5:49

click on accounts, find the statements,

5:51

um, click on these two accounts,

5:52

download them as CSVs, and then put them

5:54

into this folder. And it was able to do

5:56

that on one shot. I didn't have to give

5:57

it any feedback, and it downloaded them

5:59

and saved that whole flow as a skill.

6:02

Now, this is super helpful because now I

6:03

could, at the end of the month, just set

6:04

this up as a scheduled routine because I

6:06

could set a scheduled routine here and

6:08

say, "Hey, use this skill on the most

6:10

recent, you know, month statement in

6:13

Relay." And for the most part, Codex is

6:15

going to save your sessions when you're

6:16

logged in. However, there are some

6:18

limitations like the fact that Relay is

6:19

going to pretty much boot you out

6:21

because it's, you know, it's a banking

6:23

platform. It's going to be secure and

6:25

boot you out. So, look at this here. I

6:27

asked it to do this again, and it said,

6:28

"Hey, Relay is open at the login screen,

6:30

but you have to sign in again." Now,

6:31

what's really interesting that you can

6:32

do is you can actually open up your

6:34

settings. You can have a password

6:36

manager inside of Codex. And you can see

6:37

this goes to your browser, and now you

6:39

can go to import them so that Codex can

6:42

actually fill that out for you. And that

6:43

way you're not putting it into the chat

6:45

history or anything like that. You're

6:46

saving it more securely. So, all you

6:47

have to do for that is create a CSV file

6:49

that looks like this. I've got the name,

6:50

the URL, the username, the password, and

6:52

any notes, which in this case are empty.

6:54

I imported that here, right here as a

6:56

file, a CSV file, and now we have

6:58

relay.com password. Go ahead and try

7:01

this again in the password manager for

7:03

the in-app Codex browser. You should be

7:05

able to log in now. And then run the

7:07

skill on June. All right, so over here

7:09

now you can see the browser has taken

7:10

control, and it is going to put in my

7:12

email and my password. Now, if this asks

7:14

me for a two-factor authentication code

7:16

from my phone or something, then I'm

7:17

obviously not going to be able to do

7:18

that. I'd have to do that manually. But

7:19

hey, if I was on a Mac and my text went

7:22

there and I used computer use to do

7:24

that, technically you would be able to

7:25

automate that whole thing, but I would

7:27

say you probably want to be careful

7:29

about stuff like that. But as you can

7:30

see, it was able to pull the passwords

7:31

from my password manager, paste those

7:33

in, and now it should be able to log in

7:34

to Relay for me. You can see here that

7:36

the mouse has moved over to accounts.

7:38

It's then going over to statements, and

7:40

it's going to be able to download these

7:41

two statements for us. So, you can see

7:43

it's pretty cool that it was able to

7:44

figure all this out. Remember, I've only

7:45

ran this skill once and all I did was

7:47

instruct it with natural language to do

7:49

this. Now, realistically, is that the

7:52

way you should do it, especially with

7:53

something like a bank account? No. What

7:55

you should do is

7:56

have it do the skill, you sit here and

7:58

watch it, and have it do that like

8:00

at least 10 times where you're very

8:02

closely watching it to make sure nothing

8:04

goes wrong, and you're making sure the

8:05

script in the skill is super strict.

8:07

Because at the end of the day, it's

8:08

using vision, it's using AI, and that's

8:10

how it's deciding what to click and what

8:11

to do. So, really the way that I think

8:13

about it is when you want to automate

8:15

something, your first call should always

8:17

be to go for an API. Like, that's what

8:19

you want to do. It's the fastest, it's

8:21

usually the cheapest, it's usually just

8:22

the most consistent. Now, if there's no

8:25

API, that's where you would maybe say,

8:27

"Okay, cool. Well, now let me see if I

8:28

could use like a deterministic, you

8:31

know, like a macro script, right?" And

8:33

that's where you'd basically be able to

8:34

have, yes, you know, it'd be more of a

8:36

browser use because you have a mouse

8:38

that's clicking around, but it's not

8:40

like your mouse needs reasoning. Your

8:41

mouse doesn't need to look at the

8:43

screen, figure out where to click, and

8:44

based on the next UI that pops up, where

8:46

do I click again? This is basically more

8:48

like, "Okay,

8:49

I have to click on this pixel, then this

8:51

pixel, then this pixel every single

8:54

time." That's something that you could

8:55

do for much cheaper and much safer by

8:58

just programming a macro. And that's not

9:00

that hard either. You just ask Claude or

9:01

Codex, "Hey, help me figure out how to

9:03

make like a deterministic macro script

9:05

to just click on these pixels in that

9:06

order every single time." But, if you

9:08

can't do that every single time because

9:09

there is an element of vision or

9:11

reasoning, then you need to come in here

9:13

and you need to do something like some

9:14

browser use. And luckily, Codex has your

9:16

back. I've tested a lot of different

9:18

apps and different CLIs and different

9:20

things for browser use, and so far Codex

9:21

is just really solid. So, as you can see

9:24

now, in my downloads, this is the

9:25

statements that it just downloaded as a

9:27

zip file, and it basically took these

9:29

and it put it into my Hercule project.

9:30

If I go in here and I scroll down to

9:32

statements, and I go to probably June,

9:35

yes, June, you can see right here it put

9:37

those in here in the correct spot. Okay,

9:40

so that was an example where we needed

9:42

to save a password somewhere, right? But

9:44

I've done this with things like school,

9:46

I've done this with things like X, I've

9:47

even done this with things like

9:48

Instagram that don't make you constantly

9:50

resign in because it saves that session.

9:52

So let me show you another example. I'm

9:53

going to go over here to YouTube and I'm

9:55

going to grab the URL from the video

9:56

that I uploaded earlier today. I'm going

9:58

to go back into Codex, paste in that URL

10:00

and say,

10:01

"I need you to turn this YouTube video

10:03

into an X article for me and then use

10:05

browser use to go ahead and um format it

10:08

and put it into an X article as a draft

10:11

and then I will go ahead and review it."

10:12

So I know some of you guys are probably

10:14

thinking, "Why don't you just use X's

10:15

API for that?" You can, but it's a

10:18

little bit weird with the formatting.

10:19

It's also weird with sometimes adding

10:20

things like media or thumbnails for some

10:22

reason, at least with my testing that I

10:24

used to do, and it's also it charges

10:27

you, right? Because you have to pay X

10:30

per usage in order to do something like

10:31

that. And so because this is just part

10:33

of my Codex subscription, I can just run

10:35

this skill and it will go ahead and

10:36

format everything for me and take care

10:38

of it. I even have the skill worked out

10:39

where it can go ahead and open up my

10:40

YouTube videos and it can take

10:41

screenshots of specific moments and put

10:44

those into the article, too, where it

10:45

makes sense. So obviously this is a bit

10:47

of a loaded process because it has to do

10:48

a few steps before it actually has the

10:50

article ready to put into X. So while

10:52

that's happening, let's just open up a

10:53

new chat and let's see if our X session

10:55

is there.

10:56

"Can you open up a browser? I want to

10:58

see it right here in the app and just go

11:00

to X and scroll around a little bit and

11:02

show me if there's anything new that has

11:03

just come out um like in the AI space.

11:06

Just scroll on my X feed." And so

11:07

because this was a brand new session, we

11:09

can actually see if X is already signed

11:11

in and if it's able to interact with

11:13

things.

11:17

Cool. So it just opened up X. Let me get

11:18

my mouse out of here. You can see

11:20

clearly that this is already signed in

11:21

as me and my feed is visible and now it

11:24

is going ahead and it's starting to

11:25

scroll on the X page. So just start to

11:27

think about already, how could you

11:29

create some sort of skills out of this

11:31

and then schedule these as tasks? So,

11:33

maybe every morning if you wanted it to

11:34

scroll through Instagram and things like

11:36

that, or scroll through your own

11:37

profile, or mark things as read or

11:40

unread, you could totally go ahead and

11:41

do that, no problem, because you can set

11:43

up the schedule tasks right in here and

11:45

use browser use skills that you've

11:46

already set up. So, you can see the two

11:47

things that it already noticed was Grok

11:49

Bot and Grok 4.6. So, this is an

11:51

interesting case though, because there

11:52

is an API, but using the browser use is

11:55

actually cheaper. So, that's why in this

11:57

case we decided to do it and test it

11:59

out. Now, what about computer use that

12:01

you guys have also probably heard? These

12:03

are basically the same thing. It's still

12:04

going to use vision. It's still going to

12:05

use AI, but the difference here is that

12:07

the browser is basically things that are

12:09

on the internet, or things that you

12:11

would type into a browser like Chrome,

12:13

but the computer use are things that

12:14

might be on your actual computer. So,

12:16

changing your computer settings, opening

12:18

up desktop apps of things. So, let's go

12:20

back into Codex. While we're waiting for

12:21

this X article to finish up, let's see

12:23

how this is going. It is still breaking

12:25

down and turning it into an article.

12:27

Let's see if we can do some computer

12:28

use. So, one thing you have to do though

12:29

is you have to go to plugins, and you'll

12:31

have to actually install computer use,

12:32

which is just as simple as typing in

12:33

computer use and hitting install, like

12:35

literally 2 seconds. And then,

12:38

Hey Codex, I need you to use computer

12:39

use to open up my desktop app for Glydo,

12:41

and I just want you to go to the

12:42

settings and turn on the ability for it

12:45

to allow and use like Bluetooth devices.

12:47

So, just go ahead and get that set up

12:49

for me. And so, this is a little bit

12:50

different, right? Because it's not going

12:51

to open up your in-app browser. It's

12:53

going to actually kind of show your

12:55

screen being taken over. It said, "I'll

12:57

stop if Windows asks for an admin

12:59

password or presents a security

13:01

sensitive permission that it can't

13:02

safely interpret." It shouldn't run into

13:04

any of that, but it's good that it sort

13:05

of has that guardrail baked in.

13:07

Interesting. Okay, so it said,

13:08

"Bluetooth access is a privacy

13:10

permission, so the computer use won't

13:12

let me actually do that." But, what it

13:13

said it's going to do is open up Glydo.

13:14

As you can see, I didn't open that up.

13:16

It opened up Glydo for me. And it

13:17

doesn't look like on my recording you

13:19

guys can actually see this right now,

13:20

but my screen kind of has like a big

13:22

blue hue around the edges. So, you will

13:25

clearly know if Codex is using your

13:27

computer.

13:33

And there you go. You can see that it

13:34

actually turned on the include Bluetooth

13:36

devices button. Now, I think it got

13:38

confused because I think at first it

13:39

thought that I was messing with my

13:40

actual computer settings rather than

13:42

like an in-app setting. So, it thought

13:43

that that was a security thing. But, it

13:45

looks like it figured that out on the

13:46

way. And I think the computer use thing

13:48

is pretty interesting because you can

13:49

remote into your Codex from your phone.

13:52

So, I can control all of these sessions

13:54

on my phone. And what I could do is on

13:56

my phone I could say, "Hey, computer

13:57

use, go ahead and change my settings or

13:59

go ahead and do something on a desktop

14:01

app that's on my PC that I'm actually

14:03

not sitting in front of." And then you

14:04

could control your computer from your

14:06

phone. So, anyways, we're back in the

14:08

session where it's creating that X

14:09

article. You can see that it is actually

14:11

creating some thumbnails for the actual

14:13

article. So, hopefully we're getting

14:14

close to the point where it's going to

14:15

actually open up X and start to put

14:17

stuff in there. But, I'll check with you

14:19

guys when we get there. Okay, so it just

14:20

decided to open up X. You can see that

14:22

it's going to the drafts in my signed in

14:25

X and then it's going to I'll go ahead

14:27

and format this new article and then

14:28

we'll take a look. It's clicking on the

14:30

plus button. Man, I think it even just

14:32

like resized the page a little bit. It

14:33

was getting a little picky. In goes the

14:35

title. And then it just pasted in the

14:37

entire content of the article. Now, it's

14:39

uploading this thumbnail which I don't

14:41

think looks very good. That part of the

14:43

skill has not been refined at all. But,

14:44

it's still cool that it was able to make

14:46

that thumbnail and then go ahead and put

14:48

it in there. So, I literally just have

14:49

to make sure that it looks okay and I

14:50

like it and I can publish it. Now, I

14:52

know what a lot of you guys are probably

14:53

thinking when it comes to this browser

14:55

use stuff. It's like,

14:56

"What's the point, Nate? Like, where's

14:57

the value?" And honestly, I understand

14:59

where you're coming from. And that's why

15:00

I wanted to show you guys this piece

15:01

where it's like

15:02

pretty much API will handle 95% of your

15:05

automation things that you need. And

15:07

then a lot of the browser use

15:07

opportunities might just be a script.

15:09

But, then when you need it, if you have

15:11

this in your back pocket, it's good. And

15:13

you also might think, "Okay, cool. But,

15:14

like, how much time does this really

15:15

save?" This probably saves me like maybe

15:17

5 to 10 minutes. Like, realistically.

15:19

But, what's cool about it is that I can

15:21

basically have this pipeline now where I

15:23

have a bunch of these browser use

15:24

skills, and so when I need to do

15:26

something, I just open up Codex, I shoot

15:27

off that skill, and I can go back to the

15:28

other task that I was working on. It

15:30

kind of just like helps you do more

15:32

without context switching as much. And

15:34

as you start to stack more and more of

15:35

these pipelines and combine them

15:36

together, you can get some really cool

15:38

things to happen when you start to chain

15:40

together these skills. And look, this

15:41

one actually did it. It basically did

15:43

the thing where I said it opened up the

15:44

YouTube video, it took screenshots of

15:46

it, and now it's going ahead and it's

15:47

planting some screenshots from my video

15:50

in the article. So, if that doesn't

15:51

prove the capabilities of this thing

15:52

being intelligent, take screenshots,

15:54

look at the page, know where to click,

15:55

know what to do, then I don't know what

15:57

else will. Besides the fact that I've

15:58

had this thing play chess, I've had it

16:00

play games, I've had it set a world

16:01

record on Tetris, it can do a lot of

16:03

cool things.

16:05

Look at this. You can see that it even

16:08

took the image and drug it all the way

16:09

down to where it actually fits in the

16:11

article. I haven't actually seen it do

16:13

that before live. That was pretty cool.

16:14

And this thing is just still going at

16:16

it. But that is going to do it for

16:17

today's video. Hope you guys enjoyed it.

16:19

Hope you learned something new, and if

16:20

you did, please give it a like. It helps

16:21

me out a ton. As always, I appreciate

16:23

you guys making it to the end of the

16:24

video, and I'll see you on the next one.

16:26

Thanks, everyone.

Interactive Summary

The video explores the capabilities of Codex's browser and computer use features, demonstrating how AI agents can automate complex tasks by interacting with web browsers and desktop applications. The narrator showcases practical use cases, such as automated UI testing for bug detection, executing multi-step workflows like downloading bank statements, and content creation tasks like repurposing YouTube videos into social media drafts. The video also highlights the difference between headless and headed browser operation, the importance of secure password management within the tool, and the distinction between AI-driven browser/computer use versus traditional deterministic macros.

Suggested questions

4 ready-made prompts