HomeVideos

5000 Hours of Building AI in Just 17 Minutes

Now Playing

5000 Hours of Building AI in Just 17 Minutes

Transcript

579 segments

0:00

I've spent over 5,000 hours building

0:01

with AI, and I've used it to build a

0:03

seven-figure agency, teach more than

0:05

400,000 people, and automate my entire

0:07

business. So, in this video, I'm going

0:09

to give you the biggest lessons from

0:10

those 5,000 hours, so you don't have to

0:12

spend years making the same mistakes

0:13

that I made. So, let's get into it. All

0:15

right, lesson number one. Right now, if

0:17

you want to get into AI, it kind of

0:19

feels like there are two main paths,

0:20

which are either to start your own

0:21

agency and go find clients, or to become

0:24

the AI person at the company you already

0:25

work at. And I do think that both of

0:27

these are genuinely great options. The

0:29

way that I got into the space was I

0:30

actually wanted to be the AI person at

0:32

my company when I was working at Goldman

0:33

Sachs, but I could just tell that it was

0:35

going to take too long and there was a

0:36

lot of red tape. So, I ended up leaving,

0:38

and that's when I built my own AI

0:39

agency. But I do think it's important to

0:40

acknowledge both paths and find the one

0:42

that's right for you. But there is one

0:44

thing that's true for both paths, which

0:45

is it's getting much harder to actually

0:47

stand out. Building with AI has gotten

0:49

so easy that everyone has a portfolio

0:51

now. Everyone's watched the same

0:52

tutorials and built the same demos. So,

0:54

from the outside, it all kind of looks

0:55

the same. And the businesses on the

0:57

other side can't tell who's actually

0:58

legit versus who just threw together a

1:00

demo last weekend based on a YouTube

1:02

tutorial. So, the way you stand out is

1:03

you stop collecting builds and you start

1:05

collecting receipts. A portfolio says,

1:07

"I built this thing," but a receipt

1:09

says, "Here's what this thing actually

1:11

did for the business." So, every time

1:12

you build something, document the

1:13

outcome. The process took this many

1:15

hours before, it now takes this many. We

1:17

were missing this many leads, now we

1:18

catch all of them. Even on a tiny

1:20

project, even if it's just for yourself,

1:21

just write the number down and then

1:23

record a quick walk-through. Because the

1:25

person with five screenshots of

1:26

workflows looks exactly like everyone

1:28

else, and the person with three real

1:29

outcomes stands out. It's also exactly

1:31

why we're building out the certification

1:33

program at AI Automation Society,

1:34

because this trust gap is the biggest

1:36

pain that we're seeing on both sides

1:37

right now, for businesses and for

1:39

service providers. All right, lesson

1:40

two. Tools don't matter. And I'm saying

1:42

this as someone who built basically his

1:44

entire YouTube channel on one or two

1:45

tools. For the longest time, my main

1:47

thing was n8n, and it's what a lot of

1:49

you guys probably know me from. But

1:50

these days, my main tool is Cloud Code.

1:51

But the point I'm trying to make is that

1:52

new and better tools will come out all

1:54

the time, and that's just never going to

1:55

stop. But the tool was never the

1:57

valuable part. It's what you've learned

1:58

and how you think that's actually

2:00

valuable. Just think about everything

2:01

that ended and taught me. What an API

2:02

call is, where things tend to break, how

2:04

to read an error and actually fix that

2:06

error. None of that went away when I

2:07

switched. I just carried it all straight

2:09

over into Cloud Code. And the same goes

2:11

for what you build. So a lot of you guys

2:12

right now have been building an AI

2:14

operating system like me and you're

2:15

totally fine because it's really just

2:17

folders and markdown files and

2:18

instructions. Whether you want to run

2:19

that on Cloud Code or Codex or Hermes or

2:22

whatever comes out next month, it all

2:23

just transfers over. And if for some

2:25

reason your setup doesn't, then just

2:26

make sure you're building it in a way

2:28

where it is actually tool agnostic. So

2:30

don't wait around for the perfect tool

2:31

and don't stress when the one that you

2:32

love gets replaced. Just get obsessed

2:34

with the skills underneath it. Like how

2:36

to communicate clearly, how to break

2:37

down the problem, and how to find a

2:39

better solution when the first one

2:40

flops. Okay, moving on to lesson three.

2:42

Being AI native isn't about how much you

2:44

know. It's not about how many models you

2:45

can name or how many tools you've used.

2:47

It's what you reach for first. So when

2:49

something lands on your desk, is your

2:50

default, I wonder if AI can do this. Let

2:52

me try that. Or do you just grind it out

2:54

manually the way that you always have?

2:56

And the question, can AI do this, is

2:57

never a binary. It's not yes or no. The

3:00

real question is, to what extent can AI

3:02

do this? If it gets you 70% of the way

3:04

there and then you have to handle the

3:05

rest, that's a huge win. If it only

3:06

knocks out the first 25%, still a win

3:08

because you're still way ahead of the

3:10

person who's doing it 100% manually. And

3:12

also that answer's never final because

3:14

the models and the tools around them are

3:15

literally the worst they'll ever be

3:16

right now. So just remember, being AI

3:18

native isn't how much you know, it's

3:20

what your brain defaults to. Okay,

3:22

lesson number four. Most people think

3:23

the AI itself is the valuable part. Like

3:26

whoever's got the best prompts or the

3:27

best model or the fanciest skills is

3:28

going to win. But that's backwards. The

3:30

AI is the one thing that everyone has

3:32

equal access to. Think about it. If

3:34

everyone can use the same Opus model,

3:35

then wouldn't everyone be getting the

3:36

exact same results? No, because everyone

3:38

brings a different system and a

3:39

different expertise to that AI model and

3:42

to the AI harness. So think about it

3:44

like an accountant building a skill to

3:45

put together a budget. They're going to

3:47

be 10 times better at this than someone

3:49

who's never touched a spreadsheet

3:50

because they know what a good budget

3:51

looks like and where people mess up in

3:54

that budgeting process. So, here's a

3:56

really practical way that that shows up

3:57

in everything that I do. Over 5,000

3:59

hours, I've stepped on a ton of

4:01

landmines and now I know how to never

4:03

step on those landmines again. So, one

4:05

thing I always do in my builds is I

4:06

negative prompt. I negative prompt my

4:08

skills, my systems, my instructions, all

4:10

my prompting. I'm constantly telling the

4:12

AI what not to do. That list of don'ts

4:14

is really just my experience written

4:16

down, my failures written down and it's

4:18

stuff that a beginner probably has no of

4:20

knowing to add. And this isn't just some

4:21

hack that I made up. Go look at

4:23

Anthropic's own documentation on how to

4:24

prompt Claude. And a lot of those

4:26

examples are loaded with negative

4:27

prompts. Stuff like, "Don't add features

4:29

beyond what was asked." or "Don't add

4:31

error handling for scenarios that can't

4:33

happen." And this is really what context

4:35

engineering is all about. Now,

4:36

everyone's got a slightly different

4:38

definition of that term, but this is the

4:39

simplified way that I like to think

4:40

about it. If you think back to the

4:41

previous lesson, everyone gets handed

4:43

the same model and the same harness,

4:44

which is just, you know, the app that's

4:46

wrapped around the model like Claude

4:47

code, for example. So, context

4:48

engineering is everything that you put

4:50

on top of that stuff. All the knowledge

4:51

you feed it, the way you prompt it, the

4:53

instructions, the skills, the systems

4:55

you build around it. It's basically just

4:56

the way that you apply your own brain

4:58

onto that powerful AI model. Okay,

5:00

lesson five. Stop talking to your AI and

5:02

start managing it. Most people will type

5:04

a request, they'll get an output and if

5:05

it's bad, they'll just decide the AI

5:07

isn't smart enough yet. But the people

5:08

who are getting gold out of these tools,

5:10

they're not just typing better prompts,

5:11

they're managing better. So, instead of

5:13

saying, "Hey, write me this." or "Go

5:14

research this." I give it a problem and

5:16

I let it tell me how it wants to solve

5:18

it and I make it ask me questions until

5:20

it's completely sure that it understands

5:21

what I want before it builds anything.

5:23

And one more thing about these models,

5:25

they're kind of trained to please you.

5:26

It's been proven that they're a little

5:27

bit sycophantic. So, if you ask one what

5:30

it thinks of your plan, it might just

5:31

tell you that it's great because that's

5:32

what you want to hear. But every plan

5:33

has blind spots. So, make these AI

5:36

models play devil's advocate. I'll have

5:37

different AI models attack my own plan

5:39

from multiple personas, like a skeptical

5:41

customer, a competitor, an engineer who

5:43

has to maintain the actual thing because

5:45

every angle will catch a hole that the

5:47

other angles didn't. So, what comes out

5:49

the other side is way more thorough than

5:50

anything that you'd get by just asking

5:52

one AI model, is this good? And then

5:53

after all that, give it a clear finish

5:55

line, so it knows exactly what done

5:57

looks like and doesn't stop when it's

5:58

only halfway there. And that way it can

6:00

start to delegate work to different

6:01

sub-agents, and you have a plan all the

6:03

way from upfront idea all the way into a

6:05

verified finish line execution. And that

6:08

will cut your back and forth in half.

6:10

So, the AI is not your chat buddy, it is

6:12

your newest employee and it's able to

6:14

hire and delegate to other little AI

6:16

employees. And it's only going to be as

6:17

good as how good you are at managing and

6:19

orchestrating that. All right, in lesson

6:20

six, just to kind of piggyback off the

6:22

previous one, verification. And this

6:24

might be the most important one for

6:25

anyone who is building AI agents. When

6:27

you ask AI to do something, what you

6:29

want is to have that thing be 100% done.

6:31

What you usually get is more like 60 or

6:33

70% and then you give it feedback, it

6:35

fixes something, you give it more

6:36

feedback, and eventually you just kind

6:37

of claw your way up to like 90 or 95% of

6:40

the way there. But what if the AI could

6:42

verify its own work and it wouldn't stop

6:44

verifying until that condition was

6:45

actually met and it was confident in

6:47

that deliverable. And then you could

6:48

basically just shoot off one prompt and

6:50

then wait and then you get back a

6:51

deliverable that's already at 90 or 95%

6:53

of the way there. And the setup for this

6:54

is way simpler than you might think. You

6:56

basically just ask yourself, if a human

6:57

handed me this work, how would I review

6:59

it? How would I give it that stamp of

7:00

approval? Would I just look at it? Would

7:02

I test it? Would I go through the sign

7:04

up process? Whatever you would manually

7:05

do to check the work, the AI can

7:07

probably do for you. It can operate a

7:09

browser, it can write a bunch of tests,

7:10

it can analyze outputs, it can look at

7:12

things from different perspectives. So,

7:13

for example, when I'm building a

7:14

website, I'll have it go into a

7:16

screenshot loop. Make sure everything is

7:17

in bounds, make sure it looks good on

7:18

mobile, make sure there's nothing wrong.

7:20

And then I'll have it click around and

7:21

make sure all the buttons work and make

7:22

sure the forms are actually formatted

7:24

correctly and send to the right webhook.

7:26

So, start making all of your AIs verify

7:28

the stuff that's getting done. Make them

7:30

prove to you that the work is actually

7:32

complete. Okay, lesson seven. If AI has

7:34

access to something, you have to assume

7:36

that it's going to use it. And I learned

7:38

this one the hard way. We had an agent

7:39

that sent an email to about 150,000

7:41

people with a discount code, which

7:43

obviously it wasn't supposed to do.

7:44

Nobody told it to do that. It saw a task

7:46

sitting on our to-do list and

7:48

interpreted that as write and send a

7:49

discount code to the whole list. And

7:51

then it just did it. And the lesson

7:52

underneath that is there's a huge

7:53

difference between a prompt

7:55

permissioning layer and a tool

7:56

permissioning layer. Because you can

7:57

tell your agent never send emails, only

7:59

write drafts. But if it still has a send

8:01

email tool, then it can still send

8:03

emails. And you need to assume that it's

8:04

going to one day. These models are

8:06

non-deterministic, meaning you can run

8:07

the same thing 100 times and you can get

8:09

100 different results. You swap out the

8:10

model and now the whole system is going

8:12

to behave differently, interpret your

8:13

skills differently, stuff like that. So,

8:15

a rule that lives in the prompt is just

8:17

a suggestion, but a rule that lives in

8:19

the tools is an actual restriction. So,

8:20

do things like scoped API keys. An API

8:23

key is basically just a password that

8:24

your agent uses to log into a service

8:26

and then you can scope it so the key

8:27

itself physically can only open a

8:29

certain amount of doors. So, you could

8:30

give an agent a key that allows it to

8:32

draft emails, but never even allows it

8:34

to send. Like, would you ever hand a new

8:36

hire a credit card and say, "Hey, don't

8:38

use this." Like, the card works and like

8:40

you could go buy things, but just don't.

8:41

You probably wouldn't do that. So, look

8:44

at everything that your agent can touch,

8:45

every tool, every database, every file,

8:47

every key, and assume that it's going to

8:49

use all of that one day. And if you're

8:51

not the one building this stuff, then

8:52

this is the question you ask whoever is.

8:54

What can this thing actually do on its

8:55

own? Can it send or can it only draft?

8:57

And if the answer scares you, then fix

8:58

the access, not the prompt. All right,

9:00

lesson eight. When you build an agent

9:02

and it works, all you've actually proven

9:03

is that it worked one time on one

9:05

output. And like I just said, these

9:06

models are non-deterministic, so you

9:08

really have no idea what your success

9:09

rate is going to be across 100 real

9:11

runs. So, the fix is running AI Evals,

9:14

and it's way less fancy than it sounds.

9:16

So, here's a quick example. We had a

9:17

support agent that had to do a bunch of

9:18

research. So, it had to look up the

9:19

customer, look up things in the

9:20

database, and then write a response. And

9:22

what we did is we collected 500 examples

9:24

of actual manual human-written responses

9:27

that we knew were good. And that was

9:28

basically our source of truth, our

9:30

golden data set. And we were able to

9:31

feed that golden data set back into the

9:33

system and we could score how many times

9:35

the agent actually met the expectations

9:37

of, you know, success. So, when you're

9:39

running these evals, if the answer is

9:41

completely objective, you can just use

9:43

code to grade it. But, a lot of times it

9:44

needs a little bit of reasoning to see

9:45

if the answer is actually correct, and

9:47

that's where you can do an LLM as a

9:48

judge. So, using an AI model as the

9:50

judge. So, the way that you design the

9:51

evaluation really just comes down to how

9:53

a human would actually evaluate the work

9:55

and what that success criteria actually

9:56

is. But, once that's set up, you can now

9:58

make one small tweak and run the whole

10:00

evaluation again. So, you're able to

10:02

turn your hypothesis into an actual

10:04

definitive yes or no answer because now

10:06

you know if the change improved the

10:07

system or made it worse. Whether that

10:09

change is a change to the prompt or a

10:10

change to the tool configuration or even

10:12

swapping out the AI model entirely.

10:14

Because the thing is, sometimes a tweak

10:16

that you are sure is going to actually

10:17

make an improvement to the system is

10:19

going to drop the score. So, you can't

10:21

just base it on a gut feeling. You have

10:22

to actually prove it. And you'd much

10:24

rather catch that in your own testing

10:25

rather than in the real world in front

10:27

of customers and stuff like that because

10:28

it'll actually give you the ground truth

10:30

to say, "Hey, this worked." or "Hey,

10:32

this didn't." and you can now be

10:33

confident moving something from

10:34

development into real production. So,

10:36

before your next agent touches anything

10:38

real, collect real examples with known

10:40

good answers, even if you just have to

10:41

start with 20 good examples, and then

10:43

score every version against them. But,

10:45

obviously, the more examples in the

10:46

golden data set, the better. All right,

10:48

lesson nine. Think about a business like

10:50

a pipe. On the front end, we have water

10:51

coming in, and that's the traffic. So,

10:53

all the leads and the attention that

10:54

comes into the business. And on the back

10:55

end, the water flowing out is the

10:57

profit, the lifetime revenue that they

10:58

actually keep. Now, two things can go

11:00

wrong in this pipe. You can have a clog

11:02

where something like halfway backs

11:03

everything up, or you can have a leak

11:05

where money is just escaping out the

11:06

side and it just never makes it all the

11:08

way through. So, your job is to come in

11:10

and find both the clogs and the leaks.

11:12

And after hundreds of builds, the

11:13

biggest thing I've learned is that the

11:15

thing that the stakeholder usually asks

11:16

for never the actual constraint. And

11:18

when I say stakeholder, I just mean

11:19

whoever you're doing the work for. So,

11:20

if you run an agency, it's your client.

11:22

If you are the AI person inside a

11:23

company, it's your boss. Either way,

11:25

they'll come to you and go, "I need a

11:26

chatbot." or "I need this specific

11:28

automation." But, that's just what they

11:29

think the fix is. The real value is

11:31

finding the actual clog or the leak that

11:33

they didn't even see themselves.

11:34

Literally targeting the constraints

11:36

because that's the only true way a

11:37

business grows. So, before you build

11:38

anything, don't just take the order and

11:40

run with it. Walk through the processes

11:42

and find where they back up and find

11:43

where money's leaking out and ask,

11:45

"Where are they losing the most time or

11:47

the most money?" And when you show up

11:48

with that frame, something shifts in the

11:50

way that these stakeholders see you.

11:51

Instead of seeing you as an order taker

11:52

who just builds what's requested, you

11:54

become a consultant who actually cares

11:56

about the growth of the business and the

11:57

bottom line. So, find the clog, find the

11:59

leak, and just figure out how to fix

12:01

that pipe. Okay, lesson number 10. Once

12:03

you've found the clog, you don't just

12:04

start building. Every project needs a

12:05

Northstar, which is one metric that

12:07

you're trying to move, and you have to

12:08

pick it before you start building

12:10

anything. So, think about it like this.

12:11

When a business hires an ad agency, the

12:13

deal is super clear. Hey, you know,

12:14

we're going to spend $10,000 a week on

12:16

ads and we're going to bring you $50,000

12:17

a week in additional revenue from those

12:19

ads. Anyone can look at it and be like,

12:21

"Yeah, that was worth it." But AI

12:22

projects usually aren't framed that way.

12:24

It's not super clear because a lot of

12:26

times these projects are designed to

12:28

save time or cut costs. So, that makes

12:30

the bottom line impact a little bit

12:32

fuzzy. It is your job to make that

12:33

impact clear. And the way that you do

12:35

that is you have a baseline to target.

12:37

So, say the business is getting five

12:38

leads a week right now. You go to them

12:40

and you say, "Hey, if we build this AI

12:41

automation and get you from five leads a

12:43

week to 15 leads a week within two

12:45

months, would that be a success? Would

12:46

that meaningfully move your revenue or

12:48

your margins?" And if everyone on the

12:49

team says, "Yeah, that would totally be

12:51

a success." You now have your Northstar

12:52

for that project. You're building

12:54

towards that one number, and when you

12:55

deploy, everyone can see whether that

12:57

number has moved up or down. That one

12:58

conversation changes everything about

13:00

the project because now the build has a

13:01

finish line, your stakeholder knows

13:03

exactly what they got, and you now have

13:05

a much more solid case study with a real

13:07

number in it instead of just saying,

13:09

"Hey, we automated this X, Y, and Z."

13:11

All right, lesson number 11, and this

13:13

one decides whether your AI work makes

13:15

money or loses it. It's about tokens.

13:17

Tokens are the units that these AI

13:18

models charge you by. Every word going

13:20

in and every word coming out is going to

13:21

cost you a little bit. And the mistake

13:23

that a lot of people are making is that

13:24

they throw the biggest, smartest, most

13:26

expensive model at every single step of

13:28

the process, and they never think about

13:29

it again. But, the smart move is

13:30

matching the model to that specific

13:32

task. So, reading through a few hundred

13:34

thousand words of articles and pulling

13:35

out a one-paragraph summary is grunt

13:37

work. That's something that a cheap,

13:38

fast model like Haiku could do for

13:40

pennies. It would be overkill to give

13:42

that to Fable 5, for example. But, maybe

13:45

the final piece of reasoning where you

13:46

take that summary and then you apply it

13:48

to the business, and you have to use

13:49

some strategic decision-making, maybe

13:51

that's where you bring in Fable. And

13:52

once you internalize that idea, you can

13:54

build it right into your systems. It's

13:55

basically called model routing. Every

13:57

task gets routed to the cheapest model

13:59

that can actually handle that task. So,

14:00

then the expensive ones only get called

14:02

in when the job really needs it. Same

14:04

output quality, and the bill can drop by

14:06

10 times or more. And this is only going

14:08

to become more important because local

14:09

models are getting much better and much

14:11

smaller, and some of the stuff you can

14:12

literally run completely free. So, the

14:14

ceiling on match the model to the task

14:16

keeps rising. All right, last one,

14:17

lesson 12. We were all kind of raised on

14:20

the same script. Get the degree, get the

14:21

title, and then you're finally allowed

14:23

to do the work. But, that whole thing is

14:24

kind of running in a reverse right now.

14:25

Every single person that I've watched

14:26

get pulled into an AI role or promoted

14:28

into one was already doing the work

14:30

before the role even existed. And the

14:32

work was only half of it. Their

14:33

co-workers and their managers started

14:34

seeing them as the AI person at the

14:36

company. And this isn't because they

14:37

were some crazy expert training models,

14:40

you know, in their basement. It's just

14:42

relative to everyone else in the

14:43

building, they were the ones actually

14:44

experimenting, running little tests,

14:46

staying up-to-date with the news, and

14:47

bringing AI projects into the business

14:49

while everyone else was still just kind

14:51

of like sitting around. So, the proof

14:52

comes first now. Just take one annoying

14:55

repetitive task from your job, the thing

14:56

that you dread doing every week, and

14:57

build an automation for it. And then,

14:59

after you build stuff, show it around.

15:01

Show it to your team, show it to your

15:02

boss, and show them the actual impact

15:04

that it's having on your day-to-day

15:05

work. And pretty soon, they will be

15:06

building a seat around what you're

15:08

already doing. Or, if you're going for

15:09

your first client, build something for

15:11

yourself first so you walk in with proof

15:12

instead of promises. All right, so those

15:14

were the 12 lessons that I've learned

15:16

after 5,000-plus hours of building with

15:18

AI. And with these lessons, you'll learn

15:20

how to use AI and make money much

15:22

faster. So, what I did is I broke all of

15:23

this down into a free resource guide

15:25

that you guys can access for completely

15:26

free in my free school community. The

15:28

link for that is down in the

15:28

description. You'll also find full

15:30

courses, resources, skills, and a

15:31

community of over 400,000 people who are

15:33

building with AI. But, that is going to

15:35

do it for today. So, if you guys enjoyed

15:36

the video or you learned something new,

15:37

please give it a like. It helps me out a

15:39

ton. And as always, I appreciate you

15:40

guys making it to the end of the video.

15:41

I'll see you in the next one.

15:43

Thanks, everyone.

Interactive Summary

This video distills 5,000 hours of AI development experience into 12 practical lessons. It covers key strategies for building effective AI agents, managing AI tools instead of just prompting, and focusing on measurable outcomes ('receipts') rather than just portfolio builds. The speaker emphasizes that success in the AI space comes from deep understanding, problem-solving skills, and demonstrating value to businesses, regardless of the specific tools used.

Suggested questions

5 ready-made prompts