5000 Hours of Building AI in Just 17 Minutes
579 segments
I've spent over 5,000 hours building
with AI, and I've used it to build a
seven-figure agency, teach more than
400,000 people, and automate my entire
business. So, in this video, I'm going
to give you the biggest lessons from
those 5,000 hours, so you don't have to
spend years making the same mistakes
that I made. So, let's get into it. All
right, lesson number one. Right now, if
you want to get into AI, it kind of
feels like there are two main paths,
which are either to start your own
agency and go find clients, or to become
the AI person at the company you already
work at. And I do think that both of
these are genuinely great options. The
way that I got into the space was I
actually wanted to be the AI person at
my company when I was working at Goldman
Sachs, but I could just tell that it was
going to take too long and there was a
lot of red tape. So, I ended up leaving,
and that's when I built my own AI
agency. But I do think it's important to
acknowledge both paths and find the one
that's right for you. But there is one
thing that's true for both paths, which
is it's getting much harder to actually
stand out. Building with AI has gotten
so easy that everyone has a portfolio
now. Everyone's watched the same
tutorials and built the same demos. So,
from the outside, it all kind of looks
the same. And the businesses on the
other side can't tell who's actually
legit versus who just threw together a
demo last weekend based on a YouTube
tutorial. So, the way you stand out is
you stop collecting builds and you start
collecting receipts. A portfolio says,
"I built this thing," but a receipt
says, "Here's what this thing actually
did for the business." So, every time
you build something, document the
outcome. The process took this many
hours before, it now takes this many. We
were missing this many leads, now we
catch all of them. Even on a tiny
project, even if it's just for yourself,
just write the number down and then
record a quick walk-through. Because the
person with five screenshots of
workflows looks exactly like everyone
else, and the person with three real
outcomes stands out. It's also exactly
why we're building out the certification
program at AI Automation Society,
because this trust gap is the biggest
pain that we're seeing on both sides
right now, for businesses and for
service providers. All right, lesson
two. Tools don't matter. And I'm saying
this as someone who built basically his
entire YouTube channel on one or two
tools. For the longest time, my main
thing was n8n, and it's what a lot of
you guys probably know me from. But
these days, my main tool is Cloud Code.
But the point I'm trying to make is that
new and better tools will come out all
the time, and that's just never going to
stop. But the tool was never the
valuable part. It's what you've learned
and how you think that's actually
valuable. Just think about everything
that ended and taught me. What an API
call is, where things tend to break, how
to read an error and actually fix that
error. None of that went away when I
switched. I just carried it all straight
over into Cloud Code. And the same goes
for what you build. So a lot of you guys
right now have been building an AI
operating system like me and you're
totally fine because it's really just
folders and markdown files and
instructions. Whether you want to run
that on Cloud Code or Codex or Hermes or
whatever comes out next month, it all
just transfers over. And if for some
reason your setup doesn't, then just
make sure you're building it in a way
where it is actually tool agnostic. So
don't wait around for the perfect tool
and don't stress when the one that you
love gets replaced. Just get obsessed
with the skills underneath it. Like how
to communicate clearly, how to break
down the problem, and how to find a
better solution when the first one
flops. Okay, moving on to lesson three.
Being AI native isn't about how much you
know. It's not about how many models you
can name or how many tools you've used.
It's what you reach for first. So when
something lands on your desk, is your
default, I wonder if AI can do this. Let
me try that. Or do you just grind it out
manually the way that you always have?
And the question, can AI do this, is
never a binary. It's not yes or no. The
real question is, to what extent can AI
do this? If it gets you 70% of the way
there and then you have to handle the
rest, that's a huge win. If it only
knocks out the first 25%, still a win
because you're still way ahead of the
person who's doing it 100% manually. And
also that answer's never final because
the models and the tools around them are
literally the worst they'll ever be
right now. So just remember, being AI
native isn't how much you know, it's
what your brain defaults to. Okay,
lesson number four. Most people think
the AI itself is the valuable part. Like
whoever's got the best prompts or the
best model or the fanciest skills is
going to win. But that's backwards. The
AI is the one thing that everyone has
equal access to. Think about it. If
everyone can use the same Opus model,
then wouldn't everyone be getting the
exact same results? No, because everyone
brings a different system and a
different expertise to that AI model and
to the AI harness. So think about it
like an accountant building a skill to
put together a budget. They're going to
be 10 times better at this than someone
who's never touched a spreadsheet
because they know what a good budget
looks like and where people mess up in
that budgeting process. So, here's a
really practical way that that shows up
in everything that I do. Over 5,000
hours, I've stepped on a ton of
landmines and now I know how to never
step on those landmines again. So, one
thing I always do in my builds is I
negative prompt. I negative prompt my
skills, my systems, my instructions, all
my prompting. I'm constantly telling the
AI what not to do. That list of don'ts
is really just my experience written
down, my failures written down and it's
stuff that a beginner probably has no of
knowing to add. And this isn't just some
hack that I made up. Go look at
Anthropic's own documentation on how to
prompt Claude. And a lot of those
examples are loaded with negative
prompts. Stuff like, "Don't add features
beyond what was asked." or "Don't add
error handling for scenarios that can't
happen." And this is really what context
engineering is all about. Now,
everyone's got a slightly different
definition of that term, but this is the
simplified way that I like to think
about it. If you think back to the
previous lesson, everyone gets handed
the same model and the same harness,
which is just, you know, the app that's
wrapped around the model like Claude
code, for example. So, context
engineering is everything that you put
on top of that stuff. All the knowledge
you feed it, the way you prompt it, the
instructions, the skills, the systems
you build around it. It's basically just
the way that you apply your own brain
onto that powerful AI model. Okay,
lesson five. Stop talking to your AI and
start managing it. Most people will type
a request, they'll get an output and if
it's bad, they'll just decide the AI
isn't smart enough yet. But the people
who are getting gold out of these tools,
they're not just typing better prompts,
they're managing better. So, instead of
saying, "Hey, write me this." or "Go
research this." I give it a problem and
I let it tell me how it wants to solve
it and I make it ask me questions until
it's completely sure that it understands
what I want before it builds anything.
And one more thing about these models,
they're kind of trained to please you.
It's been proven that they're a little
bit sycophantic. So, if you ask one what
it thinks of your plan, it might just
tell you that it's great because that's
what you want to hear. But every plan
has blind spots. So, make these AI
models play devil's advocate. I'll have
different AI models attack my own plan
from multiple personas, like a skeptical
customer, a competitor, an engineer who
has to maintain the actual thing because
every angle will catch a hole that the
other angles didn't. So, what comes out
the other side is way more thorough than
anything that you'd get by just asking
one AI model, is this good? And then
after all that, give it a clear finish
line, so it knows exactly what done
looks like and doesn't stop when it's
only halfway there. And that way it can
start to delegate work to different
sub-agents, and you have a plan all the
way from upfront idea all the way into a
verified finish line execution. And that
will cut your back and forth in half.
So, the AI is not your chat buddy, it is
your newest employee and it's able to
hire and delegate to other little AI
employees. And it's only going to be as
good as how good you are at managing and
orchestrating that. All right, in lesson
six, just to kind of piggyback off the
previous one, verification. And this
might be the most important one for
anyone who is building AI agents. When
you ask AI to do something, what you
want is to have that thing be 100% done.
What you usually get is more like 60 or
70% and then you give it feedback, it
fixes something, you give it more
feedback, and eventually you just kind
of claw your way up to like 90 or 95% of
the way there. But what if the AI could
verify its own work and it wouldn't stop
verifying until that condition was
actually met and it was confident in
that deliverable. And then you could
basically just shoot off one prompt and
then wait and then you get back a
deliverable that's already at 90 or 95%
of the way there. And the setup for this
is way simpler than you might think. You
basically just ask yourself, if a human
handed me this work, how would I review
it? How would I give it that stamp of
approval? Would I just look at it? Would
I test it? Would I go through the sign
up process? Whatever you would manually
do to check the work, the AI can
probably do for you. It can operate a
browser, it can write a bunch of tests,
it can analyze outputs, it can look at
things from different perspectives. So,
for example, when I'm building a
website, I'll have it go into a
screenshot loop. Make sure everything is
in bounds, make sure it looks good on
mobile, make sure there's nothing wrong.
And then I'll have it click around and
make sure all the buttons work and make
sure the forms are actually formatted
correctly and send to the right webhook.
So, start making all of your AIs verify
the stuff that's getting done. Make them
prove to you that the work is actually
complete. Okay, lesson seven. If AI has
access to something, you have to assume
that it's going to use it. And I learned
this one the hard way. We had an agent
that sent an email to about 150,000
people with a discount code, which
obviously it wasn't supposed to do.
Nobody told it to do that. It saw a task
sitting on our to-do list and
interpreted that as write and send a
discount code to the whole list. And
then it just did it. And the lesson
underneath that is there's a huge
difference between a prompt
permissioning layer and a tool
permissioning layer. Because you can
tell your agent never send emails, only
write drafts. But if it still has a send
email tool, then it can still send
emails. And you need to assume that it's
going to one day. These models are
non-deterministic, meaning you can run
the same thing 100 times and you can get
100 different results. You swap out the
model and now the whole system is going
to behave differently, interpret your
skills differently, stuff like that. So,
a rule that lives in the prompt is just
a suggestion, but a rule that lives in
the tools is an actual restriction. So,
do things like scoped API keys. An API
key is basically just a password that
your agent uses to log into a service
and then you can scope it so the key
itself physically can only open a
certain amount of doors. So, you could
give an agent a key that allows it to
draft emails, but never even allows it
to send. Like, would you ever hand a new
hire a credit card and say, "Hey, don't
use this." Like, the card works and like
you could go buy things, but just don't.
You probably wouldn't do that. So, look
at everything that your agent can touch,
every tool, every database, every file,
every key, and assume that it's going to
use all of that one day. And if you're
not the one building this stuff, then
this is the question you ask whoever is.
What can this thing actually do on its
own? Can it send or can it only draft?
And if the answer scares you, then fix
the access, not the prompt. All right,
lesson eight. When you build an agent
and it works, all you've actually proven
is that it worked one time on one
output. And like I just said, these
models are non-deterministic, so you
really have no idea what your success
rate is going to be across 100 real
runs. So, the fix is running AI Evals,
and it's way less fancy than it sounds.
So, here's a quick example. We had a
support agent that had to do a bunch of
research. So, it had to look up the
customer, look up things in the
database, and then write a response. And
what we did is we collected 500 examples
of actual manual human-written responses
that we knew were good. And that was
basically our source of truth, our
golden data set. And we were able to
feed that golden data set back into the
system and we could score how many times
the agent actually met the expectations
of, you know, success. So, when you're
running these evals, if the answer is
completely objective, you can just use
code to grade it. But, a lot of times it
needs a little bit of reasoning to see
if the answer is actually correct, and
that's where you can do an LLM as a
judge. So, using an AI model as the
judge. So, the way that you design the
evaluation really just comes down to how
a human would actually evaluate the work
and what that success criteria actually
is. But, once that's set up, you can now
make one small tweak and run the whole
evaluation again. So, you're able to
turn your hypothesis into an actual
definitive yes or no answer because now
you know if the change improved the
system or made it worse. Whether that
change is a change to the prompt or a
change to the tool configuration or even
swapping out the AI model entirely.
Because the thing is, sometimes a tweak
that you are sure is going to actually
make an improvement to the system is
going to drop the score. So, you can't
just base it on a gut feeling. You have
to actually prove it. And you'd much
rather catch that in your own testing
rather than in the real world in front
of customers and stuff like that because
it'll actually give you the ground truth
to say, "Hey, this worked." or "Hey,
this didn't." and you can now be
confident moving something from
development into real production. So,
before your next agent touches anything
real, collect real examples with known
good answers, even if you just have to
start with 20 good examples, and then
score every version against them. But,
obviously, the more examples in the
golden data set, the better. All right,
lesson nine. Think about a business like
a pipe. On the front end, we have water
coming in, and that's the traffic. So,
all the leads and the attention that
comes into the business. And on the back
end, the water flowing out is the
profit, the lifetime revenue that they
actually keep. Now, two things can go
wrong in this pipe. You can have a clog
where something like halfway backs
everything up, or you can have a leak
where money is just escaping out the
side and it just never makes it all the
way through. So, your job is to come in
and find both the clogs and the leaks.
And after hundreds of builds, the
biggest thing I've learned is that the
thing that the stakeholder usually asks
for never the actual constraint. And
when I say stakeholder, I just mean
whoever you're doing the work for. So,
if you run an agency, it's your client.
If you are the AI person inside a
company, it's your boss. Either way,
they'll come to you and go, "I need a
chatbot." or "I need this specific
automation." But, that's just what they
think the fix is. The real value is
finding the actual clog or the leak that
they didn't even see themselves.
Literally targeting the constraints
because that's the only true way a
business grows. So, before you build
anything, don't just take the order and
run with it. Walk through the processes
and find where they back up and find
where money's leaking out and ask,
"Where are they losing the most time or
the most money?" And when you show up
with that frame, something shifts in the
way that these stakeholders see you.
Instead of seeing you as an order taker
who just builds what's requested, you
become a consultant who actually cares
about the growth of the business and the
bottom line. So, find the clog, find the
leak, and just figure out how to fix
that pipe. Okay, lesson number 10. Once
you've found the clog, you don't just
start building. Every project needs a
Northstar, which is one metric that
you're trying to move, and you have to
pick it before you start building
anything. So, think about it like this.
When a business hires an ad agency, the
deal is super clear. Hey, you know,
we're going to spend $10,000 a week on
ads and we're going to bring you $50,000
a week in additional revenue from those
ads. Anyone can look at it and be like,
"Yeah, that was worth it." But AI
projects usually aren't framed that way.
It's not super clear because a lot of
times these projects are designed to
save time or cut costs. So, that makes
the bottom line impact a little bit
fuzzy. It is your job to make that
impact clear. And the way that you do
that is you have a baseline to target.
So, say the business is getting five
leads a week right now. You go to them
and you say, "Hey, if we build this AI
automation and get you from five leads a
week to 15 leads a week within two
months, would that be a success? Would
that meaningfully move your revenue or
your margins?" And if everyone on the
team says, "Yeah, that would totally be
a success." You now have your Northstar
for that project. You're building
towards that one number, and when you
deploy, everyone can see whether that
number has moved up or down. That one
conversation changes everything about
the project because now the build has a
finish line, your stakeholder knows
exactly what they got, and you now have
a much more solid case study with a real
number in it instead of just saying,
"Hey, we automated this X, Y, and Z."
All right, lesson number 11, and this
one decides whether your AI work makes
money or loses it. It's about tokens.
Tokens are the units that these AI
models charge you by. Every word going
in and every word coming out is going to
cost you a little bit. And the mistake
that a lot of people are making is that
they throw the biggest, smartest, most
expensive model at every single step of
the process, and they never think about
it again. But, the smart move is
matching the model to that specific
task. So, reading through a few hundred
thousand words of articles and pulling
out a one-paragraph summary is grunt
work. That's something that a cheap,
fast model like Haiku could do for
pennies. It would be overkill to give
that to Fable 5, for example. But, maybe
the final piece of reasoning where you
take that summary and then you apply it
to the business, and you have to use
some strategic decision-making, maybe
that's where you bring in Fable. And
once you internalize that idea, you can
build it right into your systems. It's
basically called model routing. Every
task gets routed to the cheapest model
that can actually handle that task. So,
then the expensive ones only get called
in when the job really needs it. Same
output quality, and the bill can drop by
10 times or more. And this is only going
to become more important because local
models are getting much better and much
smaller, and some of the stuff you can
literally run completely free. So, the
ceiling on match the model to the task
keeps rising. All right, last one,
lesson 12. We were all kind of raised on
the same script. Get the degree, get the
title, and then you're finally allowed
to do the work. But, that whole thing is
kind of running in a reverse right now.
Every single person that I've watched
get pulled into an AI role or promoted
into one was already doing the work
before the role even existed. And the
work was only half of it. Their
co-workers and their managers started
seeing them as the AI person at the
company. And this isn't because they
were some crazy expert training models,
you know, in their basement. It's just
relative to everyone else in the
building, they were the ones actually
experimenting, running little tests,
staying up-to-date with the news, and
bringing AI projects into the business
while everyone else was still just kind
of like sitting around. So, the proof
comes first now. Just take one annoying
repetitive task from your job, the thing
that you dread doing every week, and
build an automation for it. And then,
after you build stuff, show it around.
Show it to your team, show it to your
boss, and show them the actual impact
that it's having on your day-to-day
work. And pretty soon, they will be
building a seat around what you're
already doing. Or, if you're going for
your first client, build something for
yourself first so you walk in with proof
instead of promises. All right, so those
were the 12 lessons that I've learned
after 5,000-plus hours of building with
AI. And with these lessons, you'll learn
how to use AI and make money much
faster. So, what I did is I broke all of
this down into a free resource guide
that you guys can access for completely
free in my free school community. The
link for that is down in the
description. You'll also find full
courses, resources, skills, and a
community of over 400,000 people who are
building with AI. But, that is going to
do it for today. So, if you guys enjoyed
the video or you learned something new,
please give it a like. It helps me out a
ton. And as always, I appreciate you
guys making it to the end of the video.
I'll see you in the next one.
Thanks, everyone.
Ask follow-up questions or revisit key timestamps.
This video distills 5,000 hours of AI development experience into 12 practical lessons. It covers key strategies for building effective AI agents, managing AI tools instead of just prompting, and focusing on measurable outcomes ('receipts') rather than just portfolio builds. The speaker emphasizes that success in the AI space comes from deep understanding, problem-solving skills, and demonstrating value to businesses, regardless of the specific tools used.
Videos recently processed by our community