How to Create Your Own AI Assistant (It's So Easy)
609 segments
Hello there, Jaryy. How's it going?
>> Hey there. It's going smoothly on my
side.
>> Jary, can you please open the menu?
>> Sure. Let me click.
>> Hey, can you actually change the
background to pink?
>> Okay, switching the background.
>> Hey JY, show me the moods you can be in.
>> Nice. Let's pop open the mood options.
>> Hey Jary, can you change to angry?
>> Okay, switching to an angry mood now.
>> Thank you. Close the menu.
>> Got it. Closing the menu and tucking it
away. Jary, can you actually go into
computer use mode?
>> All right, switching into computer
control mode now.
>> Hey Jary, this is pretty cool. Yeah, I
want you to open up the uh browser.
>> Okay, I'll open your default browser so
we can use it. The browser is open and
ready for your next step.
>> Hey, can you please open up Codeex?
>> Okay, let's bring up Codeex for you now.
>> All right, so this right here is my mini
Jarvis that I created in three prompts
using cursor. And this AI is connected
to GPT realtime 2, which is OpenAI's new
voice model. In this video, we're going
to be building a custom Jarvis. And this
is going to be a full voice agent that
can use your tools and it will be able
to fully control your computer. And for
this, the tool that we're going to be
using is Cursor. And you might be
wondering, how much coding experience is
required? Well, the answer is 0 out of
10. You can be a complete novice to
coding and you'll be able to create a
voice agent for your personal use or for
your business. But the really magical
technology behind Jarvis is OpenAI's
brand new GPT real time 2. And it is so
realistic to talk to. The AI agent will
just speak to you in real time. And
we're about to create that right now. So
in order to get started, you should
download Cursor if you haven't already.
and you can find it at cursor.com
and you can very easily download it for
Mac OS. And then you're just going to
sign in. When you first open up Cursor,
it may not look like this. At the top
right, it'll say open agents window.
That's what I'm going to be using. It's
just a different view of the same
technology. And we're going to be using
the agents window. So, it should look
something like this. And all we're going
to do to start out is create a new
project. I am going to come to this
workspaces tab. And here you won't see
anything if it's your first time using
cursor. And I'm just going to hover over
this and click open workspace. And I'm
going to hit open folder. I'm going to
go into my desktop for now. And we're
just going to create Riley Custom
Jarvis. And I'm going to create this
folder Riley Custom Jarvis and press
open. Now I am creating an app at this
folder location on my computer. Riley
custom Jarvis. Okay. So now it's time to
plan our first prompt. We are going to
give a really long prompt to cursor and
it's just going to build Jarvis for us.
And we can use any AI model to create
it. What's important for us here is we
need to give it an OpenAI API key. And
in order to find your OpenAI API key,
you need to go to openai.com/api-keys.
And so we are going to create an API
key. And remember, keep this absolutely
safe. And I'm just going to create a new
secret key now. Create new secret key.
Jarvis key. And we can create it in the
default project. Now I'm going to hit
create secret key. Okay. So I copied
this key to my clipboard. And don't
worry, I'm going to delete it right
after I film this video. But make sure
this stays completely safe and private.
And so when I paste my prompt, I'm
actually going to run it with my API
key. I'll paste it right there. And so
the prompt is relatively simple. Build a
Jarvis-like desktop AI companion named
Jary. Except let's go ahead and call
Jary. Let's make Ricky. We're going to
make one that's a completely different
vibe than the one that I showed you
earlier. The goal is to create a desktop
AI companion that I can talk to
naturally while it does useful work in
the background. It should feel like a
personal Jarvis, conversational, fast,
visual, and able to use tools. Read the
docs on GPT Realtime 2. This is going to
be a desktop application. We're using
the GPT Realtime 2 for live voice
conversation. Let me interrupt, ask, uh,
etc., etc. Give Ricky a minimal animated
face. I'm going to say high quality
animations, but just eyes and mouth,
nothing else. Natural blinking,
expressive moods, and speech syncs mouth
movement. We're going to keep the UI
simple, a small companion window, plus
an artifact panel for anything visual
and structured. Don't worry, I'm posting
this whole prompt in the description if
you want to try it out. When using GPT
Realtime 2, it is a voice model that can
use tools. And so all you have to do is
ask cursor to create and add tools to
the agent and it'll be able to just do
all of these really cool things. And so
one of the tools will be web search. And
for this we're going to be using Exa.
We're also going to be generating
images. So this voice agent will be able
to generate and edit images with GBT. I
also want it to be able to create
mermaid charts. This is a coding
language that animates or it renders
highquality charts. And so this is going
to be really cool. I'm just going to
copy this entire prompt. Now I'm going
to go into cursor. Now what I'm going to
do is I'm going to use GPT 5.5 high and
I'm going to paste this prompt and I am
now going to run it. So we are running
this prompt and now cursor is going to
build it. So now Cursor is probably
going to take 10 to 15 minutes building
this app. I'm going to go get a little
bit of dinner and come back and we'll
see what Cursor cooked up. All right,
guys. So I just came back and it is now
done and it created an Electron desktop
app and here it is open on my computer.
This is Ricky. He blinks. Wait a sec.
Wait for it. There he is. This is Ricky
and let's try it. Hey Ricky, how are
you? Ricky, are you there?
>> Yeah, I'm here. Loud and clear. What do
you need?
>> Wow, this is cool. Can you list out the
things that you can do?
>> Let's map out what I can handle for you.
I can search the web, show results,
notes, tables, charts, and images in the
artifact panel, and keep simple local
nodes and records. I can also help
>> Can you display this in a mermaid
diagram in the artifacts panel? Is that
possible?
>> I'll sketch the capabilities as a simple
mermaid diagram in the artifact panel.
Done. It did not render correctly. It
says parse error on line 8.
>> I'll simplify the node labels and
rerender the diagram. So it parses.
>> Okay. So it can create little diagrams.
All right. Very very cool. This is
amazing. Um can you search the internet?
>> Yes. Give me a specific topic or
question.
>> Search um the best uh places to eat in
New York.
>> I'll pull a few reputable lists and
recent roundups that summarize the
common standouts. I can't access web
search right now because the search
>> Okay, I just realized I forgot to give
you the exa API key. There's a lot I
want to edit here. Hey, for the past few
weeks, we've been helping companies get
useful agents set up in their
businesses. And these are agents that
can be shared with your entire team. So,
our team can come in, analyze all of
your workflows, find where agents can
save you time, and be incredibly useful
across ads, research, content,
reporting, internal docs, whatever it
may be. We can locate where agents can
help your business and then together we
can build sharable agents that everyone
in your company can use. These agents
will have the right context, the right
tools and most importantly the right
permissions. So your whole team could
just use agents where you already work
because we can put these agents in Slack
and we can even put them in iMessage or
wherever else you communicate as a team.
So if you want help becoming an
agentnative company, the link is in my
description. Reach out there. I'm going
to disconnect from Ricky. So, we
disconnected. Now, I need to make some
changes on cursor. Okay. So, I'm going
to list out a bunch of things that I
want you to change right now. The first
thing that I want you to change is I
want you to fix the mermaid diagrams.
There was a parse error and I want to
make sure that never happens again. I
want you to fix that. The next thing
that I want you to fix, I am going to
give you my Exa API key and that's how I
want you to make the web search work
every single time. So, I'm going to
paste that at the bottom of this prompt.
And then for the notes, I noticed that
it saves them locally, which is awesome.
And then overall, I want a design
overhaul. I want the site to have less
like components. You know, there's too
many like rounded edges on this thing.
When I look at this, it's just kind of
ugly. I want it to just have a simpler
interface and have the buttons. We don't
like those buttons like connect, voice,
display, computer use, artifacts. Those
can all be very simple buttons down at
the bottom of the left panel. make the
face take up like the left half of the
screen like 50/50 and then the artifacts
are just on the right. Have like a nice
line that separates the left and the
right. Okay, huge prompt. We're making
like 10 changes here. Let's see if it
does it. Right before I enter this
prompt, I'm going to enter my Exa API
key. So, here we are. We're using Exa.
Exa is just a web search API. The same
way we're using an API key to use the
GPT real-time voice, we're also using
one for Exa. In my opinion, it is the
best web search API you can give any
agent the ability to search the web
really fast and at a really high
quality. Name this project. Let's call
this Ricky. We're going to create a key.
And now I'm going to copy the key. And
now I'm going to go ahead and paste it.
And we are going to run the prompt. All
right. So now let's wait for cursor to
finish. Okay. So it is now done. Let's
go ahead and check out ah our new
updated Ricky. So, let's go ahead and I
think I can just click this. Hey, Ricky
boy, can you please create a mermaid
chart for me explaining how artificial
intelligence works?
>> Okay, I'll sketch a simple flow of an AI
system and display it as a mermaid
chart.
>> Hey, can you actually edit it to be a
little more detailed and have like
little complete sentence subtitles on
it?
>> I'll update the chart with clearer steps
and short sentence subtitles. Done. The
updated version has short sentence
subtitles for each step, so it reads
more like a process description.
>> Very, very cool. I want you to please do
a little quick web search for me. Tell
me about the latest news in the World
Cup and then explain it to me simply
because I don't really understand
soccer.
>> I'll check the latest World Cup news and
then break it down in plain language.
Here's the simple version. The
tournament is in the knockout phase.
>> Wait, pause for a sec. Don't respond. I
need to change you. Okay. So, I need to
go to cursor for the web search. I want
it to render the web search as this like
clean markdown. I want it to stream out
and look really cool and futuristic when
it when it does that. And right now, you
just kind of pull up a bunch of tables.
I want AI to like whenever I ask for a
web search, I want it to render out in
in text, you know, make it look good.
Okay, I'll fire that prompt off. Hey,
one thing I want you to do as well, I
want to be able to ask you to show me
the menu. So, I'm going to say, "Show me
the menu," which allow just lists all
the things that I can do just to keep
track of all the things I can do. And I
also want you to put some more time and
effort into making the mouth move more
realistically based on the different way
the the sound comes out, making the uh
animations look better and higher
quality. So, it is done. Let's give it a
try. So, let's go ahead and Hey, buddy.
Uh, show me the menu. Hello, you there?
Show me the menu.
>> Menu is up in the Artifact panel. Want
me to walk you through a quick starter
like web search notes?
>> Um, actually search the internet. Tell
me about the latest about the World Cup.
>> Got it. I'll pull a quick current
overview from the web and summarize
what's new. Here's
>> Nice. Good job. Generate a photo of a
fox.
>> Nice idea. I'll create that image and
pop it into the artifact panel for you.
>> Let's go.
>> Oh, nice work.
>> Thanks. The image generation is still
running in the background and it should
appear in the artifact panel as soon as
it finishes.
>> You're doing a great job, buddy. Thank
you.
>> You got it. Happy to help anytime,
buddy.
>> Hey, can you please show me the menu
again?
>> I'll pop open the capability menu in the
panel for you. The menu is up in the
artifact panel.
>> Okay. Can you actually enter computer
use mode?
>> Sure. Switching you into computer use
mode now so I can control the desktop.
You're in computer use mode now.
>> Can you open codeex?
>> Got it. I'm opening codeex now.
It works. Let's go. Good job, Ricky.
>> Love to hear it. All set and standing by
for the next move. What do you want to
do in Codex?
>> Can you type into Codeex to make the
script better for the intro? I just want
you to type it into the box there.
>> I can do that, but I need your explicit
confirmation before typing into an app.
>> Yeah, I I give you my explicit
confirmation.
>> Okay, I'll type that into codeex now.
>> Enter it in.
>> I'll enter that now. If by enter it in,
you mean press enter to submit that
text? I need your explicit approval.
>> Let's do it. Why do you need my explicit
approval? Go.
>> Okay. Submitting it now. Done. It's
submitted.
>> It didn't submit.
>> It still didn't.
>> When I go to use the computer use mode
and I want it to type something, it asks
me for approval every time. Is there a
way to turn off that explicit approval
mode? Are you able to do that? If so, do
it. Let me see. I'll look at the
computer use. I found the exact blocker.
Computer type text reject sensitive
monoton. Okay. Let's see if this works.
When I enter computer use mode, I want
it to the window to get really small and
take up the bottom left. I want it to be
like really small on the bottom left.
And it should just show the little
character on the bottom left. And the
background of the whole app should be
translucent. So, it should just be the
little character, just the little circle
at the bottom with a little button at
the top right of it. That's brings it
back to normal mode, like computer mode,
because I want to see whatever's on my
computer. When I go into computer mode,
it should just get really small and it
should just be the little character at
the bottom left. Okay. So, let's give
this a try here. Let's see. Hey pal, how
are you?
>> Steady and sharp. Ready?
>> Nice. Nice. I need you to enter computer
use mode.
>> Got it. Switching modes so I can operate
the desktop directly.
>> Look at him. All right. Um, please can
you open up codeex?
>> I'll open codeex now from the desktop.
>> Nice. Can you please uh type in a prompt
here and say, can you please make this a
little shorter and then fire off that
prompt for me?
>> Sure. I'll type that prompt and send it
now. Done.
>> Nice work. And that's amazing. Okay, I
just got a really good idea. I want you
to be able to generate thumbnails of me
using the GBT realtime image one. And
then I want to be able to edit images.
And so the way I want this to work is
I'm going to give you an image of me and
that should be used as an image
reference. Look up the GBT image 2 or
the the whatever the latest one is. Look
up the documentation. When I want to
create an image of me, basically I'll
ask you to create a thumbnail and then I
want to be able to edit the image. So I
can say like edit it so that it does
this. And then you should be able to
edit it and change it just with voice.
And I want to be able to generate
multiple versions in parallel. So like
what you'll do is when I say generate an
image, I want you to generate them in a
grid. So like I'm thinking like a three
wide grid and it always generates in
position one, the top left one. And then
when I tell you to generate another one,
they should all be numbered. And so I
could say edit number two and make it
blah blah blah. And then that one, the
newest one, will always show up in
position one. And so I can continuously
edit these images. This is kind of this
continuous process for editing
thumbnails. This will be sick if we get
it right. Let's see how cursor does
here. Wait. And I just actually entered
a photo of me. I realized it's going to
need a photo of me. All right, let's see
if this works. Let's open up our homie
here. Hey, pal. How you doing? You
ready?
>> Ready. What do you mean?
>> All right. I want you to generate a
thumbnail image of me standing in front
of a tree. Cinematic view.
>> I'm generating a single new thumbnail on
the board with your exact scene in mind.
>> Okay. Generate an image of me holding a
iPhone. The iPhone screen's facing the
camera, so it's visible. And it shows a
really cool app on the screen, and it
looks like a camera, but with like a
cool yellow border around it.
>> All right, I'm curing a clean new 16x9
thumbnail with your exact iPhone setup.
>> Cool. Generate an image of me holding an
a MacBook except the screen is visible
showing a little blue character with
cute white eyes.
>> Number 14 is being generated now with
the iPhone screen facing the camera and
the yellowboard. Nice. I'll create a
single new thumbnail with that visible
MacBook screen and the blue character.
>> Amazing. Okay. Can you pull up uh photo
13?
>> Sure. I'm opening number 13 so you can
see it full screen.
>> Nice. Can you please give him blue
sunglasses?
>> Got it. I'll make a quick edit to that
selected thumbnail with blue sunglasses.
>> Amazing. Hey, can you please pull up
photo 15?
>> Got it. I'll pull up number 15 full.
>> Can you add text here that says cute
character?
>> Got it. I'll add that exact text to the
selected thumbnail.
>> Can you please pull up number 14?
>> Sure. I'll open number 14 so you can
view it full screen.
>> Amazing. Can you please make the
background outside like in the forest
instead of like the blue to orange
gradient?
>> I'll update the background to a forest
while keeping everything else
consistent.
>> Can you pull up 16?
>> The thumbnails are still 16 by
>> pull up 16.
>> Got it. Pulling up number 16 full screen
so you can see it clearly.
>> Nice. Can you generate another one with
um brown sunglasses?
>> Nice. I'll create a new version with
brown sunglasses using this selected.
>> Very cool. And yeah, I want to do a web
search real quick.
>> I'll pop open the menu. Then you can
give me the web search query. Menu is
up. Tell me what you want to search for.
>> Okay. I want you to please search and
tell me the latest on the rumors about
Nano Banana. I just saw Logan Kilpatrick
tweeted about it. Tell me what you can
say about that.
>> I'll pull together the latest reliable
info about those Nano Banana rumors and
summarize what's actually supported.
Short version, it's mostly speculation.
I want you to show me a mermaid diagram
of how our thumbnail tool and like like
generating images work. I want to kind
of see a diagram of how that works.
Mermaid chart.
>> Let me sketch a clear flow of the
thumbnail and image steps in a simple
diagram.
>> Very cool. Thank you for showing me
that. Can we go into computer mode?
>> Switching to computer use mode now.
>> Nice. Nice. There you go. So we created
this Jarvis and we created it in one
sitting using cursor with GPT real time.
It also has the GPT image model built in
and you can just ask it to create stuff.
That's what I'm trying to show you in
this video is you can just talk to AI,
get it to create your own little custom
Jarvis and you can even make a useful
workflow. I actually enjoyed using that
thumbnail image generator. It could use
a little bit of work, but it's so much
fun to create this little Jarvis. And uh
yeah, it's just fun to use and it can
control my computer. It can do a lot of
things. Hey, Jarvis, go back. Uh leave
computer mode. Actually, don't don't do
it.
>> Got it. Staying in computer mode.
>> No, leave computer mode.
>> Okay, leave in computer mode now. You're
back in display mode.
>> Nice. Good job. All right. Well, I'm
going to take this entire project. I'm
going to upload it on GitHub. And so
you're going to be able to take the
link, the GitHub link below and get feed
it to cursor and say clone this and you
could create this exact Jarvis or you
can look below in the description and
get the prompt and you can create your
own version of Jarvis. And uh yeah, let
me know what you guys create. I'm
genuinely want to see what people can
create and what types of Jarvises people
are creating because you could just
create Jarvis. I one thing I would love
to see is someone create it connect it
to their email and someone would connect
it to just some some of their other
tools. I think that would be really
cool. And by the way, guys, I'm here in
New York City. I have my entire team is
here. Basically, all we've been doing is
helping companies implement AI agents
into their business. We will help you
set up AI agents in your company. We
will come into your Slack and we will
add agents. You can set up OpenAI's
codecs and quad you and you can just
switch between the different models. You
can also use the open source models if
you want. We'll take a look at all of
the workflows inside your business and
we can help you create agents. The most
useful agents that we are creating for
companies right now are marketing
agents, specifically ones that can like
scrape high-quality ads and content from
every single platform and then create
versions for your company, controlling
your ads and running your ads and
managing that. And so Claude Code,
Cursor, and Codeex are all individual
tools. We want to come in and help you
build agents for your entire team. And
that means creating skills that anyone
in your company Slack can use. And so we
will help you set that up. And if that
interests you, the link is in the
description. Anyway, thank you guys so
much for watching.
Ask follow-up questions or revisit key timestamps.
The video demonstrates how to create a custom AI voice assistant, referred to as a 'Jarvis', using Cursor and OpenAI's GPT-4o real-time voice model. The host guides the viewer through the process, from initial setup and project configuration to crafting detailed prompts for functionality, such as web searching, generating mermaid diagrams, and even controlling the computer. By the end, a fully functional desktop application is built, capable of managing complex tasks like image generation, editing, and workflow automation, with the project files and prompts made available for viewers to create their own versions.
Videos recently processed by our community