hi everyone so in this video I would

like to continue our general audience

series on large language models like

chpd now in the previous video deep dive

into llms that you can find on my

YouTube we went into a lot of the

underhood fundamentals of how these

models are trained and how you should

think about their cognition or

psychology now in this video I want to

go into more practical applications of

these tools I want to show you lots of

examples I want to take you through all

the different settings that are

available and I want to show you how I

use these tools and how you can also use

them uh in your own life and work so

let's dive in okay so first of all the

web page that I have pulled up here is

chp.com now as you might know chpt it

was developed by openai and deployed in

2022 so this was the first time that

people could actually just kind of like

talk to a large language model through a

text interface and this went viral and

over all over the place on the internet

and uh this was huge now since then

though the ecosystem has grown a lot so

I'm going to be showing you a lot of

examples of Chachi PT specifically but

now in

2025 uh there's many other apps that are

kind of like Chachi PT like and this is

now a much bigger and richer ecosystem

so in particular I think Chachi PT by

openai is this Original Gangster

incumbent it's most popular and most

featur rich also because it's been

around the longest but there are many

other kind of clones available I would

say I don't think it's too unfair to say

but in some cases there are kind of like

unique experiences that are not found in

chashi p and we're going to see examples

of

those so for example big Tech has

followed with a lot of uh kind of chat

GPT like experiences so for example

Gemini met and co-pilot from Google meta

and Microsoft respectively and there's

also a number of startups so for example

anthropic uh has Claud which is kind of

like a chasht equivalent xai which is

elon's company has Gro uh and there's

many others so all of these here are

from the United States um companies

basically deep seek is a Chinese company

and lchat is a French company

Mistral now where can you find these and

how can you keep track of them well

number one on the internet somewhere but

there are some leaderboards and in the

previous video I've shown you uh chatbot

arena is one of them so here you can

come to some ranking of different models

and you can see sort of their strength

or ELO score and so this is one place

where you can keep track of them I would

say like another place maybe is this um

seal Le leaderboard from scale and so

here you can also see different kinds of

eval

and different kinds of models and how

well they rank and you can also come

here to see which models are currently

performing the best on a wide variety of

tasks so understand that the ecosystem

is fairly rich but for now I'm going to

start with open AI because it is the

incumbent and is most feature Rich but

I'm going to show you others over time

as well so let's start with chachy PT

what is this text box text box and what

do we put in here okay so the most basic

form of interaction with the language

model is that we give it text and then

we get some typ text back in response so

as an example we can ask to get a ha cou

about what it's like to be a large

language model so uh this is a good kind

of example askas for a language model

because these models are really good at

writing so writing haikus or poems or

cover letters or resumés or email

replies they're just good at writing so

when we ask for something like this what

happens looks as follows the model

basically responds um words flow like a

stream endless Echo never mind ghost of

thought

unseen okay it's pretty dramatic but

what we're seeing here in chashi PT is

something that looks a bit like a

conversation that you would have with a

friend these are kind of like chat

bubbles now we saw in the previous video

is that what's going on under the hood

here is that this is what we call a user

query this piece of text and this piece

of text and also the response from the

model this piece of text is chopped up

into little text chunks that we call

tokens so these this sequence of text is

under the hood a token sequence

onedimensional token sequence now the

way we can see those tokens is we can

use an app like for example Tik

tokenizer so making sure that GPT 40 is

selected I can paste my text here and

this is actually what the model sees

Under the Hood my piece of text to the

model looks like a sequence of exactly

15 tokens and these are the little text

chunks that the model

sees now there's a vocabulary here of

200,000 roughly of possible tokens and

then these are the token IDs

corresponding to all these little text

chunks that are part of my query and you

can play with this and update and you

can see that for example this is Skate

sensitive you would get different tokens

and you can kind of edit it and see live

how the token sequence changes so our

query was 15 tokens and then the model

response is right here and it responded

back to us with a sequence of exactly 19

tokens so that Hau is this sequence of

19

tokens now

so we said 15 tokens and it said 19

tokens back now because this is a

conversation and we want to actually

maintain a lot of the metadata that

actually makes up a conversation object

this is not all that's going on under

under the hood and we saw in the

previous video a little bit about the um

conversation format um so it gets a

little bit more complicated in that we

have to take our user query and we have

to actually use this a chat format so

let me delete the system message I don't

think it's very important for the

purposes of understanding what's going

on let me paste my message as the user

and then let me paste the model response

as an assistant and then let me crop it

here properly the tool doesn't do that

properly so here we have it as it

actually happens under the hood there

are all these special tokens that

basically begin a message from the user

and then the user says and this is the

content of what we said and then the

user ends and then the assistant begins

and says this Etc now the precise

details of the conversation format are

not important what I want to get across

here is that what looks to you and I as

little chat bubbles going back and forth

under the hood we are collaborating with

the model and we're both writing into a

token

stream and these two bubbles back and

forth were in sequence of exactly 42

tokens under the hood I contributed some

of the first tokens and then the model

continued the sequence of tokens with

its response

and we could alternate and continue

adding tokens here and together we're

are building out a token window a

onedimensional tokens onedimensional

sequence of tokens okay so let's come

back to chpt now what we are seeing here

is kind of like little bubbles going

back and forth between us and the model

under the hood we are building out a

one-dimensional token sequence when I

click new chat here that wipes the token

window that resets the tokens to

basically zero again and restarts the

conversation from scratch now the

cartoon diagram that I have in my mind

when I'm speaking to a model looks

something like this when we click new

chat we begin a token sequence so this

is a onedimensional sequence of tokens

the user we can write tokens into this

stream and then when we hit enter we

transfer control over to the language

model and the language model responds

with its own token streams and then the

language to model has a special token

that basically says something along the

lines of I'm done so when it emits that

token the chat GPT application transfers

control back to us and we can take turns

together we are building out the token

the token stream which we also call the

context window so the context window is

kind of like this working memory of

tokens and anything that is inside this

context window is kind of like in the

working memory of this conversation and

is very directly accessible by the

model now what is this entity here that

we are talking to and how should we

think about it well this language model

here we saw that the way it is trained

in the previous video we saw there are

two major stages the pre-training stage

and the post-training stage the

pre-training stage is kind of like

taking all of Internet chopping it up

into tokens and then compressing it into

a single kind of like zip file but the

zip file is not exact the zip file is

lossy and probabilistic zip file because

we can't possibly represent all of

internet in just one one sort of like

say terabyte of uh of zip file um

because there's just way too much

information so we just kind of get the

gal or The Vibes inside this um zip

file now what actually inside the zip

file are the parameters of a neural

network and so for example a one tbte

zip file would correspond to roughly say

one trillion parameters inside this

neural

network and when this neural network is

trying to to do is it's trying to

basically take tokens and it's trying to

predict the next token in a sequence but

it's doing that on internet documents so

it's kind of like this internet document

generator right um and in the process of

predicting the next token on a sequence

on internet the neural network gains a

huge amount of knowledge about the world

and this knowledge is all represented

and stuffed and compressed inside the

one trillion parameters roughly of this

language model now this pre-training

stage also we saw is fairly costly so

this can be many tens of millions of

dollars say like three months of

training and so on um so this is a

costly long phase for that reason this

phase is not done that often so for

example gbt 40 uh this model was

pre-trained uh

probably many months ago maybe like even

a year ago by now and so that's why

these models are a little bit out of

date they have what's called a knowledge

cutof because that knowledge cut off

corresponds to when the model was

pre-trained and its knowledge only goes

up to that point

now some knowledge can come into the

model through the post-training fa phase

which we'll talk about in a second but

roughly speaking you should think of

these uh models is kind of like a little

bit out of date because pre- training is

way too expensive and happens

infrequently so any kind of recent

information like if you wanted to talk

to your model about something that

happened last week or so on we're going

to need other ways of providing that

information to the model model because

it's not stored in the knowledge of the

model so we're going to have various

tool use to give that information to the

model now after pre-training there's a

second stage goes post-training and

post-training Stage is really attaching

a smiley face to this ZIP file because

we don't want to generate internet

documents we want this thing to take on

the Persona of an assistant that

responds to user queries and that's done

in a process of post training where we

swap out the data set for a data set of

conversations that are built out by

humans so this is basically where the

model takes on this Persona and that

actually so that we can like ask

questions and it responds with answers

so it takes on the style of the of an

assistant that's post trainining but it

has the knowledge of all of internet and

that's by

pre-training so these two are combined

in this

artifact um now the important thing to

understand here I think for this section

is that what you are talking to to is a

fully self-contained entity by default

this language model think of it as a one

tbte file on a dis secretly that

represents one trillion parameters and

their precise settings inside the neural

network that's trying to give you the

next token in the

sequence but this is the fully

selfcontained entity there's no

calculator there's no computer and

python interpreter there's no worldwide

web browsing there's none of that

there's no tool use yet in what we've

talked about so far you're talking to a

zip file if you stream tokens to it it

will respond with tokens back and this

ZIP file has the knowledge from

pre-training and it has the style and

form from posttraining

and uh so that's roughly how you can

think about this entity okay so if I had

to summarize what we talked about so far

I would probably do it in the form of an

introduction of Chach PT in a way that I

think you should think about it so the

introduction would be hi I'm Chach PT I

am a one tab zip file my knowledge comes

from the internet which I read in its

entirety about six months ago and I only

remember vaguely okay and my winning

personality was programmed by example by

human labelers at open AI so the

personality is programmed in

post-training and the knowledge comes

from compressing the internet during

pre-training and this knowledge is a

little bit out of date and it's a

probabilistic and slightly vague some of

the things that uh probably are

mentioned very frequently on the

internet I will have a lot better better

recollection of than some of the things

that are discussed very rarely very

similar to what you might expect with a

human so let's not talk about some of

the repercussions of this entity and how

we can talk to it and what kinds of

things we can expect from it now I'd

like to use real examples when we

actually go through this so for example

this morning I asked Chachi the

following how much caffeine is in one

shot of Americana and I was curious

because I was comparing it to matcha now

chashi PT will tell me that this is

roughly 63 Mig of caffeine or so now the

reason I'm asking chash HPT this

question that I think this is okay is

number one I'm not asking about any

knowledge that is very recent so I do

expect that the model has sort of read

about how much caffeine there is in one

shot this I don't think this information

has changed too much and number two I

think this information is extremely

frequent on the internet this kind of a

question and this kind of information

has occurred all over the place on the

internet and because there was so many

mentions of it I expect a model to have

good memory of it in its knowledge so

there's no tool use and the model the

zip file responded that there's roughly

63 Mig now I'm not guaranteed that this

is the correct answer uh this is just

its vague recollection of the internet

but I can go to primary sources and

maybe I can look up okay uh caffeine and

uh Americano and I could verify that

yeah it looks to be about 63 is roughly

right and you can look at primary

sources to decide if this is true or not

so I'm not strictly speaking guaranteed

that this is true but I think probably

this is the kind of thing that chpt

would know here's an example of a

conversation I had two days ago actually

um and there's another example of a

knowledge based conversation and things

that I'm comfortable asking of Chach PT

with some caveats so I'm a bit sick I

have runny nose and I want to get meds

that help with that so it told me a

bunch of stuff um and um I want my nose

to not be runny so I gave it a

clarification based on what it said and

then it kind of gave me some of the

things that might be helpful with that

and then I looked at some of the meds

that I have at home and I said does

daycool or night call work

and it went off and it kind of like went

over the ingredients of Dil and NYL and

whether or not they um helped mitigate

Ronnie nose now when these ingredients

are coming here again remember we are

talking to a zip file that has a

recollection of the internet I'm not

guaranteed that these ingredients are

correct and in fact I actually took out

the box and I looked at the ingredients

and I made sure that NY ingredients are

exactly these ingredients um and I'm

doing that because I don't always fully

trust what's coming out here right this

is just a probabilistic statistical

recollection of the internet but that

said conversations of DayQuil and NyQuil

these are very common meds uh probably

there's tons of information about a lot

of this on the internet and this is the

kind of things that the model have

pretty good uh recollection of so

actually these were all correct and then

I said okay well I have nyel um how far

how fast would it act roughly and it

kind of tells

me and then is a basically a tal and

says yes so this is a good example of

how chipt was useful to me it is a

knowledge based query this knowledge uh

sort of isn't recent knowledge U this is

all coming from the knowledge of the

model I think this is common information

this is not a high stakes situation I'm

checking Chach PT a little bit uh but

also this is not a high Stak situation

so no big deal so I popped an iol and

indeed it helped um but that's roughly

how I'm thinking about what's going back

here okay so at this point I want to

make two notes the first note I want to

make is that naturally as you interact

with these models you'll see that your

conversations are growing longer right

anytime you are switching topic I

encourage you to always start a new chat

when you start a new chat as we talked

about you are wiping the context window

of tokens and resetting it back to zero

if it is the case that those tokens are

not any more useful to your next query I

encourage you to do this because these

tokens in this window are expensive and

they're expensive in kind of like two

ways number one if you have lots of

tokens here then the model can actually

find it a little bit distracting uh so

if this was a lot of tokens um the model

might this is kind of like the working

memory of the model the model might be

distracted by all the tokens in the in

the past when it is trying to sample

tokens much later on so it could be

distracting and it could actually

decrease the accuracy of of the model

and of its performance and number two

the more tokens are in the window uh the

more expensive it is by a little bit not

by too much but by a little bit to

sample the next token in the sequence so

your model is actually slightly slowing

down it's becoming more expensive to

calculate the next token and uh the more

tokens there are

here and so think of the tokens in the

context window as a precious resource um

think of that as the working memory of

the model and don't overload it with

irrelevant information and keep it as

short as you can and you can expect that

to work faster and slightly better of

course if the if the information

actually is related to your task you may

want to keep it in there but I encourage

you to as often as as you can um

basically start a new chat whenever you

are switching topic the second thing is

that I always encourage you to keep in

mind what model you are actually using

so here in the top left we can drop down

and we can see that we are currently

using GPT 40 now there are many

different models of many different

flavors and there are too many actually

but we'll go through some of these over

time so we are using GPT 40 right now

and in everything that I've shown you

this is GPD 40 now when I open a new

incognito window so if I go to chat

gt.com and I'm not logged in the model

that I'm talking to here so if I just

say hello uh the model that I'm talking

to here might not be GPT 40 it might be

a smaller version uh now unfortunately

opening ey does not tell me when I'm not

logged in what model I'm using which is

kind of unfortunate but it's possible

that you are using a smaller kind of

Dumber model so if we go to the chipt

pricing page

here we see that they have three basic

tiers for individuals the free plus and

pro and in the free tier you have access

to what's called GPT 40 mini and this is

a smaller version of GPT 40 it is

smaller model with a smaller number of

parameters it's not going to be as

creative like it's writing might not be

as good its knowledge is not going to be

as good it's going to probably

hallucinate a bit more Etc uh but it is

kind of like the free offering the free

tier they do say that you have limited

access to 40 and3 mini but I'm not

actually 100% sure like it didn't tell

us which model we were using so we just

fundamentally don't know

now when you pay for $20 per month even

though it doesn't say this I I think

basically like they're screwing up on

how they're describing this but if you

go to fine print limits apply we can see

that the plus users get 80 messages

every 3 hours for GPT 40 so that's the

flagship biggest model that's currently

available as of today um that's

available and that's what we want to be

using so if you pay $20 per month you

have that with some limits and then if

you pay for2 $100 per month you get the

pro and there's a bunch of additional

goodies as well as unlimited GPD foro

and we're going to go into some of this

because I do pay for pro

subscription now the whole takeaway I

want you to get from this is be mindful

of the models that you're using

typically with these companies the

bigger models are more expensive to uh

calculate and so therefore uh the

companies charge more for the bigger

models and so make those tradeoffs for

yourself depending on your usage of llms

um have a look at you can get away with

the cheaper offerings and if the

intelligence is not good enough for you

and you're using this professionally you

may really want to consider paying for

the top tier models that are available

from these companies in my case in my

professional work I do a lot of coding

and a lot of things like that and this

is still very cheap for me so I pay this

very gladly uh because I get access to

some really powerful models that I'll

show you in a bit um so yeah keep track

of what model you're using and make

those decisions for yourself I also want

to show you that all the other llm

providers will all have different

pricing teams TI with different models

at different tiers that you can pay for

so for example if we go to Claude from

anthropic you'll see that I am paying

for the professional plan and that gives

me access to Claude 3.5 Sonet and if you

are not paying for a Pro Plan then

probably you only have access to maybe

ha cou or something like that um and so

use the most powerful model that uh kind

of like works for you here's an example

of me using Claud a while back I was

asking for just a travel advice uh so I

was asking for a cool City to go to and

Claud told me that zerat in Switzerland

is really cool so I ended up going there

for a New Year's break following claud's

advice but this is just an example of

another thing that I find these models

pretty useful for is travel advice and

ideation and giving getting pointers

that you can research further um here we

also have an example of gemini.com so

this is from Google I got Gemini's

opinion on the matter and I asked it for

a cool City to go to and it also

recommended zerat so uh that was nice so

I like to go between different models

and asking them similar questions and

seeing what they think about and for

Gemini also on the top left we also have

a model selector so you can pay for the

more advanced tiers and use those models

same thing goes for grock just released

we don't want to be asking Gro 2

questions because we know that grock 3

is the most advanced model so I want to

make sure that I pay enough and such

that I have grock 3 access um so for all

these different providers find the one

that works best for you experiment with

different providers experiment with

different pricing tiers for the problems

that you are working on and uh that's

kind of and often I end up personally

just paying for a lot of them and then

asking all all of them uh the same

question and I kind of refer to all

these models as my llm Council so

they're kind of like the Council of

language models if I'm trying to figure

out where to go on a vacation I will ask

all of them and uh so you can also do

that for yourself if that works for you

okay the next topic I want to now turn

to is that of thinking models qu unquote

so we saw in the previous video that

there are multiple stages of training

pre-training goes to supervised fine

tuning goes to reinforcement learning

and reinforcement learning is where the

model gets to practice um on a large

collection of problems that resemble the

practice problems in the textbook and it

gets to practice on a lot of math en

code

problems um and in the process of

reinforcement learning the model

discovers thinking strategies that lead

to good outcomes and these thinking

strategies when you look at them they

very much resemble kind of the inner

monologue you have when you go through

problem solving so the model will try

out different ideas uh it will backtrack

it will revisit assumptions and it will

do things like that now a lot of these

strategies are very difficult to

hardcode as a human labeler because it's

not clear what the thinking process

should be it's only in the reinforcement

learning that the model can try out lots

of stuff and it can find the thinking

process that works for it with its

knowledge and its

capabilities so so this is the third

stage of uh training these models this

stage is relatively recent so only a

year or two ago and all of the different

llm Labs have been experimenting with

these models over the last year and this

is kind of like seen as a large

breakthrough

recently and here we looked at the paper

from Deep seek that was the first to uh

basically talk about it publicly and

they had a nice paper about

incentivizing reasoning capabilities in

llms Via reinforcement learning so

that's the paper that we looked at in

the previous video so we now have to

adjust our cartoon a little bit because

uh basically what it looks like is our

Emoji now has this optional thinking

bubble and when you are using a thinking

model which will do additional thinking

you are using the model that has been

additionally tuned with reinforcement

learning and qualitatively what does

this look like well qualitatively the

model will do a lot more thinking and

what you can expect is that you will get

higher accuracies especially on problems

that are for example math code and

things that require a lot of thinking

things that are very simple like uh

might not actually benefit from this but

things that are actually deep and hard

might benefit a lot and so um but

basically what you're paying for it is

that the models will do thinking and

that can sometimes take multiple minutes

because the models will emit tons and

tons of tokens over a period of many

minutes and you have to wait uh because

the model is thinking just like a human

would think but in situations where you

have very difficult problems this might

Translate to higher accuracy so let's

take a look at some examples so here's a

concrete example when I was stuck on a

programming problem recently so uh

something called the gradient check

fails and I'm not sure why and I copy

pasted the model uh my code uh so the

details of the code are not important

but this is basically um an optimization

of a multier perceptron and details are

not important it's a bunch of code that

I wrote and there was a bug because my

gradient check didn't work and I was

just asking for advice and GPT 40 which

is the blackship most powerful model for

open AI but without thinking uh just

kind of like uh went into a bunch of uh

things that it thought were issues or

that I should double check but actually

didn't really solve the problem like all

of the things that it gave me here are

not the core issue of the problem so the

model didn't really solve the issue um

and it tells me about how to debug it

and so on but then what I did was here

in the drop down I turned to one of the

thinking models now for open

all of these models that start with o

are thinking models 01 O3 mini O3 mini

high and 01 Pro promote are all thinking

models and uh they're not very good at

naming their models uh but uh that is

the case and so here they will say

something like uses Advanced reasoning

or uh good at COD and Logics and stuff

like that but these are basically all

tuned with reinforcement learning and

the because I am paying for $200 per

month I have have access to O Pro mode

which is best at

reasoning um but you might want to try

some of the other ones if depending on

your pricing tier and when I gave the

same model the same prompt to 01 Pro

which is the best at reasoning model and

you have to pay $200 per month for this

one then the exact same prompt it went

off and it thought for 1 minute and it

went through a sequence of thoughts and

opening eye doesn't fully show you the

exact thoughts they just kind of give

you little summaries of the thoughts but

it thought about the code for a while

and then it actually came to get came

back with the correct solution it

noticed that the parameters are

mismatched and how I pack and unpack

them and Etc so this actually solved my

problem and I tried out giving the exact

same prompt to a bunch of other llms so

for example

Claud I gave Claude the same problem and

it actually noticed the correct issue

and solved it and it did that even with

uh sonnet which is not a thinking model

so claw 3.5 Sonet to my knowledge is not

a thinking model and to my knowledge

anthropic as of today doesn't have a

thinking model deployed but this might

change by the time you watch this video

um but even without thinking this model

actually solved the issue uh when I went

to Gemini I asked it um and it also

solved the issue even though I also

could have tried the a thinking model

but it wasn't

necessary I also gave it to grock uh

grock 3 in this case and grock 3 also

solved the problem after a bunch of

stuff um so so it also solved the issue

and then finally I went to uh perplexity

doai and the reason I like perplexity is

because when you go to the model

dropdown one of the models that they

host is this deep seek R1 so this has

the reasoning with the Deep seek R1

model which is the model that we saw uh

over here uh this is the paper so

perplexity just hosts it and makes it

very easy to use so I copy pasted it

there and I ran it and uh I think they

render they like really render it

terribly

but down here you can see the raw

thoughts of the

model uh even though you have to expand

them but you see like okay the user is

having trouble with the gradient check

and then it tries out a bunch of stuff

and then it says but wait when they

accumulate the gradients they're doing

the thing incorrectly let's check the

order the parameters are packed as this

and then it notices the issue and then

it kind of like um says that's a

critical mistake and so it kind of like

thinks through it and you have to wait a

few minutes and then also comes up with

the correct answer so basically long

story short what do I want to show you

there exist a class of models that we

call thinking models all the different

providers may or may not have a thinking

model these models are most effective

for difficult problems in math and code

and things like that and in those kinds

of cases they can push up the accuracy

of your performance in many cases like

if if you're asking for travel advice or

something like that you're not going to

benefit out of a thinking model there's

no need to wait for one minute for it to

think about uh some destinations that

you might want to go to so for myself I

usually try out the non-thinking models

because their responses are really fast

but when I suspect the response is not

as good as it could have been and I want

to give the opportunity to the model to

think a bit longer about it I will

change it to a thinking model depending

on whichever one you have available to

you now when you go to Gro for example

when I start a new conversation with

grock

um when you put the question here like

hello you should put something important

here you see here think so let the model

take its time so turn on think and then

click go and when you click think grock

under the hood switches to the thinking

model and all the different LM providers

will kind of like have some kind of a

selector for whether or not you want the

model to think or whether it's okay to

just like go um with the previous kind

of generation of the models okay now the

next section I want to continue to is to

Tool use uh so far we've only talked to

the language model through text and this

language model is again this ZIP file in

a folder it's inert it's closed off it's

got no tools it's just um a neural

network that can emit

tokens so what we want to do now though

is we want to go beyond that and we want

to give the model the ability to use a

bunch of tools and one of the most

useful tools is an internet search and

so let's take a look at how we can make

models use internet search so for

example again using uh concrete examples

from my own life a few days ago I was

watching White Lotus season 3 um and I

watched the first episode and I love

this TV show by the way and I was

curious when the episode two was coming

out uh and so in the old world you would

imagine you go to Google or something

like that you put in like new episodes

of white lot of season 3 and then you

start clicking on these links and maybe

open a few of

them or something like that right and

you start like searching through it and

trying to figure it out and sometimes

you lock out and you get a

schedule um but many times you might get

really crazy ads there's a bunch of

random stuff going on and it's just kind

of like an unpleasant experience right

so wouldn't it be great if a model could

do this kind of a search for you visit

all the web pages and then take all

those web

pages take all their content and stuff

it into the context window and then

basically give you the response and

that's what we're going to do now

basically we haven't a mechanism or a

way we introduce a mechanism for for the

model to emit a special token that is

some kind of a searchy internet token

and when the model emits the searchd

internet token the Chach PT application

or whatever llm application it is you're

using will stop sampling from the model

and it will take the query that the

model model gave it goes off it does a

search it visits web pages it takes all

of their text and it puts everything

into the context window so now you have

this internet search

tool that itself can also contribute

tokens into our context window and in

this case it would be like lots of

internet web pages and maybe there's 10

of them and maybe it just puts it all

together and this could be thousands of

tokens coming from these web pages just

as we were looking at them ourselves and

then after it has inserted all those web

pages into the Contex window it will

reference back to your question as to

hey what when is this Mo when is this

season getting released and it will be

able to reference the text and give you

the correct answer and notice that this

is a really good example of why we would

need internet search without the

internet search this model has no chance

to actually give us the correct answer

because like I mentioned this model was

trained a few months ago the schedule

probably was not known back then and so

when uh White load of season 3 is coming

out is not part of the real knowledge of

the model and it's not in the zip file

most likely uh because this is something

that was presumably decided on in the

last few weeks and so the model has to

basically go off and do internet search

to learn this knowledge and it learns it

from the web pages just like you and I

would without it and then it can answer

the question once that information is in

the context window and remember again

that the context window is this working

memory so once we load the

Articles once all of these articles

think of their text as being coped copy

pasted into the context window now

they're in a working memory and the

model can actually answer those

questions because it's in the context

window so basically long story short

don't do this manually but use tools

like perplexity as an

example so perplexity doai had a really

nice sort of uh llm that was doing

internet search um and I think it was

like the first app that really

convincingly did this more recently

chashi PT also introduced a search

button says search the web so we're

going to take a look at that in a second

for now when are new episodes of wi

Lotus season 3 getting released you can

just ask and instead of having to do the

work manually we just hit enter and the

model will visit these web pages it will

create all the queries and then it will

give you the answer so it just kind of

did a ton of the work for you um and

then you can uh usually there will be

citations so you can actually visit

those web pages yourself and you can

make sure that these are not

hallucinations from the model and you

can actually like double check that this

is actually correct because it's not in

principle guaranteed it's just um you

know something that may or may not work

if we take this we can also go to for

example chat GPT say the same thing but

now when we put this question in without

actually selecting search I'm not

actually 100% sure what the model will

do in some cases the model will actually

like know that this is recent knowledge

and that it probably doesn't know and it

will create a search in some cases we

have to declare that we want to do the

search in my own personal use I would

know that the model doesn't know and so

I would just select search but let's see

first uh let's see if uh what

happens okay searching the web and then

it prints stuff and then it sites so the

model actually detected itself that it

needs to search the web because it

understands that this is some kind of a

recent information Etc so this was

correct alternatively if I create a new

conversation I could have also select it

search because I know I need to search

enter and then it does the same thing

searching the web and and that's the the

result so basically when you're using

these LM look for this for example

grock excuse

me let's try grock without it without

selecting search Okay so the model does

some search uh just knowing that it

needs to search and gives you the answer

so

basically uh let's see what cloud

does you see so CLA does actually have

the Search tool available so it will say

as of my last update in April

2024 this last update is when the model

went through

pre-training and so Claud is just saying

as of my last update the knowledge cut

off of April

2024 uh it was announced but it doesn't

know so Claud doesn't have the internet

search integrated as an option and will

not give you the answer I expect that

this is something that anthropic might

be working on let's try Gemini and let's

see what it

says unfortunately no official release

date for white loto season 3 yet so um

Gemini 2.0 pro experimental does not

have access to Internet search and

doesn't know uh we could try some of the

other ones like 2.0 flash let me try

that okay so this model seems to know

but it doesn't give citations oh wait

okay there we go sources and related

content so we see how 2.0 flash actually

has the internet search tool but I'm

guessing that the 2.0 pro which is uh

the most powerful model that they have

this one actually does not have access

and it in here it actually tells us 2.0

pro experimental lacks access to

real-time info and some Gemini features

so this model is not fully wired with

internet search so long story short we

can get models to perform Google

searches for us visit the web page just

pull in the information to the context

window and answer questions and uh this

is a very very cool feature but

different models possibly different apps

have different amount of integration of

this capability and so you have to be

kind of on the lookout for that and

sometimes the model will automatically

detect that they need to do search and

sometimes you're better off uh telling

the model that you want it to do the

search so when I'm doing GPT 40 and I

know that this requires to search you

probably will not tick that box

so uh that's uh search tools I wanted to

show you a few more examples of how I

use the search tool in my own work so

what are the kinds of queries that I use

and this is fairly easy for me to do

because usually for these kinds of cases

I go to perplexity just out of habit

even though chat GPT today can do this

kind of stuff as well uh as do probably

many other services as well but I happen

to use perplexity for these kinds of

search queries so whenever I expect that

the answer can be achieved by doing

basically something like Google search

and visiting a few of the top links and

the answer is somewhere in those top

links whenever that is the case I expect

to use the search tool and I come to

perplexity so here are some examples is

the market open today um and uh this was

unprecedent day I wasn't 100% sure so uh

perplexity understands what it's today

it will do the search and it will figure

out that I'm President's Day this was

closed where's White Lotus season 3

filmed again this is something that I

wasn't sure that a model would know in

its knowledge this is something Niche so

maybe there's not that many mentions of

it on the internet and also this is more

recent so I don't expect a model to know

uh by default so uh this was a good a

fit for the Search tool does versel

offer post equal database so this was a

good example of this because I this kind

of stuff changes over time and the

offerings of verel which is accompany

uh may change over time and I want the

latest and whenever something is latest

or something changes I prefer to use the

search tool so I come to

proplex uh when is what do the Apple

launch tomorrow and what are some of the

rumors so again this is something

recent uh where is the singles Inferno

season 4 cast uh must know uh so this is

again a good example because this is

very fresh

information why is the paler stock going

up what is driving the

enthusiasm when is civilization 7 coming

out

exactly um this is an example also like

has Brian Johnson talked about the

toothpaste uses um and I was curious

basically I like what Brian does and

again it has the two features number one

it's a little bit esoteric so I'm not

100% sure if this is at scale on the

internet and would be part of like

knowledge of a model and number two this

might change over time so I want to know

what toothpaste he uses most recently

and so this is good fit again for a

Search tool is it safe to travel to

Vietnam uh this can potentially change

over time and then I saw a bunch of

stuff on Twitter about a USA ID and I

wanted to know kind of like what's the

deal uh so I searched about that and

then you can kind of like dive in in a

bunch of ways here but this use case

here is kind of along the lines of I see

something trending and I'm kind of

curious what's happening like what is

the gist of it and so I very often just

quickly bring up a search of like what's

happening and then get a model to kind

of just give me a gist of roughly what

happened um because a lot of the IND

idual tweets or posts might not have the

full context just by itself so these are

examples of how I use a Search tool okay

next up I would like to tell you about

this capability called Deep research and

this is fairly recent only as of like a

month or two ago uh but I think it's

incredibly cool and really interesting

and kind of went under the radar for a

lot of people even though I think it

shouldn't have so when we go to chipt

pricing here we notice that deep

research is listed here under Pro so it

currently requires $200 per month so

this is the top tier

uh however I think it's incredibly cool

so let me show you by example um in what

kinds of scenarios you might want to use

it roughly speaking uh deep research is

a combination of internet search and

thinking and rolled out for a long time

so the model will go off and it will

spend tens of minutes doing what deep

research um and a first sort of company

that announced this was CH GPT as part

of its Pro offering uh very recently

like a month ago so here's an

example recently I was on the internet

buying supplements which I know is kind

of crazy but Brian Johnson has this

starter pack and I was kind of curious

about it and there's this thing called

Longevity mix right and it's got a bunch

of health actives and I want to know

what these things are right and of

course like so like ca AKG like like

what the hell is this Boost energy

production for sustained Vitality like

what does that mean so one thing you

could of course do is you could open up

Google search uh and look at the

Wikipedia page or something like that

and do everything that you're kind of

used to but deep research allows you to

uh basically take an an alternate route

and it kind of like processes a lot of

this information for you and explains it

a lot better so as an example we can do

something like this this is my example

prompt C AKG is one Health one of the

health actives in Brian Johnson's

blueprint at 2.5 grams per serving can

you do research on CG tell me why um

tell me about why it might be found in

the longevity mix it's possible

efficency in humans or animal models its

potential mechanism of action any

potential concerns or toxicity or

anything like that now here I have this

button available to you to me and you

won't unless you pay $200 per month

right now but I can turn on deep

research so let me copy paste this and

hit

go um and now the model will say okay

I'm going to research this and then

sometimes it likes to ask clarifying

questions before it goes off so a focus

on human clinical studies animal models

are both so let's say both specific

sources uh all of all sources I don't

know comparison to other longevity

compounds uh not

needed comparison just

AKG uh we can be pretty brief the model

understands uh and we hit

go and then okay I'll research AKG

starting research and so now we have to

wait for probably about 10 minutes or so

and if you'd like to click on it you can

get a bunch of preview of what the model

is doing on a high level

so this will go off and it will do a

combination of like I said thinking and

internet search but it will issue many

internet searches it will go through

lots of papers it will look at papers

and it will think and it will come back

10 minutes from now so this will run for

a while uh meanwhile while this is

running uh I'd like to show you

equivalence of it in the industry so

inspired by this a lot of people were

interested in cloning it and so one

example is for example perplexity so

complexity when you go to the model drop

down has something called Deep research

and so you can issue the same queries

here and we can give this to perplexity

and then grock as well has something

called Deep search instead of deep

research but I think that grock's deep

search is kind of like deep research but

I'm not 100% sure so we can issue grock

deep search as well grock 3 deep search

go and uh this model is going to go off

as well now

I

think uh where is my Chachi PT so Chachi

PT is kind of like maybe a quarter

done perplexity is going to be down soon

okay still thinking and Gro is still

going as

well I like grock's interface the most

it seems like okay so basically it's

looking up all kinds of papers Web MD

browsing results and it's kind of just

getting all this now while this is all

going on of course it's accumulating a

giant cont text window and it's

processing all that information trying

to kind of create a report for us so key

points uh what is C CG and why is it in

longevity mix how is it Associated to

longevity Etc and so it will do

citations and it will kind of like tell

you all about it and so this is not a

simple and short response this is a kind

of like almost like a custom research

paper on any topic you would like and so

this is really cool and it gives a lot

of references potentially for you to go

off and do some of your own reading and

maybe ask some clarifying questions

afterwards but it's actually really

incredible that it gives you all these

like different citations and processes

the information for you a little bit

let's see if perplexity finished okay

perplexity is still still researching

and chat PT is also researching so let's

uh briefly pause the video and um I'll

come back when this is done okay so

perplexity finished and we can see some

of the report that it wrote

up uh so there's some references here

and some uh basically description and

then chashi he also finished and it also

thought for 5 minutes looked at 27

sources and produced a

report so here it talked about uh

research in worms dropa in mice and in

human trials that are ongoing and then a

proposed mechanism of action and some

safety and potential

concerns and references which you can

dive uh deeper into so usually in my own

work right now I've only used this maybe

for like 10 to 20 queries so far

something like that usually I find that

the chash PT offering is currently the

best it is the most thorough it reads

the best it is the longest uh it makes

most sense when I read it um and I think

the perplexity and the gro are a little

bit uh a little bit shorter and a little

bit briefer and don't quite get into the

same detail as uh as the Deep research

from Google uh from Chach right now I

will say that everything that is given

to you here again keep in mind that even

though it is doing research and it's

pulling

in there are no guarantees that there

are no hallucinations here uh any of

this can be hallucinated at any point in

time it can be totally made up

fabricated misunderstood by the model so

that's why these citations are really

important treat this as your first draft

treat this as papers to look at um but

don't take this as uh definitely true so

here what I would do now is I would

actually go into these papers and I

would try to understand uh is the is

chat understanding it correctly and

maybe I have some follow-up questions

Etc so you can do all that but still

incredibly useful to see these reports

once in a while to get a bunch of

sources that you might want to descend

into afterwards okay so just like before

I wanted to show a few brief examples of

how how I've used deep research so for

example I was uh trying to change

browser um because Chrome was not uh

Chrome upset me and so it deleted all my

tabs so I was looking at either Brave or

Arc and I I was most interested in which

one is more private and uh basically

Chach BT compil this report for me and I

this was actually quite helpful and I

went into some of the sources and I sort

of understood why Brave is basically

tldr significantly better and that's why

for example here I'm using brave because

I switched to it now and so this is an

example of um basically researching

different kinds of products and

comparing them I think that's a good fit

for deep research uh here I wanted to

know about a life extension in mice so

it kind of gave me a very long reading

but basically mice are an animal model

for longevity and uh different Labs have

tried to extend it with various

techniques and then here I wanted to

explore llm labs in the USA and I wanted

a table of how large they are how much

funding they've had Etc so this is the

table that It produced now this table is

basically hit and miss unfortunately so

I wanted to show it as an example of a

failure um I think some of these numbers

I didn't fully check them but they don't

seem way too wrong some of this looks

wrong um but the bigger Mission I

definitely see is that xai is not here

which I think is a really major emission

and then also conversely hugging phase

should probably not be here because I

asked specifically about llm labs in the

USA and also a Luther AI I don't think

should count as a major llm lab um due

to mostly its resources and so I think

it's kind of a hit and miss things are

missing I don't fully trust these

numbers I have to actually look at them

and so again use it as a first draft

don't fully trust it still very helpful

that's it so what's really happening

here that is interesting is that we are

providing the llm with additional

concrete documents that it can reference

inside its context window so the model

is not just relying on the knowledge the

hazy knowledge of the world through its

parameters and what it knows in its

brain we're actually giving it concrete

documents it's as if you and I reference

specific documents like on the Internet

or something like that while we are um

kind of producing some answer for some

question

now we can do that through an internet

search or like a tool like this but we

can also provide these llms with

concrete documents ourselves through a

file upload and I find this

functionality pretty helpful in many

ways so as an example uh let's look at

Cloud because they just released Cloud

3.7 while I was filming this video so

this is a new Cloud Model that is now

the

state-of-the-art and notice here that we

have thinking mode now as of 3.7 and so

normal is what we looked at so far but

they just release extended best for Math

and coding challenges and what they're

not saying but is actually true under

the hood probably most likely is that

this was trained with reinforcement

learning in a similar way that all the

other thinking models were produced so

what we can do now is we can uploaded

documents that we wanted to reference

inside its context window so as an

example uh there's this paper that came

out that I was kind of interested in

it's from Arc Institute and it's

basically um a language model trained on

DNA and so I was kind of curious ious I

mean I'm not from biology but I was kind

of curious what this is and this is a

perfect example of um what is what LMS

are extremely good for because you can

upload these documents to the llm and

you can load this PDF into the context

window and then ask questions about it

and uh basically read the document

together with an llm and ask questions

off it so the way you do that is you

basically just drag and drop so we can

take that PDF and just drop it

here um this is about 30 megabytes now

when Claude gets this document it is

very likely that they actually discard a

lot of the images and that kind of

information I don't actually know

exactly what they do under the hood and

they don't really talk about it but it's

likely that the images are thrown away

or if they are there they may not be as

as um as well understood as you and I

would understand them potentially and

it's very likely that what's happening

under the hood is that this PDF is

basically converted to a text file and

that text file is loaded into the token

window and once it's in the token window

it's in the working memory and we can

ask questions of it so typically when I

start reading papers together with any

of these llms I just ask for can you uh

give me a

summary uh summary of this

paper let's see what cloud 3.7

says uh okay I'm exceeding the length

limit of this chat

oh god really oh damn okay well let's

try

chbt

uh can you summarize this

paper and we're using gbt 40 and we're

not using thinking

um which is okay we don't we can start

by not thinking

reading documents summary of the paper

genome modeling and design across all

domains of life so this paper introduces

Evo 2 large scale biological Foundation

model and then key

features and so on so I personally find

this pretty helpful and then we can kind

of go back and forth and as I'm reading

through the abstract and the

introduction Etc I am asking questions

of the llm and it's kind of like uh

making it easier for me to understand

the paper another way that I like to use

this functionality extensively is when

I'm reading books it is rarely ever the

case anymore that I read books just by

myself I always involve an LM to help me

read a book so a good example of that

recently is The Wealth of Nations uh

which I was reading recently and it is a

book from 1776 written by Adam Smith and

it's kind of like the foundation of

classical economics and it's a really

good book and it's kind of just very

interesting to me that it was written so

long ago but it has a lot of modern day

kind of like uh it's just got a lot of

insights um that I think are very timely

even today so the way I read books now

as an example is uh you basically pull

up the book and you have to get uh

access to like the raw content of that

information in the case of Wealth of

Nations this is easy because it is from

1776 so you can just find it on wealth

Project Gutenberg as an example and then

basically find the chapter that you are

currently reading so as an example let's

read this chapter from book one and this

chapter uh I was reading recently and it

kind of goes into the division of labor

and how it is limited by the extent of

the market roughly speaking if your

Market is very small then people can't

specialize and specialization is what um

is basically huge uh specialization is

extremely important for wealth creation

um because you can have experts who

specialize in their simple little task

but you can only do that at scale uh

because without the scale you don't have

a large enough market to sell to uh your

specialization so what we do is we copy

paste this book uh this chapter at least

uh this is how I like to do it we go to

say Claud and um we say something like

we are reading The Wealth of

Nations now remember Claude has kind has

knowledge of The Wealth of Nations but

probably doesn't remember exactly the uh

content of this chapter so it wouldn't

make sense to ask Claud questions about

this chapter directly uh because it

probably doesn't remember remember what

this chapter is about but we can remind

Claud by loading this into the context

window so we reading the weal of Nations

uh please summarize this chapter to

start and then what I do here is I copy

paste um now in Cloud when you copy

paste they don't actually show all the

text inside the text box they create a

little text attachment uh when it is

over uh some size and so we can click

enter and uh we just kind of like start

off usually I like to start off with a

summary of what this chapter is about

just so I have a rough idea and then I

go in and I start reading the chapter

and uh any point we have any questions

then we just come in and just ask our

question and I find that basically going

hand inand with llms uh dramatically

creases my retention my understanding of

these chapters and I find that this is

especially the case when you're reading

for example uh documents from other

fields like for example biology or for

example documents from a long time ago

like 1776 where you sort of need a

little bit of help of even understanding

what uh the basics of the language or

for example I would feel a lot more

courage approaching a very old text that

is outside of my area of expertise maybe

I'm reading Shakespeare or I'm reading

things like that I feel like llms make a

lot of reading very dramatically more

accessible than it used to be before

because you're not just right away

confused you can actually kind of go

slowly through it and figure it out

together with the llm in hand so I use

this extensively and I think it's

extremely helpful I'm not aware of tools

unfortunately that make this very easy

for you today I do this clunky back and

forth so literally I will find uh the

book somewhere and I will copy paste

stuff around and I'm going back and

forth and it's extremely awkward and

clunky and unfortunately I'm not aware

of a tool that makes this very easy for

you but obviously what you want is as

you're reading a book you just want to

highlight the passage and ask questions

about it this currently as far as I know

does not exist um but this is extremely

helpful I encourage you to experiment

with it and uh don't read books alone

okay the next very powerful tool that I

now want to turn to is the use of a

python interpreter or basically giving

the ability to the llm to use and write

computer programs so instead of the llm

giving you an answer directly it has the

ability now to write a computer program

and to emit special tokens that the chpt

application recognizes as hey this is

not for the human this is uh basically

saying that whatever I output it here uh

is actually a computer program please go

off and run it and give me the result of

running that computer

program so uh it is the integration of

the language model with a programming

language here like python so uh this is

extremely powerful let's see the

simplest example of where this would be

uh used and what this would look like so

if I go go to chpt and I give it some

kind of a multiplication problem problem

let's say 30 * 9 or something like

that then this is a fairly simple

multiplication and you and I can

probably do something like this in our

head right like 30 * 9 you can just come

up with the result of 270 right so let's

see what happens okay so llm did exactly

what I just did it calculated the result

of this multiplication to be 270 but

it's actually not really doing math it's

actually more like almost memory work uh

but it's easy enough to do in your head

um so there was no tool use involved

here all that happened here was just the

zip file uh doing next token prediction

and uh gave the correct result here in

its head the problem now is what if we

want something more more complicated so

what is this

times this and now of course this if I

asked you to calculate this you would

give up instantly because you know that

you can't possibly do this in your head

and you would be looking for a

calculator and that's exactly what the

llm does now too and opening ey has

trained chat GPT to recognize problems

that it cannot do in its head and to

rely on tools instead so what I expect

jpt to do for this kind of a query is to

turn to Tool use so let's see what it

looks

like okay there we go so what's opened

up here is What's called the python

interpreter and python is basically a

little programming language and instead

of the llm telling you directly what the

result is the llm writes a program and

then not shown here are special tokens

that tell the chipd application to

please run the program and then the llm

pauses

execution instead the Python program

runs creates a result and then passes

this this result back to the language

model as text and the language model

takes over and tells you that the result

of this is that so this is Tulu

incredibly powerful and open a has

trained chpt to kind of like know in

what situations to on tools and they've

taught it to do that by example so uh

human labelers are involved in curating

data sets that um kind of tell the model

by example in what kinds of situations

it should lean on tools and how but

basically we have a python interpreter

and uh this is just an example of

multiplication uh but uh this is

significantly more powerful so let's see

uh what we can actually do inside

programming languages before we move on

I just wanted to make the point that

unfortunately um you have to kind of

keep track of which llms that you're

talking to have different kinds of tools

available to them because different llms

might not have all the same tools and in

particular LMS that do not have access

to the python interpreter or programming

language or are unwilling to use it

might not give you correct results in

some of these harder problems so as an

example here we saw that um chasht

correctly used a programming language

and didn't do this in its head grock 3

actually I believe does not have access

to a programming language uh like like a

python interpreter and here it actually

does this in its head and gets

remarkably close but if you actually

look closely at it uh it gets it wrong

this should be one 120 instead of

060 so grock 3 will just hallucinate

through this multiplication and uh do it

in its head and get it wrong but

actually like remarkably close uh then I

tried Claud and Claude actually wrote In

this case not python code but it wrote

JavaScript code but uh JavaScript is

also a programming l language and get

gets the correct result then I came to

Gemini and I asked uh 2.0 pro and uh

Gemini did not seem to be using any

tools there's no indication of that and

yet it gave me what I think is the

correct result which actually kind of

surprised me so Gemini I think actually

calculated this in its head correctly

and the way we can tell that this is uh

which is kind of incredible the way we

can tell that it's not using tools is we

can just try something harder what is we

have to make it harder for it

okay so it gives us some result and then

I can use uh my calculator here and it's

wrong right so this is using my MacBook

Pro calculator and uh two it's it's not

correct but it's like remarkably close

but it's not correct but it will just

hallucinate the answer so um I guess

like my point is unfortunately the state

of the llms right now is such that

different llms have different tools

available to them and you kind of have

to keep track of it and if they don't

have the tools available they'll just do

their best uh which means that they

might hallucinate a result for you so

that's something to look out for okay so

one practical setting where this can be

quite powerful is what's called Chach

Advanced Data analysis and as far as I

know this is quite unique to chpt itself

and it basically um gets chpt to be kind

of like a junior data analyst uh who you

can uh kind of collaborate with so let

me show you a concrete example without

going into the full detail so first we

need to get some data that we can

analyze and plot and chart Etc so here

in this case I said uh let's research

openi evaluation as an example and I

explicitly asked Chachi to use the

search tool because I know that under

the hood such a thing exists and I don't

want it to be hallucinating data to me I

wanted to actually look it up and back

it up and create a table where each year

have we have the valuation so these are

the open evaluations over time notice

how in 2015 it's not applicable

so uh the valuation is like unknown then

I said now plot this use lock scale for

y- axis and so this is where this gets

powerful Chachi PT goes off and writes a

program that plots the data over here so

it cre a little figure for us and it uh

sort of uh ran it and showed it to us so

this can be quite uh nice and valuable

because it's very easy way to basically

collect data upload data in a

spreadsheet and visualize it Etc I will

note some of the things here so as an

example notice that we had na for 2015

but Chachi PT when I was writing the

code and again I would always encourage

you to scrutinize the code it put in 0.1

for 2015 and so basically it implicitly

assumed that uh it made the Assumption

here in code that the valuation of 2015

was 100

million uh and because it put in 0.1 and

it's kind of like did it without telling

us so it's a little bit sneaky and uh

that's why you kind of have to pay

attention little bit to the code so I'm

Amil with the code and I always read it

um but I think I would be hesitant to

potentially recommend the use of these

tools uh if people aren't able to like

read it and verify it a little bit for

themselves um now fit a trend line and

extrapolate until the year 2030 Mark the

expected valuation in 2030 so it went

off and it basically did a linear fit

and it's using cciis curve

fit and it did this and came up with a

plot and uh

it told me that the valuation based on

the trend in 2030 is approximately 1.7

trillion which sounds amazing except uh

here I became suspicious because I see

that Chach PT is telling me it's 1.7

trillion but when I look here at 2030

it's printing 2027 1.7 B so its

extrapolation when it's printing the

variable is inconsistent with 1.7

trillion uh this makes it look like that

valuation should be about 20 trillion

and so that's what I said print this

variable directly by itself what is it

and then it sort of like rewrote the

code and uh gave me the variable itself

and as we see in the label here it is

indeed

2271 Etc so in 2030 the true exponential

Trend extrapolation would be a valuation

of 20

trillion um so I was like I was trying

to confront Chach and I was like you

lied to me right and it's like yeah

sorry I messed up

so I guess I I I like this example

because number one it shows the power of

the tool in that it can create these

figures for you and it's very nice but I

think number two it shows the um

trickiness of it where for example here

it made an implicit assumption and here

it actually told me something uh it told

me just the wrong it hallucinated 1.7

trillion so again it is kind of like a

very very Junior data analyst it's

amazing that it can plot figures

but you have to kind of still know what

this code is doing and you have to be

careful and scrutinize it and make sure

that you are really watching very

closely because your Junior analyst is a

little bit uh absent minded and uh not

quite right all the time so really

powerful but also be careful with this

um I won't go into full details of

Advanced Data analysis but uh there were

many videos made on this topic so if you

would like to use some of this in your

work uh then I encourage you to look at

at some of these videos I'm not going to

go into the full detail so a lot of

promise but be careful okay so I've

introduced you to Chach PT and Advanced

Data analysis which is one powerful way

to basically have LMS interact with code

and add some UI elements like showing of

figures and things like that I would now

like to uh introduce you to one more

related tool and that is uh specific to

cloud and it's called

artifacts so let me show you by example

what this is so I have a conversation

with Claude and I'm asking generate 20

flash cards from the following

text um and for the text itself I just

came to the Adam Smith Wikipedia page

for example and I copy pasted this

introduction here so I copy pasted this

here and asked for flash cards and

Claude responds with 20 flash cards so

for example when was Adam Smith baptized

on June 16th Etc when did he die what

was his nationality Etc so once we have

the flash cards we actually want to

practice these flashcards and so this is

where I continue the conversation and I

say now use the artifacts feature to

write a flashcards app to test these

flashcards and so clot goes off and

writes code for an app that uh basically

formats all of this into flashcards and

that looks like this so what Claude

wrote specifically was this C code here

so it uses a react library and then

basically creates all these components

it hardcodes the Q&A into this app and

then all the other functionality of it

and then the cloud interface basically

is able to load these react components

directly in your browser and so you end

up with an app so when was Adam Smith

baptized and you can click to reveal the

answer and then you can say whether you

got it correct or not when did he

die uh what was his nationality Etc so

you can imagine doing this and then

maybe we can reset the progress or

Shuffle the cards Etc so what happened

here is that Claude wrote us a super

duper custom app just for us uh right

here and um typically what we're used to

is some software Engineers write apps

they make them available and then they

give you maybe some way to customize

them or maybe to upload flashcards like

for example in the eny app you can

import flash cards and all this kind of

stuff this is a very different Paradigm

because in this Paradigm Claud just

writes the app just for you and deploys

it here in your browser now keep in mind

that a lot of apps you will find on the

internet they have entire backends Etc

there's none of that here there's no

database or anything like that but these

are like local apps that can run in your

browser and uh they can get fairly

sophisticated and useful in some

cases uh so that's Cloud artifacts now

to be honest I'm not actually a daily

user of artifacts I use it once in a

while I do know that a large number of

people are experimenting with it and you

can find a lot of artifact showcasing

cases because they're easy to share so

these are a lot of things that people

have developed um various timers and

games and things like that um but the

one use case that I did find very useful

in my own work is basically uh the use

of diagrams diagram generation so as an

example let's go back to the book

chapter of Adam Smith that we were

looking at what I do sometimes is we are

reading The Wealth of Nations by Adam

Smith I'm attaching chapter 3 and book

one please create a conceptual diagram

of this chapter

and when Claude hears conceptual diagram

of this chapter very often it will write

a code that looks like

this and if you're not familiar with

this this is using the mermaid library

to basically create or Define a graph

and then uh this is plotting that

mermaid diagram and so Claud analyzes

the chapter and figures out that okay

the key principle that's being

communicated here is as follows that

basically the division of labor is

related to the extent of the market the

size of it and then these are the pieces

of the chapter so there's the

comparative example um of trade and how

much easier it is to do on land and on

water and the specific example that's

used and that Geographic factors

actually make a huge difference here and

then the comparison of land transport

versus water transport and how much

easier water transport

is and then here we have some early

civilizations that have all benefited

from basically the availability of water

water transport and have flourished as a

result of it because they support

specialization so it's if you're a

conceptual kind of like visual thinker

and I think I'm a little bit like that

as well I like to lay out information

and like as like a tree like this and it

helps me remember what that chapter is

about very easily and I just really

enjoy these diagrams and like kind of

getting a sense of like okay what is the

layout of the argument how is it

arranged spatially and so on and so if

you're like me then you will definitely

enjoy this and you can make diagrams of