hi everyone so I've wanted to make this

video for a while it is a comprehensive

but General audience introduction to

large language models like Chachi PT and

what I'm hoping to achieve in this video

is to give you kind of mental models for

thinking through what it is that this

tool is it is obviously magical and

amazing in some respects it's uh really

good at some things not very good at

other things and there's also a lot of

sharp edges to be aware of so what is

behind this text box you can put

anything in there and press enter but uh

what should we be putting there and what

are these words generated back how does

this work and what what are you talking

to exactly so I'm hoping to get at all

those topics in this video we're going

to go through the entire pipeline of how

this stuff is built but I'm going to

keep everything uh sort of accessible to

a general audience so let's take a look

at first how you build something like

chpt and along the way I'm going to talk

about um you know some of the sort of

cognitive psychological implications of

the tools okay so let's build Chachi PT

so there's going to be multiple stages

arranged sequentially the first stage is

called the pre-training stage and the

first step of the pre-training stage is

to download and process the internet now

to get a sense of what this roughly

looks like I recommend looking at this

URL here so um this company called

hugging face uh collected and created

and curated this data set called Fine

web and they go into a lot of detail on

this block post on how how they

constructed the fine web data set and

all of the major llm providers like open

AI anthropic and Google and so on will

have some equivalent internally of

something like the fine web data set so

roughly what are we trying to achieve

here we're trying to get ton of text

from the internet from publicly

available sources so we're trying to

have a huge quantity of very high

quality documents and we also want very

large diversity of documents because we

want to have a lot of knowledge inside

these models so we want large diversity

of high quality documents and we want

many many of them and achieving this is

uh quite complicated and as you can see

here takes multiple stages to do well so

let's take a look at what some of these

stages look like in a bit for now I'd

like to just like to note that for

example the fine web data set which is

fairly representative what you would see

in a production grade application

actually ends up being only about 44

terabyt of dis space um you can get a

USB stick for like a terabyte very

easily or I think this could fit on a

single hard drive almost today so this

is not a huge amount of data at the end

of the day even though the internet is

very very large we're working with text

and we're also filtering it aggressively

so we end up with about 44 terabytes in

this example so let's take a look at uh

kind of what this data looks like and

what some of these stages uh also are so

the starting point for a lot of these

efforts and something that contributes

most of the data by the end of it is

Data from common crawl so common craw is

an organization that has been basically

scouring the internet since 2007 so as

of 2024 for example common CW has

indexed 2.7 billion web

pages uh and uh they have all these

crawlers going around the internet and

what you end up doing basically is you

start with a few seed web pages and then

you follow all the links and you just

keep following links and you keep

indexing all the information and you end

up with a ton of data of the internet

over time so this is usually the

starting point for a lot of the uh for a

lot of these efforts now this common C

data is quite raw and is filtered in

many many different ways

so here they Pro they document this is

the same diagram they document a little

bit the kind of processing that happens

in these stages so the first thing here

is something called URL

filtering so what that is referring to

is that there's these block

lists of uh basically URLs that are or

domains that uh you don't want to be

getting data from so usually this

includes things like U malware websites

spam websites marketing websites uh

racist websites adult sites and things

like that so there's a ton of different

types of websites that are just

eliminated at this stage because we

don't want them in our data set um the

second part is text extraction you have

to remember that all these web pages

this is the raw HTML of these web pages

that are being saved by these crawlers

so when I go to inspect

here this is what the raw HTML actually

looks like you'll notice that it's got

all this markup uh like lists and stuff

like that and there's CSS and all this

kind of stuff so this is um computer

code almost for these web pages but what

we really want is we just want this text

right we just want the text of this web

page and we don't want the navigation

and things like that so there's a lot of

filtering and processing uh and heris

that go into uh adequately filtering for

just their uh good content of these web

pages the next stage here is language

filtering so for example fine web

filters uh using a language classifier

they try to guess what language every

single web page is in and then they only

keep web pages that have more than 65%

of English as an

example and so you can get a sense that

this is like a design decision that

different companies can uh can uh take

for themselves what fraction of all

different types of languages are we

going to include in our data set because

for example if we filter out all of the

Spanish as an example then you might

imagine that our model later will not be

very good at Spanish because it's just

never seen that much data of that

language and so different companies can

focus on multilingual performance to uh

to a different degree as an example so

fine web is quite focused on English and

so their language model if they end up

training one later will be very good at

English but not may be very good at

other

languages after language filtering

there's a few other filtering steps and

D duplication and things like that um

finishing with for example the pii

removal this is personally identifiable

information so as an example addresses

Social Security numbers and things like

that you would try to detect them and

you would try to filter out those kinds

of web pages from the the data set as

well so there's a lot of stages here and

I won't go into full detail but it is a

fairly extensive part of the

pre-processing and you end up with for

example the fine web data set so when

you click in on it uh you can see some

examples here of what this actually ends

up looking like and anyone can download

this on the huging phase web page and so

here are some examples of the final text

that ends up in the training set so this

is some article about tornadoes in

2012 um so there's some t tadoes in 2020

in 2012 and what

happened uh this next one is something

about did you know you have two little

yellow 9vt battery sized adrenal glands

in your body okay so this is some kind

of a odd medical

article so just think of these as

basically uh web pages on the internet

filtered just for the text in various

ways and now we have a ton of text 40

terabytes off it and that now is the

starting point for the next step of this

stage now I wanted to give you an

intuitive sense of where we are right

now so I took the first 200 web pages

here and remember we have tons of them

and I just take all that text and I just

put it all together concatenate it and

so this is what we end up with we just

get this just just raw text raw internet

text and there's a ton of it even in

these 200 web pages so I can continue

zooming out here and we just have this

like massive tapestry of Text data and

this text data has all these p patterns

and what we want to do now is we want to

start training neural networks on this

data so the neural networks can

internalize and model how this text

flows right so we just have this giant

texture of text and now we want to get

neural Nets that mimic it okay now

before we plug text into neural networks

we have to decide how we're going to

represent this text uh and how we're

going to feed it in now the way our

technology works for these neuron Lots

is that they expect

a one-dimensional sequence of symbols

and they want a finite set of symbols

that are possible and so we have to

decide what are the symbols and then we

have to represent our data as

one-dimensional sequence of those

symbols so right now what we have is a

onedimensional sequence of text it

starts here and it goes here and then it

comes here Etc so this is a

onedimensional sequence even though on

my monitor of course it's laid out in a

two-dimensional way but it goes from

left to right and top to bottom right so

it's a one-dimensional sequence of text

now this being computers of course

there's an underlying representation

here so if I do what's called utf8 uh

encode this text then I can get the raw

bits that correspond to this text in the

computer and that's what uh that looks

like this so it turns out that for

example this very first bar here is the

first uh eight bits as an

example so what is this thing right this

is um representation that we are looking

for uh in in a certain sense we have

exactly two possible symbols zero and

one and we have a very long sequence of

it right now as it turns out um this

sequence length is actually going to be

very finite and precious resource uh in

our neural network and we actually don't

want extremely long sequences of just

two symbols instead what we want is we

want to trade off uh this um symbol

size uh of this vocabulary as we call it

and the resulting sequence length so we

don't want just two symbols and

extremely long sequences we're going to

want more symbols and shorter sequences

okay so one naive way of compressing or

decreasing the length of our sequence

here is to basically uh consider some

group of consecutive bits for example

eight bits and group them into a single

what's called bite so because uh these

bits are either on or off if we take a

group of eight of them there turns out

to be only 256 possible combinations of

how these bits could be on or off and so

therefore we can re repesent this

sequence into a sequence of bytes

instead so this sequence of bytes will

be eight times shorter but now we have

256 possible symbols so every number

here goes from 0 to

255 now I really encourage you to think

of these not as numbers but as unique

IDs or like unique symbols so maybe it's

a bit more maybe it's better to actually

think of these to replace every one of

these with a unique Emoji you'd get

something like this so um we basically

have a sequence of emojis and there's

256 possible emojis you can think of it

that way now it turns out that in

production for state-of-the-art language

models uh you actually want to go even

Beyond this you want to continue to

shrink the length of the sequence uh

because again it is a precious resource

in return for more symbols in your

vocabulary and the way this is done is

done by running what's called The Bite

pair encoding algorithm and the way this

works is we're basically looking for

consecutive bytes or symbols that are

very common so for example turns out

that the sequence 116 followed by 32 is

quite common and occurs very frequently

so what we're going to do is we're going

to group uh this um pair into a new

symbol so we're going to Mint a symbol

with an ID 256 and we're going to

rewrite every single uh pair 11632 with

this new symbol and then can we can

iterate this algorithm as many times as

we wish and each time when we mint a new

symbol we're decreasing the length and

we're increasing the symbol size and in

practice it turns out that a pretty good

setting of um the basically the

vocabulary size turns out to be about

100,000 possible symbols so in

particular GPT 4 uses

100,

277 symbols

um and this process of converting from

raw text into these symbols or as we

call them tokens is the process called

tokenization so let's now take a look at

how gp4 performs tokenization conting

from text to tokens and from tokens back

to text and what this actually looks

like so one website I like to use to

explore these token representations is

called tick tokenizer and so come here

to the drop down and select CL 100 a

base which is the gp4 base model

tokenizer and here on the left you can

put in text and it shows you the

tokenization of that text so for example

heo space

world so hello world turns out to be

exactly two Tokens The Token hello which

is the token with ID

15339 and the token space

world that is the token 1

1917 so um hello space world now if I

was to join these two for example I'm

going to get again two tokens but it's

the token H followed by the token L

world without the

H um if I put in two Spa two spaces here

between hello and world it's again a

different uh tokenization there's a new

token 220

here okay so you can play with this and

see what happens here also keep in mind

this is not uh this is case sensitive so

if this is a capital H it is something

else or if it's uh hello world then

actually this ends up being three tokens

instead of just two

tokens yeah so you can play with this

and get an sort of like an intuitive

sense of uh what these tokens work like

we're actually going to loop around to

tokenization a bit later in the video

for now I just wanted to show you the

website and I wanted to uh show you that

this text basically at the end of the

day so for example if I take one line

here this is what GT4 will see it as so

this text will be a sequence of length

62 this is the sequence here and this is

how the chunks of text correspond to

these symbols and again there's 100,

27777 possible symbols and we now have

one-dimensional sequences of those

symbols so um yeah we're going to come

back to tokenization but that's uh for

now where we are okay so what I've done

now is I've taken this uh sequence of

text that we have here in the data set

and I have re-represented it using our

tokenizer into a sequence of tokens and

this is what that looks like now so for

example when we go back to the Fine web

data set they mentioned that not only is

this 44 terab of dis space but this is

about a 15 trillion token sequence of um

in this data set and so here these are

just some of the first uh one or two or

three or a few thousand here I think uh

tokens of this data set but there's 15

trillion here uh to keep in mind and

again keep in mind one more time that

all of these represent little text

chunks they're all just like atoms of

these sequences and the numbers here

don't make any sense they're just uh

they're just unique IDs okay so now we

get to the fun part which is the uh

neural network training and this is

where a lot of the heavy lifting happens

computationally when you're training

these neural networks so what we do here

in this this step is we want to model

the statistical relationships of how

these tokens follow each other in the

sequence so what we do is we come into

the data and we take Windows of tokens

so we take a window of tokens uh from

this data fairly

randomly and um the windows length can

range anywhere anywhere between uh zero

tokens actually all the way up to some

maximum size that we decide on uh so for

example in practice you could see a

token with Windows of say 8,000 tokens

now in principle we can use arbitrary

window lengths of tokens uh but uh

processing very long uh basically U

window sequences would just be very

computationally expensive so we just

kind of decide that say 8,000 is a good

number or 4,000 or 16,000 and we crop it

there now in this example I'm going to

be uh taking the first four tokens just

so everything fits nicely so these

tokens

we're going to take a window of four

tokens this bar view in and space single

which are these token

IDs and now what we're trying to do here

is we're trying to basically predict the

token that comes next in the sequence so

3962 comes next right so what we do now

here is that we call this the context

these four tokens are context and they

feed into a neural

network and this is the input to the

neural network

now I'm going to go into the detail of

what's inside this neural network in a

little bit for now it's important to

understand is the input and the output

of the neural net so the input are

sequences of tokens of variable length

anywhere between zero and some maximum

size like 8,000 the output now is a

prediction for what comes next so

because our vocabulary has

100277 possible tokens the neural

network is going to Output exactly that

many numbers

and all of those numbers correspond to

the probability of that token as coming

next in the sequence so it's making

guesses about what comes

next um in the beginning this neural

network is randomly initialized so um

and we're going to see in a little bit

what that means but it's a it's a it's a

random transformation so these

probabilities in the very beginning of

the training are also going to be kind

of random uh so here I have three

examples but keep in mind that there's

100,000 numbers here um so the

probability of this token space

Direction neural network is saying that

this is 4% likely right now 11799 is 2%

and then here the probility of 3962

which is post is 3% now of course we've

sampled this window from our data set so

we know what comes next we know and

that's the label we know that the

correct answer is that 3962 actually

comes next in the sequence so now what

we have is this mathematical process for

doing an update to the neural network we

have the way of tuning it and uh we're

going to go into a little bit of of

detail in a bit but basically we know

that this probability here of 3% we want

this probability to be higher and we

want the probabilities of all the other

tokens to be

lower and so we have a way of

mathematically calculating how to adjust

and update the neural network so that

the correct answer has a slightly higher

probability so if I do an update to the

neural network now the next time I Fe

this particular sequence of four tokens

into neural network the neural network

will be slightly adjusted now and it

will say Okay post is maybe 4% and case

now maybe is

1% and uh Direction could become 2% or

something like that and so we have a way

of nudging of slightly updating the

neuronet to um basically give a higher

probability to the correct token that

comes next in the sequence and now you

just have to remember that this process

happens not just for uh this um token

here where these four fed in and

predicted this one this process happens

at the same time for all of these tokens

in the entire data set and so in

practice we sample little windows little

batches of Windows and then at every

single one of these tokens we want to

adjust our neural network so that the

probability of that token becomes

slightly higher and this all happens in

parallel in large batches of these

tokens and this is the process of

training the neural network it's a

sequence of updating it so that it's

predictions match up the statistics of

what actually happens in your training

set and its probabilities become

consistent with the uh statistical

patterns of how these tokens follow each

other in the data so let's now briefly

get into the internals of these neural

networks just to give you a sense of

what's inside so neural network

internals so as I mentioned we have

these inputs uh that are sequences of

tokens in this case this is four input

tokens but this can be anywhere between

zero up to let's say 8,000 tokens in

principle this can be an infinite number

of tokens we just uh it would just be

too computationally expensive to process

an infinite number of tokens so we just

crop it at a certain length and that

becomes the maximum context length of

that uh

model now these inputs X are mixed up in

a giant mathematical expression together

with the parameters or the weights of

these neural networks so here I'm

showing six example parameters and their

setting but in practice these uh um

modern neural networks will have

billions of these uh parameters and in

the beginning these parameters are

completely randomly set now with a

random setting of parameters you might

expect that this uh this neural network

would make random predictions and it

does in the beginning it's totally

random predictions but it's through this

process of iteratively updating the

network uh as and we call that process

training a neural network so uh that the

setting of these parameters gets

adjusted such that the outputs of our

neural network becomes consistent with

the patterns seen in our training

set so think of these parameters as kind

of like knobs on a DJ set and as you're

twiddling these knobs you're getting

different uh predictions for every

possible uh token sequence input and

training in neural network just means

discovering a setting of parameters that

seems to be consistent with the

statistics of the training

set now let me just give you an example

what this giant mathematical expression

looks like just to give you a sense and

modern networks are massive expressions

with trillions of terms probably but let

me just show you a simple example here

it would look something like this I mean

these are the kinds of Expressions just

to show you that it's not very scary we

have inputs x uh like X1 x2 in this case

two example inputs and they get mixed up

with the weights of the network w0 W1 2

3 Etc and this mixing is simple things

like multiplication addition addition

exponentiation division Etc and it is

the subject of neural network

architecture research to design

effective mathematical Expressions uh

that have a lot of uh kind of convenient

characteristics they are expressive

they're optimizable they're paralyzable

Etc and so but uh at the end of the day

these are these are not complex

expressions and basically they mix up

the inputs with the parameters to make

predictions and we're optimizing uh the

parameters of this neural network so

that the predictions come out consistent

with the training set now I would like

to show you an actual production grade

example of what these neural networks

look like so for that I encourage you to

go to this website that has a very nice

visualization of one of these

networks so this is what you will find

on this website and this neural network

here that is used in production settings

has this special kind of structure this

network is called the Transformer and

this particular one as an example has 8

5,000 roughly

parameters now here on the top we take

the inputs which are the token

sequences and then information flows

through the neural network until the

output which here are the logit softmax

but these are the predictions for what

comes next what token comes

next and then here there's a sequence of

Transformations and all these

intermediate values that get produced

inside this mathematical expression s it

is sort of predicting what comes next so

as an example these tokens are embedded

into kind of like this distributed

representation as it's called so every

possible token has kind of like a vector

that represents it inside the neural

network so first we embed the tokens and

then those values uh kind of like flow

through this diagram and these are all

very simple mathematical Expressions

individually so we have layer norms and

Matrix multiplications and uh soft Maxes

and so on so here kind of like the

attention block of this Transformer and

then information kind of flows through

into the multi-layer perceptron block

and so on and all these numbers here

these are the intermediate values of the

expression and uh you can almost think

of these as kind of like the firing

rates of these synthetic neurons but I

would caution you to uh not um kind of

think of it too much like neurons

because these are extremely simple

neurons compared to the neurons you

would find in your brain your biological

neurons are very complex dynamical

processes that have memory and so on

there's no memory in this expression

it's a fixed mathematical expression

from input to Output with no memory it's

just a

stateless so these are very simple

neurons in comparison to biological

neurons but you can still kind of

loosely think of this as like a

synthetic piece of uh brain tissue if

you if you like uh to think about it

that way so information flows through

all these neurons fire until we get to

the predictions now I'm not actually

going to dwell too much on the precise

kind of like mathematical details of all

these Transformations honestly I don't

think it's that important to get into

what's really important to understand is

that this is a mathematical function it

is uh parameterized by some fixed set of

parameters like say 85,000 of them and

it is a way of transforming inputs into

outputs and as we twiddle the parameters

we are getting uh different kinds of

predictions and then we need to find a

good setting of these parameters so that

the predictions uh sort of match up with

the patterns seen in training set

so that's the Transformer okay so I've

shown you the internals of the neural

network and we talked a bit about the

process of training it I want to cover

one more major stage of working with

these networks and that is the stage

called inference so in inference what

we're doing is we're generating new data

from the model and so uh we want to

basically see what kind of patterns it

has internalized in the parameters of

its Network so to generate from the

model is relatively straightforward

we start with some tokens that are

basically your prefix like what you want

to start with so say we want to start

with the token 91 well we feed it into

the

network and remember that the network

gives us probabilities right it gives us

this probability Vector here so what we

can do now is we can basically flip a

biased coin so um we can sample uh

basically a token based on this

probability distribution so the tokens

that are given High probability by the

model are more likely to be sampled when

you flip this biased coin you can think

of it that way so we sample from the

distribution to get a single unique

token so for example token 860 comes

next uh so 860 in this case when we're

generating from model could come next

now 860 is a relatively likely token it

might not be the only possible token in

this case there could be many other

tokens that could have been sampled but

we could see that 86c is a relatively

likely token as an example and indeed in

our training examp example here 860 does

follow 91 so let's now say that we um

continue the process so after 91 there's

a60 we append it and we again ask what

is the third token let's sample and

let's just say that it's 287 exactly as

here let's do that again we come back in

now we have a sequence of three and we

ask what is the likely fourth token and

we sample from that and get this one and

now let's say we do it one more time we

take those four we sample and we get

this one and this

13659 uh this is not actually uh 3962 as

we had before so this token is the token

article uh instead so viewing a single

article and so in this case we didn't

exactly reproduce the sequence that we

saw here in the training data so keep in

mind that these systems are stochastic

they have um we're sampling and we're

flipping coins and sometimes we lock out

and we reproduce some like small chunk

of the text and training set but

sometimes we're uh we're getting a token

that was not verbatim part of any of the

documents in the training data so we're

going to get sort of like remixes of the

data that we saw in the training because

at every step of the way we can flip and

get a slightly different token and then

once that token makes it in if you

sample the next one and so on you very

quickly uh start to generate token

streams that are very different from the

token streams that UR

in the training documents so

statistically they will have similar

properties but um they are not identical

to your training data they're kind of

like inspired by the training data and

so in this case we got a slightly

different sequence and why would we get

article you might imagine that article

is a relatively likely token in the

context of bar viewing single Etc and

you can imagine that the word article

followed this context window somewhere

in the training documents uh to some

extent and we just happen to sample it

here at that stage so basically

inference is just uh predicting from

these distributions one at a time we

continue feeding back tokens and getting

the next one and we uh we're always

flipping these coins and depending on

how lucky or unlucky we get um we might

get very different kinds of patterns

depending on how we sample from these

probability distributions so that's

inference so in most common scenarios uh

basically downloading the internet and

tokenizing it is is a pre-processing

step you do that a single time and then

uh once you have your token sequence we

can start training networks and in

Practical cases you would try to train

many different networks of different

kinds of uh settings and different kinds

of arrangements and different kinds of

sizes and so you''ll be doing a lot of

neural network training and um then once

you have a neural network and you train

it and you have some specific set of

parameters that you're happy with um

then you can take the model and you can

do inference and you can actually uh

generate data from the model and when

you're on chat GPT and you're talking

with a model uh that model is trained

and has been trained by open aai many

months ago probably and they have a

specific set of Weights that work well

and when you're talking to the model all

of that is just inference there's no

more training those parameters are held

fixed and you're just talking to the

model sort of uh you're giving it some

of the tokens and it's kind of

completing token sequences and that's

what you're seeing uh generated when you

actually use the model on CH GPT so that

model then just does inference alone so

let's now look at an example of training

an inference that is kind of concrete

and gives you a sense of what this

actually looks like uh when these models

are trained now the example that I would

like to work with and that I'm

particularly fond of is that of opening

eyes gpt2 so GPT uh stands for

generatively pre-trained Transformer and

this is the second iteration of the GPT

series by open AI when you are talking

to chat GPT today the model that is

underlying all of the magic of that

interaction is GPT 4 so the fourth

iteration of that series now gpt2 was

published in 2019 by openi in this paper

that I have right here and the reason I

like gpt2 is that it is the first time

that a recognizably modern stack came

together so um all of the pieces of gpd2

are recognizable today by modern

standards it's just everything has

gotten bigger now I'm not going to be

able to go into the full details of this

paper of course because it is a

technical publication but some of the

details that I would like to highlight

are as follows gpt2 was a Transformer

neural network just like you were just

like the neural networks you would work

with today it was it had 1.6 billion

parameters right so these are the

parameters that we looked at here it

would have 1.6 billion of them today

modern Transformers would have a lot

closer to a trillion or several hundred

billion

probably the maximum context length here

was 1,24 tokens so it is when we are

sampling chunks of Windows of tokens

from the data set we're never taking

more than 1,24 tokens and so when you

are trying to predict the next token in

a sequence you will never have more than

1,24 tokens uh kind of in your context

in order to make that prediction now

this is also tiny by modern standards

today the token uh the context lengths

would be a lot closer to um couple

hundred thousand or maybe even a million

and so you have a lot more context a lot

more tokens in history history and you

can make a lot better prediction about

the next token in the sequence in that

way and finally gpt2 was trained on

approximately 100 billion tokens and

this is also fairly small by modern

standards as I mentioned the fine web

data set that we looked at here the fine

web data set has 15 trillion tokens uh

so 100 billion is is quite

small

now uh I actually tried to reproduce uh

gpt2 for fun as part of this project

called lm. C so you can see my rup of

doing that in this post on GitHub under

the lm. C repository so in particular

the cost of training gpd2 in 2019 what

was estimated to be approximately

$40,000 but today you can do

significantly better than that and in

particular here it took about one day

and about

$600 uh but this wasn't even trying too

hard I think you could really bring this

down to about $100 today now why is it

that the costs have come down so much

well number one these data sets have

gotten a lot better and the way we

filter them extract them and prepare

them has gotten a lot more refined and

so the data set is of just a lot higher

quality so that's one thing but really

the biggest difference is that our

computers have gotten much faster in

terms of the hardware and we're going to

look at that in a second and also the

software for uh running these models and

really squeezing out all all the speed

from the hardware as it is possible uh

that software has also gotten much

better as as everyone has focused on

these models and try to run them very

very

quickly now I'm not going to be able to

go into the full detail of this gpd2

reproduction and this is a long

technical post but I would like to still

give you an intuitive sense for what it

looks like to actually train one of

these models as a researcher like what

are you looking at and what does it look

like what does it feel like so let me

give you a sense of that a little bit

okay so this is what it looks like let

me slide this

over so what I'm doing here is I'm

training a gpt2 model right now

and um what's happening here is that

every single line here like this one is

one update to the model so remember how

here we are um basically making the

prediction better for every one of these

tokens and we are updating these weights

or parameters of the neural net so here

every single line is One update to the

neural network where we change its

parameters by a little bit so that it is

better at predicting next token and

sequence in particular every single line

here is improving the prediction on 1

million tokens in the training set so

we've basically taken 1 million tokens

out of this data set and we've tried to

improve the prediction of that token as

coming next in a sequence on all 1

million of them

simultaneously and at every single one

of these steps we are making an update

to the network for that now the number

to watch closely is this number called

loss and the loss is a single number

that is telling you how well your neural

network is performing right now and it

is created so that low loss is good so

you'll see that the loss is decreasing

as we make more updates to the neural

nut which corresponds to making better

predictions on the next token in a

sequence and so the loss is the number

that you are watching as a neural

network researcher and you are kind of

waiting you're twiddling your thumbs uh

you're drinking coffee and you're making

sure that this looks good so that with

every update your loss is improving and

the network is getting better at

prediction now here you see that we are

processing 1 million tokens per update

each update takes about 7 Seconds

roughly and here we are going to process

a total of 32,000 steps of

optimization so 32,000 steps with 1

million tokens each is about 33 billion

tokens that we are going to process and

we're currently only about 420 step 20

out of 32,000 so we are still only a bit

more than 1% done because I've only been

running this for 10 or 15 minutes or

something like

that now every 20 steps I have

configured this optimization to do

inference so what you're seeing here is

the model is predicting the next token

in a sequence and so you sort of start

it randomly and then you continue

plugging in the tokens so we're running

this inference step and this is the

model sort of predicting the next token

in the sequence and every time you see

something appear that's a new

token um so let's just look at this and

you can see that this is not yet very

coherent and keep in mind that this is

only 1% of the way through training and

so the model is not yet very good at

predicting the next token in the

sequence so what comes out is actually

kind of a little bit of gibberish right

but it still has a little bit of like

local coherence so since she is mine

it's a part of the information should

discuss my father great companions

Gordon showed me sitting over at and Etc

so I know it doesn't look very good but

let's actually scroll up and see what it

looked like when I started the

optimization so all the way here at

step

one so after 20 steps of optimization

you see that what we're getting here is

looks completely random and of course

that's because the model has only had 20

updates to its parameters and so it's

giving you random text because it's a

random Network and so you can see that

at least in comparison to this model is

starting to do much better and indeed if

we waited the entire 32,000 steps the

model will have improved the point that

it's actually uh generating fairly

coherent English uh and the tokens

stream correctly um and uh they they

kind of make up English a a lot

better

um so this has to run for about a day or

two more now and so uh at this stage we

just make sure that the loss is

decreasing everything is looking good um

and we just have to wait

and now um let me turn now to the um

story of the computation that's required

because of course I'm not running this

optimization on my laptop that would be

way too expensive uh because we have to

run this neural network and we have to

improve it and we have we need all this

data and so on so you can't run this too

well on your computer uh because the

network is just too large uh so all of

this is running on the computer that is

out there in the cloud and I want to

basically address the compute side of

the store of training these models and

what that looks like so let's take a

look okay so the computer that I'm

running this optimization on is this 8X

h100 node so there are eight h100s in a

single node or a single computer now I

am renting this computer and it is

somewhere in the cloud I'm not sure

where it is physically actually the

place I like to rent from is called

Lambda but there are many other

companies who provide this service so

when you scroll down you can see that uh

they have some on demand pricing for

um sort of computers that have these uh

h100s which are gpus and I'm going to

show you what they look like in a second

but on demand 8times Nvidia h100 uh

GPU this machine comes for $3 per GPU

per hour for example so you can rent

these and then you get a machine in a

cloud and you can uh go in and you can

train these

models and these uh gpus they look like

this so this is one h100 GPU uh this is

kind of what it looks like and you slot

this into your computer and gpus are

this uh perfect fit for training your

networks because they are very

computationally expensive but they

display a lot of parallelism in the

computation so you can have many

independent workers kind of um working

all at the same time in solving uh the

matrix multiplication that's under the

hood of training these neural

networks so this is just one of these

h100s but actually you would put them

you would put multiple of them together

so you could stack eight of them into a

single node and then you can stack

multiple nodes into an entire data

center or an entire system

so when we look at a data

center can't spell when we look at a

data center we start to see things that

look like this right so we have one GPU

goes to eight gpus goes to a single

system goes to many systems and so these

are the bigger data centers and there of

course would be much much more expensive

um and what's happening is that all the

big tech companies really desire these

gpus so they can train all these

language models because they are so

powerful and that has is fundamentally

what has driven the stock price of

Nvidia to be $3.4 trillion today as an

example and why Nvidia has kind of

exploded so this is the Gold Rush the

Gold Rush is getting the gpus getting

enough of them so they can all

collaborate to perform this optimization

and they're what are they all doing

they're all collaborating to predict the

next token on a data set like the fine

web data

set this is the computational workflow

that that basically is extremely

expensive the more gpus you have the

more tokens you can try to predict and

improve on and you're going to process

this data set faster and you can iterate

faster and get a bigger Network and

train a bigger Network and so on so this

is what all those machines are look like

are uh are doing and this is why all of

this is such a big deal and for example

this is a

article from like about a month ago or

so this is why it's a big deal that for

example Elon Musk is getting 100,000

gpus uh in a single Data Center and all

of these gpus are extremely expensive

are going to take a ton of power and all

of them are just trying to predict the

next token in the sequence and improve

the network uh by doing so and uh get

probably a lot more coherent text than

what we're seeing here a lot faster okay

so unfortunately I do not have a couple

10 or hundred million of dollars to

spend on training a really big model

like this but luckily we can turn to

some big tech companies who train these

models routinely and release some of

them once they are done training so

they've spent a huge amount of compute

to train this network and they release

the network at the end of the

optimization so it's very useful because

they've done a lot of compute for that

so there are many companies who train

these models routinely but actually not

many of them release uh these what's

called base models so the model that

comes out at the end here is is what's

called a base model what is a base model

it's a token simulator right it's an

internet text token simulator and so

that is not by itself useful yet because

what we want is what's called an

assistant we want to ask questions and

have it respond to answers these models

won't do that they just uh create sort

of remixes of the internet they dream

internet pages so the base models are

not very often released because they're

kind of just only a step one of a few

other steps that we still need to take

to get in system

however a few releases have been made so

as an example the gbt2 model released

the 1.6 billion sorry 1.5 billion model

back in 2019 and this gpt2 model is a

base model now what is a model release

what does it look like to release these

models so this is the gpt2 repository on

GitHub well you need two things

basically to release model number one we

need the um python code usually that

describes the sequence of operations in

detail that they make in their model so

um if you remember

back this

Transformer the sequence of steps that

are taken here in this neural network is

what is being described by this code so

this code is sort of implementing the

what's called forward pass of this

neural network so we need the specific

details of exactly how they wired up

that neural network so this is just

computer code and it's usually just a

couple hundred lines of code it's not

it's not that crazy and uh this is all

fairly understandable and usually fairly

standard what's not standard are the

parameters that's where the actual value

is what are the parameters of this

neural network because there's 1.6

billion of them and we need the correct

setting or a really good setting and so

that's why in addition to this source

code they release the parameters which

in this case is roughly 1.5 billion

parameters and these are just numbers so

it's one single list of 1.5 billion

numbers the precise and good setting of

all the knobs such that the tokens come

out

well so uh you need those two things to

get a base model

release

now gpt2 was released but that's

actually a fairly old model as I

mentioned so actually the model we're

going to turn to is called llama 3 and

that's the one that I would like to show

you next so llama 3 so gpt2 again was

1.6 billion parameters trained on 100

billion tokens Lama 3 is a much bigger

model and much more modern model it is

released and trained by meta and it is a

45 billion parameter model trained on 15

trillion tokens in very much the same

way just much much

bigger um and meta has also made a

release of llama 3 and that was part of

this

paper so with this paper that goes into

a lot of detail the biggest base model

that they released is the Lama 3.1 4.5

405 billion parameter model so this is

the base model and then in addition to

the base model you see here

foreshadowing for later sections of the

video they also released the instruct

model and the instruct means that this

is an assistant you can ask it questions

and it will give you answers we still

have yet to cover that part later for

now let's just look at this base model

this token simulator and let's play with

it and try to think about you know what

is this thing and how does it work and

um what do we get at the end of this

optimization if you let this run Until

the End uh for a very big neural network

on a lot of data so my favorite place to

interact with the base models is this um

company called hyperbolic which is

basically serving the base model of the

405b Llama 3.1 so when you go to the

website and I think you may have to

register and so on make sure that in the

models make sure that you are using

llama 3.1 405 billion base it must be

the base model and then here let's say

the max tokens is how many tokens we're

going to be gener rating so let's just

decrease this to be a bit less just so

we don't waste compute we just want the

next 128 tokens and leave the other

stuff alone I'm not going to go into the

full detail here um now fundamentally

what's going to happen here is identical

to what happens here during inference

for us so this is just going to continue

the token sequence of whatever you

prefix you're going to give it so I want

to first show you that this model here

is not yet an assistant so you can for

example ask it what is 2 plus 2 it's not

going to tell you oh it's four uh what

else can I help you with it's not going

to do that because what is 2 plus 2 is

going to be tokenized and then those

tokens just act as a prefix and then

what the model is going to do now is

just going to get the probability for

the next token and it's just a glorified

autocomplete it's a very very expensive

autocomplete of what comes next um

depending on the statistics of what it

saw in its training documents which are

basically web

pages so let's just uh hit enter to see

what tokens it comes up with as a

continuation okay so here it kind of

actually answered the question and

started to go off into some

philosophical territory uh let's try it

again so let me copy and paste and let's

try again from scratch what is 2 plus

two so okay so it just goes off again so

notice one more thing that I want to

stress is that the system uh I think

every time you put it in it just kind of

starts from scratch

so it doesn't uh the system here is

stochastic so for the same prefix of

tokens we're always getting a different

answer and the reason for that is that

we get this probity distribution and we

sample from it and we always get

different samples and we sort of always

go into a different territory uh

afterwards so here in this case um I

don't know what this is let's try one

more

time so it just continues on so it's

just doing the stuff that it's saw on

the internet right um and it's just kind

of like regurgitating those uh

statistical

patterns so first things it's not an

assistant yet it's a token autocomplete

and second it is a stochastic system now

the crucial thing is that even though

this model is not yet by itself very

useful for a lot of applications just

yet um it is still very useful because

in the task of predicting the next token

in the sequence the model has learned a

lot about the world and it has stored

all that knowledge in the parameters of

the network so remember that our text

looked like this right internet web

pages and now all of this is sort of

compressed in the weights of the network

so you can think of um these 405 billion

parameters is a kind of compression of

the internet you can think of the

45 billion parameters is kind of like a

zip file uh but it's not a loss less

compression it's a loss C compression

we're kind of like left with kind of a

gal of the internet and we can generate

from it right now we can elicit some of

this knowledge by prompting the base

model uh accordingly so for example

here's a prompt that might work to

elicit some of that knowledge that's

hiding in the parameters here's my top

10 list of the top landmarks to see in

the

pairs

um and I'm doing it this way because I'm

trying to Prime the model to now

continue this list so let's see if that

works when I press

enter okay so you see that it started a

list and it's now kind of giving me some

of those

landmarks and now notice that it's

trying to give a lot of information here

now you might not be able to actually

fully trust some of the information here

remember that this is all just a

recollection of some of the internet

documents and so the things that occur

very frequently in the internet data are

probably more likely to be remembered

correctly compared to things that happen

very infrequently so you can't fully

trust some of the things that and some

of the information that is here because

it's all just a vague recollection of

Internet documents because the

information is not stored explicitly in

any of the parameters it's all just the

recollection that said we did get

something that is probably approximately

correct and I don't actually have the

expertise to verify that this is roughly

correct but you see that we've elicited

a lot of the knowledge of the model and

this knowledge is not precise and exact

this knowledge is vague and

probabilistic and statistical and the

kinds of things that occur often are the

kinds of things that are more likely to

be remembered um in the model now I want

to show you a few more examples of this

model's Behavior the first thing I want

to show you is this example I went to

the Wikipedia page for zebra and let me

just copy paste the first uh even one

sentence

here and let me put it here now when I

click enter what kind of uh completion

are we going to get so let me just hit

enter there are three living species

etc etc what the model is producing here

is an exact regurgitation of this

Wikipedia entry it is reciting this

Wikipedia entry purely from memory and

this memory is stored in its parameters

and so it is possible that at some point

in these 512 tokens the model will uh

stray away from the Wikipedia entry but

you can see that it has huge chunks of

it memorized here uh let me see for

example if this sentence

occurs by now okay so this so we're

still on track let me check

here okay we're still on

track it will eventually uh stray

away okay so this thing is just recited

to a very large extent it will

eventually deviate uh because it won't

be able to remember exactly now the

reason that this happens is because

these models can be extremely good at

memorization and usually this is not

what you want in the final model and

this is something called regurgitation

and it's usually undesirable to site uh

things uh directly uh that you have

trained on now the reason that this

happens actually is because for a lot of

documents like for example Wikipedia

when these documents are deemed to be of

very high quality as a source like for

example Wikipedia it is very often uh

the case that when you train the model

you will preferentially sample from

those sources so basically the model has

probably done a few epochs on this data

meaning that it has seen this web page

like maybe probably 10 times or so and

it's a bit like you like when you read

some kind of a text many many times say

you read something a 100 times uh then

you'll be able to recite it and it's

very similar for this model if it sees

something way too often it's going to be

able to recite it later from memory

except these models can be a lot more

efficient um like per presentation than

human so probably it's only seen this

Wikipedia entry 10 times but basically

it has remembered this article exactly

in its parameters okay the next thing I

want to show you is something that the

model has definitely not seen during its

training so for example if we go to the

paper uh and then we navigate to the

pre-training data we'll see here that uh

the data set has a knowledge cut off

until the end of 2023 so it will not

have seen documents after this point and

certainly it has not seen anything about

the 2024 election and how it turned out

now if we Prime the model with the

tokens from the future it will continue

the token sequence and it will just take

its best guess according to the

knowledge that it has in its own

parameters so let's take a look at what

that could look like

so the Republican Party kit

Trump okay president of the United

States from

2017 and let's see what it says after

this point so for example the model will

have to guess at the running mate and

who it's against Etc so let's hit

enter so here thingss that Mike Pence

was the running mate instead of JD Vance

and the ticket was against Hillary

Clinton and Tim Kane so this is kind of

a interesting parallel universe

potentially of what could have happened

happened according to the LM let's get a

different sample so the identical prompt

and let's

resample so here the running mate was

Ronda santis and they ran against Joe

Biden and Camala Harris so this is again

a different parallel universe so the

model will take educated guesses and it

will continue the token sequence based

on this knowledge um and it will just

kind of like all of what we're seeing

here is what's called hallucination the

model is just taking its best guess uh

in a probalistic manner the next thing I

would like to show you is that even

though this is a base model and not yet

an assistant model it can still be

utilized in Practical applications if

you are clever with your prompt design

so here's something that we would call a

few shot

prompt so what it is here is that I have

10 words or 10 pairs and each pair is a

word of English column and then a the

translation in Korean and we have 10 of

them and what the model does here is at

the end we have teacher column and then

here's where we're going to do a

completion of say just five tokens and

these models have what we call in

context learning abilities and what

that's referring to is that as it is

reading this context it is learning sort

of in

place that there's some kind of a

algorithmic pattern going on in my data

and it knows to continue that pattern

and this is called kind of like Inc

context learning so it takes on the role

of a

translator and when we hit uh completion

we see that the teacher translation is

Sim which is correct um and so this is

how you can build apps by being clever

with your prompting even though we still

just have a base model for now and it

relies on what we call this um uh in

context learning ability and it is done

by constructing what's called a few shot

prompt okay and finally I want to show

you that there is a clever way to

actually instantiate a whole language

model assistant just by prompting and

the trick to it is that we're structure

a prompt to look like a web page that is

a conversation between a helpful AI

assistant and a human and then the model

will continue that conversation so

actually to write the prompt I turned to

chat gbt itself which is kind of meta

but I told it I want to create an llm

assistant but all I have is the base

model so can you please write my um uh

prompt and this is what it came up with

which is actually quite good so here's a

conversation between an AI assistant and

a human

the AI assistant is knowledgeable

helpful capable of answering wide

variety of questions Etc and then here

it's not enough to just give it a sort

of description it works much better if

you create this fot prompt so here's a

few terms of human assistant human

assistant and we have uh you know a few

turns of conversation and then here at

the end is we're going to be putting the

actual query that we like so let me copy

paste this into the base model prompt

and now let me do human column and this

is where we put our actual prompt why is

the sky

blue and uh let's uh

run assistant the sky appears blue due

to the phenomenon called R lights

scattering etc etc so you see that the

base model is just continuing the

sequence but because the sequence looks

like this conversation it takes on that

role but it is a little subtle because

here it just uh you know it ends the

assistant and then just you know

hallucinate Ates the next question by

the human Etc so it'll just continue

going on and on uh but you can see that

we have sort of accomplished the task

and if you just took this why is the sky

blue and if we just refresh this and put

it here then of course we don't expect

this to work with a base model right

we're just going to who knows what we're

going to get okay we're just going to

get more

questions okay so this is one way to

create an assistant even though you may

only have a base model okay so this is

the kind of brief summary of the things

we talked about over the last few

minutes now let me zoom out

here and this is kind of like what we've

talked about so far we wish to train LM

assistants like chpt we've discussed the

first stage of that which is the

pre-training stage and we saw that

really what it comes down to is we take

Internet documents we break them up into

these tokens these atoms of little text

chunks and then we predict token

sequences using neural networks the

output of this entire stage is this base

model it is the setting of The

parameters of this network and this base

model is basically an internet document

simulator on the token level so it can

just uh it can generate token sequences

that have the same kind of like

statistics as Internet documents and we

saw that we can use it in some

applications but we actually need to do

better we want an assistant we want to

be able to ask questions and we want the

model to give us answers and so we need

to now go into the second stage which is

called the post-training stage so we

take our base model our internet

document simulator and hand it off to

post training so we're now going to

discuss a few ways to do what's called

post training of these models these

stages in post training are going to be

computationally much less expensive most

of the computational work all of the

massive data centers um and all of the

sort of heavy compute and millions of

dollars are the pre-training stage but

now we go into the slightly cheaper but

still extremely important stage called

post trining where we turn this llm

model into an assistant so let's take a

look at how we can get our model to not

sample internet documents but to give

answers to questions so in other words

what we want to do is we want to start

thinking about conversations and these

are conversations that can be multi-turn

so so uh there can be multiple turns and

they are in the simplest case a

conversation between a human and an

assistant and so for example we can

imagine the conversation could look

something like this when a human says

what is 2 plus2 the assistant should re

respond with something like 2 plus 2 is

4 when a human follows up and says what

if it was star instead of a plus

assistant could respond with something

like

this um and similar here this is another

example showing that the assistant could

also have some kind of a personality

here uh that it's kind of like nice and

then here in the third example I'm

showing that when a human is asking for

something that we uh don't wish to help

with we can produce what's called

refusal we can say that we cannot help

with that so in other words what we want

to do now is we want to think through

how in a system should interact with the

human and we want to program the

assistant and Its Behavior in these

conversations now because this is neural

networks we're not going to be

programming these explicitly in code

we're not going to be able to program

the assistant in that way because this

is neural networks everything is done

through neural network training on data

sets and so because of that we are going

to be implicitly programming the

assistant by creating data sets of

conversations so these are three

independent examples of conversations in

a data dat set an actual data set and

I'm going to show you examples will be

much larger it could have hundreds of

thousands of conversations that are

multi- turn very long Etc and would

cover a diverse breath of topics but

here I'm only showing three examples but

the way this works basically is uh a

assistant is being programmed by example

and where is this data coming from like

2 * 2al 4 same as 2 plus 2 Etc where

does that come from this comes from

Human labelers so we will basically give

human labelers some conversational

context and we will ask them to um

basically give the ideal assistant

response in this situation and a human

will write out the ideal response for an

assistant in any situation and then

we're going to get the model to

basically train on this and to imitate

those kinds of

responses so the way this works then is

we are going to take our base model

which we produced in the preing stage

and this base model was trained on

internet documents we're now going to

take that data set of internet documents

and we're gonna throw it out and we're

going to substitute a new data set and

that's going to be a data set of

conversations and we're going to

continue training the model on these

conversations on this new data set of

conversations and what happens is that

the model will very rapidly adjust and

will sort of like learn the statistics

of how this assistant responds to human

queries and then later during inference

we'll be able to basically um Prime the

assistant and get the response and it

will be imitating what the humans will

human labelers would do in that

situation if that makes sense so we're

going to see examples of that and this

is going to become bit more concrete I

also wanted to mention that this

post-training stage we're going to

basically just continue training the

model but um the pre-training stage can

in practice take roughly three months of

training on many thousands of computers

the post-training stage will typically

be much shorter like 3 hours for example

um and that's because the data set of

conversations that we're going to create

here manually is much much smaller than

the data set of text on the internet and

so this training will be very short but

fundamentally we're just going to take

our base model we're going to continue

training using the exact same algorithm

the exact same everything except we're

swapping out the data set for

conversations so the questions now are

what are these conversations how do we

represent them how do we get the model

to see conversations instead of just raw

text and then what are the outcomes of

um this kind of training and what do you

get in a certain like psychological

sense uh when we talk about the model so

let's turn to those questions now so

let's start by talking about the

tokenization of conversations everything

in these models has to be turned into

tokens because everything is just about

token sequences so how do we turn

conversations into token sequences is

the question and so for that we need to

design some kind of ending coding and uh

this is kind of similar to maybe if

you're familiar you don't have to be

with for example the TCP IP packet in um

on the internet there are precise rules

and protocols for how you represent

information how everything is structured

together so that you have all this kind

of data laid out in a way that is

written out on a paper and that everyone

can agree on and so it's the same thing

now happening in llms we need some kind

of data structures and we need to have

some rules around how these data

structures like conversations get

encoded and decoded to and from tokens

and so I want to show you now how I

would

recreate uh this conversation in the

token space so if you go to Tech

tokenizer

I can take that conversation and this is

how it is represented in uh for the

language model so here we have we are

iterating a user and an assistant in

this two- turn

conversation and what you're seeing here

is it looks ugly but it's actually

relatively simple the way it gets turned

into a token sequence here at the end is

a little bit complicated but at the end

this conversation between a user and

assistant ends up being 49 tokens it is

a one-dimensional sequence of 49 tokens

and these are the tokens

okay and all the different llms will

have a slightly different format or

protocols and it's a little bit of a

wild west right now but for example GPT

40 does it in the following way you have

this special token called imore start

and this is short for IM imaginary

monologue uh the

start then you have to specify um I

don't actually know why it's called that

to be honest then you have to specify

whose turn it is so for example user

which is a token 4

28 then you have internal monologue

separator and then it's the exact

question so the tokens of the question

and then you have to close it so I am

end the end of the imaginary monologue

so

basically the question from a user of

what is 2 plus two ends up being the

token sequence of these tokens and now

the important thing to mention here is

that IM start this is not text right IM

start is a special token that gets added

it's a new token and um this token has

never been trained on so far it is a new

token that we create in a post-training

stage and we introduce and so these

special tokens like IM seep IM start Etc

are introduced and interspersed with

text so that they sort of um get the

model to learn that hey this is a the

start of a turn for who is it start of

the turn for the start of the turn is

for the user and then this is what the

user says and then the user ends and

then it's a new start of a turn and it

is by the assistant and then what does

the assistant say well these are the

tokens of what the assistant says Etc

and so this conversation is not turned

into the sequence of tokens the specific

details here are not actually that

important all I'm trying to show you in

concrete terms is that our conversations

which we think of as kind of like a

structured object end up being turned

via some encoding into onedimensional

sequences of tokens and so because this

is one dimensional sequence of tokens we

can apply all the stuff that we applied

before now it's just a sequence of

tokens and now we can train a language

model on it and so we're just predicting

the next token in a sequence uh just

like before and um we can represent and

train on conversations and then what

does it look like at test time during

inference so say we've trained a model

and we've trained a model on these kinds

of data sets of conversations and now we

want to

inference so during inference what does

this look like when you're on on chash

apt well you come to chash apt and you

have say like a dialogue with it and the

way this works is

basically um say that this was already

filled in so like what is 2 plus 2 2

plus 2 is four and now you issue what if

it was times I am end and what basically

ends up happening um on the servers of

open AI or something like that is they

put in I start assistant I amep and this

is where they end it right here so they

construct this context and now they

start sampling from the model so it's at

this stage that they will go to the

model and say okay what is a good for

sequence what is a good first token what

is a good second token what is a good

third token and this is where the LM

takes over and creates a response like

for example response that looks

something like this but it doesn't have

to be identical to this but it will have

the flavor of this if this kind of a

conversation was in the data set so um

that's roughly how the protocol Works

although the details of this protocol

are not important so again my goal is

that just to show you that everything

ends up being just a one-dimensional

token sequence so we can apply

everything we've already seen but we're

now training on conversations and we're

now uh basically generating

conversations as well okay so now I

would like to turn to what these data

sets look like in practice the first

paper that I would like to show you and

the first effort in this direction is

this paper from openai in 2022 and this

paper was called instruct GPT or the

technique that they developed and this

was the first time that opena has kind

of talked about how you can take

language models and fine-tune them on

conversations and so this paper has a

number of details that I would like to

take you through so the first stop I

would like to make is in section 3.4

where they talk about the human

contractors that they hired uh in this

case from upwork or through scale AI to

uh construct these conversations and so

there are human labelers involved whose

job it is professionally to create these

conversations and these labelers are

asked to come up with prompts and then

they are asked to also complete the

ideal assistant responses and so these

are the kinds of prompts that people

came up with so these are human labelers

so list five ideas for how to regain

enthusiasm for my career what are the

top 10 science fiction books I should

read next and there's many different

types of uh kind of prompts here so

translate this sentence from uh to

Spanish Etc and so there's many things

here that people came up with they first

come up with the prompt and then they

also uh answer that prompt and they give

the ideal assistant response now how do

they know what is the ideal assistant

response that they should write for

these prompts so when we scroll down a

little bit further we see that here we

have this excerpt of labeling

instructions uh that are given to the

human labelers so the company that is

developing the language model like for

example open AI writes up labeling

instructions for how the humans should

create ideal responses and so here for

example is an excerpt uh of these kinds

of labeling instruction instructions on

High level you're asking people to be

helpful truthful and harmless and you

can pause the video if you'd like to see

more here but on a high level basically

just just answer try to be helpful try

to be truthful and don't answer

questions that we don't want um kind of

the system to handle uh later in chat

gbt and so roughly speaking the company

comes up with the labeling instructions

usually they are not this short usually

there are hundreds of pages and people

have to study them professionally and

then they write out the ideal assistant

responses uh following those labeling

instructions so this is a very human

heavy process as it was described in

this paper now the data set for instruct

GPT was never actually released by openi

but we do have some open- Source um

reproductions that were're trying to

follow this kind of a setup and collect

their own data so one that I'm familiar

with for example is the effort of open

Assistant from a while back and this is

just one of I think many examples but I

just want to show you an example so

here's so these were people on the

internet that were asked to basically

create these conversations similar to

what um open I did with human labelers

and so here's an entry of a person who

came up with this BR can you write a

short introduction to the relevance of

the term

manop uh in economics please use

examples Etc and then the same person or

potentially a different person will

write up the response so here's the

assistant response to this and so then

the same person or different person will

actually write out this ideal

response and then this is an example of

maybe how the conversation could

continue now explain it to a dog and

then you can try to come up with a

slightly a simpler explanation or

something like that now this then

becomes the label and we end up training

on this so what happens during training

is that um of course we're not going to

have a full coverage of all the possible

questions that um the model will

encounter at test time during inference

we can't possibly cover all the possible

prompts that people are going to be

asking in the future but if we have a

like a data set of a few of these

examples then the model during training

will start to take on this Persona of

this helpful truthful harmless assistant

and it's all programmed by example and

so these are all examples of behavior

and if you have conversations of these

example behaviors and you have enough of