hi everyone so today we are going to be

continuing our Zero to Hero series and

in particular today we are going to

reproduce the gpt2 model the 124 million

version of it so when openi released

gpt2 this was 2019 and they released it

with this blog post on top of that they

released this paper and on top of that

they released this code on GitHub so

open a/

gpt2 now when we talk about reproducing

gpt2 we have to be careful because in

particular in this video we're going to

be reproducing the 124 million parameter

model so the thing to realize is that

there's always a miniseries when these

are releases are made so there are the

gpt2 miniseries made up of models at

different sizes and usually the biggest

model is called the

gpt2 but basically the reason we do that

is because you can put the model sizes

on the x-axis of plots like this and on

the Y AIS you put a lot of uh Downstream

metrics that you're interested in like

translation summarization question

answering and so on and you can chart

out these scaling laws so basically as

the model size increases you're getting

better and better at Downstream metrics

and so in particular for

gpt2 if we scroll down in paper there

are four models in the gpt2 miniseries

starting at 124 million all the way up

to 1558 million now the reason my

numbers the way I say them disagree with

this table is that this table is wrong

if you actually go to the uh gpt2 uh

GitHub repo they sort of say that um

there was an error in how they added up

the parameters but basically this is the

124 million parameter model Etc so the

124 million parameter had 12 layers in

the Transformer and it had 768 channels

in the Transformer 768 dimensions and

I'm going to be assuming some

familiarity with what these terms mean

because I covered all of this in my

previous video let's build gpt2 uh let's

build GPT from scratch so I covered that

in the previous video in this playlist

now if we do everything correctly and

everything works out well by the end of

this video we're going to see something

like this where we're looking at the

validation loss which basically um

measures how good we are at predicting

the next token in a sequence on some

validation data that the model has not

seen during training and we see that we

go from doing that task not very well

because we're initializing from scratch

all the way to doing that task quite

well um by the end of the training and

hopefully we're going to beat the gpt2

uh 124 M model

now previously when they were working on

this this is already 5 years ago so this

was probably a fairly complicated

optimization at the time and the gpus

and the compute was a lot smaller today

you can reproduce this model in roughly

an hour or probably less even and it

will cost you about 10 bucks if you want

to do this on the cloud uh Cloud Compu a

sort of computer that you can all rent

and if you pay $10 for that computer you

wait about an hour or less you can

actually achieve a model that is as good

as this model that open ey released and

uh one more thing to mention is unlike

many other models open ey did release

the weights for gpt2 so those weights

are all available in this repository but

the gpt2 paper is not always as good

with all of the details of training so

in addition to the gpt2 paper we're

going to be referencing the gpt3 paper

which is a lot more Concrete in a lot of

the hyp parameters and optimization

settings and so on um and it's not a

huge departure in the architecture from

the GPT 2 uh version of the model so

we're going to be referencing both gpt2

and gpt3 as we try to reproduce gpt2 124

M uh so let's go so the first thing I

would like to do is actually start at

the end or at the Target so in other

words let's load the GPT to 124 M model

as it was released by openi and maybe

take it for a spin let's sample some

tokens from it now the issue with that

is when you go into the code base of

gpt2 and you go into the source and you

click in on the model. pi you'll realize

that actually this is using tensorflow

so the original gpt2 code here was

written in tensor flow which is

um you know not let's just say not used

as much anymore um so we'd like to use

pytorch uh because it's a lot friendlier

easier and I just personally like a lot

more the problem with that is the

initial code is intenser flow we'd like

to use pytorch so instead uh to get the

target we're going to use the hugging

face Transformers um code which I like a

lot more so when you go into the

Transformers source Transformers models

gpt2 modeling gpt2 Pi you will see that

they have the gpt2 implementation of

that Transformer here in this

file um and it's like medium readable

but not fully readable um but what it

does is it did all the work of

converting all those weights uh from

tensor flow to pytorch Friendly and so

it's much easier to load and work with

so in particular we can look at the

gpt2 um model here and we can load it

using hugging face Transformers so

swinging over this is what that looks

like from Transformers import the DP GT2

LM head model and then from pre-train

gpt2 uh now one awkward thing about this

is that when you do gpt2 as the model

that we're loading this actually is the

124 million parameter model if you want

the actual the gpt2 the 1.5 billion then

you actually want to do- XL so this is

the 12 4 M our Target now what we're

doing is when we actually get this we're

initializing the uh pytorch NN module as

defined here in this

class from it I want to get just the

state dict which is just a raw tensors

so we just have um the tensors of that

file and by the way here this is a

jupyter notebook uh but this is jupyter

notebook running inside vs code uh so I

like to work with it all in a single

sort of interface so I like to use vs

code so this is the jupyter notebook

extension inside the es

code so when we get the state dick this

is just a dict so we can print the key

and the value which is the tensor and

let's just look at the shapes so these

are sort of

the uh different parameters inside the

gbt2 model and their shape so the W

weight for token

embedding is of size

50257 by 768 where this is coming from

is that we have

50257 tokens in the gpt2 vocabulary um

and the tokens by the way these are

exactly the tokens that we spoken about

in the previous video on my tokenization

Series so the previous videos just

before this I go into a ton of detail on

tokenization gpt2 tokenizer happens to

have this many tokens for each

token we have a 768 dimensional

embedding that is the distributed

representation that stands in for that

token so each token is a little string

piece and then the 768 numbers are the

vector that represents that

token and so this is just our lookup

table for tokens and then here we have

the lookup table for the positions so

because gbt2 has a maximum sequence

length of

1024 we have up to 1,24 positions that

each token can be attending to in the

past and every one of those positions in

gpd2 has a fixed Vector of

768 that is learned by

optimization um and so this is the

position embedding and the token

embedding um and then everything here is

just the other weights and biases and

everything else of this

Transformer so when you just take for

example the positional embeddings and

flatten it out and take just the 20

elements you can see that these are just

the parameters these are weights floats

just we can take and we can plot them so

these are the position embeddings and we

get something like this and you can see

that this has structure and it has

structure because what we what we have

here really is every Row in this

visualization is a different position a

fixed absolute position in um the range

from 0 to

1024 and each row here is the

representation of that position and so

it has structure because these

positional embeddings end up learning

these sinusoids and cosiness um that

sort of like represent each of these

positions and uh each row here stands in

for that position and is processed by

the Transformer to recover all the

relative positions and uh sort of

realize which token is where and um

attend to them depending on their

position not just their

content so when we actually just look

into an individual column inside these

and I just grabbed three random columns

you'll see that for example here we are

focusing on every every single um

Channel and we're looking

at what that channel is doing as a

function of uh position from one from Z

to

1223

really and we can see that some of these

channels basically like respond more or

less to different parts of the position

Spectrum so this green channel uh really

likes to fire for everything after 200

uh up to 800 but not less a lot less and

has a sharp drop off here near zero so

who knows what these embeddings are

doing and why they are the way they are

you can tell for example that because

they're a bit more Jagged and they're

kind of noisy you can tell that this

model was not fully trained and the more

trained this model was the more you

would expect to smooth this out and so

this is telling you that this is a

little bit of an undertrained model um

but in principle actually these curves

don't even have to be smooth this should

just be totally random noise and in fact

in the beginning of the optimization it

is complete random noise because this

position embedding table is initialized

completely at random so in the beginning

you have jaggedness and the fact that

you end up with something smooth is

already kind of impressive um that that

just falls out of the optimization

because in principle you shouldn't even

be able to get any single graph out of

this that makes sense but we actually

get something that looks a little bit

noisy but for the most part looks

sinusoidal like um in the original

Transformer um in the original

Transformer paper the attention is all

you need paper the positional embeddings

are actually initialized and fixed if I

remember correctly to sinusoids and

cosiness of uh different frequencies and

that's the positional coding and it's

fixed but in gpt2 these are just

parameters and they're trained from

scratch just like any other parameter uh

and that seems to work about as well and

so what they do is they kind of like

recover these sinusoidal like features

during the

optimization we can also look at any of

the other matrices here so here I took

the first layer of the

Transformer and looking at like one of

its weights and just the first block of

300 by 300 and you see some structure

but like again like who knows what any

of this is if you're into mechanistic

interpretability you might get a real

kick out of trying to figure out like

what is going on what is this structure

and what does this all mean but we're

not going to be doing that in this video

but we definitely see that there's some

interesting structure and that's kind of

cool what we're mostly interested in is

we've loaded the weights of this model

that was released by open Ai and now

using the hogging face Transformers we

can not just get all the raw weights but

we can also get the um what they call

Pipeline and sample from it so this is

the prefix hello I'm a language model

comma and then we're sampling uh 30

tokens and we getting five sequences and

I ran this and this is what it produced

um hell language

model but what I'm really doing is

making a human readable document there

are other languages but those are dot

dot dot so you can read through these if

you like but basically these are five

different completions of the same prefix

from this uh gbt

2124m now uh if I go here I took this

example from here and sadly even though

we are fixing the seed we are getting

different Generations from the snippet

than what they got so presumably the

code changed um but what we see though

at this stage that's important is that

we are getting coherent text so we've

loaded the model successfully we can

look at all its parameters and the keys

tell us where in the model these come

from and we want to actually write our

own gpt2 class so that we have full

understanding of what's happening there

we don't want to be working with

something like uh the modeling gpt2 Pi

because it's just too complicated we

want to write this from scratch

ourselves so we're going to be

implementing the GPT model here in

parallel and as our first task let's

load the gpt2 124 M into the class that

we're going to develop here from scratch

that's going to give us confidence that

we can load the open ey model and

therefore there's a setting of Weights

that exactly is the 124 model but then

of course what we're going to do is

we're going to initialize the model from

scratch instead and try try to train it

ourselves um on a bunch of documents

that we're going to get and we're going

to try to surpass that model so we're

going to get different weights and

everything's going to look different

hopefully better even um

but uh we're going to have a lot of

confidence that because we can load the

openi model we are in the same model

family and model class and we just have

to ReDiscover a good setting of the

weights uh but from scratch so let's now

write the gbt2 model and let's load the

weights and make sure that we can also

generate text that looks coherent okay

so let's now swing over to the attention

is all un need paper that started

everything and let's scroll over to the

model architecture the original

Transformer now remember that gpt2 is

slightly modified from the or or

Transformer in particular we do not have

uh the encoder gpt2 is a decoder only

Transformer as we call it so this entire

encoder here is missing in addition to

that this cross attention here that was

using that encoder is also missing so we

delete this entire part everything else

stays almost the same but there are some

differences that we're going to uh sort

of look at here so there are two main

differences when we go to the gb2 page

under 2.3 model we notice that first

there's a reshuffling of the layer Norms

so they change place and second an

additional layer normalization was added

here to the final self detention block

so basically all the layer Norms here

instead of being after the MLP or after

the attention they SN before it and an

additional layer Norm gets added here

right before the final

classifier so now let's Implement some

of the first sort of skeleton NN module

modules here in our GPT NN module and in

particular we're going to try to match

up this schema here that is used by

hugging face Transformers because that

will make it much easier to load these

weights from this state dict so we want

something that reflects uh this schema

here so here's what I came up with

um basically we see that the main

container here that has all the modules

is called Transformer so I'm reflecting

that with an NN module dict and this is

basically a module that allows you to

index into the subm modules using keys

just like a dictionary uh

strings within it we have the weights of

the token embeddings WT and that's an N

embedding and the weights of the

position embeddings which is also just

an N embedding and if you remember n

embedding is really just a fancy little

wrapper module around just a single um

single array of numbers a single uh

block of numbers just like this it's a

single tensor and an embedding is a

glorified um wrapper around a tensor

that allows you to access its elements

uh by indexing into the

rows now in addition to that we see here

that we have a h and then there's a this

is index using numbers instead of

indexed using strings so there's a h. 0

1 2 Etc all the way up till h. 11 and

that's because there are 12 layers here

in this Transformer so to reflect that

I'm creating also an H I think that

probably stands for hidden and instead

of a module dict this is a model list so

we can index it using integers exactly

as we see here 01 2 Etc and the modular

list has a n layer blocks and the blocks

are yet to be defined in a module in a

bit in addition to that following the

gpt2 paper we have we need an additional

final layer Norm that we're going to put

in there and then we have the final

classifier uh the language model head

which um projects from 768 the number of

embedding dimensions in this GPT all the

way to the vocab size which is

50257 and gpt2 uses no bias for this

final uh sort of projection so this is

the skeleton and you can see that it

reflects this so the wte is the token

embeddings here it's called output

embedding but it's really the token

embeddings the PE is the positional

codings uh those two pieces of

information as we saw previously are

going to add and then go into the

Transformer the H is the all the blocks

in Gray and the LNF is this new layer

that gets added here by the gpt2 model

and LM head is this linear part here so

that's the skeleton of the gpt2 we now

have to implement the block okay so

let's now recurse to the block itself so

we want to define the block um so I'll

start putting them here so the block I

like to write out like

this uh these are some of the

initializations and then this is the

actual forward pass of what this block

computes and notice here that there's a

change from the Transformer again that

is mentioned in the gpt2 paper so here

the layer normalizations are after the

application of attention or feed forward

in addition to that note that the

normalizations are inside the residual

stream you see how feed forward is

applied and this arrow goes through and

through the normalization so that means

that your residual pathway has

normalizations inside them and this is

not very good or desirable uh you

actually prefer to have a single uh

clean residual stream all the way from

supervision all the way down to the

inputs the tokens and this is very

desirable and nice because the gradients

that flow from the top if you remember

from your microad addition just

distributes gradients during the

backwards state to both of its branches

equally so addition is a branch in the

gradients and so that means that the

gradients from the top flows straight to

the inputs the tokens through the

residual Pathways unchanged but then in

addition to that the gradient also flows

through the blocks and the blocks you

know contribute their own contribution

over time and kick in and change the

optimization over time but basically

clean residual pathway is desirable from

an optimization perspective and then the

this is the pre-normalization version

where you see that RX first goes through

the layer normalization and then the

attention and then goes uh back out to

go to the L ration number two and the

multia perceptron sometimes also

referred to as a feed forward Network or

an FFN and then that goes into the

residual stream again and the one more

thing that is kind of interesting to

note is that recall that attention is a

communication operation it is where all

the tokens and there's 1,24 tokens lined

up in a sequence and this is where the

tokens communicate this is where they

exchange information so attention is a

um aggregation function it's a pooling

function it's a weighted sum function it

is a reduce operation whereas MLP this

uh MLP here happens at every single

token individually there's no

information being collected or exchanged

between the tokens so the attention is

the reduce and the MLP is the map and

what you end up with is that the

Transformer just ends up just being a

repeated application of map produce if

you want to think about it that way so

um this is where they communicate and

this is where they think individually

about the information that they gathered

and every one of these blocks uh

iteratively refines the um

representation is at the residual stream

so this is our block um slightly

modified from this picture Okay so let's

now move on to the MLP so the MLP block

uh I implemented as follows

it is relatively straightforward we

basically have two linear projections

here that are sandwiched in between the

G

nonlinearity so nn. G approximate is 10h

now when we swing on uh swing over to

the Pyro documentation this is n.g and

it has this format and it has two

versions the original version of G which

we'll step into into in a bit and the

approximate version of Galo which we can

request using

10 so as you can see just as a preview

here G is a basically like a reu except

there's no flat exactly Flat Tail here

at exactly zero but otherwise it looks

very much like a slightly smoother reu

it comes from this paper here Gan error

linear units and uh you can step through

this paper and there's some mathematical

calac reasoning that leads to an

interpretation that leads to the

specific formulation it has to do with

stochastic radial risers and the

expectation of a modification to

Adaptive dropout so you can read through

all of that if you'd like here and

there's a little bit of history as to

why there is an an approximate version

of G and that comes from this issue here

as far as I can tell and in this issue

Daniel Hendrix mentions that at the time

when they developed this nonlinearity

the Earth function which you need to

evaluate the exact G was very slow in

tensor flow so they ended up basically

developing this approximation and this

approximation that then ended up being

picked up by Bert and by GP P2 Etc but

today there's no real good reason to use

the approximate version you'd prefer to

just use the exact version um because I

my expectation is that there's no big

difference anymore and this is kind of

like a historical um kind of Quirk um

but we are trying to reproduce gpt2

exactly and gpt2 used the 10h

approximate version so we prefer to

stick with

that um now one other reason to actually

just intuitively use G instead of veru

is previously in the in videos in the

past we've spoken about the dead reu

neuron problem where in this tale of a

reu if it's exactly flat at zero any

activations that fall there will get

exactly zero gradient there's no change

there's no adaptation there's no

development of the network if any of

these activations end in this flat

region but the G always contributes a

local gradient and so there's always

going to be a change always going to be

an adaptation and sort of smoothing it

out ends up empirically working better

in practice as demonstrated in this

paper and also as demonstrated by it

being picked up by the bird paper gbt2

paper and so on so for that reason we

adopt this nonlinearity uh here in the

10 in the gbt2 reproduction now in more

modern networks also like llama 3 and so

on this nonlinearity also further

changes uh to swiglo and other variants

like that uh but for gpt2 they Ed this

approximate

G okay and finally we have the attention

operation so let me paste in my

attention

so I know this is a lot so I'm going to

go through this a bit quickly a bit

slowly but not too slowly because we

have covered this in the previous video

and I would just point you there um so

this is the attention operation now in

the previous video you will remember

this is not just attention this is um

multi-headed attention right and so in

the previous video we had this

multi-headed attention module and this

implementation made it obvious that

these heads are not actually that

complicated uh there's basically

in parallel inside every attention block

there's multiple heads and they're all

functioning in parallel and uh their

outputs are just being concatenated and

that becomes the output of the

multi-headed attention so the heads are

just kind of like parallel streams and

their outputs get

concatenated and so it was very simple

and made the head be kind of like U

fairly straightforward in terms of its

implementation what happens here is that

instead of having two separate modules

and indeed many more modules that get

concatenated all of that is just put

into a single uh self attention uh

module and instead I'm being very

careful and doing a bunch of transpose

split um tensor gymnastics to make this

very efficient in pych but fundamentally

and algorithmically nothing is different

from the implementation we saw

before um in this uh give

repository so to remind you very briefly

and I don't want to go in this uh into

this in too many in too much time but we

have these tokens lined up in a sequence

and there's 1,20 of them and then each

token at this stage of the attention

emits three vectors the query key and

the value and first what happens here um

is that the queries and the keys have to

multiply each other to get sort of the

attention um amount like how interesting

they find each other so they have to

interact multiplicatively so what we're

doing here is we're calculating the qkv

we splitting it and then there's a bunch

of gymnastics as I mentioned here and

the way this works is that we're

basically making the number of heads and

H into a batch Dimension and so it's a

batch Dimension just like B so that in

these operations that follow pytorch

treats B and NH as batches and it

applies all the operations on all of

them in parallel in both the batch and

the

heads and the operations that get

applied are number one the queries and

the keys intera to give us her attention

this is the autoaggressive mask that

makes sure that the tokens only attend

to tokens before them and never to

tokens in the

future the softmax here normalizes the

attention so it sums to one always and

then recall from the previous video that

doing the attention Matrix multiply with

the values is basically a way to do a

weighted sum of the values of the tokens

that we found interesting at every

single token and then the final

transpose conf VI and view is just

reassembling all of that again and this

actually performs the concatenation

operation so you can step through this

uh slowly if you'd like um but it is

equivalent mathematically to our

previous implementation is just more

efficient in P torch so that's why I

chose this implementation

instead now in addition to that I'm

being careful with how I name my

variables so for example cattin is the

same as seaten and so actually our keys

should basically exactly follow the

schema of the hugging face train

Transformers code and that will make it

very easy for us to now Port over all

the weights from exactly this sort of

naming conventions because all of our

variables are named the same thing but

um at this point we have finished the

gpt2 implementation and what that allows

us to do is we don't have to basically

use uh this file from hugging face which

is fairly long

um this

is uh 2,000 lines of code um instead we

just have a less than 100 lines of code

and this is the complete uh gpd2

implementation so at this stage we

should just be able to take over all the

weights set them and then do generation

so let's see what that looks like okay

so here I've also changed the GPT config

so that the numbers here the H

parameters agree with the gpt2 124 M

model so the maximum sequence length

which I call block size here is 124 the

number of tokens is 50250 257 which if

you watch my tokenizer video know that

this is 50,000 m merges BP merges 256

bite tokens the leaves of the BP tree

and one special end of text token that

delimits different documents and can

start generation as well and there are

12 layers there are 12 heads in the

attention and the dimension of the

Transformers was

768 so here's how we can now load the

parameters from hugging face to uh our

code here and initialize the GPT class

with those parameters so let me just

copy paste a bunch of code

here and I'm not going to go through

this code too slow too quickly too

slowly because um honestly it's not that

interesting it's not that exciting we're

just loading the weights so it's kind of

dry but as I mentioned there are four

models in this miniseries of gpt2 this

is some of the Jupiter code um code that

we had here on the right I'm just pting

it over these are the hyper parameters

of the gpt2 models uh we're creating the

config object and creating our own model

and then what's Happening Here is we're

creating the state dict both for our

model and for the hugging face

model um and then what we're doing here

is we're going over the hugging face

model keys and we're copying over those

tensors and in the process we are kind

of ignoring a few of the buffers they're

not parameters they're buffers so for

example attention dobias uh that's just

used for the autoaggressive mask and so

we are ignoring some of those masks and

uh that's it and then then one

additional kind of annoyance is that

this comes from the tensorflow repo and

I'm not sure how this is a little bit

annoying but some of the weights are

transposed from what pytorch would want

and so manually I hardcoded the weights

that should be transposed and then we

transpose them if that is so and then we

return this model so the from

pre-trained is a

Constructor or class method in Python

that Returns the GPT object if we just

give it the model type which in our case

is gpt2 the smallest model that we're

interested in so this is the code and

this is how you would use it and um we

can pop open the terminal here in vs

code and we can python train gbt2 pi and

fingers

crossed okay so we didn't crash and so

we can load the weights and the biases

and everything else into our Ann module

but now let's also get additional

confidence that this is working and

let's try to actually generate from this

model okay now before we can actually

generate from this model we have to be

able to forward it we didn't actually

write that code yet so here's the

forward

function so the input to the forward is

going to be our indices our tokens uh

token indices and they are always of

shape B BYT and so we have batch

dimension of B and then we have the time

dimension of up to T and the T can't be

more than the block size the block size

is is the maximum sequence length so B

BYT indices arranged is sort of like a

two-dimensional layout and remember that

basically every single row of this is of

size up to uh block size and this is T

tokens that are in a sequence and then

we have B independent sequences stacked

up in a batch so that this is

efficient now here we are forwarding the

position embeddings and the token

embeddings and this code should be very

recognizable from the previous lecture

so um we basically use uh a range which

is kind of like a version of range but

for pytorch uh and we're iterating from

Z to T and creating this uh positions uh

sort of uh indices

um and then we are making sure that

they're in the same device as idx

because we're not going to be training

on only CPU that's going to be too

inefficient we want to be training on

GPU and that's going to come in in a

bit uh then we have the position

embeddings and the token embeddings and

the addition operation of those two now

notice that the position embed are going

to be identical for every single row of

uh of input and so there's broadcasting

hidden inside this plus where we have to

create an additional Dimension here and

then these two add up because the same

position embeddings apply at every

single row of our example stacked up in

a batch then we forward the Transformer

blocks and finally the last layer norm

and the LM head so what comes out after

forward is the logits and if the input

was B BYT indices then at every single B

by T we will calculate the uh logits for

what token comes next in the sequence so

what is the token B t+1 the one on the

right of this token and B app size here

is the number of possible tokens and so

therefore this is the tensor that we're

going to obtain and these low jits are

just a softmax away from becoming

probabilities so this is the forward

pass of the network and now we can get

load and so we're going to be able to

generate from the model

imminently okay so now we're going to

try to set up the identical thing on the

left here that matches hug and face on

the right so here we've sampled from the

pipeline and we sampled five times up to

30 tokens with the prefix of hello I'm a

language model and these are the

completions that we achieved so we're

going to try to replicate that on the

left here so number turn sequences is

five max length is 30 so the first thing

we do of course is we initialize our

model then we put it into evaluation

mode now this is a good practice to put

the model into eval when you're not

going to be training it you're just

going to be using it and I don't

actually know if this is doing anything

right now for the following reason our

model up above here contains no modules

or layers that actually have a different

uh Behavior at training or evaluation

time so for example Dropout batch norm

and a bunch of other layers have this

kind of behavior but all of these layers

that we've used here should be identical

in both training and evaluation time um

so so potentially model that eval does

nothing but then I'm not actually sure

if this is the case and maybe pytorch

internals uh do some clever things

depending on the evaluation mode uh

inside here the next thing we're doing

here is we are moving the entire model

to Cuda so we're moving this all of the

tensors to GPU so I'm sshed here to a

cloud box and I have a bunch of gpus on

this box and here I'm moving the entire

model and all of its members and all of

its tensors and everything like that

everything gets shipped off to basically

a whole separate computer that is

sitting on the GPU and the GPU is

connected to the uh CPU and they can

communicate but it's basically a whole

separate computer with its own computer

architecture and it's really well

catered to parallel processing tasks

like those of running neural networks so

I'm doing this so that the model lives

on the GPU a whole separate computer and

it's just going to make our code a lot

more efficient because all of this stuff

runs a lot more efficiently on the

gpus so that's the model

itself now uh the next thing we want to

do is we want to start with this as the

prefix when we do the generation so

let's actually create those prefix

tokens so here's the code that I've

written we're going to import the tich

token library from open Ai and we're

going to get the gpt2 encoding so that's

the tokenizer for gpt2 and then we're

going to encode this string and get a

list of integers which are the tokens uh

now these integers here should actually

be fairly straightforward because we can

just copy paste this string and we can

sort of inspect what it is in tick

tokenizer so just pasting that in these

are the tokens that are going to come

out so this list of integers is what we

expect tokens to become and as you

recall if you saw my video of course all

the tokens they're just little string

chunks right so these are this is the

chunc of this string into gpt2

tokens so once we have those tokens it's

a list of integers we can create a torch

tensor out of it in this case it's eight

tokens and then we're going to replicate

these eight tokens for five times to get

five rows of eight tokens and that is

our initial um input X as I call it here

and it lives on the GPU as well so X now

is this idx that we can put into forward

to get our logits so that we know what

comes as the sixth token

uh sorry as the ninth token in every one

of these five rows okay and we are now

ready to generate so let me paste in one

more code block

here um so what's happening here in this

code block is we have this x which is of

size B BYT right so batch by time and

we're going to be in every iteration of

this loop we're going to be adding a

column of new indices into each one of

these rows right and so these are the

new indices and we're appending them to

the the sequence as we're sampling so

with each Loop iteration we get one more

column into X and all of the operations

happen in the context manager of torch.

nograd this is just telling pytorch that

we're not going to be calling that

backward on any of this so it doesn't

have to cach all the intermediate

tensors it's not going to have to

prepare in any way for a potential

backward later and this saves a lot of

space and also possibly uh some time so

we get our low jits we get the loow jits

at only the last location we throw away

all the other low jits uh we don't need

them we only care about the last columns

low jits so this is being wasteful uh

but uh this is just kind of like an

inefficient implementation of

sampling um so it's correct but

inefficient so we get the last column of

loow jits pass it through soft Max to

get our probabilities then here I'm

doing top case sampling of 50 and I'm

doing that because this is the hugging

face default so just looking at the

hugging face docks here of a pipeline um

there's a bunch of

quarks that go into hugging face and I

mean it's it's kind of a lot honestly

but I guess the important one that I

noticed is that they're using top K by

default which is 50 and what that does

is that uh so that's being used here as

well and what that does is basically we

want to take our probabilities and we

only want to keep the top 50

probabilities and anything that is lower

than the 50th probability uh we just

clamp to zero and renormalize and so

that way we are never sampling very rare

tokens uh the tokens we're going to be

sampling are always in the top 50 of

most likely tokens and this helps keep

the model kind of on track and it

doesn't blabber on and it doesn't get

lost and doesn't go off the rails as

easily uh and it kind of like um sticks

in the vicinity of likely tokens a lot

better so this is the way to do it in

pytorch and you can step through it if

you like I don't think it's super

insightful so I'll speed through it but

roughly speaking we get this new column

of of tokens we append them on x and

basically The Columns of X grow until

this y Loop gets tripped up and then

finally we have an entire X of size um 5

by 30 in this case in this example and

we can just basically print all those

individual rows so I'm getting all the

rows I'm getting all the tokens that

were sampled and I'm using the decode

function from Tik tokenizer to get back

the string which we can print and so

terminal new terminal

and let me python train

gpt2 okay so these are the generations

that we're getting hello I'm a language

model not a

program um new line new line Etc hello

I'm a language model and one of the main

things that bothers me when they create

languages is how easy it becomes to

create something that I me so this will

just like blabber on right in all these

cases now one thing you will notice is

that these Generations are not the

generations of hugging face here and I

can't find the discrepancy to be honest

and I didn't fully go through all these

options but probably there's something

else hiding in on addition to the top P

so I'm not able to match it up but just

for correctness um down here Below in

the juper notebook and using the hugging

face model so this is the hugging face

model here I was I replicated the code

and if I do this and I run that then I

am getting the same results so basically

the model internals are not wrong it's

just I'm not 100% sure what the pipeline

does in hugging face and that's why

we're not able to match them up but

otherwise the code is correct and we've

loaded all the um tensors correctly so

we're initializing the model correctly

and everything here works so long story

short uh We've Port it all the weights

we initialize the gpt2 this is the exact

opening gpt2 and it can generate

sequences and they look sensible and now

here of course we're initializing with

gbt2 model weights but now we want to

initialize from scratch from random

numbers and we want to actually train a

model that will give us sequences as

good as or better than these ones in

quality and so that's what we turn to

next so it turns out that using the

random model is actually fairly

straightforward because pytorch already

initializes our model randomly and by

default so when we create the GPT model

and the Constructor this is all um all

of these layers and modules have random

initializers that are there by default

so when these linear layers get created

and so on there's default Constructors

for example using the Javier

initialization that we saw in the past

uh to construct the weights of these

layers and so creating a random model

instead of a gpt2 model is actually

fairly straightforward and we would just

come here and instead we would create

model equals GPT and then we want to use

the default config GPT config and the

default config uses the 124 M parameters

so this is the random model

initialization and we can run

it and we should be able to get uh

results now the results here of course

are total garbage carbal and that's

because this is random model and so

we're just getting all these random

token string pieces chunked up totally

at random so that's what we have right

now uh now one more thing I wanted to

point out by the way is in case you do

not have Cuda available because you

don't have a GPU you can still follow

along with uh with what we're doing here

uh to some extent uh and probably not to

the very end because by the end we're

going to be using multiple gpus and

actually doing a serious training run uh

but for now you can actually follow

along decently okay uh so one thing that

I like to do in pytorch is I like to

autod detect the device that is

available to you so in particular you

could do that like this

so here we are trying to detect a device

to run on that has the highest compute

capability you can think about it that

way so by default we start with CPU

which of course is available everywhere

because every single computer will have

a CPU but then we can try to detect do

you have a GPU you so use a Cuda and

then if you don't have a Cuda uh do you

at least have MPS MPS is the back end

for Apple silicon so if you have a

Macbook that is fairly new you probably

have apple silicon on the inside and

then that has a GPU that is actually

fairly capable uh depending on which

MacBook you have and so you can use MPS

which will be potentially faster than

CPU and so we can print the device here

now once we have the device we can

actually use it in place of Puda so we

just swap it in and notice that here

when we call model on X if this x here

is on CPU instead of GPU then it will

work fine because here in the forward

which is where P to will come when we

create a pose we were careful to use the

device of idx to create this tensor as

well and so there won't be any mismatch

where one tensor is on CPU one is on GPU

and uh that you can't combine those but

here we are um carefully initializing on

the correct device as indicated by the

input to this model so this will autod

detect device for me this will be of

course

GPU so using device

Cuda uh but uh you can also run with um

as I mentioned another device and it's

not going to be too much slower so if I

override device here

oops if I override device equals

CPU

then we'll still print Cuda of course

but now we're actually using CPU one 2 3

4 5 6 okay about 6 seconds and actually

we're not using torch compile and stuff

like that which will speed up everything

a lot faster as well but you can follow

even on a CPU I think to a decent extent

um so that's note on that okay so I do

want to loop around eventually into what

it means to have different devices in

pytorch and what it is exactly that

pytorch does in the background for you

when you do something like module. 2

device or where you take a torch tensor

and do A2 device and what exactly

happens and how that works but for now

I'd like to get to training and I'd like

to start training the model and for now

let's just say the device makes code go

fast um and let's go into how we can

actually train the model so to train the

model we're going to need some data set

and for me the best debugging simplest

data set that I like to use is the tiny

Shakespeare data set um and it's

available at this URL so you can W get

it or you can just search tiny

Shakespeare data

set and so um I have in my file system

as just LS input.txt

so I already downloaded it and here I'm

reading the data set getting the first

1,000 characters and printing the first

100

now remember that gpt2 has uh roughly a

compression ratio the tokenizer has a

compression ratio of rly 3 to1 so th000

characters is roughly 300 tokens here uh

that will come out of this in the slice

that we're currently getting so this is

the first few uh

characters and uh if you want to get a

few more statistics on this we can do

work count on input.txt

so we can see that this is uh 40,000

lines about 200,000 words in this data

set and about 1 million bytes in this

file and knowing that this file is only

asky characters there's no crazy unic

code here as far as I know and so every

asky character is encoded with one bite

and so this is uh the same number

roughly a million characters inside this

data set so that's the data set size uh

by default very small and minimal data

set for debugging to get us off the

ground in order to tokenize this data

set we're going to get Tik token

encoding for gbt2 encode the data uh the

first um 1,000 characters and then I'm

only going to print the first 24 tokens

so these are the tokens as a list of

integers and if you can read gpt2 tokens

you will see that 198 here you'll

recognize that as the slashing character

so that is a new line and then here for

example we have two new lines so that's

198 twice here uh so this is just a

tokenization of the first 24 tokens so

what we want to do now is we want to

actually process these token sequences

and feed them into a Transformer and in

particular we want them we want to

rearrange these tokens into this idx

variable that we're going to be feeding

into the Transformer so we don't want a

single very long onedimensional sequence

we want an entire batch where each

sequence is up to uh is basically T

tokens and T cannot be larger than the

maximum sequence length and then we have

these t uh tlong uh sequences of tokens

and we have B independent examples of

sequences so how can we create a b BYT

tensor that we can feed into the forward

out of these onedimensional

sequences so here's my favorite way to

to achieve this uh so if we take torch

and then we create a tensor object out

of this list of integers and just the

first 24 tokens my favorite way to do

this is basically you do a do view of um

of uh for example 4x6 which multiply to

24 and so it's just a two-dimensional

rearrangement of these tokens and you'll

is that when you view this

onedimensional sequence as

two-dimensional 4x6 here the first six

uh tokens uh up to here end up being the

first row the next six tokens here end

up being the second row and so on and so

basically it's just going to stack up

this the um every six tokens in this

case as independent rows and it creates

a batch of tokens in this case and so

for example if we are token 25 in the

Transformer when we feed this in and

this becomes the idx this token is going

to see these three tokens and it's going

to try to predict that 198 comes

next so in this way we are able to

create this two-dimensional batch that's

that's quite nice now in terms of the

label that we're going to need for the

Target to calculate the loss function

how do we get that well we could write

some code inside the forward pass

because we know that the next uh token

in a sequence which is the label is just

to the right of us but you'll notice

that actually we for this token at the

very end 13 we don't actually have the

next correct token because we didn't

load it so uh we actually didn't get

enough information here so I'll show you

my favorite way of basically getting

these batches and I like to personally

have not just the input to the

Transformer which I like to call X but I

also like to create the labels uh tensor

which is of the exact same size as X but

contains the targets at every single

position

and so here's the way that I like to do

that I like to make sure that I fetch

plus one uh token because we need the

ground Truth for the very last token uh

for

13 and then when we're creating the

input we take everything up to the last

token not including and view it as 4x6

and when we're creating targets we do

the buffer but starting at index one not

index zero so we're skipping the first

element and we view it in the exact same

size and then when I print this

here's what happens where we see that

basically as an example for this token

25 its Target was 198 and that's now

just stored at the exact same position

in the Target tensor which is 198 and

also this last token 13 now has its

label which is 198 and that's just

because we loaded this plus one here so

basically this is the way I like to do

it you take long sequences you uh view

them in two- dimensional terms so that

you get batch of time and then we make

sure to load one additional token so we

basically load a buffer of tokens of B *

t+ one and then we sort of offset things

and view them and then we have two

tensors one of them is the input to the

Transformer and the other exactly is the

labels and so let's now reorganize this

code and um create a very simple data

loader object that tries to basically

load these tokens and um feed them to

the Transformer and calculate the loss

okay so I reshuffled the code here uh

accordingly so as you can see here I'm

temporarily overwriting U to run a CPU

and importing TI token and all of this

should look familiar we're loading a

th000 characters I'm setting BT to just

be 4 and 32 right now just because we're

debugging we just want to have a single

batch that's very small and all of this

should now look familiar and follows

what we did on the right and then here

we get the we create the model and get

the lits and so so here as you see I

already ran this only runs in a few

seconds but because we have a batch of

uh 4X 32 our lits are now of size 4X 32x

50257 so those are the lit for what

comes next at every position and now we

have the labels which are stored in y so

now is the time to calculate the loss

and then do the backward pass and then

the optimization so let's first

calculate the

loss okay so to calculate the loss we're

going to adjust the forward function of

this NN module in the model and in

particular we're not just going to be

returning logits but also we're going to

return the loss uh and we're going to

not just pass in the input in thees but

also the targets uh in y and now we will

print not Lo just. shape anymore we're

actually going to print the loss

function and then c. exit of zero so

that we skip some of the sampling logic

so now let's swing up to the forward

function which gets called there because

now we also have these optional

targets and when we get the targets we

can also calculate uh the loss and

remember that we want to basically

return uh log just loss and loss by

default is none

but

um let's put this here if uh targets is

not none then we want to calculate loss

and co-pilot is already getting excited

here and calculating the what looks to

be correct loss it is using the cross

entropy loss as is documented here uh so

this is a function in pytorch under the

functional now what is actually

happening here because it looks a little

bit scary uh basically uh the F that

cross entropy does not like

multi-dimensional inputs it can't take a

b BYT by vocap size so what's happening

here is that we are flattening out this

three-dimensional tensor into just two

Dimensions the First Dimension is going

to be calculated automatically and it's

going to be B * T and then the last

Dimension is vocap size so basically

this is uh flattening out this

three-dimensional tensor of logits to

just be two- dimensional B * T all

individual examples and vocap size on uh

in terms of the length of each row and

then it's also flattening out the

targets which are also two- dimensional

at this stage but we're going to just

flatten them out so they're just a

single tensor of B * T and this can then

pass into cross entropy to calculate a

loss which we return so this should

basically at this point run because this

is not too complicated

so let's run it and let's see if we

should be printing the

loss and here we see that we printed 11

uh roughly and so

um and notice that this is the tensor of

a single element which is this number 11

now we also want to be able to calculate

a reasonable uh kind of starting point

for a random rationalized Network so we

covered this in previous videos but our

vocabulary size is

50257 at initialization of the network

you would hope that um every vocab

element is getting roughly a uniform

probability uh so that we're not

favoring at initialization any token way

too much we're not confidently wrong at

initialization so what we're hoping is

that the probability of any arbitrary

token is roughly 1 over 50,2 57 and now

we can sanity check the loss because

remember that the cross entropy loss is

just basically the negative um log

likelihood so if we now take this

probability and we take it through the

natural logarithm and then we do the

negative that is the loss we expect at

initialization and we covered this in

previous videos so I would expect

something around 10.82 and we're seeing

something around 11 so it's not way off

this is roughly the probability I expect

at initialization so that tells me that

the at initialization or probability

distribtion is roughly diffused it's a

good starting point and we can now uh

perform the optimization and tell the

network which elements you know should

follow correctly in what order so at

this point we can do a l step backward

calculate the gradients and do an

optimization so let's get to that okay

so let's do the optimization now um so

here we

have the loss is this is how we get the

loss but now basically we want a load

for Loop here so 4 I in range let's do

50 steps or something like that uh let's

create an Optimizer object in

pytorch um and so here we are using the

atom um Optimizer which is an

alternative to the stochastic radian

descent Optimizer SGD that we were using

so SGD is a lot simpler atom is a bit

more involved and I actually

specifically like the atom W variation

because in my opinion it kind of just

like fixes a bug um so adom w is a bug

fix of atom is what I would say when we

go to the documentation for atom

W oh my

gosh we see um that it takes a bunch of

hyper parameters and it's a little bit

more complicated than the SGD we were

looking at before uh because in addition

to basically updating the parameters

with the gradient uh scaled by the

Learning rate it keeps these buffers

around and it keeps two buffers the m

and the V which it calls the first and

the second moment so something that

looks a bit like momentum and something

that looks a bit like RMS prop if you're

familiar with it but you don't have to

be it's just kind of a normalization

that happens on each gradient element

individually and speeds up the

optimization especially for language

models but I'm not going to go into the

detail right here we're going to treat

it as a bit of a black box and it just

optimizes um the objective faster than

SGD which is what we've seen in the

previous lectures so let's use it as a

black box in our case uh create the

optimizer object and

then go through the optimization

the first thing to always make sure the

co-pilot did not forget to zero the

gradients so um always remember that you

have to start with a zero gradient then

when you get your loss and you do a DOT

backward dot backward adds to gradients

so it deposits gradients it it always

does a plus equals on whatever the

gradients are which is why you must set

them to zero so this accumulates the

gradient from this loss and then we call

the step function on the optimizer to um

update the parameters and to um decrease

the

loss and then we print a step and the

loss do item is used here because loss

is a tensor with a single element do

item will actually uh convert that to a

single float and this float will live

not will will live on the CPU so this

gets to some of the internals again of

the devices but loss is a is a tensor

with a single element and it lifts on

GPU for me because I'm using gpus when

you call item P torch behind the scenes

will take that one-dimensional tensor

ship it back to the CPU uh memory and

convert it into a float that we can just

print so this is the optimization and

this should probably just

work let's see what

happens actually sorry let me instead of

using CPU override let me delete that so

this is a bit faster for me and it runs

on Cuda

oh expected all tensors to be on the

same device but found at least two

devices Cuda zero and CPU so Cuda zero

is the zeroth GPU because I actually

have eight gpus on this box uh so the

zeroth GPU in my box and CPU and model

we have moved to device but when I was

writing this code I actually introduced

a bug because buff we never moved to

device and you have to be careful

because you can't just do buff dot two

of

device um it's not stateful it doesn't

convert it to be a device it instead uh

returns pointer to a new memory which is

on the device so you see how we can just

do model that two a device that does not

apply to tensors you have to do buff

equals

um b.2 device and then this should work

okay so what do we expect to see we

expect to see a reasonable loss in the

beginning and then we continue to

optimize just the single batch and so we

want to see that we can overfit this

single batch we can we can crush this

little batch and we can perfectly

predict the indices on just this little

batch and indeed that is roughly what

we're seeing here

so um we started off at roughly 10.82 11

in this case and then as we continue

optimizing on this single batch without

loading new examples we are making sure

that we can overfit a single batch and

we are getting to very very low loss so

the Transformer is memorizing this

single individual batch and one more

thing I didn't mention is uh the

learning rate here is 3 E4 which is a

pretty good default for most uh

optimizations that you want to run at a

very early debugging stage so this is

our simple inter Loop and uh we are

overfitting a single batch and this

looks good so now what uh what comes

next is we don't just want to overfit a

single batch we actually want to do an

optimization so we actually need to

iterate these XY batches and create a

little data loader uh that makes sure

that we're always getting a fresh batch

and that we're actually optimizing a

reasonable objective so let's do that

next okay so this is what I came up with

and I wrote a little data loader

light um so what this data loader does

is we're importing the token up here

we're reading the entire text file from

this single input.txt

tokenizing it and then we're just

printing the number of tokens in total

and the number of batches in a single

Epoch of iterating over this data set so

how many unique batches do we output

before we loop back around the beginning

of the document and start reading it

again so we start off at position zero

and then we simply walk the document in

batches of B * T so we take chunks of B

* T and then always Advance by B * T and

um it's important to note that we're

always advancing our position by exactly

B * T but when we're fetching the tokens

we're actually fetching from current

position to B * t + 1 and we need that

plus one because remember uh we need the

target token

um for the last token in the current

batch and so that way we can do um the

XY exactly as we did it before and if we

are to um run out of data we'll just

loop back around to zero so this is one

way to write a very very simple data

loader um that simply just goes through

the file in chunks and is good enough

for us uh for current purposes and we're

going to complexify it later and now

we'd like to come back around here and

we'd like to actually use our data

loader so the import Tik token has moved

up and actually all of this is now

useless so instead we just want a train

loader for the training data and we want

to use the same hyper parameters for

four so B size was four and time was

32 and then here we need to get the XY

for the current batch so let's see if

copal gets it because this is simple

enough uh so we call the next batch and

then we um make sure that we have to

move our tensors from CPU to the device

so here when I converted the tokens

notice that I didn't actually move these

tokens to the GPU I left them on CPU

which is the default um and that's just

because I'm trying not to waste too much

memory on the GPU in this case this is a

tiny data set and it would fit uh but

it's fine to just uh ship it to GPU

right now for for our purposes right now

so we get the next batch we keep the

data loader simple CPU class and then

here we actually ship it to the GPU and

do all the computation and uh let's see

if this runs so python train gbt2 pi and

what do we expect to see before this

actually happens what we expect to see

is now we're actually getting the next

batch so we expect to not overfit a

single batch and so I expect our loss to

come down but not too much and that's

because I still expect it to come down

because in the

50257 tokens many of those tokens never

occur in our data set so there are some

very easy gains to be made here in the

optimization by for example taking the

biases of all the loits that never occur

and driving them to negative infinity

and that would basically just it's just

that all of these crazy unic codes or

different languages those tokens never

occur so their probability should be

very low and so the gains that we should

be seeing are along the lines of

basically deleting the usage of tokens

that never occur that's probably most of

the loss gain that we're going to see at

this scale right now uh but we shouldn't

come to a zero uh because um we are only

doing 50 iterations and I don't think

that's enough to do an eoch right now so

let's see what we

got we um we have 338,000

tokens which makes sense with our 3:1

compression ratio because there are 1

million uh characters so one Epoch with

the current setting of B and T will take

2, 600 batches and we're only doing 50

batches of optimization in

here so we start off in a familiar

territory as expected and then we seem

to come down to about 6.6 so basically

things seem to be working okay right now

with respect to our expectations so

that's good okay next I want to actually

fix a bug that we have in our code um

it's not a major bug but it is a bug

with respect to how gpt2 training uh

should

happen um

so the buck is the following we were not

being careful enough when we were

loading the weights from hugging face

and we actually missed a little detail

so if we come

here notice that um the shape of these

two tensors is the same so this one here

is the token embedding at the bottom of

the

Transformer right so and this one here

is the language modeling head at the top

of the

Transformer and both of these are

basically two-dimensional tensors and

they shape is identical so here the

first one is the output embedding the

token embedding and the second one is

this linear layer at the very top the

classifier layer both of them are of

shape

50257 X

768 um this one here is giving us our

token embeddings at the bottom and this

one here is taking the 768 channels of

the Transformer and trying to upscale

that to 50, 257 to get the Lis for the

next token so they're both the same

shape but more than that actually if you

look at um comparing their elements um

in pytorch this is an element wise

equality so then we use do all and we

see that every single element is

identical and more than that we see that

if we actually look at the data pointer

uh this is what this is a way in pytorch

to get the actual pointer to the uh data

and the storage we see that actually the

pointer is identical so not only are

these two separate tensors that happen

to have the same shape and elements

they're actually pointing to the

identical tensor so what's happening

here is that this is a common weight

tying scheme uh that actually comes from

the original

um from the original attention is all

you need paper and actually even the

reference before it so if we come

here

um eddings and softmax in the attention

is all you need paper they mentioned

that in our model we shared the same

weight Matrix between the two embedding

layers and the pre softmax linear

transformation similar to 30 um so this

is an awkward way to phrase that these

two are shared and they're tied and

they're the same Matrix and the 30

reference is this

paper um so this came out in

2017 and you can read the full paper but

basically it argues for this weight

tying scheme and I think intuitively the

idea for why you might want to do this

comes from from this paragraph here and

basically you you can observe

that um you actually want these two

matrices to behave similar in the

following sense if two tokens are very

similar semantically like maybe one of

them is all lowercase and the other one

is all uppercase or it's the same token

in a different language or something

like that if you have similarity between

two tokens presumably you would expect

that they are uh nearby in the token

embedding space but in the exact same

way you'd expect that if you have two

tokens that are similar semantically

you'd expect them to get the same

probabilities at the output of a

transformer because they are

semantically similar and so both

positions in the Transformer at the very

bottom and at the top have this property

that similar tokens should have similar

embeddings or similar weights and so

this is what motivates their exploration

here and they they kind of you know I

don't want to go through the entire

paper and and uh you can go through it

but this is what they observe they also

observe that if you look at the output

embeddings they also behave like word

embeddings um if you um if you just kind

of try to use those weights as word

embeddings um so they kind of observe

this similarity they try to tie them and

they observe that they can get much

better performance in that way and so

this was adopted and the attention is

all need paper and then it was used

again in gpt2 as well

so I couldn't find it in the

Transformers implementation I'm not sure

where they tie those embeddings but I

can find it in the original gpt2 code U

introduced by open aai so this is um

openai gpt2 Source model and here where

they are forwarding this model and this

is in tensorflow but uh that's okay we

see that they get the wte token

embeddings and then here is the incoder

of the token embeddings and the

position and then here at the bottom

they Ed the WT again to do the lits so

when they get the loits it's a math Mo

of uh this output from the Transformer

and the wte tensor is

reused um and so the wte tensor

basically is used twice on the bottom of

the Transformer and on the top of the

Transformer and in the backward pass

we'll get gradients contributions from

both branches right and these gradients

will add up um on the wte tensor um so

we'll get a contribution from the

classifier list

and then at the very end of the

Transformer we'll get a contribution at

the at the bottom of it float floating

again into the wte uh tensor so we want

to we are currently not sharing WT and

our code but we want to do

that um

so weight sharing scheme um and one way

to do this let's see if goil gets it oh

it does okay uh so this is one way to do

it

uh

basically relatively straightforward

what we're doing here is we're taking

the wte do weight and we're simply uh

redirecting it to point to the LM head

so um this basically copies the data

pointer right it copies the reference

and now the wte weight becomes orphaned

uh the old value of it and uh pytorch

will clean it up python will clean it up

and so we are only left with a single

tensor and it's going to be used twice

in the forward pass and uh this is to my

knowledge all that's required so we

should be able to use this and this

should probably train uh we're just

going to basically be using this exact

same sensor twice and

um we weren't being careful with

tracking the likelihoods but uh

according to the paper and according to

the results you'd actually expect

slightly better results doing this and

in addition to that one other reason

that this is very very nice for us is

that this is a ton of parameters right

uh what is the size here it's 768 *

50257 so This Is 40 million parameters

and this is a 124 million parameter

model so 40 divide 124 so this is like

30% of the parameters are being saved

using this weight time scheme and so

this might be one of the reasons that

this is working slightly better if

you're not training the model long

enough because of the weight tying uh

you don't have to train as many

parameters and so you become more

efficient um in terms of the training

process uh because you have fewer

parameters and you're putting in this

inductive bias that these two embeddings

should share similarities between tokens

so this is the way time scheme and we've

saved a ton of parameters and we expect

our model to work slightly better

because of the scheme okay next I would

like us to be a bit more careful with

the initialization and to try to follow

the way gpt2 initialized their model now

unfortunately the gpt2 paper and the

gpt3 paper are not very explicit about

initialization so we kind of have to

read between the lines uh and instead of

going to the paper which is quite vague

um there's a bit of information in the

code that open I released so when we go

to the model.py we see that when they

initialize their weights they are using

the standard deviation of

0.02 and that's how they they so this is

a normal distribution for the weights

and the standard deviation is

0.02 for the bias they initialize that

with

zero and then when we scroll down

here why is this not scrolling

um the token embeddings are initialized

at

0.02 and position embeddings at 0.01 for

some reason so those are the

initializations and we'd like to mirror

that in

gpt2 uh in our module here so here's a

snippet of code that I sort of came up

with very

quickly so what's happening here is at

the end of our initializer for the GPT

module we're calling the apply function

of NN module and that iterates all the

sub modules of this module and uh

applies in it weights function on them

and so what's happening here is that

we're in we're iterating all the modules

here and if they are an nn. linear

module then we're going to make sure to

initialize the weight using a normal

with the standard deviation of

0.02 if there's a bias in this layer we

will make sure to initialize that to

zero note that zero initialization for

the bias is not actually the pyto

default um by default the bias here is

initialized with a uniform so uh that's

interesting so we make sure to use zero

and for the embedding we're just going

to use 0.02 and um keep it the same um

so we're not going to change it to 0.01

for positional because it's about the

same and then if you look through our

model the only other layer that requires

initialization and that has parameters

is the layer norm and the fighter defer

initialization sets the scale in the

layer Norm to be one and the offset in

the layer Norm to be zero so that's

exactly what we want and so we're just

going to uh keep it that way and so this

is the default initialization if we are

following the um where is it the uh gpt2

uh source code that they released I

would like to point out by the way that

um typically the standard deviation here

on this initialization if you follow the

Javier initialization would be one of

over the square root of the number of

features that are incoming into this

layer but if you'll notice actually 0.02

is basically consistent with that

because the the model sizes inside these

Transformers for gpt2 are roughly 768

1600 Etc so 1 over the square root of

for example 768 gives us

0.03 if we plug in 600 1,600 we get

0.02 if we plug in three times that

0.014 Etc so basically 0.02 is roughly

in the vicinity of reasonable values for

the for um for these initializations

anyway so so it's not uh completely

crazy to be hard coding 0.02 here uh but

you'd like typically uh some something

that grows with the model size instead

but we will keep this because that is

the gpt2 initialization per their source

code but we are not fully done yet on

initialization because there's one more

caveat here so

here a mod initialization which accounts

for the accumulation on the residual

path with model depth is used we scale

the weight of residual layers of

initialization by factor of one over squ

of n where n is the number of residual

layers so this is what gbt2 paper says

so we have not implemented that yet and

uh we can do so now now I'd like to

actually kind of like motivate a little

bit what they mean here I think um so

here's roughly what they

mean if you start out with zeros in your

residual stream remember that each

residual stream is a is of this form

where we continue adding to it X is X

plus something some kind of contribution

so every single block of the residual uh

Network contributes some uh amount and

it gets added and so what ends up