hi everyone so by now you have probably

heard of chat GPT it has taken the world

and AI Community by storm and it is a

system that allows you to interact with

an AI and give it text based tasks so

for example we can ask chat GPT to write

us a small Hau about how important it is

that people understand Ai and then they

can use it to improve the world and make

it more prosperous so when we run this

AI knowledge brings prosperity for all

to see Embrace its

power okay not bad and so you could see

that chpt went from left to right and

generated all these words SE sort of

sequentially now I asked it already the

exact same prompt a little bit earlier

and it generated a slightly different

outcome ai's power to grow ignorance

holds us back learn Prosperity weights

so uh pretty good in both cases and

slightly different so you can see that

chat GPT is a probabilistic system and

for any one prompt it can give us

multiple answers sort of uh replying to

it now this is just one example of a

problem people have come up with many

many examples and there are entire

websites that index interactions with

chpt and so many of them are quite

humorous explain HTML to me like I'm a

dog uh write release notes for chess 2

write a note about Elon Musk buying a

Twitter and so on so as an example uh

please write a breaking news article

about a leaf falling from a

tree uh and a shocking turn of events a

leaf has fallen from a tree in the local

park Witnesses report that the leaf

which was previously attached to a

branch of a tree attached itself and

fell to the ground very dramatic so you

can see that this is a pretty remarkable

system and it is what we call a language

model uh because it um it models the

sequence of words or characters or

tokens more generally and it knows how

sort of words follow each other in

English language and so from its

perspective what it is doing is it is

completing the sequence so I give it the

start of a sequence and it completes the

sequence with the outcome and so it's a

language model in that sense now I would

like to focus on the under the hood of

um under the hood components of what

makes CH GPT work so what is the neural

network under the hood that models the

sequence of these words and that comes

from this paper called attention is all

you need in 2017 a landmark paper a

landmark paper in AI that produced and

proposed the Transformer

architecture so GPT is uh short for

generally generatively pre-trained

Transformer so Transformer is the neuron

nut that actually does all the heavy

lifting under the hood it comes from

this paper in 2017 now if you read this

paper this uh reads like a pretty random

machine translation paper and that's

because I think the authors didn't fully

anticipate the impact that the

Transformer would have on the field and

this architecture that they produced in

the context of machine translation in

their case actually ended up taking over

uh the rest of AI in the next 5 years

after and so this architecture with

minor changes was copy pasted into a

huge amount of applications in AI in

more recent years and that includes at

the core of chat GPT now we are not

going to what I'd like to do now is I'd

like to build out something like chat

GPT but uh we're not going to be able to

of course reproduce chat GPT this is a

very serious production grade system it

is trained on uh a good chunk of

internet and then there's a lot of uh

pre-training and fine-tuning stages to

it and so it's very complicated what I'd

like to focus on is just to train a

Transformer based language model and in

our case it's going to be a character

level language model I still think that

is uh very educational with respect to

how these systems work so I don't want

to train on the chunk of Internet we

need a smaller data set in this case I

propose that we work with uh my favorite

toy data set it's called tiny

Shakespeare and um what it is is

basically it's a concatenation of all of

the works of sh Shakespeare in my

understanding and so this is all of

Shakespeare in a single file uh this

file is about 1 megab and it's just all

of

Shakespeare and what we are going to do

now is we're going to basically model

how these characters uh follow each

other so for example given a chunk of

these characters like this uh given some

context of characters in the past the

Transformer neural network will look at

the characters that I've highlighted and

is going to predict that g is likely to

come next in the sequence and it's going

to do that because we're going to train

that Transformer on Shakespeare and it's

just going to try to produce uh

character sequences that look like this

and in that process is going to model

all the patterns inside this data so

once we've trained the system i' just

like to give you a preview we can

generate infinite Shakespeare and of

course it's a fake thing that looks kind

of like

Shakespeare

um apologies for there's some Jank that

I'm not able to resolve in in here but

um you can see how this is going

character by character and it's kind of

like predicting Shakespeare like

language so verily my Lord the sites

have left the again the king coming with

my curses with precious pale and then

tranos say something else Etc and this

is just coming out of the Transformer in

a very similar manner as it would come

out in chat GPT in our case character by

character in chat GPT uh it's coming out

on the token by token level and tokens

are these sort of like little subword

pieces so they're not Word level they're

kind of like word chunk

level um and now I've already written

this entire code uh to train these

Transformers um and it is in a GitHub

repository that you can find and it's

called nanog

GPT so nanog GPT is a repository that

you can find in my GitHub and it's a

repository for training Transformers um

on any given text and what I think is

interesting about it because there's

many ways to train Transformers but this

is a very simple implementation so it's

just two files of 300 lines of code each

one file defines the GPT model the

Transformer and one file trains it on

some given Text data set and here I'm

showing that if you train it on a open

web Text data set which is a fairly

large data set of web pages then I

reproduce the the performance of

gpt2 so gpt2 is an early version of open

AI GPT uh from 2017 if I recall

correctly and I've only so far

reproduced the the smallest 124 million

parameter model uh but basically this is

just proving that the codebase is

correctly arranged and I'm able to load

the uh neural network weights that openi

has released later so you can take a

look at the finished code here in N GPT

but what I would like to do in this

lecture is I would like to basically uh

write this repository from scratch so

we're going to begin with an empty file

and we're we're going to define a

Transformer piece by piece we're going

to train it on the tiny Shakespeare data

set and we'll see how we can then uh

generate infinite Shakespeare and of

course this can copy paste to any

arbitrary Text data set uh that you like

uh but my goal really here is to just

make you understand and appreciate uh

how under the hood chat GPT works and um

really all that's required is a

Proficiency in Python and uh some basic

understanding of um calculus and

statistics

and it would help if you also see my

previous videos on the same YouTube

channel in particular my make more

series where I um Define smaller and

simpler neural network language models

uh so multi perceptrons and so on it

really introduces the language modeling

framework and then uh here in this video

we're going to focus on the Transformer

neural network itself okay so I created

a new Google collab uh jup notebook here

and this will allow me to later easily

share this code that we're going to

develop together uh with you so you can

follow along so this will be in a video

description uh later now here I've just

done some preliminaries I downloaded the

data set the tiny Shakespeare data set

at this URL and you can see that it's

about a 1 Megabyte file then here I open

the input.txt file and just read in all

the text of the string and we see that

we are working with 1 million characters

roughly and the first 1,000 characters

if we just print them out are basically

what you would expect this is the first

1,000 characters of the tiny Shakespeare

data set roughly up to here so so far so

good next we're going to take this text

and the text is a sequence of characters

in Python so when I call the set

Constructor on it I'm just going to get

the set of all the characters that occur

in this text and then I call list on

that to create a list of those

characters instead of just a set so that

I have an ordering an arbitrary ordering

and then I sort that so basically we get

just all the characters that occur in

the entire data set and they're sorted

now the number of them is going to be

our vocabulary size these are the

possible elements of our sequences and

we see that when I print here the

characters there's 65 of them in total

there's a space character and then all

kinds of special characters and then U

capitals and lowercase letters so that's

our vocabulary and that's the sort of

like possible uh characters that the

model can see or emit okay so next we

will would like to develop some strategy

to tokenize the input text now when

people say tokenize they mean convert

the raw text as a string to some

sequence of integers According to some

uh notebook According to some vocabulary

of possible elements so as an example

here we are going to be building a

character level language model so we're

simply going to be translating

individual characters into integers so

let me show you uh a chunk of code that

sort of does that for us so we're

building both the encoder and the

decoder

and let me just talk through what's

happening

here when we encode an arbitrary text

like hi there we're going to receive a

list of integers that represents that

string so for example 46 47 Etc and then

we also have the reverse mapping so we

can take this list and decode it to get

back the exact same string so it's

really just like a translation to

integers and back for arbitrary string

and for us it is done on a character

level

now the way this was achieved is we just

iterate over all the characters here and

create a lookup table from the character

to the integer and vice versa and then

to encode some string we simply

translate all the characters

individually and to decode it back we

use the reverse mapping and concatenate

all of it now this is only one of many

possible encodings or many possible sort

of tokenizers and it's a very simple one

but there's many other schemas that

people have come up with in practice so

for example Google uses a sentence

piece uh so sentence piece will also

encode text into um integers but in a

different schema and using a different

vocabulary and sentence piece is a

subword uh sort of tokenizer and what

that means is that um you're not

encoding entire words but you're not

also encoding individual characters it's

it's a subword unit level and that's

usually what's adopted in practice for

example also openai has this Library

called tick token that uses a bite pair

encode

tokenizer um and that's what GPT uses

and you can also just encode words into

like hell world into a list of integers

so as an example I'm using the Tik token

Library here I'm getting the encoding

for gpt2 or that was used for gpt2

instead of just having 65 possible

characters or tokens they have 50,000

tokens and so when they encode the exact

same string High there we only get a

list of three integers but those

integers are not between 0 and 64 they

are between Z and 5,

5,256 so basically you can trade off the

code book size and the sequence lengths

so you can have very long sequences of

integers with very small vocabularies or

we can have short um sequences of

integers with very large vocabularies

and so typically people use in practice

these subword encodings but I'd like to

keep our token ier very simple so we're

using character level tokenizer and that

means that we have very small code books

we have very simple encode and decode

functions uh but we do get very long

sequences as a result but that's the

level at which we're going to stick with

this lecture because it's the simplest

thing okay so now that we have an

encoder and a decoder effectively a

tokenizer we can tokenize the entire

training set of Shakespeare so here's a

chunk of code that does that and I'm

going to start to use the pytorch

library and specifically the torch.

tensor from the pytorch library so we're

going to take all of the text in tiny

Shakespeare encode it and then wrap it

into a torch. tensor to get the data

tensor so here's what the data tensor

looks like when I look at just the first

1,000 characters or the 1,000 elements

of it so we see that we have a massive

sequence of integers and this sequence

of integers here is basically an

identical translation of the first

10,000 characters

here so I believe for example that zero

is a new line character and maybe one

one is a space not 100% sure but from

now on the entire data set of text is

re-represented as just it's just

stretched out as a single very large uh

sequence of

integers let me do one more thing before

we move on here I'd like to separate out

our data set into a train and a

validation split so in particular we're

going to take the first 90% of the data

set and consider that to be the training

data for the Transformer and we're going

to withhold the last 10% at the end of

it to be the validation data and this

will help us understand to what extent

our model is overfitting so we're going

to basically hide and keep the

validation data on the side because we

don't want just a perfect memorization

of this exact Shakespeare we want a

neural network that sort of creates

Shakespeare like uh text and so it

should be fairly likely for it to

produce the actual like stowed away uh

true Shakespeare text um and so we're

going to use this to uh get a sense of

the overfitting okay so now we would

like to start plugging these text

sequences or integer sequences into the

Transformer so that it can train and

learn those patterns now the important

thing to realize is we're never going to

actually feed entire text into a

Transformer all at once that would be

computationally very expensive and

prohibitive so when we actually train a

Transformer on a lot of these data sets

we only work with chunks of the data set

and when we train the Transformer we

basically sample random little chunks

out of the training set and train on

just chunks at a time and these chunks

have basically some kind of a length and

some maximum length now the maximum

length typically at least in the code I

usually write is called block size you

can you can uh find it under different

names like context length or something

like that let's start with the block

size of just eight and let me look at

the first train data characters the

first block size plus one characters

I'll explain why plus one in a

second so this is the first nine

characters in the sequence in the

training set now what I'd like to point

out is that when you sample a chunk of

data like this so say the these nine

characters out of the training set this

actually has multiple examples packed

into it and uh that's because all of

these characters follow each other and

so what this thing is going to say when

we plug it into a Transformer is we're

going to actually simultaneously train

it to make prediction at every one of

these

positions now in the in a chunk of nine

characters there's actually eight indiv

ual examples packed in there so there's

the example that when 18 when in the

context of 18 47 likely comes next in a

context of 18 and 47 56 comes next in a

context of 18 47 56 57 can come next and

so on so that's the eight individual

examples let me actually spell it out

with

code so here's a chunk of code to

illustrate X are the inputs to the

Transformer it will just be the first

block size characters y will be the uh

next block size characters so it's

offset by one and that's because y are

the targets for each position in the

input and then here I'm iterating over

all the block size of eight and the

context is always all the characters in

x uh up to T and including T and the

target is always the teth character but

in the targets array y so let me just

run

this and basically it spells out what I

said in words uh these are the eight

examples hidden in a chunk of nine

characters that we uh sampled from the

training set I want to mention one more

thing we train on all the eight examples

here with context between one all the

way up to context of block size and we

train on that not just for computational

reasons because we happen to have the

sequence already or something like that

it's not just done for efficiency it's

also done um to make the Transformer

Network be used to seeing contexts all

the way from as little as one all the

way to block size and we'd like the

transform to be used to seeing

everything in between and that's going

to be useful later during inference

because while we're sampling we can

start the sampling generation with as

little as one character of context and

the Transformer knows how to predict the

next character with all the way up to

just context of one and so then it can

predict everything up to block size and

after block size we have to start

truncating because the Transformer will

will never um receive more than block

size inputs when it's predicting the

next

character Okay so we've looked at the

time dimension of the tensors that are

going to be feeding into the Transformer

there's one more Dimension to care about

and that is the batch Dimension and so

as we're sampling these chunks of text

we're going to be actually every time

we're going to feed them into a

Transformer we're going to have many

batches of multiple chunks of text that

are all like stacked up in a single

tensor and that's just done for

efficiency just so that we can keep the

gpus busy uh because they are very good

at parallel processing of um of data and

so we just want to process multiple

chunks all at the same time but those

chunks are processed completely

independently they don't talk to each

other and so on so let me basically just

generalize this and introduce a batch

Dimension here's a chunk of

code let me just run it and then I'm

going to explain what it

does so here because we're going to

start sampling random locations in the

data set to pull chunks from I am

setting the seed so that um in the

random number generator so that the

numbers I see here are going to be the

same numbers you see later if you try to

reproduce this now the batch size here

is how many independent sequences we are

processing every forward backward pass

of the

Transformer the block size as I

explained is the maximum context length

to make those predictions so let's say B

size four block size eight and then

here's how we get batch for any

arbitrary split if the split is a

training split then we're going to look

at train data otherwise at valid data

that gives us the data array and then

when I Generate random positions to grab

a chunk out of I actually grab I

actually generate batch size number of

Random offsets so because this is four

we are ex is going to be a uh four

numbers that are randomly generated

between zero and Len of data minus block

size so it's just random offsets into

the training

set and then X's as I explained are the

first first block size characters

starting at I the Y's are the offset by

one of that so just add plus one and

then we're going to get those chunks for

every one of integers I INX and use a

torch. stack to take all those uh uh

one-dimensional tensors as we saw here

and we're going to um stack them up at

rows and so they all become a row in a

4x8 tensor

so here's where I'm printing then when I

sample a batch XB and YB the inputs to

the Transformer now are the input X is

the 4x8 tensor four uh rows of eight

columns and each one of these is a chunk

of the training

set and then the targets here are in the

associated array Y and they will come in

to the Transformer all the way at the

end uh to um create the loss function

uh so they will give us the correct

answer for every single position inside

X and then these are the four

independent

rows so spelled out as we did

before uh this 4x8 array contains a

total of 32 examples and they're

completely independent as far as the

Transformer is

concerned uh so when the input is 24 the

target is 43 or rather 43 here in the Y

array

when the input is 2443 the target is

58 uh when the input is 24 43 58 the

target is 5 Etc or like when it is a 52

581 the target is

58 right so you can sort of see this

spelled out these are the 32 independent

examples packed in to a single batch of

the input X and then the desired targets

are in y and so now this integer tensor

of um X is going to feed into the

Transformer and that Transformer is

going to simultaneously process all

these examples and then look up the

correct um integers to predict in every

one of these positions in the tensor y

okay so now that we have our batch of

input that we'd like to feed into a

Transformer let's start basically

feeding this into neural networks now

we're going to start off with the

simplest possible neural network which

in the case of language modeling in my

opinion is the Byram language model and

we've covered the Byram language model

in my make more series in a lot of depth

and so here I'm going to sort of go

faster and let's just Implement pytorch

module directly that implements the byr

language

model so I'm importing the pytorch um NN

module uh for

reproducibility and then here I'm

constructing a Byram language model

which is a subass of NN

module and then I'm calling it and I'm

passing it the inputs and the targets

and I'm just printing now when the

inputs on targets come here you see that

I'm just taking the index uh the inputs

X here which I rename to idx and I'm

just passing them into this token

embedding table so it's going on here is

that here in the Constructor we are

creating a token embedding table and it

is of size vocap size by vocap

size and we're using an. embedding which

is a very thin wrapper around basically

a tensor of shape voap size by vocab

size and what's happening here is that

when we pass idx here every single

integer in our input is going to refer

to this embedding table and it's going

to pluck out a row of that embedding

table corresponding to its index so 24

here will go into the embedding table

and we'll pluck out the 24th row and

then 43 will go here and pluck out the

43d row Etc and then pytorch is going to

arrange all of this into a batch by Time

by channel uh tensor in this case batch

is four time is eight and C which is the

channels is vocab size or 65 and so

we're just going to pluck out all those

rows arrange them in a b by T by C and

now we're going to interpret this as the

logits which are basically the scores

for the next character in the sequence

and so what's happening here is we are

predicting what comes next based on just

the individual identity of a single

token and you can do that because um I

mean currently the tokens are not

talking to each other and they're not

seeing any context except for they're

just seeing themselves so I'm a f I'm a

token number five and then I can

actually make pretty decent predictions

about what comes next just by knowing

that I'm token five because some

characters uh know um C follow other

characters in in typical scenarios so we

saw a lot of this in a lot more depth in

the make more series and here if I just

run this then we currently get the

predictions the scores the lits for

every one of the 4x8 positions now that

we've made predictions about what comes

next we'd like to evaluate the loss

function and so in make more series we

saw that a good way to measure a loss or

like a quality of the predictions is to

use the negative log likelihood loss

which is also implemented in pytorch

under the name cross entropy so what we'

like to do here is loss is the cross

entropy on the predictions and the

targets and so this measures the quality

of the logits with respect to the

Targets in other words we have the

identity of the next character so how

well are we predicting the next

character based on the lits and

intuitively the correct um the correct

dimension of low jits uh depending on

whatever the target is should have a

very high number and all the other

dimensions should be very low number

right now the issue is that this won't

actually this is what we want we want to

basically output the logits and the

loss this is what we want but

unfortunately uh this won't actually run

we get an error message but intuitively

we want to uh measure this now when we

go to the pytorch um cross entropy

documentation here um we're trying to

call the cross entropy in its functional

form uh so that means we don't have to

create like a module for it but here

when we go to the documentation you have

to look into the details of how pitor

expects these inputs and basically the

issue here is ptor expects if you have

multi-dimensional input which we do

because we have a b BYT by C tensor then

it actually really wants the channels to

be the second uh Dimension here so if

you um so basically it wants a b by C

BYT instead of a b by T by C and so it's

just the details of how P torch treats

um these kinds of inputs and so we don't

actually want to deal with that so what

we're going to do instead is we need to

basically reshape our logits so here's

what I like to do I like to take

basically give names to the dimensions

so lit. shape is B BYT by C and unpack

those numbers and then let's uh say that

logits equals lit. View and we want it

to be a b * c b * T by C so just a two-

dimensional

array right so we're going to take all

the we're going to take all of these um

positions here and we're going to uh

stretch them out in a onedimensional

sequence and uh preserve the channel

Dimension as the second

dimension so we're just kind of like

stretching out the array so it's two-

dimensional and in that case it's going

to better conform to what pytorch uh

sort of expects in its Dimensions now we

have to do the same to targets because

currently targets are um of shape B by T

and we want it to be just B * T so

onedimensional now alternatively you

could always still just do minus one

because pytor will guess what this

should be if you want to lay it out uh

but let me just be explicit and say p *

t once we've reshaped this it will match

the cross entropy case and then we

should be able to evaluate our

loss okay so that R now and we can do

loss and So currently we see that the

loss is

4.87 now because our uh we have 65

possible vocabulary elements we can

actually guess at what the loss should

be and in

particular we covered negative log

likelihood in a lot of detail we are

expecting log or lawn of um 1 over 65

and negative of that so we're expecting

the loss to be about 4.1 17 but we're

getting 4.87 and so that's telling us

that the initial predictions are not uh

super diffuse they've got a little bit

of entropy and so we're guessing wrong

uh so uh yes but actually we're I a we

are able to evaluate the loss okay so

now that we can evaluate the quality of

the model on some data we'd like to also

be able to generate from the model so

let's do the generation now I'm going to

go again a little bit faster here

because I covered all this already in

previous

videos

so here's a generate function for the

model so we take some uh we take the the

same kind of input idx here and

basically this is the current uh context

of some characters in a batch in some

batch so it's also B BYT and the job of

generate is to basically take this B BYT

and extend it to be B BYT + 1 plus 2

plus 3 and so it's just basically it

continues the generation in all the

batch dimensions in the time Dimension

So that's its job and it will do that

for Max new tokens so you can see here

on the bottom there's going to be some

stuff here but on the bottom whatever is

predicted is concatenated on top of the

previous idx along the First Dimension

which is the time Dimension to create a

b BYT + one so that becomes a new idx so

the job of generate is to take a b BYT

and make it a b BYT plus 1 plus 2 plus

three as many as we want Max new tokens

so this is the generation from the model

now inside the generation what what are

we doing we're taking the current

indices we're getting the predictions so

we get uh those are in the low jits and

then the loss here is going to be

ignored because um we're not we're not

using that and we have no targets that

are sort of ground truth targets that

we're going to be comparing with

then once we get the logits we are only

focusing on the last step so instead of

a b by T by C we're going to pluck out

the negative-1 the last element in the

time Dimension because those are the

predictions for what comes next so that

gives us the logits which we then

convert to probabilities via softmax and

then we use tor. multinomial to sample

from those probabilities and we ask

pytorch to give us one sample and so idx

next will become a b by one because in

each uh one of the batch Dimensions

we're going to have a single prediction

for what comes next so this num samples

equals one will make this be a

one and then we're going to take those

integers that come from the sampling

process according to the probability

distribution given here and those

integers got just concatenated on top of

the current sort of like running stream

of integers and this gives us a b BYT +

one and then we can return that now one

thing here is you see how I'm calling

self of idx which will end up going to

the forward function I'm not providing

any Targets So currently this would give

an error because targets is uh is uh

sort of like not given so targets has to

be optional so targets is none by

default and then if targets is none then

there's no loss to create so it's just

loss is none but else all of this

happens and we can create a loss so this

will make it so um if we have the

targets we provide them and get a loss

if we have no targets it will'll just

get the

loits so this here will generate from

the model um and let's take that for a

ride

now oops so I have another code chunk

here which will generate for the model

from the model and okay this is kind of

crazy so maybe let me let me break this

down so these are the idx

right I'm creating a batch will be just

one time will be just one so I'm

creating a little one by one tensor and

it's holding a zero and the D type the

data type is uh integer so zero is going

to be how we kick off the generation and

remember that zero is uh is the element

standing for a new line character so

it's kind of like a reasonable thing to

to feed in as the very first character

in a sequence to be the new

line um so it's going to be idx which

we're going to feed in here then we're

going to ask for 100 tokens

and then. generate will continue that

now because uh generate works on the

level of batches we we then have to

index into the zero throw to basically

unplug the um the single batch Dimension

that exists and then that gives us a um

time steps just a onedimensional array

of all the indices which we will convert

to simple python list from pytorch

tensor so that that can feed into our

decode function and uh convert those

integers into text so let me bring this

back and we're generating 100 tokens

let's

run and uh here's the generation that we

achieved so obviously it's garbage and

the reason it's garbage is because this

is a totally random model so next up

we're going to want to train this model

now one more thing I wanted to point out

here is this function is written to be

General but it's kind of like ridiculous

right now because

we're feeding in all this we're building

out this context and we're concatenating

it all and we're always feeding it all

into the model but that's kind of

ridiculous because this is just a simple

Byram model so to make for example this

prediction about K we only needed this W

but actually what we fed into the model

is we fed the entire sequence and then

we only looked at the very last piece

and predicted K so the only reason I'm

writing it in this way is because right

now this is a byr model but I'd like to

keep keep this function fixed and I'd

like it to work um later when our

characters actually um basically look

further in the history and so right now

the history is not used so this looks

silly uh but eventually the history will

be used and so that's why we want to uh

do it this way so just a quick comment

on that so now we see that this is um

random so let's train the model so it

becomes a bit less random okay let's Now

train the model so first what I'm going

to do is I'm going to create a pyour

optimization object so here we are using

the optimizer ATM W um now in a make

more series we've only ever use tastic

gradi in descent the simplest possible

Optimizer which you can get using the

SGD instead but I want to use Adam which

is a much more advanced and popular

Optimizer and it works extremely well

for uh typical good setting for the

learning rate is roughly 3 E4 uh but for

very very small networks like is the

case here you can get away with much

much higher learning rates R3 or even

higher probably but let me create the

optimizer object which will basically

take the gradients and uh update the

parameters using the

gradients and then here our batch size

up above was only four so let me

actually use something bigger let's say

32 and then for some number of steps um

we are sampling a new batch of data

we're evaluating the loss uh we're

zeroing out all the gradients from the

previous step getting the gradients for

all the parameters and then using those

gradients to up update our parameters so

typical training loop as we saw in the

make more series so let me now uh run

this for say 100 iterations and let's

see what kind of losses we're going to

get so we started around

4.7 and now we're getting to down to

like 4.6 4.5 Etc so the optimization is

definitely happening but um let's uh

sort of try to increase number of

iterations and only print at the

end because we probably want train for

longer okay so we're down to 3.6

roughly roughly down to

three this is the most janky

optimization okay it's working let's

just do

10,000 and then from here we want to

copy this and hopefully that we're going

to get something reason and of course

it's not going to be Shakespeare from a

byr model but at least we see that the

loss is improving and uh hopefully we're

expecting something a bit more

reasonable okay so we're down at about

2.5 is let's see what we get okay

dramatic improvements certainly on what

we had here so let me just increase the

number of tokens okay so we see that

we're starting to get something at least

like reasonable is

um certainly not shakes spear but uh the

model is making progress so that is the

simplest possible

model so now what I'd like to do

is obviously this is a very simple model

because the tokens are not talking to

each other so given the previous context

of whatever was generated we're only

looking at the very last character to

make the predictions about what comes

next so now these uh now these tokens

have to start talking to each other and

figuring out what is in the context so

that they can make better predictions

for what comes next and this is how

we're going to kick off the uh

Transformer okay so next I took the code

that we developed in this juper notebook

and I converted it to be a script and

I'm doing this because I just want to

simplify our intermediate work into just

the final product that we have at this

point so in the top here I put all the

hyp parameters that we to find I

introduced a few and I'm going to speak

to that in a little bit otherwise a lot

of this should be recognizable uh

reproducibility read data get the

encoder and the decoder create the train

into splits uh use the uh kind of like

data loader um that gets a batch of the

inputs and Targets this is new and I'll

talk about it in a second now this is

the Byram language model that we

developed and it can forward and give us

a logits and loss and it can

generate and then here we are creating

the optimizer and this is the training

Loop so everything here should look

pretty familiar now some of the small

things that I added number one I added

the ability to run on a GPU if you have

it so if you have a GPU then you can

this will use Cuda instead of just CPU

and everything will be a lot more faster

now when device becomes Cuda then we

need to make sure that when we load the

data we move it to

device when we create the model we want

to move uh the model parameters to

device so as an example here we have the

N an embedding table and it's got a

weight inside it which stores the uh

sort of lookup table so so that would be

moved to the GPU so that all the

calculations here happen on the GPU and

they can be a lot faster and then

finally here when I'm creating the

context that feeds in to generate I have

to make sure that I create it on the

device number two what I introduced is

uh the fact that here in the training

Loop here I was just printing the um l.

item inside the training Loop but this

is a very noisy measurement of the

current loss because every batch will be

more or less lucky and so what I want to

do usually um is uh I have an estimate

loss function and the estimate loss

basically then um goes up here and it

averages up the loss over multiple

batches so in particular we're going to

iterate eval iter times and we're going

to basically get our loss and then we're

going to get the average loss for both

splits and so this will be a lot less

noisy so here when we call the estimate

loss we're we're going to report the uh

pretty accurate train and validation

loss now when we come back up you'll

notice a few things here I'm setting the

model to evaluation phase and down here

I'm resetting it back to training phase

now right now for our model as is this

doesn't actually do anything because the

only thing inside this model is this uh

nn. embedding and um this this um

Network would behave both would behave

the same in both evaluation mode and

training mode we have no drop off layers

we have no batm layers Etc but it is a

good practice to Think Through what mode

your neural network is in because some

layers will have different Behavior Uh

at inference time or training time and

there's also this context manager torch

up nograd and this is just telling

pytorch that everything that happens

inside this function we will not call do

backward on and so pytorch can be a lot

more efficient with its memory use

because it doesn't have to store all the

intermediate variables uh because we're

never going to call backward and so it

can it can be a lot more memory

efficient in that way so also a good

practice to tpy torch when we don't

intend to do back

propagation so right now this script is

about 120 lines of code of and that's

kind of our starter code I'm calling it

b.p and I'm going to release it later

now running this

script gives us output in the terminal

and it looks something like this it

basically as I ran this code uh it was

giving me the train loss and Val loss

and we see that we convert to somewhere

around

2.5 with the pyr model and then here's

the sample that we produced at the

end and so we have everything packaged

up in the script and we're in a good

position now to iterate on this okay so

we are almost ready to start writing our

very first self attention block for

processing these uh tokens now before we

actually get there I want to get you

used to a mathematical trick that is

used in the self attention inside a

Transformer and is really just like at

the heart of an an efficient

implementation of self attention and so

I want to work with this toy example to

just get you used to this operation and

then it's going to make it much more

clear once we actually get to um to it

uh in the script

again so let's create a b BYT by C where

BT and C are just 48 and two in the toy

example and these are basically channels

and we have uh batches and we have the

time component and we have information

at each point in the sequence so

see now what we would like to do is we

would like these um tokens so we have up

to eight tokens here in a batch and

these eight tokens are currently not

talking to each other and we would like

them to talk to each other we'd like to

couple them and in particular we don't

we we want to couple them in a very

specific way so the token for example at

the fifth location it should not

communicate with tokens in the sixth

seventh and eighth location

because uh those are future tokens in

the sequence the token on the fifth

location should only talk to the one in

the fourth third second and first so

it's only so information only flows from

previous context to the current time

step and we cannot get any information

from the future because we are about to

try to predict the

future so what is the easiest way for

tokens to communicate okay the easiest

way I would say is okay if we're up to

if we're a fifth token and I'd like to

communicate with my past the simplest

way we can do that is to just do a

weight is to just do an average of all

the um of all the preceding elements so

for example if I'm the fif token I would

like to take the channels uh that make

up that are information at my step but

then also the channels from the fourth

step third step second step and the

first step I'd like to average those up

and then that would become sort of like

a feature Vector that summarizes me in

the context of my history now of course

just doing a sum or like an average is

an extremely weak form of interaction

like this communication is uh extremely

lossy we've lost a ton of information

about the spatial Arrangements of all

those tokens uh but that's okay for now

we'll see how we can bring that

information back later for now what we

would like to do is for every single

batch element independently for every

teeth token in that sequence we'd like

to now calculate the average of all the

vectors in all the previous tokens and

also at this token so let's write that

out um I have a small snippet here and

instead of just fumbling around let me

just copy paste it and talk to

it so in other words we're going to

create X and B is short for bag of words

because bag of words is um is kind of

like um a term that people use when you

are just averaging up things so this is

just a bag of words basically there's a

word stored on every one of these eight

locations and we're doing a bag of words

we're just averaging

so in the beginning we're going to say

that it's just initialized at Zero and

then I'm doing a for Loop here so we're

not being efficient yet that's coming

but for now we're just iterating over

all the batch Dimensions independently

iterating over time and then the

previous uh tokens are at this uh batch

Dimension and then everything up to and

including the teeth token okay so when

we slice out X in this way X prev

Becomes of shape um how many T elements

there were in the past and then of

course C so all the two-dimensional

information from these little tokens so

that's the previous uh sort of chunk of

um tokens from my current sequence and

then I'm just doing the average or the

mean over the zero Dimension so I'm

averaging out the time here and I'm just

going to get a little c one dimensional

Vector which I'm going to store in X bag

of words so I can run this and and uh

this is not going to be very informative

because let's see so this is X of Zer so

this is the zeroth batch element and

then expo at zero now you see how the at

the first location here you see that the

two are equal and that's because it's

we're just doing an average of this one

token but here this one is now an

average of these two and now this one is

an average of these

three and so on

so uh and this last one is the average

of all of these elements so vertical

average just averaging up all the tokens

now gives this outcome

here so this is all well and good uh but

this is very inefficient now the trick

is that we can be very very efficient

about doing this using matrix

multiplication so that's the

mathematical trick and let me show you

what I mean let's work with the toy

example here let me run it and I'll

explain I have a simple Matrix here that

is a 3X3 of all ones a matrix B of just

random numbers and it's a 3x2 and a

matrix C which will be 3x3 multip 3x2

which will give out a 3x2 so here we're

just using um matrix multiplication so a

multiply B gives us

C okay so how are these numbers in C um

achieved right so this number in the top

left is the first row of a dot product

with the First Column of B and since all

the the row of a right now is all just

ones then the do product here with with

this column of B is just going to do a

sum of these of this column so 2 + 6 + 6

is

14 the element here in the output of C

is also the first column here the first

row of a multiplied now with the second

column of B so 7 + 4 + 5 is 16 now you

see that there's repeating elements here

so this 14 again is because this row is

again all ones and it's multiplying the

First Column of B so we get 14 and this

one is and so on so this last number

here is the last row do product last

column now the trick here is uh the

following this is just a boring number

of um it's just a boring array of all

ones but torch has this function called

Trail which is short for a

triangular uh something like that and

you can wrap it in torch up once and it

will just return the lower triangular

portion of this

okay so now it will basically zero out

uh these guys here so we just get the

lower triangular part well what happens

if we do

that so now we'll have a like this and B

like this and now what are we getting

here in C well what is this number well

this is the first row times the First

Column and because this is zeros

uh these elements here are now ignored

so we just get a two and then this

number here is the first row times the

second column and because these are

zeros they get ignored and it's just

seven this seven multiplies this one but

look what happened here because this is

one and then zeros we what ended up

happening is we're just plucking out the

row of this row of B and that's what we

got now here we have one 1 Z so here 110

do product with these two columns will

now give us 2 + 6 which is 8 and 7 + 4

which is 11 and because this is 111 we

ended up with the addition of all of

them and so basically depending on how

many ones and zeros we have here we are

basically doing a sum currently of a

variable number of these rows and that

gets deposited into

C So currently we're doing sums because

these are ones but we can also do

average right and you can start to see

how we could do average uh of the rows

of B uh sort of in an incremental

fashion because we don't have to we can

basically normalize these rows so that

they sum to one and then we're going to

get an average so if we took a and then

we did aals

aide torch. sum in the um of a in the um

oneth Dimension and then let's keep them

as true so so therefore the broadcasting

will work out so if I rerun this you see

now that these rows now sum to one so

this row is one this row is 0. 5.5 Z and

here we get 1/3 and now when we do a

multiply B what are we getting here we

are just getting the first row first row

here now we are getting the average of

the first two

rows okay so 2 and six average is four

and four and seven average is

5.5 and on the bottom here we are now

getting the average of these three rows

so the average of all of elements of B

are now deposited here and so you can

see that by manipulating these uh

elements of this multiplying Matrix and

then multiplying it with any given

Matrix we can do these averages in this

incremental fashion because we just get

um and we can manipulate that based on

the elements of a okay so that's very

convenient so let's let's swing back up

here and see how we can vectorize this

and make it much more efficient using

what we've learned so in

particular we are going to produce an

array a but here I'm going to call it we

short for weights but this is our

a and this is how much of every row we

want to average up and it's going to be

an average because you can see that

these rows sum to

one so this is our a and then our B in

this example of course is X

so what's going to happen here now is

that we are going to have an expo

2 and this Expo 2 is going to be way

multiplying

RX so let's think this true way is T BYT

and this is Matrix multiplying in

pytorch a b by T by

C and it's giving us uh different what

shape so pytorch will come here and it

will see that these shapes are not the

same so it will create a batch Dimension

here and this is a batched matrix

multiply and so it will apply this

matrix multiplication in all the batch

elements um in parallel and individually

and then for each batch element there

will be a t BYT multiplying T by C

exactly as we had

below so this will now create B by T by

C and Expo 2 will now become identical

to Expo

so we can see that torch. all close of

xbo and xbo 2 should be true

now so this kind of like convinces us

that uh these are in fact um the same so

xbo and xbo 2 if I just print

them uh okay we're not going to be able

to okay we're not going to be able to

just stare it down but

um well let me try Expo basically just

at the zeroth element and Expo two at

the zeroth element so just the first

batch and we should see that this and

that should be identical which they

are right so what happened here the

trick is we were able to use batched

Matrix multiply to do this uh

aggregation really and it's a weighted

aggregation and the weights are

specified in this um T BYT array and

we're basically doing weighted sums and

uh these weighted sums are are U

according to uh the weights inside here

they take on sort of this triangular

form and so that means that a token at

the teth dimension will only get uh sort

of um information from the um tokens

perceiving it so that's exactly what we

want and finally I would like to rewrite

it in one more way and we're going to

see why that's useful so this is the

third version and it's also identical to

the first and second but let me talk

through it it uses

softmax so Trill here is this Matrix

lower triangular

ones way begins as all

zero okay so if I just print way in the

beginning it's all zero then I

used masked fill so what this is doing

is we. masked fill it's all zeros and

I'm saying for all the elements where

Trill is equal equal Z make them be

negative Infinity so all the elements

where Trill is zero will become negative

Infinity now so this is what we get and

then the final line here is

softmax so if I take a softmax along

every single so dim is negative one so

along every single row if I do softmax

what is that going to

do well softmax is um is also like a

normalization operation right and so

spoiler alert you get the exact same

Matrix let me bring back to

softmax and recall that in softmax we're

going to exponentiate every single one

of these and then we're going to divide

by the sum and so if we exponentiate

every single element here we're going to

get a one and here we're going to get uh

basically zero 0 z0 Z everywhere else

and then when we normalize we just get

one here we're going to get one one and

then zeros and then softmax will again

divide and this will give us 5.5 and so

on and so this is also the uh the same

way to produce uh this mask now the

reason that this is a bit more

interesting and the reason we're going

to end up using it in self

attention is that these weights here

begin uh with zero and you can think of

this as like an interaction strength or

like an affinity so basically it's

telling us how much of each uh token

from the past do we want to Aggregate

and average up

and then this line is saying tokens from

the past cannot communicate by setting

them to negative Infinity we're saying

that we will not aggregate anything from

those

tokens and so basically this then goes

through softmax and through the weighted

and this is the aggregation through

matrix

multiplication and so what this is now

is you can think of these as um these

zeros are currently just set by us to be

zero but a quick preview is that these

affinities between the tokens are not

going to be just constant at zero

they're going to be data dependent these

tokens are going to start looking at

each other and some tokens will find

other tokens more or less interesting

and depending on what their values are

they're going to find each other

interesting to different amounts and I'm

going to call those affinities I think

and then here we are saying the future

cannot communicate with the past we're

we're going to clamp them and then when

we normalize and sum we're going to

aggregate uh sort of their values

depending on how interesting they find

each other and so that's the preview for

self attention and basically long story

short from this entire section is that

you can do weighted aggregations of your

past

Elements by having by using matrix

multiplication of a lower triangular

fashion and then the elements here in

the lower triangular part are telling

you how much of each element uh fuses

into this position so we're going to use

this trick now to develop the self

attention block block so first let's get

some quick preliminaries out of the way

first the thing I'm kind of bothered by

is that you see how we're passing in

vocap size into the Constructor there's

no need to do that because vocap size is

already defined uh up top as a global

variable so there's no need to pass this

stuff

around next what I want to do is I don't

want to actually create I want to create

like a level of indirection here where

we don't directly go to the embedding

for the um logits but instead we go

through this intermediate phase because

we're going to start making that bigger

so let me introduce a new variable n

embed it shorted for number of embedding

Dimensions so

nbed here will be say 32 that was a

suggestion from GitHub co-pilot by the

way um it also suest 32 which is a good

number so this is an embedding table and

only 32 dimensional

embeddings so then here this is not

going to give us logits directly instead

this is going to give us token

embeddings that's I'm going to call it

and then to go from the token Tings to

the logits we're going to need a linear

layer so self. LM head let's call it

short for language modeling head is n

and linear from n ined up to vocap size

and then when we swing over here we're

actually going to get the loits by

exactly what the co-pilot says now we

have to be careful here because this C

and this C are not equal um this is nmed

C and this is vocap size so let's just

say that n ined is equal to

C and then this just creates one spous

layer of interaction through a linear

layer but uh this should basically

run so we see that this runs and uh this

currently looks kind of spous but uh

we're going to build on top of this now

next up so far we've taken these indices

and we've encoded them based on the

identity of the uh tokens in inside idx

the next thing that people very often do

is that we're not just encoding the

identity of these tokens but also their

position so we're going to have a second

position uh embedding table here so

self. position embedding table is an an

embedding of block size by an embed and

so each position from zero to block size

minus one will also get its own

embedding vector and then here first let

me decode B BYT from idx do

shape and then here we're also going to

have a pause embedding which is the

positional embedding and these are this

is to arrange so this will be basically

just integers from Z to T minus one and

all of those integers from 0 to T minus

one get embedded through the table to

create a t by

C and then here this gets renamed to

just say x and x will be the addition of

the token embeddings with the positional

embeddings and here the broadcasting

note will work out so B by T by C plus T

by C

this gets right aligned a new dimension

of one gets added and it gets

broadcasted across

batch so at this point x holds not just

the token identities but the positions

at which these tokens occur and this is

currently not that useful because of

course we just have a simple byr model

so it doesn't matter if you're in the

fifth position the second position or

wherever it's all translation invariant

at this stage uh so this information

currently wouldn't help uh but as we

work on the self attention block we'll

see that this starts to matter

okay so now we get the Crux of self

attention so this is probably the most

important part of this video to

understand we're going to implement a

small self attention for a single

individual head as they're called so we

start off with where we were so all of

this code is familiar so right now I'm

working with an example where I Chang

the number of channels from 2 to 32 so

we have a 4x8 arrangement of tokens and

each to and the information each token

is currently 32 dimensional but we just

are working with random

numbers now we saw here that the code as

we had it before does a uh simple weight

simple average of all the past tokens

and the current token so it's just the

previous information and current

information is just being mixed together

in an average and that's what this code

currently achieves and it Doo by

creating this lower triangular structure

which allows us to mask out this uh we

uh Matrix that we create so we mask it

out and then we normalize it and

currently when we initialize the

affinities between all the different

sort of tokens or nodes I'm going to use

those terms

interchangeably so when we initialize

the affinities between all the different

tokens to be zero then we see that way

gives us this um structure where every

single row has these um uniform numbers

and so that's what that's what then uh

in this Matrix multiply makes it so that

we're doing a simple

average now we don't actually want this

to be all uniform because different uh

tokens will find different other tokens

more or less interesting and we want

that to be data dependent so for example

if I'm a vowel then maybe I'm looking

for consonants in my past and maybe I

want to know what those consonants are

and I want that information to flow to

me and so I want to now gather

information from the past but I want to

do it in the data dependent way and this

is the problem that self attention

solves now the way self attention solves

this is the following every single node

or every single token at each position

will emit two vectors it will emit a

query and it will emit a

key now the query Vector roughly

speaking is what am I looking for and

the key Vector roughly speaking is what

do I

contain and then the way we get

affinities between these uh tokens now

in a sequence is we basically just do a

do product between the keys and the

queries so my query dot products with

all the keys of all the other tokens and

that dot product now becomes

wayy and so um if the key and the query

are sort of aligned they will interact

to a very high amount and then I will

get to learn more about that specific

token as opposed to any other token in

the sequence

so let's implement this

now we're going to implement a

single what's called head of self

attention so this is just one head

there's a hyper parameter involved with

these heads which is the head size and

then here I'm initializing linear

modules and I'm using bias equals false

so these are just going to apply a

matrix multiply with some fixed

weights and now let me produce a key and

q k and Q by forwarding these modules on

X so the size of this will now

become B by T by 16 because that is the

head size and the same here B by T by

16 so this being the head size so you

see here that when I forward this linear

on top of my X all the tokens in all the

positions in the B BYT Arrangement all

of them them in parallel and

independently produce a key and a query

so no communication has happened

yet but the communication comes now all

the queries will do product with all the

keys so basically what we want is we

want way now or the affinities between

these to be query multiplying key but we

have to be careful with uh we can't

Matrix multiply this we actually need to

transpose uh K but we have to be also

careful because these are when you have

The Bash Dimension so in particular we

want to transpose uh the last two

dimensions dimension1 and dimension -2

so

-21 and so this Matrix multiply now will

basically do the following B by T by

16 Matrix multiplies B by 16 by T to

give us B by T by

T right

so for every row of B we're now going to

have a t Square Matrix giving us the

affinities and these are now the way so

they're not zeros they are now coming

from this dot product between the keys

and the queries so this can now run I

can I can run this and the weighted

aggregation now is a function in a data

Bandon manner between the keys and

queries of these nodes so just

inspecting what happened

here the way takes on this form

and you see that before way was uh just

a constant so it was applied in the same

way to all the batch elements but now

every single batch elements will have

different sort of we because uh every

single batch element contains different

uh tokens at different positions and so

this is not data dependent so when we

look at just the zeroth uh Row for

example in the input these are the

weights that came out and so you can see

now that they're not just exactly

uniform um and in particular as an

example here for the last row this was

the eighth token and the eighth token

knows what content it has and it knows

at what position it's in and now the E

token based on that uh creates a query

hey I'm looking for this kind of stuff

um I'm a vowel I'm on the E position I'm

looking for any consonant at positions

up to four and then all the nodes get to

emit keys and maybe one of the channels

could be I am a I am a consonant and I

am in a position up to four and that

that key would have a high number in

that specific Channel and that's how the

query and the key when they do product

they can find each other and create a

high affinity and when they have a high

Affinity like say uh this token was

pretty interesting to uh to this eighth

token when they have a high Affinity

then through the softmax I will end up

aggregating a lot of its information

into my position and so I'll get to

learn a lot about

it now just this we're looking at way

after this has already happened um let

me erase this operation as well so let

me erase the masking and the softmax

just to show you the under the hood

internals and how that works so without

the masking in the softmax Whey comes

out like this right this is the outputs

of the do products um and these are the

raw outputs and they take on values from

negative you know two to positive two

Etc so that's the raw interactions and

raw affinities between all the nodes but

now if I'm going if I'm a fifth node I

will not want to aggregate anything from

the sixth node seventh node and the

eighth node so actually we use the upper

triangular masking so those are not

allowed to

communicate and now we actually want to

have a nice uh distribution uh so we

don't want to aggregate negative .11 of

this node that's crazy so instead we

exponentiate and normalize and now we

get a nice distribution that sums to one

and this is telling us now in the data

dependent manner how much of information

to aggregate from any of these tokens in

the

past so that's way and it's not zeros

anymore but but it's calculated in this

way now there's one more uh part to a

single self attention head and that is

that when we do the aggregation we don't

actually aggregate the tokens exactly we

aggregate we produce one more value here

and we call that the

value so in the same way that we

produced p and query we're also going to

create a value

and

then here we don't

aggregate X we calculate a v which is

just achieved by uh propagating this

linear on top of X again and then we

output way multiplied by V so V is the

elements that we aggregate or the the

vectors that we aggregate instead of the

raw

X and now of course uh this will make it

so that the output here of this single

head will be 16 dimensional because that

is the head

size so you can think of X as kind of

like private information to this token

if you if you think about it that way so

X is kind of private to this token so

I'm a fifth token at some and I have

some identity and uh my information is

kept in Vector X and now for the

purposes of the single head here's what

I'm interested in here's what I have and

if you find me interesting here's what I

will communicate to you and that's

stored in v and so V is the thing that

gets aggregated for the purposes of this

single head between the different

notes and that's uh basically the self

attention mechanism this is this is what

it does there are a few notes that I

would make like to make about attention

number one attention is a communication

mechanism you can really think about it

as a communication mechanism where you

have a number of nodes in a directed

graph where basically you have edges

pointed between noes like

this and what happens is every node has

some Vector of information and it gets

to aggregate information via a weighted

sum from all of the nodes that point to

it and this is done in a data dependent

manner so depending on whatever data is

actually stored that you should not at

any point in time now our graph doesn't

look like this our graph has a different

structure we have eight nodes because

the block size is eight and there's

always eight to

tokens and uh the first node is only

pointed to by itself the second node is

pointed to by the first node and itself

all the way up to the eighth node which

is pointed to by all the previous nodes

and itself and so that's the structure

that our directed graph has or happens

happens to have in Auto regressive sort

of scenario like language modeling but

in principle attention can be applied to

any arbitrary directed graph and it's

just a communication mechanism between

the nodes the second note is that notice

that there is no notion of space so

attention simply acts over like a set of

vectors in this graph and so by default

these nodes have no idea where they are

positioned in the space and that's why

we need to encode them positionally and

sort of give them some information that

is anchored to a specific position so

that they sort of know where they are

and this is different than for example

from convolution because if you're run

for example a convolution operation over

some input there's a very specific sort

of layout of the information in space

and the convolutional filters sort of

act in space and so it's it's not like

an attention in ATT ention is just a set

of vectors out there in space they

communicate and if you want them to have

a notion of space you need to

specifically add it which is what we've

done when we calculated the um relative

the positional encode encodings and

added that information to the vectors

the next thing that I hope is very clear

is that the elements across the batch

Dimension which are independent examples

never talk to each other they're always

processed independently and this is a

batched matrix multiply that applies

basically a matrix multiplication uh

kind of in parallel across the batch

dimension so maybe it would be more

accurate to say that in this analogy of

a directed graph we really have because

the back size is four we really have

four separate pools of eight nodes and

those eight nodes only talk to each

other but in total there's like 32 nodes

that are being processed uh but there's

um sort of four separate pools of eight

you can look at it that way the next

note is that here in the case of

language modeling uh we have this

specific uh structure of directed graph

where the future tokens will not

communicate to the Past tokens but this

doesn't necessarily have to be the

constraint in the general case and in

fact in many cases you may want to have

all of the uh noes talk to each other uh

fully so as an example if you're doing

sentiment analysis or something like

that with a Transformer you might have a

number of tokens and you may want to

have them all talk to each other fully

because later you are predicting for

example the sentiment of the sentence

and so it's okay for these NOS to talk

to each other and so in those cases you

will use an encoder block of self

attention and uh all it means that it's

an encoder block is that you will delete

this line of code allowing all the noes

to completely talk to each other what

we're implementing here is sometimes

called a decoder block and it's called a

decoder because it is sort of like a

decoding language and it's got this

autor regressive format where you have

to mask with the Triangular Matrix so

that uh nodes from the future never talk

to the Past because they would give away

the answer

and so basically in encoder blocks you

would delete this allow all the noes to

talk in decoder blocks this will always

be present so that you have this

triangular structure uh but both are

allowed and attention doesn't care

attention supports arbitrary

connectivity between nodes the next

thing I wanted to comment on is you keep

me you keep hearing me say attention

self attention Etc there's actually also

something called cross attention what is

the

difference

so basically the reason this attention

is self attention is because because the

keys queries and the values are all

coming from the same Source from X so

the same Source X produces Keys queries

and values so these nodes are self

attending but in principle attention is

much more General than that so for

example an encoder decoder Transformers

uh you can have a case where the queries

are produced from X but the keys and the

values come from a whole separate

external source and sometimes from uh

encoder blocks that encode some context

that we'd like to condition on

and so the keys and the values will

actually come from a whole separate

Source those are nodes on the side and

here we're just producing queries and

we're reading off information from the

side so cross attention is used when

there's a separate source of nodes we'd

like to pull information from into our

nodes and it's self attention if we just

have nodes that would like to look at

each other and talk to each other so

this attention here happens to be self

attention but in principle um attention

is a lot more General okay and the last

note at this stage is if we come to the

attention is all need paper here we've

already implemented attention so given

query key and value we've U multiplied

the query and a key we've soft maxed it

and then we are aggregating the values

there's one more thing that we're

missing here which is the dividing by

one / square root of the head size the

DK here is the head size why are they

doing this finds this important so they

call it the scaled attention and it's

kind of like an important normalization

to basically

have the problem is if you have unit gsh

and inputs so zero mean unit variance K

and Q are unit gashin then if you just

do we naively then you see that your we

actually will be uh the variance will be

on the order of head size which in our

case is 16 but if you multiply by one

over head size square root so this is

square root and this is one

over then the variance of we will be one

so it will be

preserved now why is this important

you'll not notice that way

here will feed into

softmax and so it's really important

especially at initialization that we be

fairly diffuse so in our case here we

sort of locked out here and we had a

fairly diffuse numbers here so um like

this now the problem is that because of

softmax if weight takes on very positive

and very negative numbers inside it

softmax will actually converge towards

one hot vectors and so I can illustrate

that here um say we are applying softmax

to a tensor of values that are very

close to zero then we're going to get a

diffuse thing out of

softmax but the moment I take the exact

same thing and I start sharpening it

making it bigger by multiplying these

numbers by eight for example you'll see

that the softmax will start to sharpen

and in fact it will sharpen towards the

max so it will sharpen towards whatever

number here is the highest and so um

basically we don't want these values to

be too extreme especially at

initialization otherwise softmax will be

way too peaky and um you're basically