hi everyone so in this video I'd like us

to cover the process of tokenization in

large language models now you see here

that I have a set face and that's

because uh tokenization is my least

favorite part of working with large

language models but unfortunately it is

necessary to understand in some detail

because it it is fairly hairy gnarly and

there's a lot of hidden foot guns to be

aware of and a lot of oddness with large

language models typically traces back to

tokenization so what is

tokenization now in my previous video

Let's Build GPT from scratch uh we

actually already did tokenization but we

did a very naive simple version of

tokenization so when you go to the

Google colab for that video uh you see

here that we loaded our training set and

our training set was this uh Shakespeare

uh data set now in the beginning the

Shakespeare data set is just a large

string in Python it's just text and so

the question is how do we plug text into

large language models and in this case

here we created a vocabulary of 65

possible characters that we saw occur in

this string these were the possible

characters and we saw that there are 65

of them and then we created a a lookup

table for converting from every possible

character a little string piece into a

token an

integer so here for example we tokenized

the string High there and we received

this sequence of

tokens and here we took the first 1,000

characters of our data set and we

encoded it into tokens and because it is

this is character level we received

1,000 tokens in a sequence so token 18

47

Etc now later we saw that the way we

plug these tokens into the language

model is by using an embedding

table and so basically if we have 65

possible tokens then this embedding

table is going to have 65 rows and

roughly speaking we're taking the

integer associated with every single

sing Le token we're using that as a

lookup into this table and we're

plucking out the corresponding row and

this row is a uh is trainable parameters

that we're going to train using back

propagation and this is the vector that

then feeds into the Transformer um and

that's how the Transformer Ser of

perceives every single

token so here we had a very naive

tokenization process that was a

character level tokenizer but in

practice in state-ofthe-art uh language

models people use a lot more complicated

schemes unfortunately

uh for constructing these uh token

vocabularies so we're not dealing on the

Character level we're dealing on chunk

level and the way these um character

chunks are constructed is using

algorithms such as for example the bik

pair in coding algorithm which we're

going to go into in detail um and cover

in this video I'd like to briefly show

you the paper that introduced a bite

level encoding as a mechanism for

tokenization in the context of large

language models and I would say that

that's probably the gpt2 paper and if

you scroll down here to the section

input representation this is where they

cover tokenization the kinds of

properties that you'd like the

tokenization to have and they conclude

here that they're going to have a

tokenizer where you have a vocabulary of

50,2 57 possible

tokens and the context size is going to

be 1,24 tokens so in the in in the

attention layer of the Transformer

neural network

every single token is attending to the

previous tokens in the sequence and it's

going to see up to 1,24 tokens so tokens

are this like fundamental unit um the

atom of uh large language models if you

will and everything is in units of

tokens everything is about tokens and

tokenization is the process for

translating strings or text into

sequences of tokens and uh vice versa

when you go into the Llama 2 paper as

well I can show you that when you search

token you're going to get get 63 hits um

and that's because tokens are again

pervasive so here they mentioned that

they trained on two trillion tokens of

data and so

on so we're going to build our own

tokenizer luckily the bite be encoding

algorithm is not uh that super

complicated and we can build it from

scratch ourselves and we'll see exactly

how this works before we dive into code

I'd like to give you a brief Taste of

some of the complexities that come from

the tokenization because I just want to

make sure that we motivate it

sufficiently for why we are doing all

this and why this is so gross so

tokenization is at the heart of a lot of

weirdness in large language models and I

would advise that you do not brush it

off a lot of the issues that may look

like just issues with the new network

architecture or the large language model

itself are actually issues with the

tokenization and fundamentally Trace uh

back to it so if you've noticed any

issues with large language models can't

you know not able to do spelling tasks

very easily that's usually due to

tokenization simple string processing

can be difficult for the large language

model to perform

natively uh non-english languages can

work much worse and to a large extent

this is due to

tokenization sometimes llms are bad at

simple arithmetic also can trace be

traced to

tokenization uh gbt2 specifically would

have had quite a bit more issues with

python than uh future versions of it due

to tokenization there's a lot of other

issues maybe you've seen weird warnings

about a trailing whites space this is a

tokenization issue um

if you had asked GPT earlier about solid

gold Magikarp and what it is you would

see the llm go totally crazy and it

would start going off about a completely

unrelated tangent topic maybe you've

been told to use yl over Json in

structure data all of that has to do

with tokenization so basically

tokenization is at the heart of many

issues I will look back around to these

at the end of the video but for now let

me just um skip over it a little bit and

let's go to this web app um the Tik

tokenizer bell.app so I have it loaded

here and what I like about this web app

is that tokenization is running a sort

of live in your browser in JavaScript so

you can just type here stuff hello world

and the whole string

rokenes so here what we see on uh the

left is a string that you put in on the

right we're currently using the gpt2

tokenizer we see that this string that I

pasted here is currently tokenizing into

300 tokens and here they are sort of uh

shown explicitly in different colors for

every single token so for example uh

this word tokenization became two tokens

the token

3,642 and

1,634 the token um space is is token 318

so be careful on the bottom you can show

white space and keep in mind that there

are spaces and uh sln new line

characters in here but you can hide them

for

clarity the token space at is token 379

the to the Token space the is 262 Etc so

you notice here that the space is part

of that uh token

chunk now so this is kind of like how

our English sentence broke up and that

seems all well and good now now here I

put in some arithmetic so we see that uh

the token 127 Plus and then token six

space 6 followed by 77 so what's

happening here is that 127 is feeding in

as a single token into the large

language model but the um number 677

will actually feed in as two separate

tokens and so the large language model

has to sort of um take account of that

and process it correctly in its Network

and see here 804 will be broken up into

two tokens and it's is all completely

arbitrary and here I have another

example of four-digit numbers and they

break up in a way that they break up and

it's totally arbitrary sometimes you

have um multiple digits single token

sometimes you have individual digits as

many tokens and it's all kind of pretty

arbitrary and coming out of the

tokenizer here's another example we have

the string egg and you see here that

this became two

tokens but for some reason when I say I

have an egg you see when it's a space

egg it's two token it's sorry it's a

single token so just egg by itself in

the beginning of a sentence is two

tokens but here as a space egg is

suddenly a single token uh for the exact

same string okay here lowercase egg

turns out to be a single token and in

particular notice that the color is

different so this is a different token

so this is case sensitive and of course

a capital egg would also be different

tokens and again um this would be two

tokens arbitrarily so so for the same

concept egg depending on if it's in the

beginning of a sentence at the end of a

sentence lowercase uppercase or mixed

all this will be uh basically very

different tokens and different IDs and

the language model has to learn from raw

data from all the internet text that

it's going to be training on that these

are actually all the exact same concept

and it has to sort of group them in the

parameters of the neural network and

understand just based on the data

patterns that these are all very similar

but maybe not almost exactly similar but

but very very similar

um after the EG demonstration here I

have um an introduction from open a eyes

chbt in Korean so manaso Pang uh Etc uh

so this is in Korean and the reason I

put this here is because you'll notice

that um non-english languages work

slightly worse in Chachi part of this is

because of course the training data set

for Chachi is much larger for English

and for everything else but the same is

true not just for the large language

model itself but also for the tokenizer

so when we train the tokenizer we're

going to see that there's a training set

as well and there's a lot more English

than non-english and what ends up

happening is that we're going to have a

lot more longer tokens for

English so how do I put this if you have

a single sentence in English and you

tokenize it you might see that it's 10

tokens or something like that but if you

translate that sentence into say Korean

or Japanese or something else you'll

typically see that the number of tokens

used is much larger and that's because

the chunks here are a lot more broken up

so we're using a lot more tokens for the

exact same thing and what this does is

it bloats up the sequence length of all

the documents so you're using up more

tokens and then in the attention of the

Transformer when these tokens try to

attend each other you are running out of

context um in the maximum context length

of that Transformer and so basically all

the non-english text is stretched out

from the perspective of the Transformer

and this just has to do with the um

trainings that used for the tokenizer

and the tokenization itself so it will

create a lot bigger tokens and a lot

larger groups in English and it will

have a lot of little boundaries for all

the other non-english text um so if we

translated this into English it would be

significantly fewer

tokens the final example I have here is

a little snippet of python for doing FS

buuz and what I'd like you to notice is

look all these individual spaces are all

separate tokens they are token

220 so uh 220 220 220 220 and then space

if is a single token and so what's going

on here is that when the Transformer is

going to consume or try to uh create

this text it needs to um handle all

these spaces individually they all feed

in one by one into the entire

Transformer in the sequence and so this

is being extremely wasteful tokenizing

it in this way and so as a result of

that gpt2 is not very good with python

and it's not anything to do with coding

or the language model itself it's just

that if he use a lot of indentation

using space in Python like we usually do

uh you just end up bloating out all the

text and it's separated across way too

much of the sequence and we are running

out of the context length in the

sequence uh that's roughly speaking

what's what's happening we're being way

too wasteful we're taking up way too

much token space now we can also scroll

up here and we can change the tokenizer

so note here that gpt2 tokenizer creates

a token count of 300 for this string

here we can change it to CL 100K base

which is the GPT for tokenizer and we

see that the token count drops to 185 so

for the exact same string we are now

roughly having the number of tokens and

roughly speaking this is because uh the

number of tokens in the GPT 4 tokenizer

is roughly double that of the number of

tokens in the gpt2 tokenizer so we went

went from roughly 50k to roughly 100K

now you can imagine that this is a good

thing because the same text is now

squished into half as many tokens so uh

this is a lot denser input to the

Transformer and in the Transformer every

single token has a finite number of

tokens before it that it's going to pay

attention to and so what this is doing

is we're roughly able to see twice as

much text as a context for what token to

predict next uh because of this change

but of course just increasing the number

of tokens is uh not strictly better

infinitely uh because as you increase

the number of tokens now your embedding

table is um sort of getting a lot larger

and also at the output we are trying to

predict the next token and there's the

soft Max there and that grows as well

we're going to go into more detail later

on this but there's some kind of a Sweet

Spot somewhere where you have a just

right number of tokens in your

vocabulary where everything is

appropriately dense and still fairly

efficient now one thing I would like you

to note specifically for the gp4

tokenizer is that the handling of the

white space for python has improved a

lot you see that here these four spaces

are represented as one single token for

the three spaces here and then the token

SPF and here seven spaces were all

grouped into a single token so we're

being a lot more efficient in how we

represent Python and this was a

deliberate Choice made by open aai when

they designed the gp4 tokenizer and they

group a lot more space into a single

character what this does is this

densifies Python and therefore we can

attend to more code before it when we're

trying to predict the next token in the

sequence and so the Improvement in the

python coding ability from gbt2 to gp4

is not just a matter of the language

model and the architecture and the

details of the optimization but a lot of

the Improvement here is also coming from

the design of the tokenizer and how it

groups characters into tokens okay so

let's now start writing some code

so remember what we want to do we want

to take strings and feed them into

language models for that we need to

somehow tokenize strings into some

integers in some fixed vocabulary and

then we will use those integers to make

a look up into a lookup table of vectors

and feed those vectors into the

Transformer as an input now the reason

this gets a little bit tricky of course

is that we don't just want to support

the simple English alphabet we want to

support different kinds of languages so

this is anango in Korean which is hello

and we also want to support many kinds

of special characters that we might find

on the internet for example

Emoji so how do we feed this text into

uh

Transformers well how's the what is this

text anyway in Python so if you go to

the documentation of a string in Python

you can see that strings are immutable

sequences of Unicode code

points okay what are Unicode code points

we can go to PDF so Unicode code points

are defined by the Unicode Consortium as

part of the Unicode standard and what

this is really is that it's just a

definition of roughly 150,000 characters

right now and roughly speaking what they

look like and what integers um represent

those characters so it says 150,000

characters across 161 scripts as of

right now so if you scroll down here you

can see that the standard is very much

alive the latest standard 15.1 in

September

2023 and basically this is just a way to

define lots of types of

characters like for example all these

characters across different scripts so

the way we can access the unic code code

Point given Single Character is by using

the or function in Python so for example

I can pass in Ord of H and I can see

that for the Single Character H the unic

code code point is

104 okay um but this can be arbitr

complicated so we can take for example

our Emoji here and we can see that the

code point for this one is

128,000 or we can take

un and this is 50,000 now keep in mind

you can't plug in strings here because

you uh this doesn't have a single code

point it only takes a single uni code

code Point character and tells you its

integer so in this way we can look

up all the um characters of this

specific string and their code points so

or of X forx in this string and we get

this encoding here now see here we've

already turned the raw code points

already have integers so why can't we

simply just use these integers and not

have any tokenization at all why can't

we just use this natively as is and just

use the code Point well one reason for

that of course is that the vocabulary in

that case would be quite long so in this

case for Unicode the this is a

vocabulary of

150,000 different code points but more

worryingly than that I think the Unicode

standard is very much alive and it keeps

changing and so it's not kind of a

stable representation necessarily that

we may want to use directly so for those

reasons we need something a bit better

so to find something better we turn to

encodings so if we go to the Wikipedia

page here we see that the Unicode

consortion defines three types of

encodings utf8 UTF 16 and UTF 32 these

encoding are the way by which we can

take Unicode text and translate it into

binary data or by streams utf8 is by far

the most common uh so this is the utf8

page now this Wikipedia page is actually

quite long but what's important for our

purposes is that utf8 takes every single

Cod point and it translates it to a by

stream and this by stream is between one

to four bytes so it's a variable length

encoding so depending on the Unicode

Point according to the schema you're

going to end up with between 1 to four

bytes for each code point on top of that

there's utf8 uh

utf16 and UTF 32 UTF 32 is nice because

it is fixed length instead of variable

length but it has many other downsides

as well so the full kind of spectrum of

pros and cons of all these different

three encodings are beyond the scope of

this video I just like to point out that

I enjoyed this block post and this block

post at the end of it also has a number

of references that can be quite useful

uh one of them is uh utf8 everywhere

Manifesto um and this Manifesto

describes the reason why utf8 is

significantly preferred and a lot nicer

than the other encodings and why it is

used a lot more prominently um on the

internet one of the major advantages

just just to give you a sense is that

utf8 is the only one of these that is

backwards compatible to the much simpler

asky encoding of text um but I'm not

going to go into the full detail in this

video so suffice to say that we like the

utf8 encoding and uh let's try to take

the string and see what we get if we

encoded into

utf8 the string class in Python actually

has do encode and you can give it the

encoding which is say utf8 now we get

out of this is not very nice because

this is the bytes is a bytes object and

it's not very nice in the way that it's

printed so I personally like to take it

through list because then we actually

get the raw B

of this uh encoding so this is the raw

byes that represent this string

according to the utf8 en coding we can

also look at utf16 we get a slightly

different by stream and we here we start

to see one of the disadvantages of utf16

you see how we have zero Z something Z

something Z something we're starting to

get a sense that this is a bit of a

wasteful encoding and indeed for simple

asky characters or English characters

here uh we just have the structure of 0

something Z something and it's not

exactly nice same for UTF 32 when we

expand this we can start to get a sense

of the wastefulness of this encoding for

our purposes you see a lot of zeros

followed by

something and so uh this is not

desirable so suffice it to say that we

would like to stick with utf8 for our

purposes however if we just use utf8

naively these are by streams so that

would imply a vocabulary length of only

256 possible tokens uh but this this

vocabulary size is very very small what

this is going to do if we just were to

use it naively is that all of our text

would be stretched out over very very

long sequences of bytes and so

um what what this does is that certainly

the embeding table is going to be tiny

and the prediction at the top at the

final layer is going to be very tiny but

our sequences are very long and remember

that we have pretty finite um context

length and the attention that we can

support in a transformer for

computational reasons and so we only

have as much context length but now we

have very very long sequences and this

is just inefficient and it's not going

to allow us to attend to sufficiently

long text uh before us for the purposes

of the next token prediction task so we

don't want to use the raw bytes of the

utf8 encoding we want to be able to

support larger vocabulary size that we

can tune as a hyper

but we want to stick with the utf8

encoding of these strings so what do we

do well the answer of course is we turn

to the bite pair encoding algorithm

which will allow us to compress these

bite sequences um to a variable amount

so we'll get to that in a bit but I just

want to briefly speak to the fact that I

would love nothing more than to be able

to feed raw bite sequences into uh

language models in fact there's a paper

about how this could potentially be done

uh from Summer last last year now the

problem is you actually have to go in

and you have to modify the Transformer

architecture because as I mentioned

you're going to have a problem where the

attention will start to become extremely

expensive because the sequences are so

long and so in this paper they propose

kind of a hierarchical structuring of

the Transformer that could allow you to

just feed in raw bites and so at the end

they say together these results

establish the viability of tokenization

free autor regressive sequence modeling

at scale so tokenization free would

indeed be amazing we would just feed B

streams directly into our models but

unfortunately I don't know that this has

really been proven out yet by

sufficiently many groups and a

sufficient scale uh but something like

this at one point would be amazing and I

hope someone comes up with it but for

now we have to come back and we can't

feed this directly into language models

and we have to compress it using the B

paare encoding algorithm so let's see

how that works so as I mentioned the B

paare encoding algorithm is not all that

complicated and the Wikipedia page is

actually quite instructive as far as the

basic idea goes go what we're doing is

we have some kind of a input sequence uh

like for example here we have only four

elements in our vocabulary a b c and d

and we have a sequence of them so

instead of bytes let's say we just have

four a vocab size of

four the sequence is too long and we'd

like to compress it so what we do is

that we iteratively find the pair of uh

tokens that occur the most

frequently and then once we've

identified that pair we repl replace

that pair with just a single new token

that we append to our vocabulary so for

example here the bite pair AA occurs

most often so we mint a new token let's

call it capital Z and we replace every

single occurrence of AA by Z so now we

have two Z's here so here we took a

sequence of 11 characters with

vocabulary size four and we've converted

it to a um sequence of only nine tokens

but now with a vocabulary of five

because we have a fifth vocabulary

element that we just created and it's Z

standing for concatination of AA and we

can again repeat this process so we

again look at the sequence and identify

the pair of tokens that are most

frequent let's say that that is now AB

well we are going to replace AB with a

new token that we meant call Y so y

becomes ab and then every single

occurrence of ab is now replaced with y

so we end up with this so now we only

have 1 2 3 4 5 6 seven characters in our

sequence but we have not just um four

vocabulary elements or five but now we

have six and for the final round we

again look through the sequence find

that the phrase zy or the pair zy is

most common and replace it one more time

with another um character let's say x so

X is z y and we replace all curses of zy

and we get this following sequence so

basically after we have gone through

this process instead of having a um

sequence of

11 uh tokens with a vocabulary length of

four we now have a sequence of 1 2 3

four five tokens but our vocabulary

length now is seven and so in this way

we can iteratively compress our sequence

I we Mint new tokens so in the in the

exact same way we start we start out

with bite sequences so we have 256

vocabulary size but we're now going to

go through these and find the bite pairs

that occur the most and we're going to

iteratively start minting new tokens

appending them to our vocabulary and

replacing things and in this way we're

going to end up with a compressed

training data set and also an algorithm

for taking any arbitrary sequence and

encoding it using this uh vocabul

and also decoding it back to Strings so

let's now Implement all that so here's

what I did I went to this block post

that I enjoyed and I took the first

paragraph and I copy pasted it here into

text so this is one very long line

here now to get the tokens as I

mentioned we just take our text and we

encode it into utf8 the tokens here at

this point will be a raw bites single

stream of bytes and just so that it's

easier to work with instead of just a

bytes object I'm going to convert all

those bytes to integers and then create

a list of it just so it's easier for us

to manipulate and work with in Python

and visualize and here I'm printing all

of that so this is the original um this

is the original paragraph and its length

is

533 uh code points and then here are the

bytes encoded in ut utf8 and we see that

this has a length of 616 bytes at this

point or 616 tokens and the reason this

is more is because a lot of these simple

asky characters or simple characters

they just become a single bite but a lot

of these Unicode more complex characters

become multiple bytes up to four and so

we are expanding that

size so now what we'd like to do as a

first step of the algorithm is we'd like

to iterate over here and find the pair

of bites that occur most frequently

because we're then going to merge it so

if you are working long on a notebook on

a side then I encourage you to basically

click on the link find this notebook and

try to write that function yourself

otherwise I'm going to come here and

Implement first the function that finds

the most common pair okay so here's what

I came up with there are many different

ways to implement this but I'm calling

the function get stats it expects a list

of integers I'm using a dictionary to

keep track of basically the counts and

then this is a pythonic way to iterate

consecutive elements of this list uh

which we covered in the previous video

and then here I'm just keeping track of

just incrementing by one um for all the

pairs so if I call this on all the

tokens here then the stats comes out

here so this is the dictionary the keys

are these topples of consecutive

elements and this is the count so just

to uh print it in a slightly better way

this is one way that I like to do that

where you it's a little bit compound

here so you can pause if you like but we

iterate all all the items the items

called on dictionary returns pairs of

key value and instead I create a list

here of value key because if it's a

value key list then I can call sort on

it and by default python will uh use the

first element which in this case will be

value to sort by if it's given tles and

then reverse so it's descending and

print that so basically it looks like

101 comma 32 was the most commonly

occurring consecutive pair and it

occurred 20 times we can double check

that that makes reasonable sense so if I

just search

10132 then you see that these are the 20

occurrences of that um pair and if we'd

like to take a look at what exactly that

pair is we can use Char which is the

opposite of or in Python so we give it a

um unic code Cod point so 101 and of 32

and we see that this is e and space so

basically there's a lot of E space here

meaning that a lot of these words seem

to end with e so here's eace as an

example so there's a lot of that going

on here and this is the most common pair

so now that we've identified the most

common pair we would like to iterate

over this sequence we're going to Mint a

new token with the ID of

256 right because these tokens currently

go from Z to 255 so when we create a new

token it will have an ID of

256 and we're going to iterate over this

entire um list and every every time we

see 101 comma 32 we're going to swap

that out for

256 so let's Implement that now and feel

free to uh do that yourself as well so

first I commented uh this just so we

don't pollute uh the notebook too much

this is a nice way of in Python

obtaining the highest ranking pair so

we're basically calling the Max on this

dictionary stats and this will return

the maximum

key and then the question is how does it

rank keys so you can provide it with a

function that ranks keys and that

function is just stats. getet uh stats.

getet would basically return the value

and so we're ranking by the value and

getting the maximum key so it's 101

comma 32 as we saw now to actually merge

10132 um this is the function that I

wrote but again there are many different

versions of it so we're going to take a

list of IDs and the the pair that we

want to replace and that pair will be

replaced with the new index

idx so iterating through IDs if we find

the pair swap it out for idx so we

create this new list and then we start

at zero and then we go through this

entire list sequentially from left to

right and here we are checking for

equality at the current position with

the

pair um so here we are checking that the

pair matches now here is a bit of a

tricky condition that you have to append

if you're trying to be careful and that

is that um you don't want this here to

be out of Bounds at the very last

position when you're on the rightmost

element of this list otherwise this

would uh give you an autof bounds error

so we have to make sure that we're not

at the very very last element so uh this

would be false for that so if we find a

match we append to this new list that

replacement index and we increment the

position by two so we skip over that

entire pair but otherwise if we we

haven't found a matching pair we just

sort of copy over the um element at that

position and increment by one then

return this so here's a very small toy

example if we have a list 566 791 and we

want to replace the occurrences of 67

with 99 then calling this on that will

give us what we're asking for so here

the 67 is replaced with

99 so now I'm going to uncomment this

for our actual use case where we want to

take our tokens we want to take the top

pair here and replace it with 256 to get

tokens to if we run this we get the

following so recall that previously we

had a length 616 in this list and now we

have a length 596 right so this

decreased by 20 which makes sense

because there are 20 occurrences

moreover we can try to find 256 here and

we see plenty of occurrences on off it

and moreover just double check there

should be no occurrence of 10132 so this

is the original array plenty of them and

in the second array there are no

occurrences of 1032 so we've

successfully merged this single pair and

now we just uh iterate this so we are

going to go over the sequence again find

the most common pair and replace it so

let me now write a y Loop that uses

these functions to do this um sort of

iteratively and how many times do we do

it four well that's totally up to us as

a hyper parameter

the more um steps we take the larger

will be our vocabulary and the shorter

will be our sequence and there is some

sweet spot that we usually find works

the best in practice and so this is kind

of a hyperparameter and we tune it and

we find good vocabulary sizes as an

example gp4 currently uses roughly

100,000 tokens and um bpark that those

are reasonable numbers currently instead

the are large language models so let me

now write uh putting putting it all

together and uh iterating these steps

okay now before we dive into the Y loop

I wanted to add one more cell here where

I went to the block post and instead of

grabbing just the first paragraph or two

I took the entire block post and I

stretched it out in a single line and

basically just using longer text will

allow us to have more representative

statistics for the bite Pairs and we'll

just get a more sensible results out of

it because it's longer text um so here

we have the raw text we encode it into

bytes using the utf8 encoding

and then here as before we are just

changing it into a list of integers in

Python just so it's easier to work with

instead of the raw byes objects and then

this is the code that I came up with uh

to actually do the merging in Loop these

two functions here are identical to what

we had above I only included them here

just so that you have the point of

reference here so uh these two are

identical and then this is the new code

that I added so the first first thing we

want to do is we want to decide on the

final vocabulary size that we want our

tokenizer to have and as I mentioned

this is a hyper parameter and you set it

in some way depending on your best

performance so let's say for us we're

going to use 276 because that way we're

going to be doing exactly 20

merges and uh 20 merges because we

already have

256 tokens for the raw bytes and to

reach 276 we have to do 20 merges uh to

add 20 new

tokens here uh this is uh one way in

Python to just create a copy of a list

so I'm taking the tokens list and by

wrapping it in a list python will

construct a new list of all the

individual elements so this is just a

copy

operation then here I'm creating a

merges uh dictionary so this merges

dictionary is going to maintain

basically the child one child two

mapping to a new uh token and so what

we're going to be building up here is a

binary tree of merges but actually it's

not exactly a tree because a tree would

have a single root node with a bunch of

leaves for us we're starting with the

leaves on the bottom which are the

individual bites those are the starting

256 tokens and then we're starting to

like merge two of them at a time and so

it's not a tree it's more like a forest

um uh as we merge these elements

so for 20 merges we're going to find the

most commonly occurring pair we're going

to Mint a new token integer for it so I

here will start at zero so we'll going

to start at 256 we're going to print

that we're merging it and we're going to

replace all of the occurrences of that

pair with the new new lied token and

we're going to record that this pair of

integers merged into this new

integer so running this gives us the

following

output so we did 20 merges and for

example the first merge was exactly as

before the

10132 um tokens merging into a new token

2556 now keep in mind that the

individual uh tokens 101 and 32 can

still occur in the sequence after

merging it's only when they occur

exactly consecutively that that becomes

256

now um and in particular the other thing

to notice here is that the token 256

which is the newly minted token is also

eligible for merging so here on the

bottom the 20th merge was a merge of 25

and 259 becoming

275 so every time we replace these

tokens they become eligible for merging

in the next round of data ration so

that's why we're building up a small

sort of binary Forest instead of a

single individual

tree one thing we can take a look at as

well is we can take a look at the

compression ratio that we've achieved so

in particular we started off with this

tokens list um so we started off with

24,000 bytes and after merging 20 times

uh we now have only

19,000 um tokens and so therefore the

compression ratio simply just dividing

the two is roughly 1.27 so that's the

amount of compression we were able to

achieve of this text with only 20

merges um and of course the more

vocabulary elements you add uh the

greater the compression ratio here would

be finally so that's kind of like um the

training of the tokenizer if you will

now 1 Point I wanted to make is that and

maybe this is a diagram that can help um

kind of illustrate is that tokenizer is

a completely separate object from the

large language model itself so

everything in this lecture we're not

really touching the llm itself uh we're

just training the tokenizer this is a

completely separate pre-processing stage

usually so the tokenizer will have its

own training set just like a large

language model has a potentially

different training set so the tokenizer

has a training set of documents on which

you're going to train the

tokenizer and then and um we're

performing The Bite pair encoding

algorithm as we saw above to train the

vocabulary of this

tokenizer so it has its own training set

it is a pre-processing stage that you

would run a single time in the beginning

um and the tokenizer is trained using

bipar coding algorithm once you have the

tokenizer once it's trained and you have

the vocabulary and you have the merges

uh we can do both encoding and decoding

so these two arrows here so the

tokenizer is a translation layer between

raw text which is as we saw the sequence

of Unicode code points it can take raw

text and turn it into a token sequence

and vice versa it can take a token

sequence and translate it back into raw

text so now that we have trained uh

tokenizer and we have these merges we

are going to turn to how we can do the

encoding and the decoding step if you

give me text here are the tokens and

vice versa if you give me tokens here's

the text once we have that we can

translate between these two Realms and

then the language model is going to be

trained as a step two afterwards and

typically in a in a sort of a

state-of-the-art application you might

take all of your training data for the

language model and you might run it

through the tokenizer and sort of

translate everything into a massive

token sequence and then you can throw

away the raw text you're just left with

the tokens themselves and those are

stored on disk and that is what the

large language model is actually reading

when it's training on them so this one

approach that you can take as a single

massive pre-processing step a

stage um so yeah basically I think the

most important thing I want to get

across is that this is completely

separate stage it usually has its own

entire uh training set you may want to

have those training sets be different

between the tokenizer and the logge

language model so for example when

you're training the tokenizer as I

mentioned we don't just care about the

performance of English text we care

about uh multi many different languages

and we also care about code or not code

so you may want to look into different

kinds of mixtures of different kinds of

languages and different amounts of code

and things like that because the amount

of different language that you have in

your tokenizer training set will

determine how many merges of it there

will be and therefore that determines

the density with which uh this type of

data is um sort of has in the token

space and so roughly speaking

intuitively if you add some amount of

data like say you have a ton of Japanese

data in your uh tokenizer training set

then that means that more Japanese

tokens will get merged

and therefore Japanese will have shorter

sequences uh and that's going to be

beneficial for the large language model

which has a finite context length on

which it can work on in in the token

space uh so hopefully that makes sense

so we're now going to turn to encoding

and decoding now that we have trained a

tokenizer so we have our merges and now

how do we do encoding and decoding okay

so let's begin with decoding which is

this Arrow over here so given a token

sequence let's go through the tokenizer

to get back a python string object so

the raw text so this is the function

that we' like to implement um we're

given the list of integers and we want

to return a python string if you'd like

uh try to implement this function

yourself it's a fun exercise otherwise

I'm going to start uh pasting in my own

solution so there are many different

ways to do it um here's one way I will

create an uh kind of pre-processing

variable that I will call

vocab and vocab is a mapping or a

dictionary in Python for from the token

uh ID to the bytes object for that token

so we begin with the raw bytes for

tokens from 0 to 255 and then we go in

order of all the merges and we sort of

uh populate this vocab list by doing an

addition here so this is the basically

the bytes representation of the first

child followed by the second one and

remember these are bytes objects so this

addition here is an addition of two

bytes objects just concatenation

so that's what we get

here one tricky thing to be careful with

by the way is that I'm iterating a

dictionary in Python using a DOT items

and uh it really matters that this runs

in the order in which we inserted items

into the merous dictionary luckily

starting with python 3.7 this is

guaranteed to be the case but before

python 3.7 this iteration may have been

out of order with respect to how we

inserted elements into merges and this

may not have worked but we are using an

um modern python so we're okay and then

here uh given the IDS the first thing

we're going to do is get the

tokens so the way I implemented this

here is I'm taking I'm iterating over

all the IDS I'm using vocap to look up

their bytes and then here this is one

way in Python to concatenate all these

bytes together to create our tokens and

then these tokens here at this point are

raw bytes so I have to decode using UTF

F now back into python strings so

previously we called that encode on a

string object to get the bytes and now

we're doing it Opposite we're taking the

bytes and calling a decode on the bytes

object to get a string in Python and

then we can return

text so um this is how we can do it now

this actually has a um issue um in the

way I implemented it and this could

actually throw an error so try to think

figure out why this code could actually

result in an error if we plug in um uh

some sequence of IDs that is

unlucky so let me demonstrate the issue

when I try to decode just something like

97 I am going to get letter A here back

so nothing too crazy happening but when

I try to decode 128 as a single element

the token 128 is what in string or in

Python object uni Cod decoder utfa can't

Decode by um 0x8 which is this in HEX in

position zero invalid start bite what

does that mean well to understand what

this means we have to go back to our

utf8 page uh that I briefly showed

earlier and this is Wikipedia utf8 and

basically there's a specific schema that

utfa bytes take so in particular if you

have a multi-te object for some of the

Unicode characters they have to have

this special sort of envelope in how the

encoding works and so what's happening

here is that invalid start pite that's

because

128 the binary representation of it is

one followed by all zeros so we have one

and then all zero and we see here that

that doesn't conform to the format

because one followed by all zero just

doesn't fit any of these rules so to

speak so it's an invalid start bite

which is byte one this one must have a

one following it and then a zero

following it and then the content of

your uni codee in x here so basically we

don't um exactly follow the utf8

standard and this cannot be decoded and

so the way to fix this um is to

use this errors equals in bytes. decode

function of python and by default errors

is strict so we will throw an error if

um it's not valid utf8 bytes encoding

but there are many different things that

you could put here on error handling

this is the full list of all the errors

that you can use and in particular

instead of strict let's change it to

replace and that will replace uh with

this special marker this replacement

character so errors equals replace and

now we just get that character

back so basically not every single by

sequence is valid

utf8 and if it happens that your large

language model for example predicts your

tokens in a bad manner then they might

not fall into valid utf8 and then we

won't be able to decode them so the

standard practice is to basically uh use

errors equals replace and this is what

you will also find in the openai um code

that they released as well but basically

whenever you see um this kind of a

character in your output in that case uh

something went wrong and the LM output

not was not valid uh sort of sequence of

tokens okay and now we're going to go

the other way so we are going to

implement

this Arrow right here where we are going

to be given a string and we want to

encode it into

tokens so this is the signature of the

function that we're interested in and um

this should basically print a list of

integers of the tokens so again uh try

to maybe implement this yourself if

you'd like a fun exercise uh and pause

here otherwise I'm going to start

putting in my

solution so again there are many ways to

do this so um this is one of the ways

that sort of I came came up with so the

first thing we're going to do is we are

going

to uh take our text encode it into utf8

to get the raw bytes and then as before

we're going to call list on the bytes

object to get a list of integers of

those bytes so those are the starting

tokens those are the raw bytes of our

sequence but now of course according to

the merges dictionary above and recall

this was the

merges some of the bytes may be merged

according to this lookup in addition to

that remember that the merges was built

from top to bottom and this is sort of

the order in which we inserted stuff

into merges and so we prefer to do all

these merges in the beginning before we

do these merges later because um for

example this merge over here relies on

the 256 which got merged here so we have

to go in the order from top to bottom

sort of if we are going to be merging

anything now we expect to be doing a few

merges so we're going to be doing W

true um and now we want to find a pair

of byes that is consecutive that we are

allowed to merge according to this in

order to reuse some of the functionality

that we've already written I'm going to

reuse the function uh get

stats so recall that get stats uh will

give us the we'll basically count up how

many times every single pair occurs in

our sequence of tokens and return that

as a dictionary and the dictionary was a

mapping from all the different uh by

pairs to the number of times that they

occur right um at this point we don't

actually care how many times they occur

in the sequence we only care what the

raw pairs are in that sequence and so

I'm only going to be using basically the

keys of the dictionary I only care about

the set of possible merge candidates if

that makes

sense now we want to identify the pair

that we're going to be merging at this

stage of the loop so what do we want we

want to find the pair or like the a key

inside stats that has the lowest index

in the merges uh dictionary because we

want to do all the early merges before

we work our way to the late

merges so again there are many different

ways to implement this but I'm going to

do something a little bit fancy

here so I'm going to be using the Min

over an iterator in Python when you call

Min on an iterator and stats here as a

dictionary we're going to be iterating

the keys of this dictionary in Python so

we're looking at all the pairs inside

stats um which are all the consecutive

Pairs and we're going to be taking the

consecutive pair inside tokens that has

the minimum what the Min takes a key

which gives us the function that is

going to return a value over which we're

going to do the Min and the one we care

about is we're we care about taking

merges and basically getting um that

pairs

index so basically for any pair inside

stats we are going to be looking into

merges at what index it has and we want

to get the pair with the Min number so

as an example if there's a pair 101 and

32 we definitely want to get that pair

uh we want to identify it here and

return it and pair would become 10132 if

it

occurs and the reason that I'm putting a

float INF here as a fall back is that in

the get function when we call uh when we

basically consider a pair that doesn't

occur in the merges then that pair is

not eligible to be merged right so if in

the token sequence there's some pair

that is not a merging pair it cannot be

merged then uh it doesn't actually occur

here and it doesn't have an index and uh

it cannot be merged which we will denote

as float INF and the reason Infinity is

nice here is because for sure we're

guaranteed that it's not going to

participate in the list of candidates

when we do the men so uh so this is one

way to do it so B basically long story

short this Returns the most eligible

merging candidate pair uh that occurs in

the tokens now one thing to be careful

with here is this uh function here might

fail in the following way if there's

nothing to merge then uh uh then there's

nothing in merges um that satisfi that

is satisfied anymore there's nothing to

merge everything just returns float imps

and then the pair I think will just

become the very first element of stats

um but this pair is not actually a

mergeable pair it just becomes the first

pair inside stats arbitrarily because

all of these pairs evaluate to float in

for the merging Criterion so basically

it could be that this this doesn't look

succeed because there's no more merging

pairs so if this pair is not in merges

that was returned then this is a signal

for us that actually there was nothing

to merge no single pair can be merged

anymore in that case we will break

out um nothing else can be

merged you may come up with a different

implementation by the way this is kind

of like really trying hard in

Python um but really we're just trying

to find a pair that can be merged with

the lowest index

here now if we did find a pair that is

inside merges with the lowest index then

we can merge it

so we're going to look into the merger

dictionary for that pair to look up the

index and we're going to now merge that

into that index so we're going to do

tokens equals and we're going to

replace the original tokens we're going

to be replacing the pair pair and we're

going to be replacing it with index idx

and this returns a new list of tokens

where every occurrence of pair is

replaced with idx so we're doing a merge

and we're going to be continuing this

until eventually nothing can be merged

we'll come out here and we'll break out

and here we just return

tokens and so that that's the

implementation I think so hopefully this

runs okay cool um yeah and this looks uh

reasonable so for example 32 is a space

in asky so that's here um so this looks

like it worked great okay so let's wrap

up this section of the video at least I

wanted to point out that this is not

quite the right implementation just yet

because we are leaving out a special

case so in particular if uh we try to do

this this would give us an error and the

issue is that um if we only have a

single character or an empty string then

stats is empty and that causes an issue

inside Min so one way to fight this is

if L of tokens is at least two because

if it's less than two it's just a single

token or no tokens then let's just uh

there's nothing to merge so we just

return so that would fix uh that

case Okay and then second I have a few

test cases here for us as well so first

let's make sure uh about or let's note

the following if we take a string and we

try to encode it and then decode it back

you'd expect to get the same string back

right is that true for all

strings so I think uh so here it is the

case and I think in general this is

probably the case um but notice that

going backwards is not is not you're not

going to have an identity going

backwards because as I mentioned us not

all token sequences are valid utf8 uh

sort of by streams and so so therefore

you're some of them can't even be

decodable um so this only goes in One

Direction but for that one direction we

can check uh here if we take the

training text which is the text that we

train to tokenizer around we can make

sure that when we encode and decode we

get the same thing back which is true

and here I took some validation data so

I went to I think this web page and I

grabbed some text so this is text that

the tokenizer has not seen and we can

make sure that this also works um okay

so that gives us some confidence that

this was correctly implemented

so those are the basics of the bite pair

encoding algorithm we saw how we can uh

take some training set train a tokenizer

the parameters of this tokenizer really

are just this dictionary of merges and

that basically creates the little binary

Forest on top of raw

bites once we have this the merges table

we can both encode and decode between

raw text and token sequences so that's

the the simplest setting of The

tokenizer what we're going to do now

though is we're going to look at some of

the St the art lar language models and

the kinds of tokenizers that they use

and we're going to see that this picture

complexifies very quickly so we're going

to go through the details of this comp

complexification one at a time so let's

kick things off by looking at the GPD

Series so in particular I have the gpt2

paper here um and this paper is from

2019 or so so 5 years ago and let's

scroll down to input representation this

is where they talk about the tokenizer

that they're using for gpd2 now this is

all fairly readable so I encourage you

to pause and um read this yourself but

this is where they motivate the use of

the bite pair encoding algorithm on the

bite level representation of utf8

encoding so this is where they motivate

it and they talk about the vocabulary

sizes and everything now everything here

is exactly as we've covered it so far

but things start to depart around here

so what they mention is that they don't

just apply the naive algorithm as we

have done it and in particular here's a

example suppose that you have common

words like dog what will happen is that

dog of course occurs very frequently in

the text and it occurs right next to all

kinds of punctuation as an example so

doc dot dog exclamation mark dog

question mark Etc and naively you might

imagine that the BP algorithm could

merge these to be single tokens and then

you end up with lots of tokens that are

just like dog with a slightly different

punctuation and so it feels like you're

clustering things that shouldn't be

clustered you're combining kind of

semantics with

uation and this uh feels suboptimal and

indeed they also say that this is

suboptimal according to some of the

experiments so what they want to do is

they want to top down in a manual way

enforce that some types of um characters

should never be merged together um so

they want to enforce these merging rules

on top of the bite PA encoding algorithm

so let's take a look um at their code

and see how they actually enforce this

and what kinds of mergy they actually do

perform so I have to to tab open here

for gpt2 under open AI on GitHub and

when we go to

Source there is an encoder thatp now I

don't personally love that they call it

encoder dopy because this is the

tokenizer and the tokenizer can do both

encode and decode uh so it feels kind of

awkward to me that it's called encoder

but that is the tokenizer and there's a

lot going on here and we're going to

step through it in detail at one point

for now I just want to focus on this

part here the create a rigix pattern

here that looks very complicated and

we're going to go through it in a bit uh

but this is the core part that allows

them to enforce rules uh for what parts

of the text Will Never Be merged for

sure now notice that re. compile here is

a little bit misleading because we're

not just doing import re which is the

python re module we're doing import reex

as re and reex is a python package that

you can install P install r x and it's

basically an extension of re so it's a

bit more powerful

re um

so let's take a look at this pattern and

what it's doing and why this is actually

doing the separation that they are

looking for okay so I've copy pasted the

pattern here to our jupit notebook where

we left off and let's take this pattern

for a spin so in the exact same way that

their code does we're going to call an

re. findall for this pattern on any

arbitrary string that we are interested

so this is the string that we want to

encode into tokens um to feed into n llm

like gpt2 so what exactly is this doing

well re. findall will take this pattern

and try to match it against a

string um the way this works is that you

are going from left to right in the

string and you're trying to match the

pattern and R.F find all will get all

the occurrences and organize them into a

list now when you look at the um when

you look at this pattern first of all

notice that this is a raw string um and

then these are three double quotes just

to start the string so really the string

itself this is the pattern itself

right and notice that it's made up of a

lot of ores so see these vertical bars

those are ores in reg X and so you go

from left to right in this pattern and

try to match it against the string

wherever you are so we have hello and

we're going to try to match it well it's

not apostrophe s it's not apostrophe t

or any of these but it is an optional

space followed by- P of uh sorry SL P of

L one or more times what is/ P of L it

is coming to some documentation that I

found um there might be other sources as

well uh SLP is a letter any kind of

letter from any language and hello is

made up of letters h e l Etc so optional

space followed by a bunch of letters one

or more letters is going to match hello

but then the match ends because a white

space is not a letter so from there on

begins a new sort of attempt to match

against the string again and starting in

here we're going to skip over all of

these again until we get to the exact

same Point again and we see that there's

an optional space this is the optional

space followed by a bunch of letters one

or more of them and so that matches so

when we run this we get a list of two

elements hello and then space world

so how are you if we add more letters we

would just get them like this now what

is this doing and why is this important

we are taking our string and instead of

directly encoding it um for

tokenization we are first splitting it

up and when you actually step through

the code and we'll do that in a bit more

detail what really is doing on a high

level is that it first splits your text

into a list of texts just like this one

and all these elements of this list are

processed independently by the tokenizer

and all of the results of that

processing are simply

concatenated so hello world oh I I

missed how hello world how are you we

have five elements of list all of these

will independent

independently go from text to a token

sequence and then that token sequence is

going to be concatenated it's all going

to be joined up and roughly speaking

what that does is you're only ever

finding merges between the elements of

this list so you can only ever consider

merges within every one of these

elements in

individually and um after you've done

all the possible merging for all of

these elements individually the results

of all that will be joined um by

concatenation and so you are basically

what what you're doing effectively is

you are never going to be merging this e

with this space because they are now

parts of the separate elements of this

list and so you are saying we are never

going to merge

eace um because we're breaking it up in

this way so basically using this regx

pattern to Chunk Up the text is just one

way of enforcing that some merges are

not to happen and we're going to go into

more of this text and we'll see that

what this is trying to do on a high

level is we're trying to not merge

across letters across numbers across

punctuation and so on so let's see in

more detail how that works so let's

continue now we have/ P ofn if you go to

the documentation SLP of n is any kind

of numeric character in any script so

it's numbers so we have an optional

space followed by numbers and those

would be separated out so letters and

numbers are being separated so if I do

Hello World 123 how are you then world

will stop matching here because one is

not a letter anymore but one is a number

so this group will match for that and

we'll get it as a separate entity

uh let's see how these apostrophes work

so here if we have

um uh Slash V or I mean apostrophe V as

an example then apostrophe here is not a

letter or a

number so hello will stop matching and

then we will exactly match this with

that so that will come out as a separate

thing so why are they doing the

apostrophes here honestly I think that

these are just like very common

apostrophes p uh that are used um

typically I don't love that they've done

this

because uh let me show you what happens

when you have uh some Unicode

apostrophes like for example you can

have if you have house then this will be

separated out because of this matching

but if you use the Unicode apostrophe

like

this then suddenly this does not work

and so this apostrophe will actually

become its own thing now and so so um

it's basically hardcoded for this

specific kind of apostrophe and uh

otherwise they become completely

separate tokens in addition to this you

can go to the gpt2 docs and here when

they Define the pattern they say should

have added re. ignore case so BP merges

can happen for capitalized versions of

contractions so what they're pointing

out is that you see how this is

apostrophe and then lowercase letters

well because they didn't do re. ignore

case then then um these rules will not

separate out the apostrophes if it's

uppercase so

house would be like this but if I did

house if I'm uppercase then notice

suddenly the apostrophe comes by

itself so the tokenization will work

differently in uppercase and lower case

inconsistently separating out these

apostrophes so it feels extremely gnarly

and slightly gross um but that's that's

how that works okay so let's come back

after trying to match a bunch of

apostrophe Expressions by the way the

other issue here is that these are quite

language specific probably so I don't

know that all the languages for example

use or don't use apostrophes but that

would be inconsistently tokenized as a

result then we try to match letters then

we try to match numbers and then if that

doesn't work we fall back to here and

what this is saying is again optional

space followed by something that is not

a letter number or a space in one or

more of that so what this is doing

effectively is this is trying to match

punctuation roughly speaking not letters

and not numbers so this group will try

to trigger for that so if I do something

like this then these parts here are not

letters or numbers but they will

actually they are uh they will actually

get caught here and so they become its

own group so we've separated out the

punctuation and finally this um this is

also a little bit confusing so this is

matching white space but this is using a

negative look ahead assertion in regex

so what this is doing is it's matching

wh space up to but not including the

last Whit space

character why is this important um this

is pretty subtle I think so you see how

the white space is always included at

the beginning of the word so um space r

space u Etc suppose we have a lot of

spaces

here what's going to happen here is that

these spaces up to not including the

last character will get caught by this

and what that will do is it will

separate out the spaces up to but not

including the last character so that the

last character can come here and join

with the um space you and the reason

that's nice is because space you is the

common token so if I didn't have these

Extra Spaces here you would just have

space you and if I add tokens if I add

spaces we still have a space view but

now we have all this extra white space

so basically the GB to tokenizer really

likes to have a space letters or numbers

um and it it preens these spaces and

this is just something that it is

consistent about so that's what that is

for and then finally we have all the the

last fallback is um whites space

characters uh so um that would be

just um if that doesn't get caught then

this thing will catch any trailing

spaces and so on I wanted to show one

more real world example here so if we

have this string which is a piece of

python code and then we try to split it

up then this is the kind of output we

get so you'll notice that the list has

many elements here and that's because we

are splitting up fairly often uh every

time sort of a category

changes um so there will never be any

merges Within These

elements and um that's what you are

seeing here now you might think that in

order to train the

tokenizer uh open AI has used this to

split up text into chunks and then run

just a BP algorithm within all the

chunks but that is not exactly what

happened and the reason is the following

notice that we have the spaces here uh

those Spaces end up being entire

elements but these spaces never actually

end up being merged by by open Ai and

the way you can tell is that if you copy

paste the exact same chunk here into Tik

token U Tik tokenizer you see that all

the spaces are kept independent and

they're all token

220 so I think opena at some point Point

en Force some rule that these spaces

would never be merged and so um there's

some additional rules on top of just

chunking and bpe that open ey is not uh

clear about now the training code for

the gpt2 tokenizer was never released so

all we have is uh the code that I've

already shown you but this code here

that they've released is only the

inference code for the tokens so this is

not the training code you can't give it

a piece of text and training tokenizer

this is just the inference code which

Tak takes the merges that we have up

above and applies them to a new piece of

text and so we don't know exactly how

opening ey trained um train the

tokenizer but it wasn't as simple as

chunk it up and BP it uh whatever it was

next I wanted to introduce you to the

Tik token library from openai which is

the official library for tokenization

from openai so this is Tik token bip

install P to Tik token and then um you

can do the tokenization in inference

this is again not training code this is

only inference code for

tokenization um I wanted to show you how

you would use it quite simple and

running this just gives us the gpt2

tokens or the GPT 4 tokens so this is

the tokenizer use for GPT 4 and so in

particular we see that the Whit space in

gpt2 remains unmerged but in GPT 4 uh

these Whit spaces merge as we also saw

in this one where here they're all

unmerged but if we go down to GPT 4 uh

they become merged

um now in the

gp4 uh tokenizer they changed the

regular expression that they use to

Chunk Up text so the way to see this is

that if you come to your the Tik token

uh library and then you go to this file

Tik token X openi public this is where

sort of like the definition of all these

different tokenizers that openi

maintains is and so uh necessarily to do

the inference they had to publish some

of the details about the strings

so this is the string that we already

saw for gpt2 it is slightly different

but it is actually equivalent uh to what

we discussed here so this pattern that

we discussed is equivalent to this

pattern this one just executes a little

bit faster so here you see a little bit

of a slightly different definition but

otherwise it's the same we're going to

go into special tokens in a bit and then

if you scroll down to CL 100k this is

the GPT 4 tokenizer you see that the

pattern has changed um and this is kind

of like the main the major change in

addition to a bunch of other special

tokens which I'll go into in a bit again

now some I'm not going to actually go

into the full detail of the pattern

change because honestly this is my

numbing uh I would just advise that you

pull out chat GPT and the regex

documentation and just step through it

but really the major changes are number

one you see this eye here that means

that the um case sensitivity this is

case insensitive match and so the

comment that we saw earlier on oh we

should have used re. uppercase uh

basically we're now going to be matching

these apostrophe s apostrophe D

apostrophe M Etc uh we're going to be

matching them both in lowercase and in

uppercase so that's fixed there's a

bunch of different like handling of the

whites space that I'm not going to go

into the full details of and then one

more thing here is you will notice that

when they match the numbers they only

match one to three numbers so so they

will never merge

numbers that are in low in more than

three digits only up to three digits of

numbers will ever be merged and uh

that's one change that they made as well

to prevent uh tokens that are very very

long number

sequences uh but again we don't really

know why they do any of this stuff uh

because none of this is documented and

uh it's just we just get the pattern so

um yeah it is what it is but those are

some of the changes that gp4 has made

and of course the vocabulary size went

from roughly 50k to roughly

100K the next thing I would like to do

very briefly is to take you through the

gpt2 encoder dopy that openi has

released uh this is the file that I

already mentioned to you briefly now

this file is uh fairly short and should

be relatively understandable to you at

this point um starting at the bottom

here they are loading two files encoder

Json and vocab bpe and they do some

light processing on it and then they

call this encoder object which is the

tokenizer now if you'd like to inspect

these two files which together

constitute their saved tokenizer then

you can do that with a piece of code

like

this um this is where you can download

these two files and you can inspect them

if you'd like and what you will find is

that this encoder as they call it in

their code is exactly equivalent to our

vocab so remember here where we have

this vocab object which allowed us us to

decode very efficiently and basically it

took us from the integer to the byes uh

for that integer so our vocab is exactly

their encoder and then their vocab bpe

confusingly is actually are merges so

their BP merges which is based on the

data inside vocab bpe ends up being

equivalent to our merges so uh basically

they are saving and loading the two uh

variables that for us are also critical

the merges variable and the vocab

variable using just these two variables

you can represent a tokenizer and you

can both do encoding and decoding once

you've trained this

tokenizer now the only thing that um is

actually slightly confusing inside what

opening ey does here is that in addition

to this encoder and a decoder they also

have something called a bite encoder and

a bite decoder and this is actually

unfortunately just

kind of a spirous implementation detail

and isn't actually deep or interesting

in any way so I'm going to skip the

discussion of it but what opening ey

does here for reasons that I don't fully

understand is that not only have they

this tokenizer which can encode and

decode but they have a whole separate

layer here in addition that is used

serially with the tokenizer and so you

first do um bite encode and then encode

and then you do decode and then bite

decode so that's the loop and they are

just stacked serial on top of each other

and and it's not that interesting so I

won't cover it and you can step through

it if you'd like otherwise this file if

you ignore the bite encoder and the bite

decoder will be algorithmically very

familiar with you and the meat of it

here is the what they call bpe function

and you should recognize this Loop here

which is very similar to our own y Loop

where they're trying to identify the

Byram uh a pair that they should be

merging next and then here just like we

had they have a for Loop trying to merge

this pair uh so they will go over all of

the sequence and they will merge the

pair whenever they find it and they keep

repeating that until they run out of

possible merges in the in the text so

that's the meat of this file and uh

there's an encode and a decode function

just like we have implemented it so long

story short what I want you to take away

at this point is that unfortunately it's

a little bit of a messy code that they

have but algorithmically it is identical

to what we've built up above and what

we've built up above if you understand

it is algorithmically what is necessary

to actually build a BP to organizer

train it and then both encode and decode

the next topic I would like to turn to

is that of special tokens so in addition

to tokens that are coming from you know

raw bytes and the BP merges we can

insert all kinds of tokens that we are

going to use to delimit different parts

of the data or introduced to create a