You might have heard people say things large language models are

trained on massive amounts of text,

or large language models predict the next token in a sequence.

But what does any of that actually mean.

How were large language models trained in practice.

In this video, we'll walk through the steps

it takes to train large language models

that behave as helpful assistants,

follow instructions, use tools, and just generally respond

with the right kind of vibe.

Step one is pre-training.

During this phase, the model is trained on a task

called next token prediction.

This essentially means the model is given a sequence of text,

and its task is to predict what comes next.

One of the advantages of model pre-training

is that it's a type of self-supervised learning,

meaning you don't need to go through the effort of labeling

the whole training data set.

Instead, you take a lot of data from publicly available sources,

and you create the pre-training data

set by chunking this data into inputs and outputs.

For example, if this document about houseplants

was part of your pre-training data set,

you could chunk it into a data set that looked something

like this.

The model's training task is to correctly predict

the next word in the sequence.

In order to do this effectively across a diverse set of inputs,

the LLM implicitly learns patterns about language,

writing styles, and even some world

knowledge how the monstera is an easy to care for houseplant.

While the labeling strategy isn't super complicated,

what is complicated about pre-training

is doing this at scale.

This phase of LLM training is a massive engineering challenge

because models are too large for a single chip.

Engineers must use complex parallelism

to split the model and data across thousands

of interconnected GPUs.

They also need to implement strategies for automated

recovery systems to handle crashes, and massive pipelines

to pre-process the petabytes of training data.

But if you do all of that successfully,

you have a model that can take in a prompt

and use its vast world knowledge,

understanding of language and sense of how different

kinds of documents work to produce a response.

Now, unless you're a developer who actually trains LLMs,

it's unlikely you've interacted directly with a model that's

only gone through pre-training.

Even though pre-trained models, particularly

the more recent and sophisticated ones,

are fairly capable, they can also

be unpredictable and unaligned.

So there's still a lot of work that

needs to happen to get the LLM to be ready for use

in a production environment.

And that is where model post-training comes in.

Post-training is the phase where we refine model quality

and focus on alignment so that the generated answers are safe,

well formatted, helpful, more accurate, and pleasant

to engage with.

Often post training involves two different training strategies

supervised fine tuning or SFT and reinforcement learning

or RL.

An SFT we train the model on gold standard interactions

that demonstrate the behavior we want it to learn.

During RL, the model generates many answers

and a separate reward model grades them.

The reward model gives points for helpful, safe, and accurate

responses.

Pushing the model towards the behaviors

we want and penalizes responses we don't want.

Let's dive into both of these in more detail.

So let's say you wanted the model

to learn how to create a friendly, conversational

response to a question, like a helpful assistant.

To do this, you might create a data set

of prompts formatted as questions

how do I make gluten free muffins.

And the corresponding answers that demonstrate

the helpful assistant style.

During SFT, the model is fed the prompt, it produces a response

and then the loss is measured between the predicted response

and the gold standard response, which is the label,

and that's used to update the model's weights.

This is basically classic ML model training,

and by showing the model these ideal examples,

it learns how to produce responses in line

with the behavior we care about.

You might follow a similar process

for teaching the model how to use tools,

creating a data set of prompts that require tools,

and the syntax of the corresponding tool call.

Or you could use SFT for improving the model's ability

to solve math problems, or really any other capability you

want the model to improve on.

Now, SFT is a good starting place,

but it teaches the model to memorize what good looks like

and doesn't always generalize to New,

unseen variations of a task.

To do that, we need a mechanism for the model

to practice producing good responses

and receiving feedback, iterating

and improving over time.

And that is where RL comes in.

In RL, you have the model generate responses

to prompts that are salient to the task you

want the model to learn.

The model's responses are scored by a separate model called

the reward model, and based on the scores,

the LLM updates its weights and improves over time.

How you train this reward model is out of scope for now,

but there are resources in the description below that

provide some more details.

One way of thinking about the differences between SFT and RL

is that SFT sets the floor.

It ensures that the model is at least competent.

It understands the basic format of a list, a code block,

or an answer to a question RL raises the ceiling.

SFT can only ever be as good as the curated training data,

but RL allows the model to surpass this data

by exploring millions of different ways

to answer a prompt and keeping only the ones that

get the highest scores.

The model can find more efficient or clearer ways

of responding to prompts than what was in the original SFT

training set.

In practice, the balance between SFT and RL

is still an open question, and there's no standard ratio

of how much to do either step.

But while the exact training recipe

will differ between different state of the art models,

the high level components are the same.

And at least for now, that means some combination

of SFT followed by RL.

So far, we've talked through the core components

of how LLMs are trained.

But there's one missing piece.

How do we measure any of this.

Anytime we train a model, we need

to have some way of quantifying how good the model is

at the task we care about.

Are the experiments we're running

actually improving the model's ability

or just wasting valuable compute.

Measuring success in pre-training

is fairly straightforward.

All we care about is whether the model is getting better

at predicting the next word.

But while pre-training focuses on how well does

the model predict the next word.

Post training focuses on questions of alignment,

how well does the model follow instructions and behave

like a safe, friendly, and helpful assistant.

And all of that can be a little harder

to measure for closed domain problems like math or coding.

You might test the model's abilities

on a data set of problems, and check if the model produced

a correct answer.

This is the score you track towards

to see if SFT and RL are actually

improving the model's skills.

The more questions in your test data

set that the model gets right, the better it's

doing for measuring things like is the model being

collaborative or insightful or helpful,

or any other task that's hard to quantify.

You first need to have a clear definition

of what those terms mean and what would

constitute a good response.

For example, here's how you might define what it means

for a model to be helpful.

When you have that defined, you can

use this criteria to evaluate the model's response.

First, you create a test set of relevant prompts.

In the case of helpfulness, pretty much any prompt

could be considered relevant because we always

want a model to respond in a helpful way.

But for measuring something like how funny is this model,

you'd want to make sure that your test set is made of prompts

where you actually expect the model to respond

in a humorous way.

You're not going to get a great signal on how good

the model is if the test prompts have nothing to do with humor.

Once you have this test set of prompts,

you pass it to the model you're evaluating

and you get back responses.

You then score those responses against the criteria

that you defined earlier.

In practice, you probably wouldn't do this completely

manually.

Instead, you would scale the process using a second LLM that

works as a judge and does the scoring for you

and have humans spot check the results.

This judge model, known as an auto rater,

evaluates the LLMs responses to a prompt

and scores it according to your criteria.

As you try out different data sets and training strategies,

you'll use this evaluation to help you determine

if you're on the right path and if the model is indeed

getting better over time.

You still definitely want to spot check the results

and do more robust testing before deploying the model

in production.

But using an auto trader with a set of relevant

prompts and defined criteria gives you

a number that quantifies the model's ability

and can be computed relatively quickly.

So you can track progress through post training.

Putting all of this together, we start with module pre-training,

where we train LLMs to do next token prediction

on large amounts of text.

This initial training phase equips the model

with a broad understanding of patterns, features,

and relationships within the data.

Then we take this pre-trained model

and we apply post-training techniques,

where we refine the quality, consistency,

and alignment of the model so that it responds

as a conversational, pleasant assistant with improved

skills in areas such as reasoning or using tools.

A lot of data experiments and compute later

and we have a trained large language model.

Let me know in the comments what you want to learn about next.

I'll read the research papers so you don't have to.