hello my name is andre

and i've been training deep neural

networks for a bit more than a decade

and in this lecture i'd like to show you

what neural network training looks like

under the hood so in particular we are

going to start with a blank jupiter

notebook and by the end of this lecture

we will define and train in neural net

and you'll get to see everything that

goes on under the hood and exactly

sort of how that works on an intuitive

level

now specifically what i would like to do

is i would like to take you through

building of micrograd now micrograd is

this library that i released on github

about two years ago but at the time i

only uploaded the source code and you'd

have to go in by yourself and really

figure out how it works

so in this lecture i will take you

through it step by step and kind of

comment on all the pieces of it so what

is micrograd and why is it interesting

good

um

micrograd is basically an autograd

engine autograd is short for automatic

gradient and really what it does is it

implements backpropagation now

backpropagation is this algorithm that

allows you to efficiently evaluate the

gradient of

some kind of a loss function with

respect to the weights of a neural

network and what that allows us to do

then is we can iteratively tune the

weights of that neural network to

minimize the loss function and therefore

improve the accuracy of the network so

back propagation would be at the

mathematical core of any modern deep

neural network library like say pytorch

or jaxx

so the functionality of microgrant is i

think best illustrated by an example so

if we just scroll down here

you'll see that micrograph basically

allows you to build out mathematical

expressions

and um here what we are doing is we have

an expression that we're building out

where you have two inputs a and b

and you'll see that a and b are negative

four and two but we are wrapping those

values into this value object that we

are going to build out as part of

micrograd

so this value object will wrap the

numbers themselves

and then we are going to build out a

mathematical expression here where a and

b are transformed into c d and

eventually e f and g

and i'm showing some of the functions

some of the functionality of micrograph

and the operations that it supports so

you can add two value objects you can

multiply them you can raise them to a

constant power you can offset by one

negate squash at zero

square divide by constant divide by it

etc

and so we're building out an expression

graph with with these two inputs a and b

and we're creating an output value of g

and micrograd will in the background

build out this entire mathematical

expression so it will for example know

that c is also a value

c was a result of an addition operation

and the

child nodes of c are a and b because the

and will maintain pointers to a and b

value objects so we'll basically know

exactly how all of this is laid out

and then not only can we do what we call

the forward pass where we actually look

at the value of g of course that's

pretty straightforward we will access

that using the dot data attribute and so

the output of the forward pass the value

of g is 24.7 it turns out but the big

deal is that we can also take this g

value object and we can call that

backward

and this will basically uh initialize

back propagation at the node g

and what backpropagation is going to do

is it's going to start at g and it's

going to go backwards through that

expression graph and it's going to

recursively apply the chain rule from

calculus

and what that allows us to do then is

we're going to evaluate basically the

derivative of g with respect to all the

internal nodes

like e d and c but also with respect to

the inputs a and b

and then we can actually query this

derivative of g with respect to a for

example that's a dot grad in this case

it happens to be 138 and the derivative

of g with respect to b

which also happens to be here 645

and this derivative we'll see soon is

very important information because it's

telling us how a and b are affecting g

through this mathematical expression so

in particular

a dot grad is 138 so if we slightly

nudge a and make it slightly larger

138 is telling us that g will grow and

the slope of that growth is going to be

138

and the slope of growth of b is going to

be 645. so that's going to tell us about

how g will respond if a and b get

tweaked a tiny amount in a positive

direction

okay

now you might be confused about what

this expression is that we built out

here and this expression by the way is

completely meaningless i just made it up

i'm just flexing about the kinds of

operations that are supported by

micrograd

what we actually really care about are

neural networks but it turns out that

neural networks are just mathematical

expressions just like this one but

actually slightly bit less crazy even

neural networks are just a mathematical

expression they take the input data as

an input and they take the weights of a

neural network as an input and it's a

mathematical expression and the output

are your predictions of your neural net

or the loss function we'll see this in a

bit but basically neural networks just

happen to be a certain class of

mathematical expressions

but back propagation is actually

significantly more general it doesn't

actually care about neural networks at

all it only tells us about arbitrary

mathematical expressions and then we

happen to use that machinery for

training of neural networks now one more

note i would like to make at this stage

is that as you see here micrograd is a

scalar valued auto grant engine so it's

working on the you know level of

individual scalars like negative four

and two and we're taking neural nets and

we're breaking them down all the way to

these atoms of individual scalars and

all the little pluses and times and it's

just excessive and so obviously you

would never be doing any of this in

production it's really just put down for

pedagogical reasons because it allows us

to not have to deal with these

n-dimensional tensors that you would use

in modern deep neural network library so

this is really done so that you

understand and refactor out back

propagation and chain rule and

understanding of neurologic training

and then if you actually want to train

bigger networks you have to be using

these tensors but none of the math

changes this is done purely for

efficiency we are basically taking scale

value

all the scale values we're packaging

them up into tensors which are just

arrays of these scalars and then because

we have these large arrays we're making

operations on those large arrays that

allows us to take advantage of the

parallelism in a computer and all those

operations can be done in parallel and

then the whole thing runs faster but

really none of the math changes and

that's done purely for efficiency so i

don't think that it's pedagogically

useful to be dealing with tensors from

scratch uh and i think and that's why i

fundamentally wrote micrograd because

you can understand how things work uh at

the fundamental level and then you can

speed it up later okay so here's the fun

part my claim is that micrograd is what

you need to train your networks and

everything else is just efficiency so

you'd think that micrograd would be a

very complex piece of code and that

turns out to not be the case

so if we just go to micrograd

and you'll see that there's only two

files here in micrograd this is the

actual engine it doesn't know anything

about neural nuts and this is the entire

neural nets library

on top of micrograd so engine and nn.pi

so the actual backpropagation autograd

engine

that gives you the power of neural

networks is literally

100 lines of code of like very simple

python

which we'll understand by the end of

this lecture

and then nn.pi

this neural network library built on top

of the autograd engine

um is like a joke it's like

we have to define what is a neuron and

then we have to define what is the layer

of neurons and then we define what is a

multi-layer perceptron which is just a

sequence of layers of neurons and so

it's just a total joke

so basically

there's a lot of power that comes from

only 150 lines of code

and that's all you need to understand to

understand neural network training and

everything else is just efficiency and

of course there's a lot to efficiency

but fundamentally that's all that's

happening okay so now let's dive right

in and implement micrograph step by step

the first thing i'd like to do is i'd

like to make sure that you have a very

good understanding intuitively of what a

derivative is and exactly what

information it gives you so let's start

with some basic imports that i copy

paste in every jupiter notebook always

and let's define a function a scalar

valued function

f of x

as follows

so i just make this up randomly i just

want to scale a valid function that

takes a single scalar x and returns a

single scalar y

and we can call this function of course

so we can pass in say 3.0 and get 20

back

now we can also plot this function to

get a sense of its shape you can tell

from the mathematical expression that

this is probably a parabola it's a

quadratic

and so if we just uh create a set of um

um

scale values that we can feed in using

for example a range from negative five

to five in steps of 0.25

so this is so axis is just from negative

5 to 5 not including 5 in steps of 0.25

and we can actually call this function

on this numpy array as well so we get a

set of y's if we call f on axis

and these y's are basically

also applying a function on every one of

these elements independently

and we can plot this using matplotlib so

plt.plot x's and y's and we get a nice

parabola so previously here we fed in

3.0 somewhere here and we received 20

back which is here the y coordinate so

now i'd like to think through

what is the derivative

of this function at any single input

point x

right so what is the derivative at

different points x of this function now

if you remember back to your calculus

class you've probably derived

derivatives so we take this mathematical

expression 3x squared minus 4x plus 5

and you would write out on a piece of

paper and you would you know apply the

product rule and all the other rules and

derive the mathematical expression of

the great derivative of the original

function and then you could plug in

different texts and see what the

derivative is

we're not going to actually do that

because no one in neural networks

actually writes out the expression for

the neural net it would be a massive

expression um it would be you know

thousands tens of thousands of terms no

one actually derives the derivative of

course and so we're not going to take

this kind of like a symbolic approach

instead what i'd like to do is i'd like

to look at the definition of derivative

and just make sure that we really

understand what derivative is measuring

what it's telling you about the function

and so if we just look up derivative

we see that

okay so this is not a very good

definition of derivative this is a

definition of what it means to be

differentiable

but if you remember from your calculus

it is the limit as h goes to zero of f

of x plus h minus f of x over h so

basically what it's saying is if you

slightly bump up you're at some point x

that you're interested in or a and if

you slightly bump up

you know you slightly increase it by

small number h

how does the function respond with what

sensitivity does it respond what is the

slope at that point does the function go

up or does it go down and by how much

and that's the slope of that function

the

the slope of that response at that point

and so we can basically evaluate

the derivative here numerically by

taking a very small h of course the

definition would ask us to take h to

zero we're just going to pick a very

small h 0.001

and let's say we're interested in point

3.0 so we can look at f of x of course

as 20

and now f of x plus h

so if we slightly nudge x in a positive

direction how is the function going to

respond

and just looking at this do you expect

do you expect f of x plus h to be

slightly greater than 20 or do you

expect to be slightly lower than 20

and since this 3 is here and this is 20

if we slightly go positively the

function will respond positively so

you'd expect this to be slightly greater

than 20. and now by how much it's

telling you the

sort of the

the strength of that slope right the the

size of the slope so f of x plus h minus

f of x this is how much the function

responded

in the positive direction and we have to

normalize by the

run so we have the rise over run to get

the slope so this of course is just a

numerical approximation of the slope

because we have to make age very very

small to converge to the exact amount

now if i'm doing too many zeros

at some point

i'm gonna get an incorrect answer

because we're using floating point

arithmetic and the representations of

all these numbers in computer memory is

finite and at some point we get into

trouble

so we can converse towards the right

answer with this approach

but basically um at 3 the slope is 14.

and you can see that by taking 3x

squared minus 4x plus 5 and

differentiating it in our head

so 3x squared would be

6 x minus 4

and then we plug in x equals 3 so that's

18 minus 4 is 14. so this is correct

so that's

at 3. now how about the slope at say

negative 3

would you expect would you expect for

the slope

now telling the exact value is really

hard but what is the sign of that slope

so at negative three

if we slightly go in the positive

direction at x the function would

actually go down and so that tells you

that the slope would be negative so

we'll get a slight number below

below 20. and so if we take the slope we

expect something negative

negative 22. okay

and at some point here of course the

slope would be zero now for this

specific function i looked it up

previously and it's at point two over

three

so at roughly two over three

uh that's somewhere here

um

this derivative be zero

so basically at that precise point

yeah

at that precise point if we nudge in a

positive direction the function doesn't

respond this stays the same almost and

so that's why the slope is zero okay now

let's look at a bit more complex case

so we're going to start you know

complexifying a bit so now we have a

function

here

with output variable d

that is a function of three scalar

inputs a b and c

so a b and c are some specific values

three inputs into our expression graph

and a single output d

and so if we just print d we get four

and now what i have to do is i'd like to

again look at the derivatives of d with

respect to a b and c

and uh think through uh again just the

intuition of what this derivative is

telling us

so in order to evaluate this derivative

we're going to get a bit hacky here

we're going to again have a very small

value of h

and then we're going to fix the inputs

at some

values that we're interested in

so these are the this is the point abc

at which we're going to be evaluating

the the

derivative of d with respect to all a b

and c at that point

so there are the inputs and now we have

d1 is that expression

and then we're going to for example look

at the derivative of d with respect to a

so we'll take a and we'll bump it by h

and then we'll get d2 to be the exact

same function

and now we're going to print um

you know f1

d1 is d1

d2 is d2

and print slope

so the derivative or slope

here will be um

of course

d2

minus d1 divide h

so d2 minus d1 is how much the function

increased

uh when we bumped

the uh

the specific input that we're interested

in by a tiny amount

and

this is then normalized by h

to get the slope

so

um

yeah

so this so if i just run this we're

going to print

d1

which we know is four

now d2 will be bumped a will be bumped

by h

so let's just think through

a little bit uh what d2 will be uh

printed out here

in particular

d1 will be four

will d2 be a number slightly greater

than four or slightly lower than four

and that's going to tell us the sl the

the sign of the derivative

so

we're bumping a by h

b as minus three c is ten

so you can just intuitively think

through this derivative and what it's

doing a will be slightly more positive

and but b is a negative number

so if a is slightly more positive

because b is negative three

we're actually going to be adding less

to d

so you'd actually expect that the value

of the function will go down

so let's just see this

yeah and so we went from 4

to 3.9996

and that tells you that the slope will

be negative

and then

uh will be a negative number

because we went down

and then

the exact number of slope will be

exact amount of slope is negative 3.

and you can also convince yourself that

negative 3 is the right answer

mathematically and analytically because

if you have a times b plus c and you are

you know you have calculus then

differentiating a times b plus c with

respect to a gives you just b

and indeed the value of b is negative 3

which is the derivative that we have so

you can tell that that's correct

so now if we do this with b

so if we bump b by a little bit in a

positive direction we'd get different

slopes so what is the influence of b on

the output d

so if we bump b by a tiny amount in a

positive direction then because a is

positive

we'll be adding more to d

right

so um and now what is the what is the

sensitivity what is the slope of that

addition

and it might not surprise you that this

should be

2

and y is a 2 because d of d

by db differentiating with respect to b

would be would give us a

and the value of a is two so that's also

working well

and then if c gets bumped a tiny amount

in h

by h

then of course a times b is unaffected

and now c becomes slightly bit higher

what does that do to the function it

makes it slightly bit higher because

we're simply adding c

and it makes it slightly bit higher by

the exact same amount that we added to c

and so that tells you that the slope is

one

that will be the

the rate at which

d will increase as we scale

c

okay so we now have some intuitive sense

of what this derivative is telling you

about the function and we'd like to move

to neural networks now as i mentioned

neural networks will be pretty massive

expressions mathematical expressions so

we need some data structures that

maintain these expressions and that's

what we're going to start to build out

now

so we're going to

build out this value object that i

showed you in the readme page of

micrograd

so let me copy paste a skeleton of the

first very simple value object

so class value takes a single

scalar value that it wraps and keeps

track of

and that's it so

we can for example do value of 2.0 and

then we can

get we can look at its content and

python will internally

use the wrapper function

to uh return

uh this string oops

like that

so this is a value object with data

equals two that we're creating here

now we'd like to do is like we'd like to

be able to

have not just like two values

but we'd like to do a bluffy right we'd

like to add them

so currently you would get an error

because python doesn't know how to add

two value objects so we have to tell it

so here's

addition

so you have to basically use these

special double underscore methods in

python to define these operators for

these objects so if we call um

the uh if we use this plus operator

python will internally call a dot add of

b

that's what will happen internally and

so b will be the other and

self will be a

and so we see that what we're going to

return is a new value object and it's

just it's going to be wrapping

the plus of

their data

but remember now because data is the

actual like numbered python number so

this operator here is just the typical

floating point plus addition now it's

not an addition of value objects

and will return a new value so now a

plus b should work and it should print

value of

negative one

because that's two plus minus three

there we go

okay let's now implement multiply

just so we can recreate this expression

here

so multiply i think it won't surprise

you will be fairly similar

so instead of add we're going to be

using mul

and then here of course we want to do

times

and so now we can create a c value

object which will be 10.0 and now we

should be able to do a times b well

let's just do a times b first

um

[Music]

that's value of negative six now

and by the way i skipped over this a

little bit suppose that i didn't have

the wrapper function here

then it's just that you'll get some kind

of an ugly expression so what wrapper is

doing is it's providing us a way to

print out like a nicer looking

expression in python

uh so we don't just have something

cryptic we actually are you know it's

value of

negative six so this gives us a times

and then this we should now be able to

add c to it because we've defined and

told the python how to do mul and add

and so this will call this will

basically be equivalent to a dot

small

of b

and then this new value object will be

dot add

of c

and so let's see if that worked

yep so that worked well that gave us

four which is what we expect from before

and i believe we can just call them

manually as well there we go so

yeah

okay so now what we are missing is the

connective tissue of this expression as

i mentioned we want to keep these

expression graphs so we need to know and

keep pointers about what values produce

what other values

so here for example we are going to

introduce a new variable which we'll

call children and by default it will be

an empty tuple

and then we're actually going to keep a

slightly different variable in the class

which we'll call underscore prev which

will be the set of children

this is how i done i did it in the

original micrograd looking at my code

here i can't remember exactly the reason

i believe it was efficiency but this

underscore children will be a tuple for

convenience but then when we actually

maintain it in the class it will be just

this set yeah i believe for efficiency

um

so now

when we are creating a value like this

with a constructor children will be

empty and prep will be the empty set but

when we're creating a value through

addition or multiplication we're going

to feed in the children of this value

which in this case is self and other

so those are the children

here

so now we can do d dot prev

and we'll see that the children of the

we now know are this value of negative 6

and value of 10 and this of course is

the value resulting from a times b and

the c value which is 10.

now the last piece of information we

don't know so we know that the children

of every single value but we don't know

what operation created this value

so we need one more element here let's

call it underscore pop

and by default this is the empty set for

leaves

and then we'll just maintain it here

and now the operation will be just a

simple string and in the case of

addition it's plus in the case of

multiplication is times

so now we

not just have d dot pref we also have a

d dot up

and we know that d was produced by an

addition of those two values and so now

we have the full

mathematical expression uh and we're

building out this data structure and we

know exactly how each value came to be

by word expression and from what other

values

now because these expressions are about

to get quite a bit larger we'd like a

way to nicely visualize these

expressions that we're building out so

for that i'm going to copy paste a bunch

of slightly scary code that's going to

visualize this these expression graphs

for us

so here's the code and i'll explain it

in a bit but first let me just show you

what this code does

basically what it does is it creates a

new function drawdot that we can call on

some root node

and then it's going to visualize it so

if we call drawdot on d

which is this final value here that is a

times b plus c

it creates something like this so this

is d

and you see that this is a times b

creating an integrated value plus c

gives us this output node d

so that's dried out of d

and i'm not going to go through this in

complete detail you can take a look at

graphless and its api uh graphis is a

open source graph visualization software

and what we're doing here is we're

building out this graph and graphis

api and

you can basically see that trace is this

helper function that enumerates all of

the nodes and edges in the graph

so that just builds a set of all the

nodes and edges and then we iterate for

all the nodes and we create special node

objects

for them in

using dot node

and then we also create edges using dot

dot edge

and the only thing that's like slightly

tricky here is you'll notice that i

basically add these fake nodes which are

these operation nodes so for example

this node here is just like a plus node

and

i create these

special op nodes here

and i connect them accordingly so these

nodes of course are not actual

nodes in the original graph

they're not actually a value object the

only value objects here are the things

in squares those are actual value

objects or representations thereof and

these op nodes are just created in this

drawdot routine so that it looks nice

let's also add labels to these graphs

just so we know what variables are where

so let's create a special underscore

label

um

or let's just do label

equals empty by default and save it in

each node

and then here we're going to do label as

a

label is the

label a c

and then

let's create a special um

e equals a times b

and e dot label will be e

it's kind of naughty

and e will be e plus c

and a d dot label will be

d

okay so nothing really changes i just

added this new e function

a new e variable

and then here when we are

printing this

i'm going to print the label here so

this will be a percent s

bar

and this will be end.label

and so now

we have the label on the left here so it

says a b creating e and then e plus c

creates d

just like we have it here

and finally let's make this expression

just one layer deeper

so d will not be the final output node

instead after d we are going to create a

new value object

called f we're going to start running

out of variables soon f will be negative

2.0

and its label will of course just be f

and then l capital l will be the output

of our graph

and l will be p times f

okay

so l will be negative eight is the

output

so

now we don't just draw a d we draw l

okay

and somehow the label of

l was undefined oops all that label has

to be explicitly sort of given to it

there we go so l is the output

so let's quickly recap what we've done

so far

we are able to build out mathematical

expressions using only plus and times so

far

they are scalar valued along the way

and we can do this forward pass

and build out a mathematical expression

so we have multiple inputs here a b c

and f

going into a mathematical expression

that produces a single output l

and this here is visualizing the forward

pass so the output of the forward pass

is negative eight that's the value

now what we'd like to do next is we'd

like to run back propagation

and in back propagation we are going to

start here at the end and we're going to

reverse

and calculate the gradient along along

all these intermediate values

and really what we're computing for

every single value here

um we're going to compute the derivative

of that node with respect to l

so

the derivative of l with respect to l is

just uh one

and then we're going to derive what is

the derivative of l with respect to f

with respect to d with respect to c with

respect to e

with respect to b and with respect to a

and in the neural network setting you'd

be very interested in the derivative of

basically this loss function l

with respect to the weights of a neural

network

and here of course we have just these

variables a b c and f

but some of these will eventually

represent the weights of a neural net

and so we'll need to know how those

weights are impacting

the loss function so we'll be interested

basically in the derivative of the

output with respect to some of its leaf

nodes and those leaf nodes will be the

weights of the neural net

and the other leaf nodes of course will

be the data itself but usually we will

not want or use the derivative of the

loss function with respect to data

because the data is fixed but the

weights will be iterated on

using the gradient information so next

we are going to create a variable inside

the value class that maintains the

derivative of l with respect to that

value

and we will call this variable grad

so there's a data and there's a

self.grad

and initially it will be zero and

remember that zero is basically means no

effect so at initialization we're

assuming that every value does not

impact does not affect the out the

output

right because if the gradient is zero

that means that changing this variable

is not changing the loss function

so by default we assume that the

gradient is zero

and then

now that we have grad and it's 0.0

we are going to be able to visualize it

here after data so here grad is 0.4 f

and this will be in that graph

and now we are going to be showing both

the data and the grad

initialized at zero

and we are just about getting ready to

calculate the back propagation

and of course this grad again as i

mentioned is representing

the derivative of the output in this

case l with respect to this value so

with respect to so this is the

derivative of l with respect to f with

respect to d and so on so let's now fill

in those gradients and actually do back

propagation manually so let's start

filling in these gradients and start all

the way at the end as i mentioned here

first we are interested to fill in this

gradient here so what is the derivative

of l with respect to l

in other words if i change l by a tiny

amount of h

how much does

l change

it changes by h so it's proportional and

therefore derivative will be one

we can of course measure these or

estimate these numerical gradients

numerically just like we've seen before

so if i take this expression

and i create a def lol function here

and put this here now the reason i'm

creating a gating function hello here is

because i don't want to pollute or mess

up the global scope here this is just

kind of like a little staging area and

as you know in python all of these will

be local variables to this function so

i'm not changing any of the global scope

here

so here l1 will be l

and then copy pasting this expression

we're going to add a small amount h

in for example a

right and this would be measuring the

derivative of l with respect to a

so here this will be l2

and then we want to print this

derivative so print

l2 minus l1 which is how much l changed

and then normalize it by h so this is

the rise over run

and we have to be careful because l is a

value node so we actually want its data

um

so that these are floats dividing by h

and this should print the derivative of

l with respect to a because a is the one

that we bumped a little bit by h

so what is the

derivative of l with respect to a

it's six

okay and obviously

if we change l by h

then that would be

here effectively

this looks really awkward but changing l

by h

you see the derivative here is 1. um

that's kind of like the base case of

what we are doing here

so basically we cannot come up here and

we can manually set l.grad to one this

is our manual back propagation

l dot grad is one and let's redraw

and we'll see that we filled in grad as

1 for l

we're now going to continue the back

propagation so let's here look at the

derivatives of l with respect to d and f

let's do a d first

so what we are interested in if i create

a markdown on here is we'd like to know

basically we have that l is d times f

and we'd like to know what is uh d

l by d d

what is that

and if you know your calculus uh l is d

times f so what is d l by d d it would

be f

and if you don't believe me we can also

just derive it because the proof would

be fairly straightforward uh we go to

the

definition of the derivative which is f

of x plus h minus f of x divide h

as a limit limit of h goes to zero of

this kind of expression so when we have

l is d times f

then increasing d by h

would give us the output of b plus h

times f

that's basically f of x plus h right

minus d times f

and then divide h and symbolically

expanding out here we would have

basically d times f plus h times f minus

t times f divide h

and then you see how the df minus df

cancels so you're left with h times f

divide h

which is f

so in the limit as h goes to zero of

you know

derivative

definition we just get f in the case of

d times f

so

symmetrically

dl by d

f will just be d

so what we have is that f dot grad

we see now is just the value of d

which is 4.

and we see that

d dot grad

is just uh the value of f

and so the value of f is negative two

so we'll set those manually

let me erase this markdown node and then

let's redraw what we have

okay

and let's just make sure that these were

correct so we seem to think that dl by

dd is negative two so let's double check

um let me erase this plus h from before

and now we want the derivative with

respect to f

so let's just come here when i create f

and let's do a plus h here and this

should print the derivative of l with

respect to f so we expect to see four

yeah and this is four up to floating

point

funkiness

and then dl by dd

should be f which is negative two

grad is negative two

so if we again come here and we change d

d dot data plus equals h right here

so we expect so we've added a little h

and then we see how l changed and we

expect to print

uh negative two

there we go

so we've numerically verified what we're

doing here is what kind of like an

inline gradient check gradient check is

when we

are deriving this like back propagation

and getting the derivative with respect

to all the intermediate results and then

numerical gradient is just you know

estimating it using small step size

now we're getting to the crux of

backpropagation so this will be the most

important node to understand because if

you understand the gradient for this

node you understand all of back

propagation and all of training of

neural nets basically

so we need to derive dl by bc

in other words the derivative of l with

respect to c

because we've computed all these other

gradients already

now we're coming here and we're

continuing the back propagation manually

so we want dl by dc and then we'll also

derive dl by de

now here's the problem

how do we derive dl

by dc

we actually know the derivative l with

respect to d so we know how l assessed

it to d

but how is l sensitive to c so if we

wiggle c how does that impact l

through d

so we know dl by dc

and we also here know how c impacts d

and so just very intuitively if you know

the impact that c is having on d and the

impact that d is having on l

then you should be able to somehow put

that information together to figure out

how c impacts l

and indeed this is what we can actually

do so in particular we know just

concentrating on d first let's look at

how what is the derivative basically of

d with respect to c so in other words

what is dd by dc

so here we know that d is c times c plus

e

that's what we know and now we're

interested in dd by dc

if you just know your calculus again and

you remember that differentiating c plus

e with respect to c you know that that

gives you

1.0

and we can also go back to the basics

and derive this because again we can go

to our f of x plus h minus f of x

divide by h

that's the definition of a derivative as

h goes to zero

and so here

focusing on c and its effect on d

we can basically do the f of x plus h

will be

c is incremented by h plus e

that's the first evaluation of our

function minus

c plus e

and then divide h

and so what is this

uh just expanding this out this will be

c plus h plus e minus c minus e

divide h and then you see here how c

minus c cancels e minus e cancels we're

left with h over h which is 1.0

and so

by symmetry also d d by d

e

will be 1.0 as well

so basically the derivative of a sum

expression is very simple and and this

is the local derivative so i call this

the local derivative because we have the

final output value all the way at the

end of this graph and we're now like a

small node here

and this is a little plus node

and it the little plus node doesn't know

anything about the rest of the graph

that it's embedded in all it knows is

that it did a plus it took a c and an e

added them and created d

and this plus note also knows the local

influence of c on d or rather rather the

derivative of d with respect to c and it

also

knows the derivative of d with respect

to e but that's not what we want that's

just a local derivative what we actually

want is d l by d c and l could l is here

just one step away but in a general case

this little plus note is could be

embedded in like a massive graph

so

again we know how l impacts d and now we

know how c and e impact d how do we put

that information together to write dl by

dc and the answer of course is the chain

rule in calculus

and so um

i pulled up a chain rule here from

kapedia

and

i'm going to go through this very

briefly so chain rule

wikipedia sometimes can be very

confusing and calculus can

can be very confusing like this is the

way i

learned

chain rule and it was very confusing

like what is happening it's just

complicated so i like this expression

much better

if a variable z depends on a variable y

which itself depends on the variable x

then z depends on x as well obviously

through the intermediate variable y

in this case the chain rule is expressed

as

if you want dz by dx

then you take the dz by dy and you

multiply it by d y by dx

so the chain rule fundamentally is

telling you

how

we chain these

uh derivatives together

correctly so to differentiate through a

function composition

we have to apply a multiplication

of

those derivatives

so that's really what chain rule is

telling us

and there's a nice little intuitive

explanation here which i also think is

kind of cute the chain rule says that

knowing the instantaneous rate of change

of z with respect to y and y relative to

x allows one to calculate the

instantaneous rate of change of z

relative to x

as a product of those two rates of

change

simply the product of those two

so here's a good one

if a car travels twice as fast as

bicycle and the bicycle is four times as

fast as walking man

then the car travels two times four

eight times as fast as demand

and so this makes it very clear that the

correct thing to do sort of

is to multiply

so

cars twice as fast as bicycle and

bicycle is four times as fast as man

so the car will be eight times as fast

as the man and so we can take these

intermediate rates of change if you will

and multiply them together

and that justifies the

chain rule intuitively so have a look at

chain rule about here really what it

means for us is there's a very simple

recipe for deriving what we want which

is dl by dc

and what we have so far

is we know

want

and we know

what is the

impact of d on l so we know d l by

d d the derivative of l with respect to

d d we know that that's negative two

and now because of this local

reasoning that we've done here we know

dd by d

c

so how does c impact d and in

particular this is a plus node so the

local derivative is simply 1.0 it's very

simple

and so

the chain rule tells us that dl by dc

going through this intermediate variable

will just be simply d l by

d

times

dd by dc

that's chain rule

so this is identical to what's happening

here

except

z is rl

y is our d and x is rc

so we literally just have to multiply

these

and because

these local derivatives like dd by dc

are just one

we basically just copy over dl by dd

because this is just times one

so what does it do so because dl by dd

is negative two what is dl by dc

well it's the local gradient 1.0 times

dl by dd which is negative two

so literally what a plus node does you

can look at it that way is it literally

just routes the gradient

because the plus nodes local derivatives

are just one and so in the chain rule

one times

dl by dd

is um

is uh is just dl by dd and so that

derivative just gets routed to both c

and to e in this case

so basically um we have that that grad

or let's start with c since that's the

one we looked at

is

negative two times one

negative two

and in the same way by symmetry e that

grad will be negative two that's the

claim so we can set those

we can redraw

and you see how we just assign negative

to negative two so this backpropagating

signal which is carrying the information

of like what is the derivative of l with

respect to all the intermediate nodes

we can imagine it almost like flowing

backwards through the graph and a plus

node will simply distribute the

derivative to all the leaf nodes sorry

to all the children nodes of it

so this is the claim and now let's

verify it so let me remove the plus h

here from before

and now instead what we're going to do

is we're going to increment c so c dot

data will be credited by h

and when i run this we expect to see

negative 2

negative 2. and then of course for e

so e dot data plus equals h and we

expect to see negative 2.

simple

so those are the derivatives of these

internal nodes

and now we're going to recurse our way

backwards again

and we're again going to apply the chain

rule so here we go our second

application of chain rule and we will

apply it all the way through the graph

we just happen to only have one more

node remaining

we have that d l

by d e

as we have just calculated is negative

two so we know that

so we know the derivative of l with

respect to e

and now we want dl

by

da

right

and the chain rule is telling us that

that's just dl by de

negative 2

times the local gradient so what is the

local gradient basically d e

by d a

we have to look at that

so i'm a little times node

inside a massive graph

and i only know that i did a times b and

i produced an e

so now what is d e by d a and d e by d b

that's the only thing that i sort of

know about that's my local gradient

so

because we have that e's a times b we're

asking what is d e by d a

and of course we just did that here we

had a

times so i'm not going to rederive it

but if you want to differentiate this

with respect to a you'll just get b

right the value of b

which in this case is negative 3.0

so

basically we have that dl by da

well let me just do it right here we

have that a dot grad and we are applying

chain rule here

is d l by d e which we see here is

negative two

times

what is d e by d a

it's the value of b which is negative 3.

that's it

and then we have b grad is again dl by

de

which is negative 2

just the same way

times

what is d e by d

um db

is the value of a which is 2.2.0

as the value of a

so these are our claimed derivatives

let's

redraw

and we see here that

a dot grad turns out to be 6 because

that is negative 2 times negative 3

and b dot grad is negative 4

times sorry is negative 2 times 2 which

is negative 4.

so those are our claims let's delete

this and let's verify them

we have

a here a dot data plus equals h

so the claim is that

a dot grad is six

let's verify

six

and we have beta data

plus equals h

so nudging b by h

and looking at what happens

we claim it's negative four

and indeed it's negative four plus minus

again float oddness

um

and uh

that's it this

that was the manual

back propagation

uh all the way from here to all the leaf

nodes and we've done it piece by piece

and really all we've done is as you saw

we iterated through all the nodes one by

one and locally applied the chain rule

we always know what is the derivative of

l with respect to this little output and

then we look at how this output was

produced this output was produced

through some operation and we have the

pointers to the children nodes of this

operation

and so in this little operation we know

what the local derivatives are and we

just multiply them onto the derivative

always

so we just go through and recursively

multiply on the local derivatives and

that's what back propagation is is just

a recursive application of chain rule

backwards through the computation graph

let's see this power in action just very

briefly what we're going to do is we're

going to

nudge our inputs to try to make l go up

so in particular what we're doing is we

want a.data we're going to change it

and if we want l to go up that means we

just have to go in the direction of the

gradient so

a

should increase in the direction of

gradient by like some small step amount

this is the step size

and we don't just want this for ba but

also for b

also for c

also for f

those are

leaf nodes which we usually have control

over

and if we nudge in direction of the

gradient we expect a positive influence

on l

so we expect l to go up

positively

so it should become less negative it

should go up to say negative you know

six or something like that

uh it's hard to tell exactly and we'd

have to rewrite the forward pass so let

me just um

do that here

um

this would be the forward pass f would

be unchanged this is effectively the

forward pass and now if we print l.data

we expect because we nudged all the

values all the inputs in the rational

gradient we expected a less negative l

we expect it to go up

so maybe it's negative six or so let's

see what happens

okay negative seven

and uh this is basically one step of an

optimization that we'll end up running

and really does gradient just give us

some power because we know how to

influence the final outcome and this

will be extremely useful for training

knowledge as well as you'll see

so now i would like to do one more uh

example of manual backpropagation using

a bit more complex and uh useful example

we are going to back propagate through a

neuron

so

we want to eventually build up neural

networks and in the simplest case these

are multilateral perceptrons as they're

called so this is a two layer neural net

and it's got these hidden layers made up

of neurons and these neurons are fully

connected to each other

now biologically neurons are very

complicated devices but we have very

simple mathematical models of them

and so this is a very simple

mathematical model of a neuron you have

some inputs axis

and then you have these synapses that

have weights on them so

the w's are weights

and then

the synapse interacts with the input to

this neuron multiplicatively so what

flows to the cell body

of this neuron is w times x

but there's multiple inputs so there's

many w times x's flowing into the cell

body

the cell body then has also like some

bias

so this is kind of like the

inert innate sort of trigger happiness

of this neuron so this bias can make it

a bit more trigger happy or a bit less

trigger happy regardless of the input

but basically we're taking all the w

times x

of all the inputs adding the bias and

then we take it through an activation

function

and this activation function is usually

some kind of a squashing function

like a sigmoid or 10h or something like

that so as an example

we're going to use the 10h in this

example

numpy has a

np.10h

so

we can call it on a range

and we can plot it

this is the 10h function and you see

that the inputs as they come in

get squashed on the y coordinate here so

um

right at zero we're going to get exactly

zero and then as you go more positive in

the input

then you'll see that the function will

only go up to one and then plateau out

and so if you pass in very positive

inputs we're gonna cap it smoothly at

one and on the negative side we're gonna

cap it smoothly to negative one

so that's 10h

and that's the squashing function or an

activation function and what comes out

of this neuron is just the activation

function applied to the dot product of

the weights and the

inputs

so let's

write one out

um

i'm going to copy paste because

i don't want to type too much

but okay so here we have the inputs

x1 x2 so this is a two-dimensional

neuron so two inputs are going to come

in

these are thought out as the weights of

this neuron

weights w1 w2 and these weights again

are the synaptic strengths for each

input

and this is the bias of the neuron

b

and now we want to do is according to

this model we need to multiply x1 times

w1

and x2 times w2

and then we need to add bias on top of

it

and it gets a little messy here but all

we are trying to do is x1 w1 plus x2 w2

plus b

and these are multiply here

except i'm doing it in small steps so

that we actually have pointers to all

these intermediate nodes so we have x1

w1 variable x times x2 w2 variable and

i'm also labeling them

so n is now

the cell body raw

raw

activation without

the activation function for now

and this should be enough to basically

plot it so draw dot of n

gives us x1 times w1 x2 times w2

being added

then the bias gets added on top of this

and this n

is this sum

so we're now going to take it through an

activation function

and let's say we use the 10h

so that we produce the output

so what we'd like to do here is we'd

like to do the output and i'll call it o

is um

n dot 10h

okay but we haven't yet written the 10h

now the reason that we need to implement

another 10h function here is that

tanh is a

hyperbolic function and we've only so

far implemented a plus and the times and

you can't make a 10h out of just pluses

and times

you also need exponentiation so 10h is

this kind of a formula here

you can use either one of these and you

see that there's exponentiation involved

which we have not implemented yet for

our low value node here so we're not

going to be able to produce 10h yet and

we have to go back up and implement

something like it

now one option here

is we could actually implement um

exponentiation

right and we could return the x of a

value instead of a 10h of a value

because if we had x then we have

everything else that we need so um

because we know how to add and we know

how to

um

we know how to add and we know how to

multiply so we'd be able to create 10h

if we knew how to x

but for the purposes of this example i

specifically wanted to

show you

that we don't necessarily need to have

the most atomic pieces

in

um

in this value object we can actually

like create functions at arbitrary

points of abstraction they can be

complicated functions but they can be

also very very simple functions like a

plus and it's totally up to us the only

thing that matters is that we know how

to differentiate through any one

function so we take some inputs and we

make an output the only thing that

matters it can be arbitrarily complex

function as long as you know how to

create the local derivative if you know

the local derivative of how the inputs

impact the output then that's all you

need so we're going to cluster up

all of this expression and we're not

going to break it down to its atomic

pieces we're just going to directly

implement tanh

so let's do that

depth nh

and then out will be a value

of

and we need this expression here so

um

let me actually

copy paste

let's grab n which is a cell.theta

and then this

i believe is the tan h

math.x of

two

no n

n minus one over

two n plus one

maybe i can call this x

just so that it matches exactly

okay and now

this will be t

and uh children of this node there's

just one child

and i'm wrapping it in a tuple so this

is a tuple of one object just self

and here the name of this operation will

be 10h

and we're going to return that

okay

so now valley should be implementing 10h

and now we can scroll all the way down

here

and we can actually do n.10 h and that's

going to return the tanhd

output of n

and now we should be able to draw it out

of o not of n

so let's see how that worked

there we go

n went through 10 h

to produce this output

so now tan h is a

sort of

our little micro grad supported node

here as an operation

and as long as we know the derivative of

10h

then we'll be able to back propagate

through it now let's see this 10h in

action currently it's not squashing too

much because the input to it is pretty

low so if the bias was increased to say

eight

then we'll see that what's flowing into

the 10h now is

two

and 10h is squashing it to 0.96 so we're

already hitting the tail of this 10h and

it will sort of smoothly go up to 1 and

then plateau out over there

okay so now i'm going to do something

slightly strange i'm going to change

this bias from 8 to this number

6.88 etc

and i'm going to do this for specific

reasons because we're about to start

back propagation

and i want to make sure that our numbers

come out nice they're not like very

crazy numbers they're nice numbers that

we can sort of understand in our head

let me also add a pose label

o is short for output here

so that's zero

okay so

0.88 flows into 10 h comes out 0.7 so on

so now we're going to do back

propagation and we're going to fill in

all the gradients

so what is the derivative o with respect

to

all the

inputs here and of course in the typical

neural network setting what we really

care about the most is the derivative of

these neurons on the weights

specifically the w2 and w1 because those

are the weights that we're going to be

changing part of the optimization

and the other thing that we have to

remember is here we have only a single

neuron but in the neural natives

typically have many neurons and they're

connected

so this is only like a one small neuron

a piece of a much bigger puzzle and

eventually there's a loss function that

sort of measures the accuracy of the

neural net and we're back propagating

with respect to that accuracy and trying

to increase it

so let's start off by propagation here

in the end

what is the derivative of o with respect

to o the base case sort of we know

always is that the gradient is just 1.0

so let me fill it in

and then let me

split out

the drawing function

here

and then here cell

clear this output here okay

so now when we draw o we'll see that oh

that grad is one

so now we're going to back propagate

through the tan h

so to back propagate through 10h we need

to know the local derivative of 10h

so if we have that

o is 10 h of

n

then what is d o by d n

now what you could do is you could come

here and you could take this expression

and you could

do your calculus derivative taking

um and that would work but we can also

just scroll down wikipedia here

into a section that hopefully tells us

that derivative uh

d by dx of 10 h of x is

any of these i like this one 1 minus 10

h square of x

so this is 1 minus 10 h

of x squared

so basically what this is saying is that

d o by d n

is

1 minus 10 h

of n

squared

and we already have 10 h of n that's

just o

so it's one minus o squared

so o is the output here so the output is

this number

data

is this number

and then

what this is saying is that do by dn is

1 minus

this squared so

one minus of that data squared

is 0.5 conveniently

so the local derivative of this 10 h

operation here is 0.5

and

so that would be d o by d n

so

we can fill in that in that grad

is 0.5 we'll just fill in

so this is exactly 0.5 one half

so now we're going to continue the back

propagation

this is 0.5 and this is a plus node

so how is backprop going to what is that

going to do here

and if you remember our previous example

a plus is just a distributor of gradient

so this gradient will simply flow to

both of these equally and that's because

the local derivative of this operation

is one for every one of its nodes so 1

times 0.5 is 0.5

so therefore we know that

this node here which we called this

its grad is just 0.5

and we know that b dot grad is also 0.5

so let's set those and let's draw

so 0.5

continuing we have another plus

0.5 again we'll just distribute it so

0.5 will flow to both of these

so we can set

theirs

x2w2 as well that grad is 0.5

and let's redraw pluses are my favorite

uh operations to back propagate through

because

it's very simple

so now it's flowing into these

expressions is 0.5 and so really again

keep in mind what the derivative is

telling us at every point in time along

here this is saying that

if we want the output of this neuron to

increase

then

the influence on these expressions is

positive on the output both of them are

positive

contribution to the output

so now back propagating to x2 and w2

first

this is a times node so we know that the

local derivative is you know the other

term

so if we want to calculate x2.grad

then

can you think through what it's going to

be

so x2.grad will be

w2.data

times this x2w2

by grad right

and

w2.grad will be

x2 that data times x2w2.grad

right so that's the local piece of chain

rule

let's set them and let's redraw

so here we see that the gradient on our

weight 2 is 0 because x2 data was 0

right but x2 will have the gradient 0.5

because data here was 1.

and so what's interesting here right is

because the input x2 was 0 then because

of the way the times works

of course this gradient will be zero and

think about intuitively why that is

derivative always tells us the influence

of

this on the final output if i wiggle w2

how is the output changing

it's not changing because we're

multiplying by zero

so because it's not changing there's no

derivative and zero is the correct

answer

because we're

squashing it at zero

and let's do it here point five should

come here and flow through this times

and so we'll have that x1.grad is

can you think through a little bit what

what

this should be

the local derivative of times

with respect to x1 is going to be w1

so w1 is data times

x1 w1 dot grad

and w1.grad will be x1.data times

x1 w2 w1 with graph

let's see what those came out to be

so this is 0.5 so this would be negative

1.5 and this would be 1.

and we've back propagated through this

expression these are the actual final

derivatives so if we want this neuron's

output to increase

we know that what's necessary is that

w2 we have no gradient w2 doesn't

actually matter to this neuron right now

but this neuron this weight should uh go

up

so if this weight goes up then this

neuron's output would have gone up and

proportionally because the gradient is

one okay so doing the back propagation

manually is obviously ridiculous so we

are now going to put an end to this

suffering and we're going to see how we

can implement uh the backward pass a bit

more automatically we're not going to be