if i were to summarize the first chapter

into two sentences there would be we

establish a framework to visualize

matrices went through the famous ones

visually they're pretty cool

the second sentence there are two

special matrices we need to remember the

diagonal and the orthogonal matrix

they are special because the

transformation on vectors they produce

have a simple visual interpretation

diagonal matrix scale or stretch each

axis

whereas the orthogonal matrix produces a

rotation

but in the harsh war of matrix most

transformations do not have an easy

visual interpretation and can hardly be

described by words

the symmetric matrix is one example

the goal of this video is by the end you

can look at any symmetric matrix and

trivially say

well its transformation may be visually

complicated but using spectral

decomposition it will just be a sequence

of three simple transformations and i

know exactly what they are

and just like any enlightenment of

knowledge a journey is required to truly

understand the theorem we need to unlock

some tasks of understanding verse

the first task what is a symmetric

matrix as you guess some form of

symmetry exists within

more specifically symmetric around the

diagonal line the numbers on each side

are equal

when the matrix is not square

aka rectangular matrix

is unfortunately never symmetrical

[Music]

the term transpose by itself does not

refer to a matrix but rather transpose

is an action you perform on the matrix

which is to take the rows of a matrix

and make them the columns instead

often times we call the resulting matrix

a transpose or transpose of a

you can likewise transpose a rectangular

matrix but notice the dimension of the

left matrix change

it went from 3x2 to 2x3 now

the action of transposing a matrix may

seem random

but it relates closely to our

protagonist today if you try to

transpose a symmetric matrix you get

exactly the same matrix back

which is why an alternative definition

of symmetric matrix is s equals to s

transpose

but what's not so obvious is that when

you transpose an orthogonal matrix

you actually get its inverse

and this is a very interesting property

that orthogonal matrix have

remember orthogonal matrix produces a

rotation

the inverse of matrix means

untransformation

the untransformation of rotation means

rotation in the reverse direction the

transpose of orthogonal matrix is also

the inverse

the transpose of orthogonal matrix is

rotation in the reverse direction

for example if i tell you orthogonal

matrix q rotates around the z axis by 25

degree

you can simply transpose that matrix

which would rotate around the z axis by

-25 degrees

what is matrix decomposition but let's

ensure we understand matrix composition

first

whenever we multiply matrices we're

essentially composing their distinct

transformations together into one

transformation

for example i can multiply the three

matrices on the right

[Music]

together one matrix on the left

and visually it looks like this

[Music]

matrix decomposition is essentially

going the reverse direction suppose you

have the transformation of one matrix

decomposing means you re-express that as

a sequence of much simpler

transformation

composition of matrices is simple we

just follow the rules of matrix algebra

and multiply them together

on the contrary decomposition is

difficult if someone just throws me a

random matrix and ask decompose this

into three simpler matrices i have no

idea where to even start would the first

matrix be a scaling

a shearing

or a rotation

i don't know

but hopefully

spectral decomposition can be the beacon

of light

if you have never heard of eigenvector

or eigenvalues then the de facto place

to learn them belongs to the essence of

linear algebra by what truly is a

legendary channel some words down in the

comment section really represents my

heart

today forgive me to present a personal

and subjective interpretation on this

concept you see after visualizing so

many different matrices their

transformation really feel like

distortions

vectors are moving to different places

scaled by different factors a skewed to

different directions

if we just pick a random matrix and

follow the movements of a few vector

we see that the linear transformation

knocks out the vector away from their

original direction

and it seems like every vector always

gets skewed away from their original

line no matter which vector we pick

every vector

or is it

actually for this particular matrix

there are two vectors hiding within

they are able to stay on their original

direction in the midst of linear

transformation

those are the eigenvectors

while every other vector deviates from

their initial direction the eigenvectors

stay on their original line despite the

distortion from the matrix

if one vector is an eigenvector all the

vectors along its linear span are also

the eigenvectors

this is a property of matrix being a

linear transformation

but we just pick one eigenvector

normally of a unit length to represent

the set of all eigenvectors along its

line

also note eigenvector is relatively

defined in respect to a matrix different

matrix and have different eigenvectors

the eigenvectors for matrix a

is probably not eigenvectors for matrix

b

and it's entirely possible for a 2x2

matrix to have less than two

eigenvectors

once we have the eigenvectors computed

normally by some software we often

denote them with vector symbol e1 e2

and then we can ask what is the

eigenvalue for each vector

you should think eigenvalue and

eigenvector always come in pairs the

eigenvalue which is a number tells how

much its corresponding eigenvector is

scaled during the matrix transformation

for example the dark blue eigenvector

has an eigenvalue of 2.7 during the

transformation it got stretched longer

by a factor of 2.7

if we watch the animation again and this

time pay attention to the light blue

eigenvector it got scale shorter because

eigenvalue is only 0.6

after seeing the visualization of

eigenvector in action

its formal definition becomes much more

straightforward which is saying the

eigenvectors are such that when

receiving a linear transformation from a

matrix this stay on their original line

only scaled by a factor of its

corresponding eigenvalue

a very strong property which symmetric

matrices have is that its eigenvector

are orthogonal and when we describe

vectors being orthogonal we mean they're

perpendicular

although this statement doesn't seem

that strong with an initial glance but

let's take a moment to appreciate just

how probabilistically unlikely that a

matrix eigenvector are actually

orthogonal

firstly we know that for m by n matrix

it's entirely possible it does not even

have an eigenvectors two by two matrix

can have less than two eigenvectors

even when it does have a full set of

eigenvectors what's the chance that the

two vectors just happen to be

perpendicular exactly 90 degree to each

other pretty unlikely right

but when the matrix is symmetric we can

guarantee to have its eigenvectors being

perpendicular

now you may ask oh yeah i know the

vectors are 90 degrees now what about it

well what else is perpendicular the

standard bases the x and y axis

this means there always exists an

orthogonal matrix which rotates the

bases to align with the eigenvectors

and the inverse of that matrix which

would rotate the eigenvector to align

with the basis

if the eigenvectors are not

perpendicular in the first place such a

rotation transformation would have been

impossible

now we have officially unlocked all

tasks of understanding

and ready to see spectral decomposition

the theorem is saying whenever you have

a symmetric matrix you can always

unconditionally decompose it into a

sequence of three simple matrices

q transpose

lambda

q

but q is an orthogonal matrix lambda is

a diagonal matrix q transpose is also an

orthogonal matrix

by this time when you hear the term

orthogonal diagonal as a decomposition

of a more complicated transformation

this should really be a reaction

precisely the column vector of matrix q

are the normalized eigenvectors of s

the matrix lambda is diagonal and the

numbers of the diagonal are the

eigenvalues of the eigenvectors

arranging from left to right we simply

transpose matrix q to get q transpose

and this decomposition can generalize

for any m by unsymmetric matrices

now we know exactly what matrix q

lambda q transpose r down to their

numerical value that means we can also

be precise about describing their

transformation visually let's start with

the easier one and assume s is 2 by 2.

matrix lambda is diagonal it means we

stretch the x-axis by lambda 1 and

y-axis by lambda 2.

how about matrix q then it's orthogonal

matrix so we know it's some form of

rotation

but what rotation precisely i sort of

gave it away earlier back it rotates our

standard bases to align with

eigenvectors consider vector 1 0 which

represents the x-axis multiplying that

with the matrix q we'll get the first

eigenvector

and likewise for vector 0 1

represents the y axis multiplication

with q

we get a second eigenvector

more generally speaking for higher

dimension matrix q apply a rotation

transformation such that the i th

standard basis moves towards the ith

eigenvector

on the other hand q transpose since it's

the same as q inverse due to a strong

property of orthogonal matrix it would

just be a rotation in the reverse

direction such that the eigenvector move

towards the eighth standard basis

now let's visualize in action

this symmetric matrix can be spectrally

decomposed into the three matrices on

the right

the linear transformation s produces

looks like this

but we know it will just be a sequence

of three simple transformations on the

right

firstly we identify the eigenvectors of

s

and rotate them onto the standard bases

secondly we scale the x-axis by 6

and y-axis

by

2. thirdly we rotate backwards from the

standard bases to the eigenvectors

likewise we can also apply a spectral

decomposition on a 3x3 symmetric matrix

let's look at what happens to the tiny

color cube under s

we also know this transformation is

exactly the same as a three sequential

transformation on the right

firstly we identify the three

perpendicular eigenvectors of matrix s

and rotate them onto the standard bases

secondly we scale the x-axis by 4.15

y-axis by -1.56

since it's a negative scaling factor

vectors are going the reverse direction

by the magnitude

and then a tiny bit stretching in the

z-axis by 1.31

in reality matrix lambda does three

scalings simultaneously i'm just

breaking them apart so it's easier to

see

and thirdly a rotation backwards from

the standard bases to the eigenvectors

and the transformations we just witness

is exactly the same as s

and now we can also look at the effects

of decomposition on image

[Music]

often times the topic of spectral

decomposition is taught in school as a

list of procedures followed by the

students which quickly turns into

tedious finger exercise and boring

algebra

while the underlying elegant geometric

interpretation is dismissed

what the theorem gives us is another

chance to express a more complicated

transformation

just as a sequence of simple rotation

scaling rotation back

in particular the rotation is guided by

the eigenvector and the scaling are told

by the eigenvalues

and i think the connection here

is quite profoundly beautiful

and this

is a visualization of spectral

decomposition

the problem is

just how often do you get a symmetric

matrix

if you just go into matrix line and grab

a matrix in random

firstly it's probably not square

if it's square it's probably not

symmetric

in reality the theorem is pretty with a

loss of generality

but would it be possible we can take the

similar spirit of decomposition and

generalize for all matrices doesn't need

to be symmetric or square but any

arbitrary matrix of any dimension

and this leads us to chapter 3 the grand

finale of linear algebra singular value

decomposition

see you there

[Music]

you