SVD, singular value decomposition, is

[music] a grand finale of linear

algebra. On one hand, it combines all

important concept of the subject into

just one theorem. On the other hand,

it's extremely relevant and applicable

in the age of data science and machine

learning. This is a one matrix

decomposition that rules them all.

[music]

On surface, the SVD is saying that any

matrix, regardless of symmetry, rank,

shape, can unconditionally be decomposed

into three very special matrices.

But, there's actually an intuitive

[music] and elegant visual

interpretation underneath.

I think it would be quite cool to see

for ourselves. Don't you?

And yep, there is a catch. [music] To

truly internalize and appreciate the

theorem, a journey is required. And

hence, the previous chapters.

For example, it's good to know that

diagonal matrix stretches each axis.

[music]

While the orthogonal matrix produces a

rotation.

I highly recommend chapter two. To be

honest, if you understand spectral

decomposition, you understand 80% of

SVD.

And there are three more tiny steps we

need to take to complete the journey.

You probably noticed all the

visualization we have seen so far comes

from square matrices.

Very sneaky on my part. However, the

primary appeal of SVD is it generalizes

to rectangular matrices as well.

Therefore, we need to understand how to

visualize such things.

But firstly, what exactly is difference

between the two vectors here?

My teacher say the left one is in R2,

the right one is in R3.

I used to think those are the same

thing. I mean, they both have one for X,

two for Y, and they got nothing for Z,

right? Even if we do represent them on

graph, they look somewhat similar.

I go ahead asking the vector on the

right about Z value. It said, "Oh,

zero."

I go ask the Z value of the left vector.

It got very confused, having no idea

what a third dimension is ever like.

In addition, I can go nudge the R3

vector around such that Z value is no

longer zero. But for the R2 vector, no

matter how I rotate, stretch, scale, it

is always devoid of Z.

Despite looking similar, in reality,

they're a completely different species

that live in different dimension of the

universe.

If those two types of vector are so

different, is there a mechanism which

can transform one to the other?

And that's the power of rectangular

matrix.

In particular, a 2 by 3 matrix has the

ability to take a vector in R3,

transforming that down to a vector in

R2.

Makes quite a lot of sense if we just

look at the matrix vector

multiplication.

A matrix of size M by N has the power to

transform a vector in the nth dimension

to a vector in the mth dimension.

This is why we say matrix apply a linear

transformation from Rn to Rm.

Remember, we can represent vector as

arrow

and a represent arrow as dot

and a collection of dots as object.

Simply illustrating the visual of a

matrix is simple, but the process of

interpretation is hard. The

visualization of rectangular matrix is

very confusing. So, it's good if we try

to understand the simplest case first,

which is this matrix. You see, it's

similar to the identity matrix, but the

rectangular version of it. For the

context of this video, let's give it a

cool name, the dimension eraser.

This matrix here represent the simplest

form of linear transformation from R3 to

R2.

It's multiplication with any vector

always preserve the X and Y, but

completely remove the Z value regardless

of the initial Z. So, what this means is

that, for example, vector 1 2 1 in R3,

we can map down to vector 1 2. But, all

the vector in the form of 1 2 any Z

transforms to 1 2.

We can also have something like

dimension adder. For example, this 3 by

2 dimension adder here basically a pin

zero as a Z value for whichever input R2

vector.

Oh, actually, I think here is a good

place for me to show you a composition

from R3 to R2.

Whenever we multiply matrix and a

matrix, we essentially combine their

distinct linear transformations. Here,

we can multiply the dimension eraser

with this diagonal matrix.

[music]

And let's call this product matrix C.

The dimension eraser takes away the

third dimension.

And then diagonal matrix stretches the X

and Y axis accordingly.

The matrix C, since it's a composition

of the two matrices on the right, it

basically encapsulates the two

transformations but in just one go.

Chapter two flashback. Symmetric matrix

is a square matrix in which on two sides

of the diagonal line, the entries are

identical. It has a very strong property

which almost no other matrices have. The

eigen vectors of symmetric matrix are

perpendicular to each other. So, that

means if we normalize the eigen vectors

and package them into a matrix, we get

an orthogonal matrix which implies

rotation.

The transpose rotates the eigen vector

to align with the standard basis.

If we don't take the transpose, that

matrix rotates the standard basis to the

eigen vector.

Symmetric matrix is nice. We got to take

advantage.

Yet, everyone knows most matrices in

nature are not symmetrical.

But, we just happen to have the ability

to artificially construct symmetry out

of nowhere. Consider this matrix here,

which is obviously not symmetrical.

If we take the transpose and multiply

them together,

we get a square matrix,

but also symmetric.

We can also put the transpose on the

left and do the multiplication. We'll

once again see a square matrix

that is also symmetrical.

It's good to take a moment here to

appreciate we just created two symmetric

matrices from a rectangular matrix A.

In general, this is true for any matrix

A, which AA transpose and A transpose A

are symmetric matrices.

See if you can prove this statement on

your own.

For now, we stick to the concrete case

when A is 2 by 3.

And let's give them some meaningful

names as well.

Since they're symmetric matrices, the

letter S better be in there.

Let's call AA transpose S left

and A transpose A

S right.

I know that you know S left and S right

are symmetric matrices.

So, S left would have two perpendicular

eigen vectors in R2 and S right would

have three perpendicular eigen vectors

in R3.

Since all those eigen vectors are

closely related to the original matrix

A, we have special names for them as

well.

The eigen vector of S left are known as

the left singular vector of A. And

likewise, the eigenvectors of S right

are the right singular vectors.

Up next, I'm about to provide two facts

without going to the detail. S left and

S right are known as PSD matrices. This

implies the eigenvalue for each

eigenvector are non-negative.

The second fact, which is not obvious,

if we sort the eigenvalues in descending

order for both set, the overlap ones are

numerically identical. The biggest

eigenvalue of S left equals to the

biggest eigenvalue of S right, and so

forth.

The leftover eigenvalue is guaranteed to

be zero.

Just like the singular vectors, those

shared eigenvalues are indirectly

derived from the very original matrix A.

If we take the square root

and my friend, those are the singular

values of matrix A.

A lot of things we did so far seem

random.

We have ambiently collected all pieces

of knowledge which we need to understand

SVD.

Now behold his entrance.

Once again, any matrix A can be

unconditionally decomposed into three

very special matrices, in which the

matrix sigma is rectangularly diagonal,

the matrix V and U are orthogonal

matrices.

But what exactly are they?

The matrix sigma would have the same

dimension as matrix A. The numbers on

the diagonal are the singular values of

matrix A arranged in descending orders.

Every other entry is zero.

The matrix U contains the normalized

eigenvectors of S left, which are

arranged in descending order of their

eigen values. Another way of saying this

is matrix U contains the left singular

vectors of matrix A.

On the other hand, matrix V contains the

normalized right singular vectors of

matrix A, also arranged in descending

order. And then we transpose that to get

V transpose.

And remember, this generalizes to all

kinds of matrix A.

The moment I've been waiting for, the

visualization.

The matrix A here, by itself, applies a

complicated linear transformation from

R3 to R2.

But we know, using SVD, it can be

perfectly understood as sequentially

applying the three simple matrices on

the right.

The blue matrix, or V transpose, is an

orthogonal matrix, which applies a

rotation such that the right singular

vectors return to the standard bases.

So, more precisely, the singular vector

with biggest singular value lands on X

axis, the singular vector with the

second biggest singular value goes on

the Y axis, and so forth. [music]

The matrix sigma is rectangularly

diagonal, and it's essentially a square

diagonal matrix composed with a

dimension eraser.

The dimension eraser is removing the

third dimension.

And through that, the diagonal matrix

stretches X and Y axis based on the

singular value.

And the final step, the matrix U,

rotates the standard bases to align with

the left singular vectors.

And boy, you bet, those transformations

composed is exactly the same as A.

So, Wikipedia said a few things about

the visual flavor of SVD, that any

matrix essentially just maps sphere into

ellipsoid across different dimensions.

Let's see for ourselves.

After singular value decomposition, the

first step of any linear transformation

is always a rotation in Rn.

If you rotate a sphere, it's still a

sphere.

Taking away the third dimension of a

sphere in R3 makes it a circle in R2.

Stretching uniformly along the X and Y

axis makes a circle an ellipse.

And a final rotation still preserves the

geometry of the ellipse, just changing

that to a different position in R2.

And by no means that was any rigorous

proof, but certainly some intuitions.

And lastly, let's look at an example of

a linear transformation from R2 to R3.

The spirit of SVD very much still holds.

Firstly, we begin with the rotation in

R2.

But next, since the matrix sigma is 3 by

2, we scale based on the singular

values.

And then, we're adding a new dimension.

And finally, we rotate the standard

basis XYZ to align with the left

singular vectors.

This video should end now.

Nevertheless, it's good to contemplate

on a question.

Is the entire purpose of SVD to

decompose a linear transformation into

those four sequential actions?

Is our visualization the correct

interpretation of SVD itself?

Not exactly. I mean, what even is a

matrix to begin with?

This mathematical object has different

interpretations under different

contexts.

The same matrix can mean very different

things to different people.

Another popular interpretation of SVD is

to view a high rank matrix as a

summation of rank one matrices.

An application of this is low rank

approximation.

We can view a picture as a massive

matrix of pixels

and then gradually approximate that

picture by adding more and more rank one

matrices.

A wise man once said, "The purpose of

computing is insight, not numbers."

I think the same goes for visualizations

as well.

SVD as a math exercise is an intense

algebraic procedure to follow.

Sometimes, after three pages of matrix

multiplication and characteristic

polynomial, I understood of nothing of

what I just did.

But decompose and visualize a matrix A

into four distinct pieces of simple

transformation gives you so much more

insights than you'd otherwise have. Each

and every single step interpretable and

makes perfect sense. We know precisely

what's going on.

In particular, that very initial step of

rotation from the right singular vector

onto the standard basis actually

captures the essence of something called

principal component analysis.

I promise a video in the future.

Even when things extend beyond the three

dimension, this type of decomposition

still give us a reasonable intuition of

linear transformation on vectors,

allowing us to say about things which we

couldn't see. We know that a rotation

does not make things bigger or smaller.

Therefore, when this matrix acts on

higher dimensional object, nothing

scale, squash, blown out of proportion,

but pristine rotation around the origin.

And after that, the last two dimensions

of the object is taken away.

Sometimes learning the final theorems of

subject is like climbing a mountain. We

get a nice scenery on the top, a higher

level overview.

But, it's nice to look afar at the other

summits. And what we see is a Fourier

transform.

I just want to say some similarity they

share about the two monumental theorems

obsession with extreme generalization

and decomposition.

The Fourier transform is saying, "Any

function can be decomposed as the sum of

sinusoidal wave functions."

And then when you see the pie creature,

you already know it's a good video. I

left link here.

And thank you for watching through the

end.