In this video, we will have a look at

how to use different AI methods for

anomaly detection. We will mainly focus

on a padding method.

We'll first see how to detect an anomaly

by creating a heat map based on the

Mahalanobis distance.

Then, we will see how padding works by

extracting channels from a pre-trained

CNN.

Next, we will have a look at some Python

code for anomaly detection by using the

anomaly package. And finally, try to

understand how the image and pixel area

under the curve is calculated.

In this video, I will use the MVTec

dataset that works as a benchmark for

different anomaly detection methods with

the focus on industrial applications.

We'll here focus only on the images of

capsules.

For example, suppose that this image

shows a capsule with good quality from

the production.

Whereas, this image shows a capsule with

bad quality.

Anomaly localization means that we

determine where in the image the

deviation occurs.

While anomaly detection means that we

have identified a capsule that deviates

from the normal case.

One way we could automatically identify

bad quality capsules is to train a

neural network based on images including

good and bad quality, so that it learns

to classify such images.

The problem is that we do not know what

kind of defect we might observe in the

future.

If the following error appears in a

production which the network has not

been trained on, the network will not

know how to classify this image.

A better alternative is therefore to use

a so-called one-class model that is

trained only on normal data so that it

learns the distribution of capsules with

good quality. This is called

unsupervised anomaly detection.

Once such a model has been trained, it

can be used to compute an anomaly score

of new images, which tells how much the

capsule deviates from the normal.

A color image consists of three

channels, one red, one green, and one

blue channel.

If you put these channels on top of each

other, where the three colors are mixed,

we'll get this image.

An image consists of pixels, and this

particular image consists of 1,000 *

1,000 pixels.

Each pixel has three values that tell

the intensity of the red, green, and

blue channels. These pixel values

usually span between 0 and 255.

Let's simplify so that the image

consists of only 20 pixels.

A white color in the image is

represented by high intensity values in

the red, green, and blue channels.

A black color is associated with low

values in the three channels.

While an orange color is represented by

a high value in the red channel, but

relatively low values in the green and

blue channels.

Suppose that we have a defect in the

capsule here.

Since the color changes from orange to

white at this location, there will be a

change in the value of the following

pixels.

The pixel values in the blue channel

will increase to 255,

which is true also for the green

channel.

The values in the red channel will not

deviate that much because the values in

the red channel were already high.

By studying the pixel values, we have

identified an anomaly in the image.

Let's say that we will make a scatter

plot of the given pixel of many images

with good quality capsules. For example,

a value of 80 in the green channel and a

value of 68 in the blue channel will

result in the following point.

If you take another image of the same

pixel location, it may have a value of

105 in the green channel and a value of

102 in the blue channel.

So, this plot shows the distribution of

values from the blue and green channels

at the given pixel in the 219 training

images from capsules with good quality

in the MVTec data set.

High intensity values likely come from

capsules where the pixel includes some

of the white text, whereas low intensity

values likely come from images where

this pixel has an orange color.

Let's calculate the average intensity of

the green and blue channels in this

pixel

based on the 219 images in the training

data set.

Then, we calculate the covariance matrix

where this value shows how much spread

there is between the images in this

pixel in the green channel.

And this is how much spread there is in

the blue channel.

And these values show how much the two

channels spread together.

The mean and the covariance matrix of

the green and blue channels in this

pixel from the images with capsules of

good quality can now be used for anomaly

detection.

Training a basic anomaly detection model

simply means that we compute the mean

and the variance of pixels in the images

with capsules of good quality.

Once the model has been trained,

we can use a new image of a capsule to

determine if it deviates from the normal

good quality capsules. We therefore like

to know if the green and blue channel in

the given pixel deviates from the values

of the images with good quality.

The green and blue values in this pixel

are 110 and 100.

Since these pixel values can be seen as

the distribution of the images with

capsules of good quality,

this pixel does not deviate from the

normality because it's within the

ellipse.

To check if this value in this pixel is

within the cloud of pixel values from

the images from good quality, we can

compute the Mahalanobis distance where

we plug in its coordinates,

the mean,

and the inverse of the covariance

matrix.

If we do the math, we see that this

pixel value deviates about 1.6

Mahalanobis distances from the mean.

In contrast, this new image has some

dark pink color in this pixel because

the pixel value in the green channel is

here only 50.

The Mahalanobis distance between this

point and the mean is 29, which

indicates that the pixel in this image

deviates a lot from the corresponding

pixels in the normal images.

This calculation can be extended to

include all three color channels, so

that we compute the Mahalanobis distance

in the three-dimensional space.

Note that if we instead would use the

Euclidean distance, the distance from

the center to the two points would be

about the same. This explains why we use

the Mahalanobis distance instead of the

Euclidean distance.

If you compute the Mahalanobis distance

between all pixels of the three channels

in this test image, then we can generate

the following heat map.

For example, a blue color means that

there is a relatively short Mahalanobis

distance between the following pixel of

the test image and the average of the

pixel values in the training images with

good quality.

Here we have detected an anomaly because

the pixel values in this region deviates

a lot from the training images.

The Mahalanobis distance in some of

these pixels are greater than 1,000.

This is how we can use the Mahalanobis

distance for anomaly detection. The main

drawback of this method is that it is

highly sensitive to colors in the

pixels.

For example, the Mahalanobis distance

will not be able to identify the

following imprinting error because the

color in these pixels will be quite

similar to the colors in the training

data.

It is also highly sensitive to the exact

position of the capsule.

In addition, it will not identify this

change in shape because the color in the

pixels is quite similar to the capsules

of good quality.

Although it identifies this area because

the image has an orange color where the

white background is visible in the

training images.

Note that any dust in the background is

also indicated in the anomaly map.

We now have a look at the PaDiM method

that was published in 2020.

It actually works very similar to the

Mahalanobis method.

During training, one computes the mean

and the covariance matrix in every

position in the images with the good

quality capsules.

Then one computes the Mahalanobis

distance between every position in the

test image and the average of the

corresponding positions in the training

data

to create this heat map.

The main difference in the PaDiM method

is that the Mahalanobis distance is not

computed based on the pixel values in

the image.

Instead, it is calculated based on

channels in the convolutional layers of

a pre-trained CNN network.

What you see here is the classic VGG16

network

that was trained on the ImageNet

dataset.

When VGG16 was trained on the ImageNet

dataset, it learned how to classify the

1,000 categories in this dataset such as

cats, dogs, and cars.

The early layers in this network have

learned to identify simple patterns,

edges, and color contrasts.

If you plug in this image in this

network,

the network will incorrectly predict

this medical capsule as a rubber eraser

with 50% probability. This incorrect

prediction is likely due to the fact

that the network has not been trained on

such type of objects.

Anyway, we'll not use the network to

predict, so we do not need to include

this part of the network.

So, to train our padding model, we plug

in the training images into the

pre-trained network based on the weights

that were optimized on ImageNet data

set, which means that we do not train

this network. We just use it to compute

the feature maps.

For example, we may extract the first

four layers from the network,

which contain the image features of one

image in the training data set. One

simple way to upsample a matrix is to

use the nearest neighbor method,

where we simply assign the empty

elements with the value of the closest

original value.

Another method is to use the bilinear

method, which gives a smoother gradient.

So, once we have doubled the size of

these feature maps,

all four layers will have the same size.

Note that layers three and four have 128

channels,

and that layers one and two have 64

channels each.

In total, we therefore have 384

channels.

In comparison to the previous

Mahalanobis distance method, where we

only had three channels,

we may have several hundred channels,

depending on which network we use and

how many layers we extract.

Since it would be too computationally

expensive to have that many channels to

compute the Mahalanobis distance in,

One method they proposed in the paper

was to randomly select 100 channels and

compute the Mahalanobis distance on

these.

One of the models they proposed in the

paper was to use 100 random channels

from ResNet 18.

Note that we do not train this network.

We just use it for feature extraction of

every image in the training data set.

So, to train our model, we plug in the

training images with good quality

capsules.

Then, we extract the computed feature

maps from each image and compute the

mean and covariance matrix of every

element in the feature maps for all

training images.

The training part therefore only

involves estimating the normal

distribution of the values in the

elements in the channels.

Once we have trained the model, we can

plug in a test image and compute the

Mahalanobis distance for every element

in the channels.

They'll produce an anomaly map.

If the test image deviates from the

training images with good quality

capsules, we have identified an anomaly.

We'll now see how to perform anomaly

detection by using the Python package

Anomalyb.

We first need to download some images,

and we will here use the MVTec data set.

In this example, we will only use the

capsule data set.

Once you click on this link, you should

see that the data is being downloaded.

After you have unzipped the downloaded

file, you should place the folder

capsule in an appropriate place on your

computer. I here place this folder on

the C AIMVTech.

In the train folder, we have 219 images

that show capsules of good quality.

These are the images that we will train

the model on.

The test folder shows different

categories of bad quality capsules.

For example, if you look into this

folder,

we'll find images that show capsules

with some defect.

Note that the test data set also

includes a folder with images of good

quality capsules.

These images are used to compute how

well the model can discriminate between

good and bad quality.

In the folder ground truth,

you will find the ground truth images

where an expert has labeled the defects.

This white region in a masked image

identifies the defect in this capsule.

You may install Anomalyb like this.

After installing Anomalyb, we can import

the following libraries.

And define the folder where we have the

capsule folder.

We here say that we like to train only

on the category capsule.

Next, we define the type of model we

like to use. We will here use PaDiM,

which is very fast to train.

You may also try other models such as

the PatchCore model.

This line sets up everything before the

training.

Here is where we train the model based

on the training data set.

Whereas this line is used to see how

well the model performs based on the

test data set and the ground truth

masks.

If you run this code, you will get the

following table. But, note that the

method includes random components. So,

you will probably not get the exact same

values as shown here.

After you run the code, you should be

able to see a folder called results in

your current Python working directory,

where you should be able to find the

resulting images

and your trained model.

We previously saw that the pixel area

under the curve was about 0.98.

We'll now see how such a value is

calculated.

This shows the pixel ROC curve for the

following image in the test data set.

The area below this curve is about

0.987.

To generate this ROC curve, we first

create an anomaly map of the image in

the test data set.

Note that pixels in anomaly map are

feature map elements.

From this anomaly map, we compute the

predicted mask based on a certain

cutoff.

The values in the anomaly map have been

normalized by the maximum Mahalanobis

distance observed in all maps in the

test data set. The values in these

anomaly maps can therefore not be

greater than one.

The dark blue color corresponds to

values around 0.3.

Whereas green or yellow color correspond

to values around 0.6.

And dark red color to values close to

one.

So, if you for example use a cutoff

value of 0.75,

we can create this predicted mask,

which can be compared to the ground

truth mask.

This means that regions in the anomaly

map that are greater than 0.75

will result in a yellow color in the

predicted mask.

Whereas values that are less than 0.75

will get a purple color.

Now, regions with the yellow color in

the predicted mask that also have a

yellow color in a ground truth mask will

be defined as true positives.

Whereas elements that have a yellow

color in the predicted mask but not in a

ground truth mask will be defined as

false positives.

If both elements have a purple color the

elements are defined as true negatives.

And if the ground truth has a yellow

color whereas the predicted mask has a

purple color

those elements are defined as false

negatives.

The true and false positive rates are

then calculated based on all pixels for

a given cutoff.

For a cutoff of 0.75,

we get a true positive rate of about

0.89

and a false positive rate of about 0.02.

If we reduce the cutoff to 0.4

we get the true positive rate of one but

an increased false positive rate.

Because the predicted mask has yellow

color in the corresponding yellow color

in the ground truth which results in

almost no false negatives.

But the predicted mask has a lot of

yellow color that is not included in the

ground truth which results in many false

positives.

In comparison, a higher cutoff value

will result in very low false positive

rate because there are almost no false

positives,

but quite many false negatives, which

will result in a low true positive rate.

The corresponding F1 score is calculated

like this.

The image area under the curve is

calculated based on good and bad images

in the test data set.

Remember that we had one folder in the

test data set with images showing good

quality capsules.

One way to calculate a single score

for an image is to compute the maximum

value from the anomaly map.

For example, this is the maximum value

in the anomaly map for an image in the

test data set with a good quality,

whereas this is the maximum anomaly

value for an image in the test data set

that has been defined as bad quality. We

can see that bad quality capsules

generally have a higher maximum value

than capsules with good quality.

If we use a cut off of about 0.7, we'll

get a true positive rate of about 0.2

and a false positive rate of zero

because we have no false positives with

this high cut off.

If we reduce the cut off to 0.5, we'll

now get a true positive rate of about

0.7 and a false positive rate of about

0.1.

With a cut off of about 0.4, we have a

false positive rate of 0.4 and a true

positive rate of one because we have no

false negatives with such a low cut off.

We'll now see how to create the

following images in AnomalyLib.

We first import the following packages

and use the predicted function to

compute the anomaly maps and the

predicted masks based on all images in

the test data set.

The results are stored in different

batches and we will here have a look at

the images in the first batch.

From this batch, we can take an

arbitrary image

and save the path to that image

and create an array of this image.

Then we do the same for the ground truth

image.

Then we extract the tensor that includes

the values from the anomaly map

and the predicted mask.

Finally, we will plot the original

image,

the anomaly map,

the predicted mask,

and the ground truth mask.

Note that there is some dust in this

image,

which explains where we identify an

anomaly here.

Also note that you should be able to

find a similar figure in the fuller

results in your current Python working

directory, which is automatically

generated once you have trained and

tested your model.

This was the end of this basic video

about anomaly detection. Thanks for

watching.