In this video, we will have a look at

object detection, which involves both

classification and bounding box

regression to identify the location of

an object in an image.

This video serves as a basic

introduction before we have a look at

the YOLO model in the next video.

I will first show how bounding box

regression works before we have a look

at some Python code.

I here assume that you are familiar with

the CNNs, transfer learning, and image

classification. If not, have a look at

this video.

In this video, we will train a

convolutional neural network to identify

heads of cats and dogs. We will also

train the model to classify if the image

contains a cat or a dog. So, to train

such a model, we first need images of

cats and dogs. Then, we need to draw a

bounding box around the head in each

image, which will serve as the ground

truth box that the network is trained

on.

This image has 800 pixels in width and

1,200 pixels in height. When we draw

this bounding box, we'll get the

location of this box in pixel

coordinates. There are a number of tools

out there that you can use to extract

such information. Here are a few

examples. The pixel coordinate of the

top left corner is 160 and 90.

The height of the bounding box is 420

and the width is 400 pixels.

We may store our data like this.

Where these columns show the size of the

image. X and Y define the coordinates of

the top left corner of the bounding box,

and these two columns show the width and

height of the box.

Whereas the last column tells if the

image contains a cat or a dog.

Then, we collect the same information

about the following image.

Note that these two columns represent

the pixel coordinates of the top left

corner of the bounding box.

In, for example, the YOLO model, one

should instead provide the coordinates

of the center of the box.

The numbers are usually normalized

before training a neural network because

the training works best if the numbers

are between 0 and 1.

To normalize the X coordinate of the top

left corner, we simply divide it by the

width of the image.

And to normalize the Y coordinate of the

top left corner, we divide it by the

height of the image and so forth.

One nice thing with normalized values

is that we can rescale the image to a

different size

and simply multiply the normalized

values by the new width and height of

the image so that we get the coordinates

to draw the bounding box in the resized

image.

So, to train a network to classify cats

and dogs and to identify the heads, we

will use an existing network, which

means that we will utilize transfer

learning.

The network you see here is the

so-called VGG-16 model.

This network was originally trained on

the ImageNet dataset to classify images

including things like cars, flowers,

birds, and fruits.

The network has 1,000 output nodes

because the ImageNet dataset had 1,000

different categories to classify.

We will modify this network so that it

is trained to only classify cats and

dogs and to identify their heads with

bounding box regression.

We first have to delete the fully

connected layers.

This bar represents a one-dimensional

vector that includes the flattened

values from the 512 channels of the

final pooling layer.

Then we add a dense or a fully connected

layer including, for example, 64 nodes.

Finally, we add a five output nodes.

Where this node will predict the

normalized X coordinate of the top-left

corner of the bounding box.

This node will predict the normalized Y

coordinate of the top-left corner of the

box.

Whereas these two nodes will predict the

width and height of the box.

The last node will predict if the image

contains a cat or a dog.

Since we have only two classes to

predict, we only need one node for the

classification.

If you have more than two classes, you

need one node for each class and use a

softmax function.

For all output nodes, we use the sigmoid

activation function, which constrains

the predicted values between zero and

one.

One simple type of loss function that

can be used for bounding box regression

is the Huber loss function.

Suppose that the network has predicted

the following bounding box.

For example, the ground truth box of the

normalized X coordinate is here 0.2.

Whereas the predicted X coordinate is

here 0.3.

The absolute difference between these

two values is 0.1.

Delta is a tuning hyperparameter that

controls how sensitive the Huber loss is

the outliers.

A larger value makes the loss behave

more like the mean squared error, which

gives a tighter fit to small errors, but

makes the model more sensitive to

outliers.

A smaller value makes it behave more

like the mean absolute error, which is

less sensitive to outliers.

Since the absolute difference is less

than one in this case,

we basically compute the squared error.

The total loss for the bounding box is

the Huber loss based on the difference

in X and the Y coordinates and the width

and height between the values of the

ground truth box and the predicted

values.

For the classification, we can use the

standard binary cross-entropy loss

function.

For example, the target value is zero if

the image contains a cat and one if the

image contains a dog.

The predicted value comes from this

output node as a probability.

Suppose that the network predicts that

the image contains a dog with 80%

probability.

Then the loss is about 1.61.

A simple way to compute the total loss

is then to add the Huber loss with the

binary cross-entropy loss function,

where you can put different weights on

these two functions.

During the training, we like to reduce

the total loss

so that the predicted box is as close as

possible to the ground truth box

and that the predicted probability for

cats and dogs is as close as possible to

their target values.

So, during training, the network might

predict the following box

and that the image contains a dog. To

correct this incorrect classification

and the difference in location and size

between the ground truth box and a

predicted box,

the network updates its weights in the

fully connected layers. One may also

unfreeze the weights in the last

convolutional layers of the existing

network to improve the performance.

Note that the output layer can be seen

as two different heads. The first head

is predicting the location and size of

the bounding box and is controlled by

the Huber loss function,

whereas the second head predicts if the

image contains a cat or a dog and is

controlled by the binary cross entropy

loss function.

All the 64 nodes in the fully connected

layer are here connected to the five

output nodes.

But it is also possible to add one or

several layers that are separate for the

heads.

For example, this layer is only

connected to head one.

This means that head one has more

weights to optimize, which is a common

architecture because the bounding box

regression sometimes requires more

layers for training.

The goal with the training is to

optimize the weights of the network so

that the predicted box is as close as

possible to the ground truth box by

minimizing this loss function and

optimize the weights for the

classification head so that the network

predicts cats and dogs as good as

possible.

To evaluate the performance of the

network, we usually split the images

into training, validation, and test

data.

The training data is used to optimize

the weights of the network, whereas the

validation data is used to optimize the

hyper parameters and try different

architectures of the network and to make

sure that we do not overfit.

Once we have developed and trained our

final optimized model, we check its

performance on completely unseen images

with bounding boxes in the test data

set.

One performance metric for the

classification part of the network is to

compute the accuracy based on the

validation data and the test data. In

this example, the network correctly

predicts the class in eight out of the

10 images, which gives an accuracy of

80%.

To evaluate the performance of the

bounding box regression, one usually

computes the intersection over union.

This is our ground truth or target

bounding box.

The area of this box covers 9,000

pixels.

When the network processes this image,

it may predict the following box given

its current weights.

The predicted box has an area of 4,096

pixels.

And the intersection covers 2,560

pixels.

We can now calculate the intersection

over union where we plug in the area of

the intersection here

and the areas of the target box and the

predicted box.

Note that we subtract the area of the

intersection from the total area because

the overlapping region would otherwise

be counted twice.

This gives us an intersection over union

score of about 0.24.

If the predicted box were over

where it does not intersect the target

box,

the intersection over union score would

be zero.

And if the predicted box would have the

same size and location as the target

box,

the intersection over union would be

one. With that, we know that the

intersection over union spans between

zero and one, and that the closer to one

we are, the better the predicted box

fits the target box.

>> So, why not use the following loss

function for the bounding box

regression?

>> If the intersection over union is one,

the loss is zero.

The problem occurs if there is no

overlap between the predicted box and

the target box.

The loss would in this case be equal to

one here,

as well as here. Such a function do not

provide any moving gradient towards the

target box, which means that the model

does not know which direction to move

the box.

One method to solve this problem is to

use the distance intersection over union

loss that was presented in the following

paper.

The distance intersection over union

loss includes the following penalty

term,

where the numerator represents the

squared Euclidean distance between the

central points of the two boxes, and C

is the diagonal length of the smallest

enclosing box covering the two boxes.

I will now show some basic code in Keras

and TensorFlow that can be used on the

examples we just discussed. Note that

this code is just for educational

purposes, which explains the basics of

object detection.

In the next video, I will show how to

train a YOLO model, which works a lot

better.

As an example, we will here use the

Oxford pet data set that you can

download from the following website.

You may download the images of cats and

dogs here.

This folder contains information about

the bounding box and segmentation.

However, we will not use that folder.

Instead, I have prepared a CSV file

based on the XML files of the data set

that you can download from my home page.

This CSV file includes the file names in

the Oxford pet data set, the width

and the height of the images.

It also includes the X coordinate of the

top left corner of the bounding box

around the heads

and the corresponding Y coordinate

as well as the width and height.

This column includes the breed, which we

will not use.

Instead, we'll use this column which

tells if the image contains a cat or a

dog.

I will now explain a simple Python code

for object detection that you can try.

This code is available on my home page.

We begin by importing the following

libraries.

Next, we define the path to our CSV file

that contains all the information about

the bounding boxes and the labels for

each image.

Note that you need to change this path

to where you have saved the file.

The information in the CSV file will be

saved as a Pandas data frame.

We also save the path to the folder

where we have the images of cats and

dogs.

Make sure that the image folder contains

the images.

Then we define the input image size for

our network.

We will here use VGG16,

which takes images of size 224 * 224.

This code creates three empty lists that

will later store the images, bounding

boxes, and labels.

Then we'll loop through all the rows in

the data frame that stores the

information in the CSV file.

Where each iteration constructs the full

path to an image file by combining the

path to the image directory with the

file name in the data frame.

Then we store the image and make sure

that the image loads successfully.

This code stores the height and width of

the loaded image,

which is used to normalize the bounding

box coordinates of the top left corner

and the width and height.

Next we code cats as zeros and dogs as

ones and store these in the variable

label.

Then we resize the image to 224 * 224

and normalize the pixel values in the

range between 0 and 1 by dividing them

by 255.

Finally, we store the image, bounding

box, and the label in the lists.

Next we convert the data into NumPy

arrays

and split the data set into training and

test data, where 20% of the images are

used as unseen test data.

We now start to build our network, where

we first import VGG16

and use its optimized weights based on

the ImageNet dataset.

By setting the parameter include top to

false, we do not include the fully

connected layers of VGG16.

Then we make sure to freeze the layers,

which means that we do not update the

weights during the training.

We will here unfreeze the last four

layers, which corresponds to all layers

in the last block, so that these are

updated during the training.

You may try to change this value to

unfreeze more or less layers.

Then we build the head for the bounding

box regression,

where we flatten the last layer from

VGG16,

and add three fully connected layers

to the four output nodes.

The output nodes involve the sigmoid

activation function to constrain the

predicted values between zero and one,

because we have normalized data, but you

may also try a linear activation

function.

Note that you may change the structure

of the head and the activation functions

to test if you get a better performance.

For this example, I tried a few existing

networks,

but VGG16 performed best for bounding

box regression, but a bit worse for the

classification.

For the classification head, we use

global average pooling of the last layer

from VGG16,

which usually works better than flatten

for classification.

We here use only one dense layer with 32

nodes, because the classification tends

to overfit if we use more. After this

line, one may also add a dropout layer

to prevent overfitting.

Since the classification only involves

two classes, it is enough to have just

one output node for the classification

head.

This line defines the final neural

network with two output heads.

Finally, we compile the model where we

say that we like to use the Adam

optimizer with a learning rate of

0.0001,

the Huber loss function for the bounding

box regression, and the binary cross

entropy loss function for the

classification.

Here we can define how much weight we

should put on the two loss functions.

I here use an equal weight, but you may

try to change these values to improve

the performance.

With this line, we can define what

performance metrics that should be shown

during the training.

This is the total training loss,

which is the sum of the bounding box

loss and the classification loss when we

use equal weights for the two loss

functions.

The mean squared error of the bounding

box regression is also shown,

as well as the classification accuracy.

After each epoch, the corresponding

validation performances will be printed.

This code defines the training of the

network on the images in the training

data

based on the target boxes and the target

classes.

We run 20 epochs, use a batch size of

eight, and 10% of the images as

validation data.

The data is shuffled before splitting

into training and validation sets, but

the split is here not stratified.

So, after each epoch, the performance

metric is shown based on this validation

data.

Finally, we check the performance

metrics based on the test data set after

the training is done.

We see that we correctly classified 97%

of the images.

And that the bounding box regression

means squared error is about 0.0047.

You may also compute the intersection of

a union to evaluate the performance of

the bounding box regression.

To see how well the predicted bounding

box fits the head of the cats and dogs,

we can pick an arbitrary image from the

test data set and multiply the pixel

values by 255 because we previously

normalized the images by dividing by

255.

The current width and height of the

images are 224 pixels.

We extract the normalized X and Y

coordinates and the width and height of

the ground truth box.

And multiply the width and height of the

image.

Then we use our trained model to predict

the class and the bounding box.

And extract the bounding box and class

probability from the prediction.

Next, we compute the X and Y coordinates

and the width and height of the

predicted box.

If the predicted probability is greater

than 0.5, we classify the image as

containing a dog. Else, as a cat.

If the ground truth label is one, we

know that it is a dog and a cat if the

target value is equal to zero.

Finally, we plot the image.

And draw the ground truth box in blue

color.

And the predicted box in green color.

And place the ground truth label.

And the predicted label here.

Here are some examples based on the

images in the test data set.

Our simple code seems to work quite

well.

Put a comment below if you managed to

improve the performance.

This was the end of this video about

object detection and bounding box

regression. Thanks for watching.