Highlights: In this post we will show how to implement a foundamental Convolutional Neural Network like \(LeNet-5\) in TensorFlow. The LeNet-5 architecture was invented by Yann LeCun in 1998 and was the first Convolutional Neural Network.

Tutorial Overview:

1. Theory recapitulation
2. Implementation in TensorFlow

1. Theory recapitulation

The goal of \(LeNet-5 \) was to recognize handwritten digits. So, it takes as an input \(32\times32\times1 \) image. It is a grayscale image, thus the number of channels is \(1 \). Below we can see an arhitecture of this network.

I. Original MNIST is nowadays considered as too easy. In the last two decades, many researchers proposed successful solutions for original MNIST. You can check direct comparison results here.

Because this network is primarily designed for MNIST dataset, it performs significantly better on it. With small changes, it can achieve that accuracy on the Fashion MNIST dataset. However, in this post we will stick to the original architecture of the network.

About LeNet-5 architecture you can read a detailed theoretical post here.

Let’us summarize layers of LeNet-5 architecture.

Layer Type	Feature Map	Size	Kernel Size	Stride	Activation
Image	1	32×32	–	–	–
Convolution	6	28×28	5×5	1	tanh
Average Pooling	6	14×14	2×2	2	–
Convolution	16	10×10	5×5	1	tanh
Average Pooling	16	5×5	2×2	2	–
Fully Connected	–	120	–	–	tanh
Fully Connected	–	84	–	–	tanh
Fully Connected	–	10	–	–	softmax

2. Implementation in TensorFlow

The interactive Colab notebook can be found at the following link   
Run in Google Colab

For practice, you can try to change a dataset by replacing fashion_mnist with mnist, cifar10 or other.

---

Let’s start with importing all necessary libraries. After imports, we can use imported module to load data. The load_data() function will automatically download and split our data into training and test sets.

We can check the shape of new data and see that our images are 28×28 pixels, so we need to add a new axis, which will represent a number of channels. Also, it is important to do one-hot encoding of labels and normalization of input images.

Now, it is time to start using TensorFlow 2.0 in order to build our convolutional neural network. The easiest way to do this is by using the Sequential API. We will wrap it in a class called LeNet. The input is an image, and the output will be a class probability vector.

In tf.keras, Sequential model represents the linear stack of layers and in this case it follows the network architecture.

After creating a model, we need to train its parameters to make it powerful. Let’s train the model for a given number of epochs. We can see the progress of training in TensorBoard.

Not bad for only 20 epochs.

After this, we can make some predictions and visualize it. With predict_classes() function we can get the exact class from the network instead of probability values. It is the same as using the numpy.argmax(). We will display the false predictions with red and the correct ones will be signified with blue.

Summary

So, here we have learned how to develop and train LeNet-5 in Tensorflow 2.0. In the next post we will continue with implementations of popular Convolutional Neural Networks and learn how to implement AlexNet in TensorFlow 2.0.