#016 CNN Network in Network – 1×1 Convolutions
Network in Network – 1×1 Convolutions
In terms of designing ConvNet architectures one of the ideas that really helps is using a 1\times 1 convolution. You might be wondering what does 1\times 1 convolution do? Isn’t that just multiplying by a number? It seems like a funny thing to do. However, it turns out that it’s not quite like that. Let’s take a look!
What does a 1\times 1 convolution do?
An example of 1\times1 convolution
We can see 1\times 1 filter which consists of just one number, number 2 . If we take this 6\times 6\times 1 image and convolve it with a 1\times 1\times 1 filter, we obtain the image and multiply it by 2 .
A convolution by a 1\times 1 filter doesn’t seem totally useful. We just multiply it by some number, but that’s the case of 6\times 6\times 1 channel images. If we have a 6\times 6\times 32 , instead of 1\times 1 , then a convolution with a 1\times 1 filter can do something that makes much more sense. Let’s look in the following picture.
An example of a 1\times 1 convolutions on a 3D images
In particular a 1\times 1 convolution will look at each of the 36 different positions (6\times 6 ), and it will take the element-wise product between 32 numbers on the left and the 32 numbers in the filter. Then, a ReLU non-linearity will be applied. Looking at one out of the 36 positions, maybe one slice through this volume, we take these 32 numbers, multiply it by 1\times 1 slice through the volume, and we get a single number.
In fact, one way to think about the 32 numbers we have in this 1 \times 1 \times 32 filter is that if we have one neuron. That is, taking as input 32 numbers, multiplying them by 32 weights and then applying a ReLU non-linearity to it, and then obtaining the corresponding result as the output.
More generally, if we have not just one filter but multiple filters, then it’s as if we have not just one unit but multiple units that are taking as inputs all the numbers in one slice, and then building them up into an output there the 6\times 6\times number \enspace of \enspace filters . One way to think about the 1\times 1 convolution is that it is basically like having a fully connected neural network that applies to each of the 36 different positions. What this fully connected neural network does, it has a 32 dimensional input whereas the number of outputs equals the number of 1\times 1 filters applied. Doing this every 36 positions we end up with an output that is 6\times 6 \times number \enspace of \enspace filters . This can carry out a pretty non-trivial computation on our input volume. This idea is often called a 1\times 1 convolution, but sometimes it’s also called a Network\enspace in\enspace Network . This idea has been very influential. It has influenced many other neural network architectures, including the Inception\enspace network which we’ll see in the next posts.
Let’s see an example where a 1\times 1 convolution is useful.
Using 1\times 1 convolutions
An example of how we can reduce a number of channels witx 1\times 1 convolution
If we want to shrink the height and width we can use a pooling layer, and we know how to do that. But what if the number of channels has gotten too big and we want to shrink that? How do we shrink that into a 28\times 28 \times 32 dimensional volume?
What we can do is use 32 filters that are 1\times 1 , and technically each filter would be of dimension 1\times 1 \times192 because the number of channels in our filter has to match the number of channels in input volume. But we use 32 filters and the output of this process will be a 28\times 28 \times 32 volume. This is a way to let us shrink n_{c} (number of channels). We’ll see later how this idea of 1\times 1 convolutions allows us to shrink the number of channels and therefore save on computations and networks. But of course, if we want to keep the number of channels to the 192 that’s fine too. The effect of a 1\times 1 convolution is that we apply non-linearity that allows us to learn the more complex function. Adding another layer also helps us to learn more complex functions. So, we can have the input which is 28 \times 28 \times 192 dimensional and the output is 28 \times 28 \times 192
That’s how a 1\times 1 convolutional layer is actually doing something pretty non-trivial. It has a non-linearity too, in our network and allow us to decrease or keep the same, or increase the number of channels in our volumes. In our next post we’ll see that this is actually very useful for building the Inception\enspace network . To conclude, a 1\times 1 convolution operation is actually doing a pretty non-trivial operation and allows us to shrink the number of channels in our volumes or keep it the same or even increase it if we want.
