Neural networks need to map inputs to outputs. It seems simple enough, but in most useful cases this means building a network with millions of parameters, which look at millions or billions of relationships hidden in the input data. Do we need all of these relationships? Is all of this information necessary? Probably not. That's where neural network pooling layers can help. In this post, we're going to dive into the deep end and learn how pooling layers can reduce the size of your network while producing highly accurate models.
The example image above shows how max pooling would work on a 4x4 input space, with a 2x2 filter with a skip (or stride) of 2. Max pooling does exactly what you would think: it takes the largest value within the filter space and uses that as the result of the pooling operation. It's easy to see the how dramatically pooling can reduce the size of the data. When the filter contains n data elements, the pooling layer can reduce the data propagated to the next layer by a factor of n.
The one-dimensional max pooling function produces a pooling layer which will return the max value in a filter of pool_size with a stride of strides.
from keras.model import Sequential
from keras.layers import MaxPooling1D
model = Sequential()
model.add(MaxPooling1D(pool_size=5, strides=2))
This particular pooling layer will take in a one-dimensional input, emit the max value of the elements [0..4], then emit the max value of the elements [2..6], and so on. In this way, the amount of data is reduced, but because the stride is smaller than the filter size, there is an overlap in the filters which allows extremely important data points to be represented more than once.
In the end, pooling is another layer in the neural network designer's toolbox, along with dense, convolutional, dropout, normalization, and a host of other layers and neuron types that we will be discussing over the course of this series.
What are Neural Network Pooling Layers?
Neural network pooling layers allow the network to reduce the data in the previous layer by sampling, aggregating, or applying any other mathematical function to a localized region of the input space. This local region is usually specified using a filter, which consists of a size (typically in one or two dimensions) and a skip. The dimensions of the filter determine the size of the sample for the reduction operation, and the skip determines how much overlap occurs when applying the filter. |
| An example of max pooling with a two-dimensional filter (Wikipedia). |
The example image above shows how max pooling would work on a 4x4 input space, with a 2x2 filter with a skip (or stride) of 2. Max pooling does exactly what you would think: it takes the largest value within the filter space and uses that as the result of the pooling operation. It's easy to see the how dramatically pooling can reduce the size of the data. When the filter contains n data elements, the pooling layer can reduce the data propagated to the next layer by a factor of n.
When to use Neural Network Pooling Layers
Neural network pooling layers are most commonly used in networks which have a dramatic reduction in size from input to output, such as image classification networks. Image classifiers take in large data elements (images) and produces a classification (usually a one-hot vector where each element corresponds to a particular classification label). Historically they were used along with convolutional layers in these image classification networks, although more recent image classification networks bypass the pooling layers by using convolutional filters with larger stride values as a way of reducing the data propagated through the network.How to Build Neural Network Pooling Layers in Keras
Keras provides a very simple, layer-wise abstraction for building neural networks (if you're new to Keras, check out my post on how to build a simple digit classifier using Keras), and has a straightforward set of pooling functions in 1, 2, and 3 dimensions for max pooling average pooling, and global average pooling.The one-dimensional max pooling function produces a pooling layer which will return the max value in a filter of pool_size with a stride of strides.
from keras.model import Sequential
from keras.layers import MaxPooling1D
model = Sequential()
model.add(MaxPooling1D(pool_size=5, strides=2))
This particular pooling layer will take in a one-dimensional input, emit the max value of the elements [0..4], then emit the max value of the elements [2..6], and so on. In this way, the amount of data is reduced, but because the stride is smaller than the filter size, there is an overlap in the filters which allows extremely important data points to be represented more than once.
 |
| The MaxPooling1D layer converts the input tensor (top) into the result tensor (bottom). |
Another Layer in the Toolbox
In many cases, reducing the size of the data moving from layer to layer in a neural network is critical to making the network not only trainable, but also usable on low-performance hardware (such as in a phone or on an autonomous drone). Pooling provides a simple way of reducing the data that propagates through the network using common statistical sampling techniques such as max and average. The goal is to try to keep the essence or critical information of the full-sized data stream without having to process the parts that might not matter.In the end, pooling is another layer in the neural network designer's toolbox, along with dense, convolutional, dropout, normalization, and a host of other layers and neuron types that we will be discussing over the course of this series.