Ahmed Menshawy - Deep Learning By Example: A hands-on guide to implementing advanced machine learning algorithms and neural networks

This is the second and more important type of error in data science. The whole purpose of building learning systems is the ability to get a smaller generalization error on the test set; in other words, to get the model to work well on a set of observation/samples that haven't been used in the training phase. If you still consider the class scenario from the previous section, you can think of generalization error as the ability to solve exam problems that werent necessarily similar to the problems you solved in the classroom to learn and get familiar with the subject. So, generalization performance is the model's ability to use the skills (parameters) that it learned in the training phase in order to correctly predict the outcome/output of unseen data.

In Figure 13, the light blue line represents the generalization error. You can see that as you increase the model complexity, the generalization error will be reduced, until some point when the model will start to lose its increasing power and the generalization error will decrease. This part of the curve where you get the generalization error to lose its increasing generalization power, is called overfitting.

The takeaway message from this section is to minimize the generalization error as much as you can.

It's now time to build the core of our classification application, which is the computational graph of this CNN architecture, but to maximize the benefits of this implementation, we aren't going to use the TensorFlow layers API. Instead, we are going to use the TensorFlow neural network version of it.

So, let's start off by defining the model input placeholders which will input the images, target classes, and the keep probability parameter of the dropout layer (this helps us to reduce the complexity of the architecture by dropping some connections and hence reducing the chances of overfitting):

# Defining the model inputs
def images_input(img_shape):
return tf.placeholder(tf.float32, (None, ) + img_shape, name="input_images")
def target_input(num_classes):
target_input = tf.placeholder(tf.int32, (None, num_classes), name="input_images_target")
return target_input
#define a function for the dropout layer keep probability
def keep_prob_input():
return tf.placeholder(tf.float32, name="keep_prob")

Next up, we need to use the TensorFlow neural network implementation version to build up our convolution layers with max pooling:

# Applying a convolution operation to the input tensor followed by max pooling
def conv2d_layer(input_tensor, conv_layer_num_outputs, conv_kernel_size, conv_layer_strides, pool_kernel_size, pool_layer_strides):
input_depth = input_tensor.get_shape()[3].value
weight_shape = conv_kernel_size + (input_depth, conv_layer_num_outputs,)
#Defining layer weights and biases
weights = tf.Variable(tf.random_normal(weight_shape))
biases = tf.Variable(tf.random_normal((conv_layer_num_outputs,)))
#Considering the biase variable
conv_strides = (1,) + conv_layer_strides + (1,)
conv_layer = tf.nn.conv2d(input_tensor, weights, strides=conv_strides, padding='SAME')
conv_layer = tf.nn.bias_add(conv_layer, biases)
conv_kernel_size = (1,) + conv_kernel_size + (1,)
pool_strides = (1,) + pool_layer_strides + (1,)
pool_layer = tf.nn.max_pool(conv_layer, ksize=conv_kernel_size, strides=pool_strides, padding='SAME')
return pool_layer

As you have probably seen in the previous chapter, the output of the max pooling operation is a 4D tensor, which is not compatible with the required input format for the fully connected layers. So, we need to implement a flattened layer to convert the output of the max pooling layer from 4D to 2D tensor:

#Flatten the output of max pooling layer to be fing to the fully connected layer which only accepts the output
# to be in 2D
def flatten_layer(input_tensor):
return tf.contrib.layers.flatten(input_tensor)

Next up, we need to define a helper function that will enable us to add a fully connected layer to our architecture:

#Define the fully connected layer that will use the flattened output of the stacked convolution layers
#to do the actuall classification
def fully_connected_layer(input_tensor, num_outputs):
return tf.layers.dense(input_tensor, num_outputs)

Finally, before using these helper functions to create the entire architecture, we need to create another one that will take the output of the fully connected layer and produce 10 real-valued corresponding to the number of classes that we have in the dataset:

#Defining the output function
def output_layer(input_tensor, num_outputs):
return tf.layers.dense(input_tensor, num_outputs)

So, let's go ahead and define the function that will put all these bits and pieces together and create a CNN with three convolution layers. Each one of them is followed by max pooling operations. We'll also have two fully connected layers, where each one of them is followed by a dropout layer to reduce the model complexity and prevent overfitting. Finally, we'll have the output layer to produce 10 real-valued vectors, where each value represents a score for each class being the correct one:

def build_convolution_net(image_data, keep_prob):
# Applying 3 convolution layers followed by max pooling layers
conv_layer_1 = conv2d_layer(image_data, 32, (3,3), (1,1), (3,3), (3,3))
conv_layer_2 = conv2d_layer(conv_layer_1, 64, (3,3), (1,1), (3,3), (3,3))
conv_layer_3 = conv2d_layer(conv_layer_2, 128, (3,3), (1,1), (3,3), (3,3))
# Flatten the output from 4D to 2D to be fed to the fully connected layer
flatten_output = flatten_layer(conv_layer_3)
# Applying 2 fully connected layers with drop out
fully_connected_layer_1 = fully_connected_layer(flatten_output, 64)
fully_connected_layer_1 = tf.nn.dropout(fully_connected_layer_1, keep_prob)
fully_connected_layer_2 = fully_connected_layer(fully_connected_layer_1, 32)
fully_connected_layer_2 = tf.nn.dropout(fully_connected_layer_2, keep_prob)
#Applying the output layer while the output size will be the number of categories that we have
#in CIFAR-10 dataset
output_logits = output_layer(fully_connected_layer_2, 10)
#returning output
return output_logits

Let's call the preceding helper functions to build the network and define its loss and optimization criteria:

#Using the helper function above to build the network
#First off, let's remove all the previous inputs, weights, biases form the previous runs
tf.reset_default_graph()
# Defining the input placeholders to the convolution neural network
input_images = images_input((32, 32, 3))
input_images_target = target_input(10)
keep_prob = keep_prob_input()
# Building the models
logits_values = build_convolution_net(input_images, keep_prob)
# Name logits Tensor, so that is can be loaded from disk after training
logits_values = tf.identity(logits_values, name='logits')
# defining the model loss
model_cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits_values, labels=input_images_target))
# Defining the model optimizer
model_optimizer = tf.train.AdamOptimizer().minimize(model_cost)
# Calculating and averaging the model accuracy
correct_prediction = tf.equal(tf.argmax(logits_values, 1), tf.argmax(input_images_target, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name='model_accuracy')
tests.test_conv_net(build_convolution_net)

Now that we have built the computational architecture of this network, it's time to kick off the training process and see some results.

There are lots of libraries out there in Python that you can use to read, transform, or write data. One of these libraries is pandas (http://pandas.pydata.org/). Pandas is an open source library and has great functionality and tools for data analysis as well as very easy-to-use data structures.