Keras, Eager and TensorFlow 2.0 – a new TF paradigm

A recent announcement from the TensorFlow development team has informed the world that some major new changes are coming to TensorFlow, resulting in a new major version TensorFlow 2.0. The three biggest changes include:

  • TensorFlow Eager execution to be standard
  • The cleaning up of a lot of redundant APIs
  • The removal of tf.contrib with select components being transferred into the TensorFlow core

The most important change is the shift from static graph definitions to the imperative Eager framework as standard. If you aren’t familiar with the Eager way of doing things, I’d recommend checking out my introduction to Eager. It will still be possible to create static graph definitions in TensorFlow 2.0, but the push to standardize Eager is a substantial change and will alter the way TensorFlow is used. However, it is already possible to use TensorFlow in this new paradigm before the transition to TensorFlow 2.0. With the newest releases of TensorFlow (i.e. version 1.11), we can already preview what the new TensorFlow 2.0 will be like. The major new TensorFlow paradigm will include the biggest APIs already available – the Dataset API, the Keras API and Eager.

This post will give you an overview of the approach that (I believe) the TensorFlow developers are pushing, and the most effective way of building and training networks in this new and upcoming TensorFlow 2.0 paradigm.

Recommended online course: If you’re interesting in learning more about TensorFlow, and you are more of a video learner, check out this inexpensive online course: Complete Guide to TensorFlow for Deep Learning with Python

Bringing together Keras, Dataset and Eager

Many people know and love the Keras deep learning framework. In recent versions, Keras has been extensively integrated into the core TensorFlow package. This is a good thing – gone are the days of “manually” constructing common deep learning layers such as convolutional layers, LSTM and other recurrent layers, max pooling, batch normalization and so on. For an introduction into the “bare” Keras framework, see my Keras tutorial. The Keras layers API  makes all of this really straight-forward, and the good news is that Keras layers integrate with Eager execution. These two factors combined make rapid model development and easy debugging a reality in TensorFlow. 

A final piece of the puzzle is the flexible and effective Dataset API. This API streamlines the pre-processing, batching and consumption of data efficiently into your deep learning model. For more on the capabilities of this API, check out my TensorFlow Dataset tutorial. The good news is that Datasets are able to be consumed in the Keras fit function. This means that all three components (Dataset, Keras and Eager) now fit together seamlessly. In this post, I’ll give an example of what I believe will be an easy, clear and efficient way of developing your deep learning models in the new TensorFlow 2.0 (and existing TensorFlow 1.11).  This example will involve creating a CIFAR-10 convolutional neural network image classifier. 

Building an image classifier in TensorFlow 2.0

In this example, I’ll show you how to build a TensorFlow image classifier using the convolutional neural network deep learning architecture. If you’re not familiar with CNNs, check out my convolutional neural network tutorial. The structure of the network will consist of the following:

  • Conv2D layer – 64 filters, 5 x 5 filter, 1 x 1 strides – ReLU activation
  • Max pooling – 3 x 3 window, 2 x 2 strides
  • Batch normalization
  • Conv2D layer – 64 filters, 5 x 5 filter, 1 x 1 strides – ReLU activation
  • Max pooling – 3 x 3 window, 2 x 2 strides
  • Batch normalization
  • Flatten layer
  • Dense layer – 750 nodes, with dropout
  • 10 node output layer with Softmax activation

I’ll go through each step of the example, and discuss the new way of doing things as I go. 


1
2
3
4
5
import tensorflow as tf
from tensorflow import keras
import datetime as dt

tf.enable_eager_execution()

First things first, in TensorFlow 2.0 it is not expected that the tf.enable_eager_execution() line will need to be executed. For the time being however, in TensorFlow 1.10+ we still need to enable the Eager execution mode. In the next code segment, I setup the training dataset using the Dataset API in TensorFlow, and extracting the CIFAR10 data from the Keras datasets library:


1
2
3
4
5
6
(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()

train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(32).shuffle(10000)
train_dataset = train_dataset.map(lambda x, y: (tf.div(tf.cast(x, tf.float32), 255.0), tf.reshape(tf.one_hot(y, 10), (-1, 10))))
train_dataset = train_dataset.map(lambda x, y: (tf.image.random_flip_left_right(x), y))
train_dataset = train_dataset.repeat()

The lines of code above show how useful and streamlined the Dataset API can be. First, numpy arrays extracted from the Keras dataset library are loaded directly into the Dataset object using the from_tensor_slices method. Batching and shuffling operations are then added to the Dataset pipeline. On the following line, the Dataset map method is used to scale the input images x to between 0 and 1, and to transform the labels into a one-hot vector of length 10. After this a random distortion (randomly flipping the image) is applied to the pre-processing pipeline – this effectively increases the number of image training samples. Finally, the dataset is set to repeat indefinitely i.e. it does not halt extraction after all the dataset has been fed through the model – rather it allows the dataset to be resampled.

The following code shows the same treatment applied to the validation dataset, however with a larger batch size and no random distortion of the images:


1
2
3
valid_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(1000).shuffle(10000)
valid_dataset = valid_dataset.map(lambda x, y: (tf.div(tf.cast(x, tf.float32),255.0), tf.reshape(tf.one_hot(y, 10), (-1, 10))))
valid_dataset = valid_dataset.repeat()

The lines above perform a significant amount of data preprocessing in a streamlined fashion, showing the advantages of the Dataset API.

Now that the input data pipeline has been constructed, it is time to create the convolutional neural network to classify the images:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
class CIFAR10Model(keras.Model):
    def __init__(self):
        super(CIFAR10Model, self).__init__(name='cifar_cnn')
        self.conv1 = keras.layers.Conv2D(64, 5,
                                         padding='same',
                                         activation=tf.nn.relu,
                                         kernel_initializer=tf.initializers.variance_scaling,
                                         kernel_regularizer=keras.regularizers.l2(l=0.001))
        self.max_pool2d = keras.layers.MaxPooling2D((3, 3), (2, 2), padding='same')
        self.max_norm = keras.layers.BatchNormalization()
        self.conv2 = keras.layers.Conv2D(64, 5,
                                         padding='same',
                                         activation=tf.nn.relu,
                                         kernel_initializer=tf.initializers.variance_scaling,
                                         kernel_regularizer=keras.regularizers.l2(l=0.001))
        self.flatten = keras.layers.Flatten()
        self.fc1 = keras.layers.Dense(750, activation=tf.nn.relu,
                                      kernel_initializer=tf.initializers.variance_scaling,
                                      kernel_regularizer=keras.regularizers.l2(l=0.001))
        self.dropout = keras.layers.Dropout(0.5)
        self.fc2 = keras.layers.Dense(10)
        self.softmax = keras.layers.Softmax()

    def call(self, x):
        x = self.max_pool2d(self.conv1(x))
        x = self.max_norm(x)
        x = self.max_pool2d(self.conv2(x))
        x = self.max_norm(x)
        x = self.flatten(x)
        x = self.dropout(self.fc1(x))
        x = self.fc2(x)
        return self.softmax(x)

The code above shows how Keras can be now integrated into the Eager framework. Notice the class definition inherits from the Keras.Model class. This is called model subclassing. The structure of such a model definition involves first creating layer definitions in the class __init__ function. These layer definitions are then utilized in the call method of the class. Note, this way of doing things allows easy layer sharing – for instance, two different inputs could use the same layer definitions, which would also involving sharing weights. This occurs in networks such as Siamese Networks and others. In this case, I’ve utilized the same max pooling definition for both convolutional layers. The feed forward pass during the Keras fit function will automatically use the call method of the model passed to it. The Keras compile function can also be called directly from the instantiated model.

This is one way of doing things. Another way is by utilizing the Keras Sequential model type. This is possible in Eager due to the fact that placeholder functions are not required, which don’t work in Eager mode (given that they operate on a deferred execution basis). However, the Keras Functional API cannot currently be used in Eager mode, given that the Functional API requires some form of placeholder inputs. The Sequential model type in Keras is handy but not overly flexible, so it may be best to stick with the model subclassing approach shown above – but it really depends on your application.

After the model definition, it is then time to instantiate the model class, compile in Keras and run the training:


1
2
3
4
5
6
7
8
9
10
11
model = CIFAR10Model()
model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='categorical_crossentropy',
              metrics=['accuracy'])
callbacks = [
  # Write TensorBoard logs to `./logs` directory
  keras.callbacks.TensorBoard(log_dir='./log/{}'.format(dt.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")))
]
model.fit(train_dataset, epochs=100, steps_per_epoch=1500,
          validation_data=valid_dataset,
          validation_steps=3, callbacks=callbacks)

In this example, we are using the TensorFlow Adam Optimizer and the Keras categorical cross-entropy loss to train the network. Rather than having to define common metrics such as accuracy in TensorFlow, we can simply use the existing Keras metrics. Keras can also log to TensorBoard easily using the TensorBoard callback.  Finally, in the Keras fit method, you can observe that it is possible to simply supply the Dataset objects, train_dataset and the valid_dataset, directly to the Keras function. 

Running the above code in Google Colaboratory on a Tesla K80 GPU yields a training accuracy of around 78% and a validation accuracy of around 60% after 200 epochs. This result could be improved with some more iterations and tuning. 

The plots below from TensorBoard show the progress of the per-iteration training accuracy and the per-epoch validation accuracy:

Keras TensorFlow 2.0 - CIFAR-10 training accuracy
Batch CIFAR-10 training accuracy
Keras TensorFlow 2.0 - CIFAR-10 validation accuracy
Epoch CIFAR-10 validation accuracy

In this tutorial, I’ve presented what I believe to be the direction the TensorFlow developers are heading in with respect to the forthcoming release of TensorFlow 2.0. This direction includes three key themes which are already available – the Dataset API, the Keras API and Eager execution. Because these themes are already available for use in TensorFlow 1.10+, this post will hopefully aid you in preparing for the release of TensorFlow 2.0. 

Happy coding

Recommended online course: If you’re interesting in learning more about TensorFlow, and you are more of a video learner, check out this inexpensive online course: Complete Guide to TensorFlow for Deep Learning with Python

1 Comment

  1. Hello Andy!
    Your tutorials and articles are top-notch, always help me understand and implement, and come through various difficulties. Great Work, keep it up!
    Also, I wanted to suggest something that please arrange all these tutorials in a manner such that we can easily find one after the other, a tutorial structural form, it would be good for newcomers to understand. The work you are doing is very fruitful, keep it up!

Leave a Reply

Your email address will not be published.


*