The Ultimate Beginner Guide to TensorFlow

How To implement linear regression and gradient descent from scratch!

Why TensorFlow? We already have Keras!

When we build a machine learning model, for example, a convolutional neural network for classifying images, we usually design our network on top of high-level libraries such as Keras.

At the end of the day, a Keras model is converted into a TensorFlow program.

What is TensorFlow?

TensorFlow, open sourced to the public by Google in 2015, is the result of years of lessons learned from a dilemma: should we attempt to do research with inflexible libraries so that we don’t have to reimplement code, or should we use one library for research and a completely different library for production?

TensorFlow was made to be flexible, efficient, extensible, and portable (source). Computers of any shape and size can run it, from smartphones all the way up to huge computing clusters. TensorFlow embraces both open source communities and stability of a large corporation.

While TensorFlow is primarily used to refer to the API used to build and train machine learning models, TensorFlow is, in fact, a bundle of software:

• TensorFlow API which is accessed through a user-friendly environment in Python Keras, while the actual computation is written in C++ for efficiency.
• TensorBoard, a graph visualization software that can give insight into a model’s behavior. This is very useful for analyzing training and for debugging TensorFlow code.
• TensorFlow Serving, a high-performance lightweight server that can take input data, pass it to the trained model, and return the output from the model. Moreover, it can seamlessly switch out old models with new ones, without any downtime for end-users.

Why TensorFlow, and not Keras?

Now, we were asking, why should you use TensorFlow directly, when you already have Keras? Per analogy, why should you develop HTML code when you have Wordpress for building websites?

Well, if you are researching new machine learning architectures, TensorFlow is incredibly flexible and useful for your imagination. Usually, you take models from recent research literature and implement them in TensorFlow, not in Keras.

Moreover, if you are in an agile environment, that requires your models to go to production fast and in short iterations, TensorFlow can give your team the ability to get your product off the ground while scaling up over many devices, including Android devices and mini-computers such as the Raspberry Pi. You can easily deploy your models to run inside web pages on client’s web browser!

Computation Graphs

A model in TensorFlow is represented by computation graph, that is like a series of functions chained together, each passing its output to other functions further along in the chain.

Nodes of the graph are circles, ovals, or boxes, representing some sort of operation. Edges are the actual values that get passed to and from operations, and are typically drawn as arrows. Below, you see a simple graph where the data is traveling from left to right to compute (5*3) + (5+3).

Data is referred to as tensors, which are simply the n-dimensional abstraction of matrices.

Below you can see what it looks like in TensorFlow code.

TensorFlow allows us to create multiple computation graphs, and run them inside a session. We can also define placeholders, which are Tensor objects, with their values specified at runtime.

For example, instead of taking two discrete scalar inputs, we might want the model to take in a single vector of any length. We might also want to accumulate the total value of all outputs over time as we use the graph. This is where placeholders are extremely useful.

We can also segment the graph nicely into named scopes. After each run, we might also want to save the output of the graph, the accumulated total of all outputs, and the average value of all outputs to disk for use in TensorBoard.

This article does not aim at explaining everything you should know about TensorFlow. Instead we will cover a practical example in the next section.

Linear Regression with TensorFlow

Let’s implement and train a single neuron network using TensorFlow API for linear regression.

The linear regression task is solved during training by identifying the weight w and bias b of the straight line y_pred = 𝑤 ∗ 𝑥 + 𝑏 that minimize the squared mean error between y_pred and training data y.

We first need to define placeholders for our regression data X and Y. The weight and bias are represented by tensor variables, which are mutable and will be changed during training loop. The loss function is computed as the mean of squared errors for any two values y_pred and y.

Gradient descent is implemented by defining a placeholder for the derivatives of the loss function as well as a learning step. The weight and bias will be updated relatively to gradient and learning rate during training.

After defining the placeholders, we can now feed our tensor graph with the tensor data previously prepared in the form of a dictionary. Synthetic training data is created by randomly generating a noisy line.

In a session, we loop through the epochs. At each epoch, TensorFlow executes the computation graph. This updates the variables, i.e. the weight and bias we need to estimate. We print their value and the error as illustrated below.

After the graph has run, we can fetch the last weight and bias value, and plot the resulting regression line.

Our linear regression in TensorFlow with a single neuron has resulted in 87% R2 score!

Conclusion

TensorFlow is an opensource software library that focuses on using data flow graph structures. Nodes represent mathematical operations and the edges represent multidimensional data arrays (tensors) flowing between nodes.

Keras is a high level API built on the top of TensorFlow.

While Keras is more user-friendly, TensorFlow offers low-level flexibility, especially when you need to implement custom features. For example, TensorFlow is very useful for objects detection, combined with OpenCV.

Keras can also run on the top of other ecosystems such as Theano or CNTK. While you may find some Theano tutorials, it is no longer in active development. Caffe, an alternative to Keras, lacks flexibility, while Torch uses Lua. MXNet, Chainer, and CNTK are currently not widely popular. You can find a benchmark between TensorFlow and Torch here.

In this article, we demonstrated how easy it is to build, train and evaluate a TensorFlow model for linear regression. If you are interested in learning how to do linear regression with Keras, my article below could be your next exciting read.