Linear Algebra with TensorFlow

TensorFlow provides easy-to-use built-in functions for linear algebra

Deep learning and machine learning algorithms use matrix operations in their computations. Most of the matrix operations happen behind the scenes when we train the models in TensorFlow. However, it is worth knowing how to perform linear algebra with TensorFlow. TensorFlow provides easy-to-use built-in functions for this.

Prerequisites

The knowledge of installing and importing TensorFlow, creating tensors, identifying tensor types and attributes is highly recommended. If you’re still not familiar with these topics, feel free to read my post by visiting the following link:

Let’s Create Tensors like NumPy Arrays

Tensors are nothing but matrix representations of multi-dimensional arrays

medium.com

In this post, we’ll discuss how to perform the following matrix operations with TensorFlow.

Matrix multiplication
Element-wise multiplication
Dot product
Tensor addition
Argmax operation
Creating the identity matrix
Trace
Transpose
Determinant
Inverse
Cholesky decomposition
LU decomposition
Eigenvalues and eigenvectors

In addition to that, we’ll also discuss how to reshape a tensor using the tf.reshape() function. This is very useful to match the dimensions of tensors when working with neural networks.

Enough intro! Let’s go into the details of each matrix operation.

Matrix multiplication

In TensorFlow, matrix multiplication can be done using the matmul() function. The general syntax is:

import tensorflow as tf
mat_mul = tf.linalg.matmul(a, b)

Here is an example:

import tensorflow as tf

a = tf.Variable([[1, 2],
                 [3, 4]])

b = tf.Variable([[10, 11],
                 [12, 13]])

mat_mul = tf.linalg.matmul(a, b)

print(a)
print("\n", b)
print("\n", mat_mul)

The matmul() function provides additional arguments to perform specific operations before the multiplication. For example,

mat_mul = tf.linalg.matmul(a, b,
                           transpose_a=True)

This time, the matrix a will be transposed before the multiplication.

Element-wise multiplication

Element-wise multiplication is completely different from matrix multiplication. In TensorFlow, this can be done using the tf.multiply() function.

ele_mul = tf.multiply(a, b)
print(ele_mul)

The star operator (*) also performs the same element-wise multiplication.

print(a * b)

Scalar multiplication is a type of element-wise multiplication.

a = tf.Variable([[1, 2],
                 [3, 4]])

sclr_mul = a * 10
# OR
# sclr_mul = tf.multiply(a, 10)
print(a)
print("\n", sclr_mul)

Dot product

The tensordot() function can be used to calculate the dot product. The general syntax is:

tf.linalg.tensordot(a, b, axes)

When a, b are two vectors (one-dimensional tensors) of equal size and axes=1, the function returns a single number (scalar).

a = tf.Variable([1, 2, 3, 4])
b = tf.Variable([10, 11, 12, 13])

tf.linalg.tensordot(a, b, axes=1)

This returns 120.

When a, b are two matrices (two-dimensional tensors) and axes=1, the function returns the matrix multiplication which is the same as the output of the matmul() function.

a = tf.Variable([[1, 2],
                 [3, 4]])

b = tf.Variable([[10, 11],
                 [12, 13]])

tf.linalg.tensordot(a, b, axes=1)

Tensor addition

Tensor addition can be done in two ways:

With the tf.add() function
With the plus (+) operator

Both ways give the same result.

a = tf.Variable([[1, 2],
                 [3, 4]])

b = tf.Variable([[10, 11],
                 [12, 13]])

add = tf.add(a, b)
print(add)

add = a + b
print(add)

Scalar addition uses broadcasting behind the scenes.

a = tf.Variable([[1, 2],
                 [3, 4]])
print(a + 5)

The number 5 was added to every element in a. This is equivalent to:

a = tf.Variable([[1, 2],
                 [3, 4]])

five = tf.Variable([[5, 5],
                    [5, 5]])
print(a + five)

This is what broadcasting did behind the scenes.

Argmax operation

The argmax operation can be done by using the tf.argmax() which returns the index of the maximum value along an axis in a tensor. The general syntax is:

tf.argmax(tensor, axis=None)

We do not need to specify the axis parameter for one-dimensional tensors (vectors) where we can find only one axis.

c = tf.Variable([1, 2, 10, 9])
arg = tf.argmax(c)
tf.print(arg)

This will return 2 which is the index of the maximum value (10) of c.

For two-dimensional tensors (matrices), we need to specify the axis parameter. When axis=0, the operation is done along with the columns. In other words, the function returns indices of the maximum values in each column in the tensor.

d = tf.Variable([[0, 2, 1],
                 [10, 1, 3]])

arg = tf.argmax(d, axis=0)
tf.print(arg)

When axis=1, the operation is done across the rows. In other words, the function returns indices of the maximum values in each row in the tensor.

d = tf.Variable([[0, 2, 1],
                 [10, 1, 3]])

arg = tf.argmax(d, axis=1)
tf.print(arg)

Creating the identity matrix

The function we use here is the diag() function.

tf.linalg.diag([1, 1, 1])

Trace

Trace is the sum of diagonal elements in a tensor. It can be calculated using the trace() function.

a = tf.Variable([[1, 2],
                 [3, 4]])

tf.print(tf.linalg.trace(a))

This returns 5 (4 + 1).

Transpose

We can use the tf.transpose() function to get the transpose.

a = tf.Variable([[1, 2],
                 [3, 4]])

tf.transpose(a)

Determinant

The determinant can be calculated by using the det() function.

a = tf.Variable([[1, 2],
                 [3, 4]], 
                dtype=tf.float32)

tf.print(tf.linalg.det(a))

This returns -2.

Inverse

The inverse operation can be done by using the inv() function.

a = tf.Variable([[1, 2],
                 [3, 4]],  
                dtype=tf.float32)

tf.linalg.inv(a)

When we try to get the inverse of a singular matrix where the determinant is zero, we’ll get an error message.

a = tf.Variable([[1, 2],
                 [2, 4]],
                dtype=tf.float32)

tf.linalg.inv(a)

Cholesky decomposition

The Cholesky decomposition is defined for square matrices. The matrix should be symmetric (or Hermitian) and positive definite. The cholesky() function can be used to perform the Cholesky decomposition.

a = tf.Variable([[4, 12, -16],
                 [12, 37, -43],
                 [-16, -43, 98]],
                dtype=tf.float32)

cho_decp = tf.linalg.cholesky(a)
print(a)
print("\nCholesky decomposition:")
tf.print(cho_decp)

LU decomposition

The LU decomposition is defined for square matrices. The matrix need not be symmetric and positive definite. However, it should be invertible. The lu() function can be used to perform the LU decomposition.

a = tf.Variable([[1, 2, 5],
                 [4, 7, 3],
                 [6, 3, 9]],
                dtype=tf.float32)

lu_decp = tf.linalg.lu(a)
print(a)
print("\nLU decomposition:")
tf.print(lu_decp)

Eigenvalues and eigenvectors

We can use the eigh() function to calculate the eigenvalues and eigenvectors. It returns an eigendecomposition form where it includes two tensors of eigenvalues and eigenvectors. The first tensor which can be accessed by using index 0 includes the eigenvalues. The second tensor which can be accessed by using index 1 includes the eigenvectors.

a = tf.Variable([[4, 12, -16],
                 [12, 37, -43],
                 [-16, -43, 98]],
                dtype=tf.float32)

eig_decp = tf.linalg.eigh(a)
print(a)
print("\nEigenvalues:")
tf.print(eig_decp[0])
print("\nEigenvectors:")
tf.print(eig_decp[1])

Here are some interesting facts about the eigenvalues.

The matrix a we used to calculate the eigenvalues and eigenvectors is positive definite. Therefore, its eigenvalues are positive (see the output).
The trace of a is the sum of the eigenvalues.

tf.print("Sum of eigenvalues:",
         tf.reduce_sum(eig_decp[0]))

tf.print("Trace:", tf.linalg.trace(a))

The product of the eigenvalues is equal to the determinant of a.

tf.print("Product of eigenvalues:",
         eig_decp[0][0] * eig_decp[0][1] * eig_decp[0][2])

tf.print("Determinant:", tf.linalg.det(a))

The tf.reshape() function (Optional)

This is an optional, but useful part. The tf.reshape() function is used to change the shape of a tensor. The general syntax is:

import tensorflow as tf

tf.reshape(tensor, shape)

The shape should be specified using an array-like object.

The most important thing is that we cannot get any shape. The number of elements in the original shape matrix should be equal to the number of elements in the new shape matrix. For example, a vector of 6 elements can be reshaped into a 2 x 3 or 3 x 2 matrix. The other thing is that the tf.reshape() function does not modify the original tensor unless we assign the new shape to it.

The tf.reshape() function can be used to change the dimension of a matrix. This is very useful when we want to match the dimension of the matrices in calculations.

a = tf.Variable([[1, 2, 3],
                 [3, 4, 6]])

print("Original matrix")
tf.print(a)
print("\nReshaped matrix")
tf.print(tf.reshape(a, [3, 2]))

The tf.reshape() function can be used to add another dimension. This is very useful when working with image data.

a = tf.Variable([[1, 2, 3],
                 [3, 4, 6]])

print("Original matrix")
tf.print(a)
print("Shape", a.shape)
tf.print("Dimensions:", tf.rank(a))

b = tf.reshape(a, [2, 3, 1])

print("\nReshaped matrix")
tf.print(b)
print("Shape", b.shape)
tf.print("Dimensions:", tf.rank(b))

Summary

Linear algebra is often used when working with deep learning algorithms. TensorFlow provides easy-to-use built-in functions for linear algebra. The most important thing is that we should provide valid inputs for those functions. For example, if we provide integer data type tensors for det() and inv() functions, you’ll get an error message. The best practice is that you refer to the documentation of those functions. For example, if you want to use the inv() function, just type “tf.inv” in Google and search. You will get the documentation!

The promise to deliver 0% plagiarism content

Yes, the above content has 0% plagiarism! Before publishing, I’ve checked this using Grammarly.

Writing unique contents that add value for readers in both context and appearance is my own responsibility. To ensure the uniqueness, I made this content with 0% plagiarism.

Until next time, happy learning to everyone! Meanwhile, you can read my other posts at:

https://rukshanpramoditha.medium.com

Special credit goes to Bryan Colosky on Unsplash, who provides me with the cover image for this post. The author copyrights the written content, code samples, other images and content links included in this post.

Rukshan Pramoditha, 2021–07–02