Real-World Examples of 0D, 1D, 2D, 3D, 4D and 5D Tensors
Data representation in neural networks: Neural Networks and Deep Learning Course: Part 4
This is Part 4 of our Neural Networks and Deep Learning Course as introduced here. Read Part 1, Part 2 and Part 3 if you haven't.
Introduction
Tensors are the basic data structure in machine learning and deep learning models. A tensor can be considered as a container for numerical data (numbers).
In neural networks, data is represented by using tensors. The input layer of a neural network holds text, speech, audio, images, videos or any other kind of data as tensors that takes numbers. We perform tensor operations (tensor addition, multiplication, reshaping, etc.) throughout the network with these tensors of numerical data.
What is a tensor?
The term “tensor” is a technical term in the context of deep learning. The deep learning library “TensorFlow” was named keeping tensors in mind.
Tensors are nothing but multi-dimensional NumPy arrays that we’re already familiar with!
Note: If you want to get hands-on experience in creating tensors with NumPy and TensorFlow, read my “Let’s Create Tensors like NumPy Arrays” article. Today, more emphasis will be given to describing the real-world examples of tensors with different dimensions.
Dimensions, axes and rank of a tensor
Dimensions
A tensor can contain an arbitrary number of dimensions. Therefore, a tensor can be 0D (no dimension!), 1D, 2D, 3D, 4D, 5D and so on. Specific names are given to tensors depending on the number of dimensions they have.
Therefore, a 0D tensor is specifically known as a Scalar. A 1D tensor is specifically known as a Vector. A 2D tensor is specifically known as a Matrix. 3D and higher-dimensional arrays are just tensors!
In some contexts, only the 3D and higher-dimensional arrays are considered as tensors. But in deep learning, “tensor” is a general term that is used to refer to an array of any dimension.
Axes
The term “axis” (plural: axes) is another way to refer to a dimension of a tensor. For example, a 1D tensor (i.e. a vector) has a single axis. Likewise, a 2D tensor (i.e. a matrix) has two axes and so on.
Rank
The number of axes (dimensions) of a tensor is called its rank. For example, a 0D tensor (i.e. scalar) is a rank-0 tensor. Likewise, a 1D tensor (i.e. a vector) is a rank-1 tensor and so on.
Top takeaway: The dimensionality or rank denotes the number of axes in a tensor.
Properties of a tensor
After a tensor is created in NumPy or TensorFlow, it has the following properties.
- Rank: As we defined earlier, the rank is the number of dimensions (axes) of a tensor. In TensorFlow, the rank of a tensor is accessed by using the
tf.rank()function. In NumPy, it is accessed by using the.ndimattribute. - Shape: The shape of a tensor returns a tuple of integers indicating that the number of elements in each dimension (axis). For example, in the case of a matrix with
shape = (2, 3), it means that one axis has 2 elements and the other one has 3 elements. The total elements are 6 (2 x 3) which is the multiplication of the two integers inside the tuple. In both NumPy and TensorFlow, the shape of a tensor is obtained by calling the.shapeattribute. - Data Type: This refers to the type of data inside a tensor. This will return data types such as
int32,int64,float32,float64, etc. The 32-bit data types are default in TensorFlow. The 64-bit data types are default in NumPy.
Note: A scalar has an empty shape of
(). A vector has a shape like(5,).
Real-world examples of tensors
We’ll start with rank-0 tensors and end up with rank-5 tensors. We’ll give real-world examples for each type of tensor along with what each axis means in the case of rank-2 and higher tensors.
Note: To understand most of the examples described below, you need to know the difference between the array representations of RGB color images and grayscale images. You can read this post written by me to learn more about that.
Rank-0 tensors (scalars or 0D tensors)
A rank-0 tensor contains only one number. It is just a scalar!
Example: A pixel value in a grayscale image.
Let’s consider a pixel value of 10 that represents a color close to black.
Rank-1 tensors (vectors or 1D tensors)
A rank-1 tensor contains a set of numbers in a single axis.
Example 1: A pixel value in an RGB image.
Let’s consider an RGB pixel value of [255, 255, 0] that represents the yellow color.
Example 2: A flattened grayscale image.
A grayscale image can be flattened into a rank-1 tensor as in the following diagram.
Rank-2 tensors (matrices or 2D tensors)
A rank-2 tensor is an array of vectors. It has two axes.
Example 1: A tabular dataset with rows and columns.
In this case, the two axes denote (samples, features) which is equivalent to rows and columns. For example, (5, 4) means that the dataset has 5 observations (rows or samples) and 4 variables (columns or features).
Example 2: A single non-flattened grayscale image.
In this case, the two axes denote (height, width) which is equivalent to the height and width of an image in pixels. For example, (28, 28) means that the array holds a single grayscale image of size 28 x 28 in pixels.
Example 3: A batch of flattened grayscale images.
In this case, the two axes denote (samples, features) which is equivalent to the number of flattened grayscale images and pixel values of each image. For example, (3, 9) means that the array contains 3 flattened grayscale images each having 9 pixels.
Rank-3 tensors (3D tensors)
A rank-3 tensor is an array of several matrices. It has 3 axes.
Example 1: A batch of grayscale images.
In this case, the three axes denote (samples, height, width). For example, (1000, 28, 28) means that the array contains 1000 grayscale images of size 28 x 28 in pixels.
Example 2: A single RGB image.
In this case, the three axes denote (height, width, color_channels). For example, (28, 28, 3) means that the array contains a single RGB image of size 28 x 28. The color_channels axis always takes the value 3 in the case of an RGB (Red, Green, Blue) image.
Rank-4 tensors (4D tensors)
A rank-4 tensor is created by arranging several 3D tensors into a new array. It has 4 axes.
Example 1: A batch of RGB images.
In this case, the four axes denote (samples, height, width, color_channels). For example, (1000, 28, 28, 3) means that the array holds 1000 RGB images of size 28 x 28 in pixels. The color_channels axis always takes the value 3 in the case of an RGB (Red, Green, Blue) image.
Example 2: A single video.
In this case, the four axes denote (frames, height, width, color_channels). For example, (240, 720, 1280, 3) means that the array holds a 240-frame HD (720 x 1280) video. Each frame in a video is a color image. Therefore, a frame is denoted by (height, width, color_channels). A video can be considered as a sequence of frames. A 240-frame video is sampled at 4 (240/60) frames per second. In an HD video, each frame is 720 x 1280 size in pixels.
Rank-5 tensors (5D tensors)
A rank-5 tensor is created by arranging several 4D tensors into a new array. It has 5 axes.
Example: A batch of videos.
In this case, the five axes denote (samples, frames, height, width, color_channels). For example, (5, 240, 720, 1280, 3) means that the array holds a batch of five 240-frame HD (720 x 1280) videos.
The samples axis (samples dimension)
In the above notations such as (samples, features) , (samples, height, width) , (samples, height, width, color_channels) , (samples, frames, heights, width, color_channels) , the first axis (i.e. the axis 0 in index notation) is always the samples axis. In a tabular dataset, the term “samples” means observations. In image data, the term “samples” means the number of images. In video data, the term “samples” means the number of videos.
The batch axis (batch dimension)
When we consider a small batch (part) of the dataset instead of considering the whole dataset, the sample axis is referred to as the batch axis. This is just a technical term often found in the context of deep learning.
This is the end of today’s post.
Never miss a great story again by subscribing to my email list. You’ll receive every story in your inbox as soon as I hit the publish button.
If you’d like, you can sign up for a membership to get full access to every story I write and I will receive a portion of your membership fee.
As always, happy learning to everyone!
Rukshan Pramoditha 2022–01–10
