Summary

The web content provides an in-depth analysis of the Transformer architecture, focusing on positional encoding, self-attention, multi-head attention, and masked multi-head attention, and contrasts absolute and relative positional encoding methods.

Abstract

The article "Inside Transformers: An In-Depth Look at the Game-Changing Machine Learning Architecture — Part 2" delves into the intricacies of the Transformer model, a pivotal architecture in machine learning for natural language processing tasks. It explains the necessity and methodology of positional encoding to incorporate sequence order information, which the Transformer lacks inherently due to its non-recurrent, non-convolutional structure. The original Transformer model uses absolute positional encoding with sine and cosine functions of varying frequencies, allowing the model to learn the position of each word. The article also introduces relative positional encoding, which adjusts attention scores based on the relative distances between words, potentially offering advantages for tasks sensitive to word order and sequence length. The author illustrates these concepts with examples and concludes by hinting at further exploration in subsequent posts.

Opinions

The author believes that positional encoding is a crucial component for the Transformer architecture to handle the order of words in a sequence.
Absolute positional encoding, as introduced by Vaswani et al., is presented as an effective method for providing positional information, with the sine and cosine functions facilitating the learning of relative positions.
Relative positional encoding is suggested to be beneficial for certain tasks, particularly where the relative position of words is more relevant than their absolute positions.
The author implies that relative positional encodings could be more efficient for very long sequences, addressing potential limitations of absolute positional encodings.
The article conveys that the Transformer's self-attention mechanism alone does not consider word order, and positional encoding is essential to mitigate this.
By providing a detailed explanation and examples, the author expresses confidence in the reader's ability to understand the complex concepts being discussed.

Inside Transformers: An In-depth Look at the Game-Changing Machine Learning Architecture — Part 2

Let’s dig deeper into some of these components now. I will explain positional encoding, self-attention, multi-head attention, and masked multi-head attention. The rest are simple and the above explanations are enough.

Positional Encoding

Positional encoding is a key component of the Transformer architecture that allows it to consider the order of words in a sequence. The transformer architecture doesn’t inherently understand the order of the sequence because it doesn’t have recurrence like RNNs or convolutions like CNNs. Positional encoding provides a way of injecting information about each token’s position in the sequence into the model.

Absolute Positional Encoding: In the original Transformer model, Vaswani et al. used a specific function to add a vector to the embedding of each token, providing absolute position information. For a given position, the elements of the positional encoding vector are computed as follows:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Here, pos is the position, and i ranges over the dimensions of the encoding (from 0 to d_model/2-1, where d_model is the dimensionality of the embeddings). The authors used sine for even indices and cosine for odd indices to create a positional encoding that would allow the model to easily learn to attend by relative position, as detailed in the paper.

The intuition here is that these functions create unique positional encodings that the model can easily distinguish and use to learn the position of each word in the sequence.

Sure, let’s look at how the Transformer uses positional encoding with an example.

Consider a simple sentence: “I love cats.” After tokenization, we might represent this sentence as a sequence of integers, where each integer is an index that corresponds to a word in our vocabulary: [9, 27, 301]. Each word is then transformed into a dense vector using a learned embedding. Let’s assume we’re using a 6-dimensional embedding space for simplicity. For example, the word “I” might be represented as a vector [0.1, 0.3, 0.9, -0.4, 0.5, 0.8], “love” as [-0.3, 0.8, -0.6, 0.1, 0.7, -0.8], and “cats” as [0.2, -0.9, 0.6, 0.3, -0.4, -0.5]. The Transformer then adds positional encodings to these embeddings to give position information.

For the absolute positional encoding, we would calculate the encoding as described in the original “Attention is All You Need” paper. For a 6-dimensional embedding, and for position ‘pos’, the positional encoding would look like this:

PE(pos, 0) = sin(pos / 10000^(0/6)) = sin(pos) PE(pos, 1) = cos(pos / 10000^(0/6)) = cos(pos) PE(pos, 2) = sin(pos / 10000^(2/6)) PE(pos, 3) = cos(pos / 10000^(2/6)) PE(pos, 4) = sin(pos / 10000^(4/6)) PE(pos, 5) = cos(pos / 10000^(4/6))

For position 1 (word “I”), we might get a positional encoding like [0.84, 0.54, 0.15, 0.99, 0.01, 1.0]. Similarly, we can calculate encodings for position 2 (“love”) and 3 (“cats”).

These positional encodings are then element-wise added to the word embeddings before they are fed into the encoder layers.

Relative Positional Encoding:

Relative positional encoding was later introduced to improve Transformer’s performance on certain tasks, where the relative position of words (how far apart they are) is more important than their absolute position in the sentence.

In this case, rather than adding a positional encoding to the embeddings before the attention calculation, the attention scores are adjusted based on the relative positions of words. Specifically, a bias is added to the attention scores that is based on a learned embedding for each possible relative position.

The exact details of how this is implemented can vary. For example, the paper “Attention is All you Need” introduced a “skewing” operation to efficiently calculate these relative positions.

In practice, relative positional encodings often perform similarly to absolute positional encodings. However, they might have advantages in certain settings, such as when dealing with very long sequences or when the relative position of words is particularly important.

Let’s do our example for relative positional encoding. In this case, the model learns a separate embedding for each possible relative position. For example, in a sequence of length 3, there are 3 possible relative positions: -1 (for a word one position earlier), 0 (for the same word), and +1 (for a word one position later). Each of these relative positions would have a separate learned embedding, which could be, for example:

PE(-1) = [0.2, -0.4, 0.3, -0.1, 0.5, -0.3] PE(0) = [0.1, -0.2, 0.3, -0.1, 0.2, -0.3] PE(+1) = [-0.1, 0.3, -0.2, 0.4, -0.3, 0.1]

In relative positional encoding, the embeddings are learned for each possible relative position. They’re not tied to any specific words, but rather indicate the relative positions between words.

For example, in a sequence of length 10, you’ll have possible relative positions ranging from -9 to +9 (a total of 19 relative positions, including 0). Each of these relative positions will have a learned embedding.

In practice, there’s often a maximum sequence length that the model can handle (due to memory and computational constraints), and we’d have a learned embedding for each possible relative position within that maximum length.

The embeddings are the same for all words and they’re learned based on relative positions, not the specific words at those positions. Once the embeddings are learned during the training process, they’re used to represent the same relative position in any context.

So, the learned relative positional embeddings represent the “distance” or “difference” between positions, and they are used to adjust the attention scores in the transformer model. This provides the model with information about the order of words in the sequence, which is crucial in many language processing tasks.

These relative positional embeddings would be used to adjust the attention scores, rather than being added to the word embeddings directly.

In both cases, the idea is to give the model some information about the order of the words in the sequence, since the self-attention mechanism by itself doesn’t have any way of considering word order.

We will continue in the next post.

Thank you for taking the time to read my post. If you found it helpful or enjoyable, please consider giving it a like and sharing it with your friends. Your support means the world to me and helps me to continue creating valuable content for you.