avatarPranjal Khadka

Summary

The transformer neural network, introduced in the paper "Attention is All You Need," is an influential architecture designed for machine translation that has expanded to various fields, featuring an encoder-decoder structure with self-attention and positional encoding mechanisms.

Abstract

The transformer architecture, a cornerstone in neural network design for natural language processing, was introduced in 2017 and has since revolutionized the field of machine translation and beyond. It comprises an encoder that processes input sentences and generates context-aware word embeddings using self-attention, and a decoder that generates the translated output. Key components include positional encoding to incorporate word order information, multi-head attention for parallel processing of attention mechanisms, and a unique training approach using batches and padding with dummy tokens for consistent input sizes. The transformer model's innovative use of self-attention allows it to weigh the importance of different words within a sentence, and its multi-head attention mechanism enables it to focus on different parts of the input sequence simultaneously, enhancing its ability to capture context.

Opinions

  • The transformer architecture is considered groundbreaking for sequence-to-sequence tasks, particularly machine translation.
  • The use of self-attention and multi-head attention is seen as a significant improvement over previous sequence modeling and translation techniques.
  • Positional encoding is deemed crucial for maintaining the order of words in a sequence, which is essential for understanding the context in language tasks.
  • The author expresses that batch processing and padding with dummy tokens are effective techniques for stabilizing and accelerating the learning process during training.
  • The transformer's design, which includes skip connections and layer normalization, is believed to mitigate the vanishing gradient problem prevalent in deep neural networks.
  • The cross-attention mechanism in the decoder is highlighted as a powerful method for aligning the words of the input sentence with those of the output sentence during translation.

Transformer Neural Network

Transformer architecture

The transformer architecture has gained widespread popularity since the publication of the influential paper “Attention is All You Need” in 2017. This groundbreaking neural network design originally conceived for Natural Language Processing has found applications in various fields beyond its initial domain. In this article, I explore the fundamentals of the transformer architecture, providing a beginner-friendly explanation of how it operates.

The transformer architecture was originally designed for Machine translation task of translating from one language to another. It consists of two section, an encoder and a decoder. The encoder decoder architecture is another remarkable innovation that proved to be extremely good for sequence to sequence tasks like Machine translation. The encoder section of transformer takes in words from a sentence that we are going to translate from simultaneously and generate vectors that are context aware meaning these vectors are aware on where they need to have their attention on. This is a result of the self attention mechanism which is the crux of entire transformer neural network.

The decoder section takes in words from the sentence that we are going to translate to but along with the words, it also takes a start token to indicate start of the sentence and end token to indicate the end of a sentence. The context vectors generated at the encoder are also the input for decoder. Using these inputs, the decoder generates the translated words one by one. This means given the start token, the first word is generated, given start token and first word, second word is generated and so on.

ENCODER

Since algorithms work with numbers and not natural language, we need a way to represent the words in a sentence as numbers. This is achieved through embedding that represent each word in the sentence as vectors. In the context of neural network, these are represented using tensors. Hence, the input could be represented with a tensor of shape 10*300*512 where 10 is the batch size, 300 is the maximum number of words in a single sentence and 512 represents the dimension of each word of a sentence. Passing the input in batch is crucial than passing sentences one at a time because of advantages provided by batch gradient descent where weights are updated after every batch and not after every sentence is passed through the entire network. This makes learning faster and stable. Not every sentence will be of 300 words so in such cases sentences in both encoder and decoder are padded with dummy tokens to make them of length 300.

Self attention mechanism on its own doesn’t take into account the position of words in a sentence. So an additional step known as Positional encoding is performed that adds a vector of shape same as the input to the original vector representation of sentence to make sure the positions matter in a sequence of input. It is calculated by taking a sine and cosine function of some operations between position of words in a sequence, index of dimension and dimension length. Sine and Cosine functions are used since they are periodic and repeat after certain time.

The resultant vector from positional encoding is then mapped to three different vector, KEY, QUERY and VALUE with the help of a feed forward neural network. Since every 512 dimensional words are now represented by three different vectors, the shape changes to 10*300*(512*3) which is 10*300*1536. The transformer architecture uses something called as a multi head attention instead of a simple self attention mechanism. Multi head attention is where a set of attention blocks operate in parallel all of which get same input but produce different context aware vectors that are concatenated at the end to get the final context aware vector which is of the same shape as input vector. The attention mechanism in the transformer neural network is known as self attention because every single word in a sentences attends to every other word in that sentence.

The multi head attention is also masked so that during back propagation the gradients are not calculated for the dummy tokens that were added to make all sentences of equal length. Multi head attention consists of 8 heads meaning 8 different self attention block run in parallel and generate 8 different context vectors. As discussed earlier, these are concatenated at the end to get the final context vector. For input to the multi head attention, each KEY, QUERY and VALUE is broken into 8 different parts where each part will be 64(512/8) dimensional vector. Each of those 64 dimensional part of KEY, QUERY and VALUE are stacked with a shape of 10*300*192(64*3) representing every word for one head. Now, the QUERY and KEY vectors are multiplied along the last two dimension to get a tensor of shape 10*300*300. The tensor is then scaled with the square root of dimension of the KEY vector to not let them go out of bound. The padding mask matrix is now added with the result where the mask matrix is represented by zeroes for passing and negative infinity for masking. This is because of the softmax activation that will be used later which converts zeroes to ones and negative infinities to zeroes. In reality, a very small negative number is used instead of negative infinity to avoid division by zero error. Finally, the tensor is passed through a softmax activation and multiplied with the VALUE vector to get the context aware vectors. Since 8 different heads are running in parallel, we get 8 of these context aware vectors that are concatenated to get a tensor of shape 10*300*512.

This tensor is now added with a skip connection of same shape to avoid the vanishing gradient problem during back propagation which is a common problem for deep neural networks. Now we perform something called as layer normalization that ensures stable training by making sure the updates during back propagation are not too large. This is achieved by subtracting the values from mean and dividing by standard deviation. Now, the tensor is again passed through some feed forward layers, added with skip connections like before and normalized to get the final attention matrix of shape 10*300*512 where each word is represented as 512 dimensional vector and are now highly context aware.

DECODER

The decoder initially takes in the sentence that we are translating to along with start token, end token and padding token to enure consistency in length like before. In the same way like the encoder, a positional encoding is added, the resultant vector is mapped to KEY, QUERY and VALUE vectors and multi head attention is performed. The only difference is along with a padding mask, an extra layer of masking is done to ensure that during prediction, only the current and previously generated words are considered for generating next word. This is because in real world scenario, we do not have labels like we have during training.

The next step is multi head cross attention where the KEY and VALUE vectors come from the context aware vectors generated by the encoder and only the QUERY vector is propagated from our previous step. The cross attention mechanism maps every word in the target language sentence to every source word in the language that we are translating from. The operations performed inside the cross attention block is exactly same as multi head attention with padding mask. At the end of cross attention, we get a concatenated tensor of shape 10*300*512 which is added with a skip connection, layer normalized and passed through a feed forward layer to get the context aware vectors.

This tensor is now passed through a feed forward layer to expand to the size of vocabulary of the language that we are translating to and passed through a softmax activation to get the probability distribution across all words. Hence, the predicted word will be the word with maximum probability in the distribution. The generated output are compared with actual label words, loss is calculated and back propagation is performed throughout the entire network to update the weights and that is how a transformer neural network is trained for the task of machine translation.

Transformers
Machine Translation
Machine Learning
Artificial Intelligence
Deep Learning
Recommended from ReadMedium
avatarGeetansh Kalra
Attention Networks: A simple way to understand Self-Attention

“Every once in a while, a revolutionary product comes along that changes everything.” — Steve Jobs

8 min read
avatarLorentz Yeung
A Study of Llama 3’s Rotary Position Embeddings

Exploring the Advanced Mechanism Behind LLaMA 3’s Superior Positional Encoding for Natural Language Processing

4 min read
avatarLM Po
A Brief History of AI with Deep Learning

Artificial intelligence (AI) and deep learning have seen remarkable progress over the past several decades, transforming fields like…

24 min read
avatarManyi
In-depth understanding of Llama Tokenizer

Llama is a family of large language models released by Meta AI starting in February 2023.

5 min read
avatarPrem Vishnoi(cloudvala)
Example LLM: GPT-2 with code

An example of a Large Language Model (LLM) is GPT-2, which is part of the GPT (Generative Pre-trained Transformer) series developed by…

6 min read
avatarXu Zhao
Deep Dive into LLaMa 3

LLaMa 3 is the LLM open sourced by Meta. The author has reversed engineer the model architecture from the source code. The purpose of the…

7 min read