What are the differences in Pre-Trained Transformer-base models like BERT, DistilBERT, XLNet, GPT, XLNet, …

This article is a cheat sheet of well-known Transformer-based models and tries to explain their uniqueness (while they are all based on the same architecture).

The combination of Transformer architecture and transfer learning is dominating the Natural Language Processing world. There are numerous pre-trained models (Huggingface alone has 40+) which might look the same at first glance because they are all using the same blocks. Make sure to have a basic understanding of the Transformer blocks to get the most out of this piece. (I highly recommend Jay Alamar “The Illustrated Transformer” post)

It is easy to correlate each model’s strengths with the number of parameters it has, but there must be more to it. I will review a couple of famous models and discuss their differences, like the overall architecture, pre-training objectives, and the training process. In general, try to answer the question What makes them unique?

BERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [1]

Can we even talk about NLP + Transformer and do not mention the almighty BERT? Where it all started. The model is based on the Transformer’s encoder block stacking on top of each other. They used the Semi-Supervised learning [2] idea to leverage the unlabeled datasets (e.g., books, Wikipedia, …) for pre-training models to use it as a transfer-learning approach in NLP. The Masked Language Model (MLM) method was chosen to deconstruct the input as an Autoencoding approach. Also, the Next Sentence Prediction task was used to reinforce the text-similarity measurement ability.

• Masked Language Model: The model masks some random words [from the input] and tries to predict the missing tokens. As reported in the paper, a total of 15% of the words will be chosen for masking. Out of them, 1) 80% of the chosen words will be replaced by [MASK] token; 2) There is a 10% chance to replace the word with a random word, and 3) The remaining 10% words will remain unchanged.
• Next Sentence Prediction: The objective is to understand the relationship between two sentences. The process is to feed sentences in a pair of two [and separate them with a special separator token] and measure how likely is it for the 2nd sentence to be the follow-up sequence of the 1st sentence. (As illustrated in Figure 1)

BERT uses these methods for pre-training a model to learn the basics of the language. We can fine-tune it for any downstream task like classification or summarization later on. There are a few problems with like:

1. It assumes all the masked tokens are independent. If we have a sequence like “New York is a beautiful city” and mask both “New” and “York” words, the model will predict each of them without seeing the other one.
2. The mask token does not appear in real-world data. But we are conditioning our model based on it during the pre-training. In other words, a generalized model should not depend on data corruption.
3. The BERT model is trained on a fixed 512 tokens input limitation. So it is hard to capture long-term dependencies.

BERT can generate a pretty good representation for any input sequence by taking the context into account. But you need to attach different heads to it for various tasks (like classification, translation, and summarization) so you can use its embeddings.

DistilBERT

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter [3]

The BERT model might not seem to be a large network by today’s standards. But its large edition still has 340M and is not easy to load and use in every setup. So, there are some attempts to make the model smaller while preserving its language understanding capabilities.

The DistilBERT model used the knowledge distilation method to train a model with 97% of the BERT’s ability but 40% smaller in size (66M parameters compared to BERT-based’s 110M) and 60% faster. The process consists of having a teacher and student model, and trying to make the student model results close to the teacher one. It is not a model with State-of-the-art (SOTA) results, but it is faster and competitive.

RoBERTa

RoBERTa: A Robustly Optimized BERT Pretraining Approach [4]

This study proved that BERT was not pre-trained with the best hyperparameters, and there was room for improvements. The fundamental changes are 1) removing the Next Sentence Prediction pre-training objective, 2) use mini-batches with a larger size, 3) with a higher learning rate, 4) for more epochs, 5) much much more data, and lastly, 6) they also changed the tokenization method to byte-level BPE.

These changes resulted in a model with 125M parameters (RoBERTa-base) with a SOTA score on a couple of NLP downstream tasks. They basically pointed out the BERT’s flawed design choices [by increasing all the hyperparameter, duh!!]. (There is also a DistilRoBERTa[-base] edition of the model with 82M parameters)

GPT Family

GPT: Improving Language Understanding by Generative Pre-Training [5] GPT-2: Language models are unsupervised multitask learners [6] GPT-3: Language Models are Few-Shot Learners [7]

GPT is known to train huge models with billions of parameters; for example, GPT-3’s largest edition has 175B parameters. Their architecture is based on the Transformer’s decoder block. The encoder-decoder cross attention part of the block is removed because there is no encoder, and the self-attention part is replaced with the masked self-attention.

They chose an autoregressive pre-training objective by introducing Causal Language Modeling. It means we will feed all the whole input tokens to the model and expect it to predict the next token at each timestep. (Then we have a loop of feeding back the newly generated tokens to the model to get the next timestep’s token prediction) The masked self-attention method will prevent the model from cheating and looking forward at each timestep by masking out the future tokens.

As you see in figure 2, we pass the whole sequence to the model, but the model at timestep 5 tries to predict the next token by only looking at the previously generated tokens. The mentioned architecture stayed relatively the same through the different iterations of the GPT model with minor changes in normalization layers, residual connections, number of input tokens, … and one significant change to the number of layers and parameters. Honestly, it is hard to say which one of these changes is responsible for the model’s success!!!!

It is a generative model and can do different tasks with linear layers on top. It also uses special tokens for each task to pass both input and target sequences jointly (like “articlesummary”) to the model so it can understand the task and make the prediction accordingly.

MASS

MASS: Masked Sequence to Sequence Pre-training for Language Generation [8]

This model tried to pre-train a complete Transformer encoder-decoder architecture jointly. (Finally!) Because it actually makes sense if you think about it. The encoder is responsible for making a representation from the input sequence, and the decoder will try to choose the best output token based on this representation. Why shouldn’t they pre-train together?

Their pre-training objective is called Masked Sequence to Sequence. In this approach, we mask out k consecutive tokens from the input sequence and only predict those tokens at the output. If the k = 1, then it is the same objective as BERT; if k=m (where m is the total number of input tokens), it is the same as GPT, so it can act as both. As illustrated in figure 3.

The model performs very well and outperforms BERT/GPT-2 on generative tasks like translation and summarization.

BART

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension [9]

It is not fair to BART if I do not mention the paper because it is published later than MASS. BART did a large-scale experiment on the complete encoder-decoder Transformer architecture. The paper defines the model as “[it] can be seen as generalizing BERT, GPT, and many other more recent pretraining schemes” just like MASS. The experiment uses different noising functions to measure their impact.

They used noising functions like Token Masking, Token Deletion, Token Infilling, Sentence Shuffling, and Document Rotation to prepare the encoder’s noisy input. Then an autoregressive decoder tries to recreate the original input (or denoise it). It is a really impactful experiment, because it reveals the effect of each method for Autoencoding regarding each task. For example, the paper shows that a combination of Text Infilling + Sentence Shuffling will result in SOTA accuracy in SQuAD task.

XLNet

XLNet: Generalized Autoregressive Pretraining for Language Understanding [10]

It is another encoder-only model and pre-trained based on the idea that corrupting input data (like BERT) is not a good idea because we will lose information and dependencies. Instead, a Permutation Language Model pre-training objective was introduced to consider all the possible permutations of an input sequence.

I will not go through all the details in the paper like Two-Stream Self-Attention or the Memory that they borrowed from Transformer-XL[11]. Just keep in mind that there is more to their success than just a new pre-training objective. They achieved SOTA accuracy and outperformed BERT on ~20 NLP tasks.

Final Words

I tried to make a timeline of some notable papers after the introduction of the Transformers architecture. The main focus was on the pre-training objective of each model and pointed out their differences without going into many details. There are already lots of articles explaining each one of these models in great detail. Just google their names!

I hope you liked it and let me know what you think.

I send out a weekly newsletter for NLP nerds. Consider subscribing if you like to stay up-to-date on the latest developments in Natural Language Processing. Read more and subscribe — join the cool kids club and sign up now!

References

[1] Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. [2] Dai, A. M., & Le, Q. V. (2015). Semi-supervised sequence learning. arXiv preprint arXiv:1511.01432. [3] Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. [4] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., … & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [5] Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. [6] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9. [7] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … & Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [8] Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. Y. (2019). Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450. [9] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., … & Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. [10] Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. [11] Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860. Chicago