Carefully Curated Papers to Study NLP in 2023 for Beginners (Part 1)
Natural Language Processing (NLP) has been a rapidly evolving field over the last decade, with groundbreaking research and advancements in various subdomains. In this blog, I would like to make a list of papers that are in my opinion must-read if you’re studying NLP in 2023. We will explore a curated list of NLP papers that every enthusiast should study to stay at the forefront of this exciting field.
Fundamentals
“Linear Classifier: An Often-Forgotten Baseline for Text Classification”
- This paper does not propose novel architectures yet shows the simplest baselines, like Linear Classifier with TF-IDF features, can outperform highly-established models like BERT.
- This paper is good for beginners to realize that sometimes bigger models do not guarantee the best results. In fact, over-parametrization is the real deal and there are many works that tackle this issue, such as distillation learning, pruning and etc.
“Attention is All You Need”
This paper introduced the Transformer architecture, which revolutionized the field of sequence modeling. The self-attention mechanism proposed in this paper replaced recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as the go-to methods for many NLP tasks.
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”
- This paper introduced the BERT (Bidirectional Encoder Representations from Transformers) model, which laid the foundation for the modern era of NLP.
- The architecture is based on Transformer’s encoder architecture, so it shouldn’t be difficult after understanding Transformers.
- It demonstrated the effectiveness of pre-training a deep bidirectional Transformer model on a large corpus and paved the way for numerous downstream NLP tasks, including question-answering, sentiment analysis, and named entity recognition.
- Outside of that, understanding BERT is fundamental for sub-fields like representation learning, self-supervised learning, and multi-modal learning in NLP.
Attention
“FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”.
- The FlashAttention algorithm rearranges the computation of attention and uses traditional techniques such as tiling and recomputation to greatly increase its speed and decrease the amount of memory used, going from a quadratic to a linear relationship with the length of the sequence.
- Performance-wise, it produces the exact same attention maps. However, due to the implementation, it is much faster than the original implementation.
“Linformer: Self-Attention with Linear Complexity”
The paper discusses the challenges of training and deploying large transformer models for long sequences due to the high computational and memory costs of the standard self-attention mechanism. The authors propose a new self-attention mechanism called Linformer that approximates the self-attention matrix as a low-rank matrix. This new mechanism reduces the overall complexity of self-attention from quadratic to linear in both time and space.
- They hypothesize that self-attention maps are low-rank matrices (provided long sequences). Therefore, projecting sequences into smaller spaces won’t result in major information loss.
Local Windowed Attention
- The following paper empirically demonstrates that language models might not need full global attention in all layers.
- In fact, a transformer needs local attention in the bottom layers, with the top layers reserved for global attention to integrate the findings of previous layer.
BERT variants
Note: reading the following papers will get you familiar with pretraining strategies and architectural details that can be leveraged in the future when you will be building your own models.
“RoBERTa: A Robustly Optimized BERT Pretraining Approach”
- RoBERTa is trained on a large corpus of unlabeled text from the internet, similar to BERT.
- RoBERTa has the same architecture as BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a different pretraining scheme.
- It uses a masked language modeling objective, where it predicts missing words in a sentence. RoBERTa also incorporates a next sentence prediction task during pre-training.
- It has a larger training dataset and longer training time compared to BERT, which leads to improved performance.
- RoBERTa outperforms previous models on a wide range of natural language processing tasks, including text classification, named entity recognition, and question answering.
“ALBERT: A Lite BERT for Self-supervised Learning of Language Representations”
- ALBERT uses repeating layers which results in a small memory footprint yet computationally remains similar to a BERT.
- They also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs.
- Next sentence prediction is replaced by a sentence ordering prediction.
“DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”
- Compared to RoBERTa models, which provide results with improved performances, DistilBERT aims to reduce computation time. To compress the model size, DistilBERT applies the “teacher-student” framework also referred to as knowledge distillation where a larger model or the “teacher” network is trained and the knowledge is passed on to the smaller model also known as the “student” network.
- DistilBERT retains 97% performance of BERT with 40% fewer parameters and faster inference time.
Some Last Words
As we look ahead to the year 2023, the field of NLP is expected to witness continued growth and innovation. By studying the papers mentioned in this blog, aspiring NLP researchers and practitioners can gain valuable insights into the foundational concepts, state-of-the-art models, and emerging trends in the field.
I might have missed some good papers but I believe these papers are important not only to understand the model architectures but also pretraining strategies and attention mechanisms. If you have any papers to suggest, please feel free to share them in the comment section.
I will be writing part 2 soon where I will be extending this list up until we reach Large Language Models (LLM). Thank you for your time reading this piece! Cheers!