The data that those large language models were built on

Why can Large Language Models (LLMs) answer questions, do book reports, draft notes, or summarize a document? An important reason is the data that they were trained on, or fine-tuned with. This post helps you to understand those widely used datasets that are known in the LLM community. While you research LLMs, you may be curious about the data. If you are thinking about fine-tuning a pre-trained LLM for your specific purpose, it is important to know the data sources that a pre-trained LLM was trained on.

There are many LLMs and datasets. In this post, I focus on the LLMs that can provide enough breath for the datasets that they were developed and evaluated against. They are:

• GPT-2 (2019), GPT-3 (2020), chatGPT (2022), GPT-4 (2023),
• T-5 (2019), Flan-T5 (2022),
• BERT (2018), RoBERTa (2019), DeBERTa (2019), DistilBERT (2020)
• MPT-7B-StoryWriter-65k+.

In the above list, MPT-7B-StoryWriter-65k+ is a model designed to read and write stories with super long context lengths — a context length of 65k tokens. The BERT models are listed as the representatives of the large BERT family.

From these LLMs, I will highlight the modeling datasets including CommonCrawl, WebText, C4, SQuAD 1.0 and SQuAD 2.0, SWAG, and BookCorpus. I also described the nine datasets of GLUE that are used as the benchmark datasets for model performance. GLUE is also used for model fine-tuning (e.g. T5). Although the datasets are readily available in hugging face, I look into the sources of the datasets. Hopefully, this will benefit your understanding quickly. If you are new to LLMs and do not know where to start, you are advised to use the Huggingface platform https://huggingface.co/.

Modeling Datasets

I categorize the datasets into “News/Wikipedia”, “Web crawling”, “Questions and Answers (Q&A)”, “Books”, and “Reading comprehension” in Table (A). This categorization is by no means mutually exclusive because, for example, “web crawling” can contain “news/Wikipedia”. This categorization features the primary sourcing method and the data contents. I provide representative models for each dataset in the table so you know the significance of each dataset.

Many people know LLMs are trained on data on the world wide web. So let’s start with the web crawled data.

(1) CommonCrawl Data — The most common place

The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling since 2008. The corpus contains raw web page data, metadata extracts, and text extracts. This page contains the monthly index files archives. Each index file contains just the indices but not the actual web pages. Even the index files are still very large. For example, the 2022–05 crawl has a total compressed size of 205 GB, and the actual data is over 72TB. You can access the Common Crawl corpus for free on AWS S3. Many projects have used the CommonCrawl corpus, including the GPTs. Handling web crawling efficiently requires specialized engineering and data management.

Web crawling returns massive data. Unfortunately, the majority of web-crawled data is not natural language but largely comprises gibberish, error messages, or duplicate text. How about human curation and filtering? Based on the CommonCrawl, researchers see the need to manually curate the data for better quality. This effort was done in WebText.

(2) WebText — An internal OpenAI corpus

The WebText dataset is a curated data file. It was created by scraping 45 million outbound links from Reddit, which resulted in over 8 million documents for a total of 40 GB of text. It removed Wikipedia documents because many documents refer to the same Wikipedia documents. To maintain the quality of the data, it was curated and filtered manually by humans. This dataset was used to train GPT-2, GPT-3, chatGPT, and GPT-4. Because the dataset is about web crawling, I categorize it into “Web crawl”.

Let’s read some examples. The paper “Language Models are Unsupervised Multitask Learners.” [GPT-2] shows a few examples of English-to-French and French-to-English translation in the WebText training set:

”I’m not the cleverest man in the world, but like they say in French: Je ne suis pas un imbecile [I’m not a fool].

In a now-deleted post from Aug. 16, Soheil Eid, Tory candidate in the riding of Joliette, wrote in French: ”Mentez mentez, il en restera toujours quelque chose,” which translates as,

”Lie lie and something will always remain.” “I hate the word ‘perfume,”’ Burr says. ‘It’s somewhat better in French: ‘parfum.’

If listened carefully at 29:55, a conversation can be heard between two guys in French: “-Comment on fait pour aller de l’autre cote ́? -Quel autre cote ́?”, which means “- How do you get to the other side? — What side?”.

If this sounds like a bit of a stretch, consider this ques- tion in French: As-tu aller au cine ́ma?, or Did you go to the movies?, which literally translates as Have-you to go to movies/theater?

“Brevet Sans Garantie Du Gouvernement”, translated to English: “Patented without government warranty”.

Since we talk about manual curation and filtering in the creation of the data, let me introduce SQuAD 1.1 and SQuAD 2.0 that aim at curated questions and answers (Q&A) data.

(3) SQuAD 1.1 — Q&A

The Stanford Question Answering Dataset has 100,000 pairs of questions and answers. Let’s see how the questions and answers were generated. First, the research team sourced from 10,000 articles in English Wikipedia to get 536 articles. Second, the research team distributed the articles to crowd workers to create questions and answers. The crowd workers were tasked to provide up to 5 questions on the content of that paragraph. They were asked to spend 4 minutes on every paragraph and paid $9 per hour for the number of hours required to complete the article. This results in highly-curated Q&A data. Interesting, isn’t it? Let’s take a look at one of its articles about “Super Bowl 50”:

Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24–10 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi’s Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the “golden anniversary” with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as “Super Bowl L”), so that the logo could prominently feature the Arabic numerals 50.

There are several questions for the above article. One question is “Who won Super Bowl 50?” The answer, or call the ground truth, is “Denver Broncos”. Another question is “What venue did Super Bowl 50 take place in?” The answer is “Santa Clara”. Another question is “If Roman numerals were used, what would Super Bowl 50 have been called? The answer is “Super Bowl L”. The answers to all these questions can be found in the article. In the example, the ‘Prediction’ was the answer predicted by an LLM.

Let’s talk about the weakness in its implication. Because SQuAD includes only answerable questions that a correct answer is guaranteed to exist in the context document, models only need to select the most relevant answers. This is not yet true reading comprehension. How can this be improved? That prompts the creation of SQuAD 2.0!

(4) SQuAD 2.0 — Reading Comprehension

SQuAD 2.0 adds to SQuAD data with over 50,000 unanswerable questions written adversarially to look similar to answerable ones. LLMs must answer questions when possible but also have to determine when no answer is supported by the paragraph and decline to answer. This challenges LLMs to a better language understanding. For this reason, I classify SQuAD 2.0 into the ‘reading comprehension’ category. Let’s take a look at one of the contexts about the “1973 oil crisis”:

The 1973 oil crisis began in October 1973 when the members of the Organization of Arab Petroleum Exporting Countries (OAPEC, consisting of the Arab members of OPEC plus Egypt and Syria) proclaimed an oil embargo. By the end of the embargo in March 1974, the price of oil had risen from US$3 per barrel to nearly $12 globally; US prices were significantly higher. The embargo caused an oil crisis, or “shock”, with many short- and long-term effects on global politics and the global economy. It was later called the “first oil shock”, followed by the 1979 oil crisis, termed the “second oil shock.”

SQuAD 2.0 has several questions for the above article. One question is “Who proclaimed the oil embargo”. This is an easy one! The answer, or call the ground truth answer, is “members of the Organization of Arab Petroleum Exporting Countries”. SQuAD 2.0 also contains very similar questions that cannot be answered. Another question is “When did OPEC begin?”. The article does not mention the beginning of OPEC so the ground truth is <No Answer>. Similarly, another question is “What was the price of oil in 1979 before it rose?”. The article mentions there is a second oil crisis in 1979, but did not refer to the price in 1979 before it rose. So the ground truth is . (In the example, the ‘Prediction’ was the answers predicted by an LLM.)

Continue the idea of human curation and filtering to get high quality datasets, let’s see the next large dataset: C4.

(5) C4 — Colossal Clean Crawled Corpus

C4 is a dataset consisting of hundreds of gigabytes of curated English text scraped from the web. It is an important effort together with the creation of the milestone T5 (Text-to-Text Transfer Transformer) model. To describe C4, we have to first describe T5.

Probably some of you have seen the T5 exhibit below. It is a unified LLM that can answer questions, translate from one language to another, classify texts, or summarize an article. The authors of T5 treat every text processing problem as a “text-to-text” problem. The text-to-text framework affords one model to directly apply the same model, objective, training procedure, and decoding process to every task. So a text summarization problem can be considered as a text-to-text problem in that the input text is an article, and the output text is the summary.

There is one more point to mention in T5: Transfer learning. It is a common modeling practice that a model is first pre-trained on a very large dataset, then fine-tuned on a downstream task. Transfer learning techniques have been applied in many data science domains including NLP, image classification, and video classification. You are highly recommended to check “Transfer Learning for Image Classification”.

With the Common Crawl web archive, the team of T5 applied many heuristic rules to clean Common Crawl’s web-extracted texts. For example, they only retained lines that ended in a terminal punctuation mark, discarded any page with fewer than 5 sentences, or removed pages containing “List of Dirty, Naughty, Obscene or Otherwise Bad Words”. Web pages that contain gibberish words like “lorem ipsum” text were removed. This large effort produced the cleaned C4 dataset.

The above datasets applied a great effort of human curation and filtering. The next dataset “SWAG” takes a different approach. It focuses on “commonsense reasoning”. My children often hear me saying “When common sense makes sense, seek no other sense.” We often exercise common sense but computers cannot. How is SWAG created? Let me introduce it.

(6) SWAG (Situations With Adversarial Generations)

The purpose of SWAG is commonsense reasoning as its paper presented, “A Large-Scale Adversarial Dataset for Grounded Commonsense Inference”. Commonsense reasoning is an important aspect of human conversations. For example, if a partial sentence is “she opened the hood of the car,” our common sense tells us it is probably about checking the car's condition. Then we will say the next texts are likely “then, she examined the engine”. Let’s talk about another example. Suppose there is a woman giving a piano performance in front of an audience. What’s her next move? She will most likely put her fingers on the piano keys and start playing. This is the commonsense reasoning that we have possessed naturally.

What types of text data can best record humans' activities and common sense reasoning? The research team of SWAG thought of video captions. In a video, the conversations are usually interactive and implicative. They employed the video caption data from ActivityNet Captions (Krishna et al., 2017; Heilbron et al., 2015), and the Large Scale Movie Description Challenge (LSMDC; Rohrbach et al., 2017). ActivityNet contains 20k YouTube clips and LSMDC consists of 128k movie captions (audio descriptions and scripts). From the two datasets, they derived 113k multiple choice questions (73k training, 20k validation, 20k test).

Although datasets with human curation and filtering present better quality, recent work has found that curated datasets are susceptible to annotation artifacts. It is an unintended consequence that human curation gives out clues for the target labels (Gururangan et al., 2018; Poliak et al., 2018). As a result, models trained on the curated datasets run the risk of overfitting the training data but perform poorly on out-of-domain examples (Wang et al., 2018; Glockner et al., 2018). An important contribution of this paper is to correct such biases through their Adversarial Filtering (AF) procedure, which I will not entail here because it is out of the scope of this post.

(7) BooksCorpus

The BooksCorpus dataset is a collection of 11,038 books from 18 different genres such as Romance (2,865 books), Fantasy (1,479 books), Science fiction (786 books), Tens (430 books), etc. All selected books should have more than 20K words. The 11,038 books have 74,004,228 sentences and 984,846,257 words, and 1,315,420 unique words. You can access the data by clicking here. I am sure I have not read that many books. The dataset was used by GPTs and BERTs.

(8) Pile — Books3

Recent LLM research shows that increased training dataset diversity can improve LLM cross-domain knowledge and downstream generalization. So The Pile — Books3, an 825 GiB English text corpus, was constructed from 22 various high-quality subsets. This dataset contains 197,000 books processed in exactly the same way as bookcorpusopen (a.k.a. books1). It contains the title and the content for each book in plain .txt.

Benchmark Datasets

If an LLM is expected to perform Natural Language Understanding (NLU), it is critical that an LLM should be able to execute a range of linguistic tasks across different domains. GLUE (General Language Understanding Evaluation) and Super GLUE are developed to be the benchmarks for model evaluation. It is not a single dataset but a collection of nine datasets that were previously developed. Many of the above LLMs have used GLUE and Super GLUE to benchmark their performances.

(9) The GLUE Benchmark (General Language Understanding Evaluation)

I show the categories of the nine datasets in Table (B).

(9.1) CoLA (Corpus of Linguistic Acceptability)

CoLA is a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. The sentences are labeled as grammatically “acceptable” or “unacceptable”. Below are a few grammatically “acceptable” sentences in the dataset:

• I said that my father, he was tight as a hoot-owl.
• The jeweler inscribed the ring with the name.
• They can sing.
• The men would have been all working.
• The gardener planted roses in the garden.
• I wrote Blair a letter, but I tore it up before I sent it.

Below are sentence examples that are labeled as “unacceptable”:

• The more books I ask to whom he will give, the more he reads.
• Usually, any lion is majestic.
• Much evidence was provided.
• Who do you think will question Seamus first?

(9.1) SST-2 (Stanford Sentiment Treebank)

An important aspect of Natural Language Understanding (NLU) is the tone or the “sentiment” of a document. Some words already carry strong sentiments like “awesome” or “angry” and can be easily put into the positive or negative range. But not all words are like this. Actually, the research on sentiment accuracies, even just for binary positive/negative classification, for single sentences has not exceeded 80% for several years.

Stanford Sentiment Treebank was created to include labels for every syntactically plausible phrase in thousands of sentences. It allows researchers to train and evaluate compositional models. The tree diagram below shows a sentence that the sentiment of every word is captured and classified into 5 sentiment classes from very negative to very positive (– –, –, 0, +, + +). The sliding bars show the scale for the phrase “nerdy folks” from “very negative”, “negative”, “somewhat negative”, “neutral”, “somewhat positive”, “positive”, and “very positive”. SST includes fine-grained sentiment labels for 215,154 phrases in the parse trees of 11,855 sentences.

There are SST-2 to SST-5. SST-2 is the SST data with binary labels (“positive”, “negative”); and SST-5 is with 5 labels.

(9.3) MRPC (Microsoft Research Paraphrase Corpus)

Automatic paraphrasing tests if an LLM truly understands the meaning of a document and can paraphrase correctly. MRPC was created to test if an LLM can perform paraphrasing accurately. It contains 5,801 sentence pairs. Each pair is labeled manually with a binary judgment of whether the pair constitutes a paraphrase.

(9.4) STS-B (Semantic Textual Similarities Benchmark)

Semantic Evaluation is an important topic in NLU and has been the product of International Workshops on Semantic Evaluation. Its goal is to create high-quality annotated datasets in a range of increasingly challenging problems in natural language semantics. STS-B is the dataset that consists of a selection of the English datasets used between 2012 and 2017. The selection of datasets includes text from image captions, news headlines, and user forums.

(9.5) QQP (Quora Question Pairs)

You may have got your questions answered from the website Quora. Quora is a popular community question-answering website. The Quora Question Pairs dataset is a collection of question pairs. It is created to determine whether a pair of questions are semantically equivalent.

(9.6) MNLI (Multi-Genre Natural Language Inference)

The MNLI corpus is a crowdsourced collection of sentence pairs with textual entailment annotations. It has the original text (“text1”), and the corresponding text (“text2”), and the label showing the two texts are “neutral”, “entailment”, or “contradiction”. In other words, the labels show whether text 2 is irrelevant, a deduction of text 1, or a contradiction of text 1. Let’s see one example that text 2 is the entailment of text 1:

(Text 1) “you know during the season and i guess at your level uh you lose them to the next level if they decide to recall the parent team the Braves decide to call to recall a guy from triple-A then a double-A guy goes up to replace him and a single A guy goes up to replace him”

(Text 2) “You lose the things to the following level if the people recall.”

This example shows “text 2” is a contradiction to “text 1”:

(Text 1) “I don’t mean to be glib about your concerns, but if I were you, I might be more concerned about the near-term rate implications of this $1.”

(Text 2) “I am concerned more about your issues than the near-term rate implications.”

(9.7) QNLI (Question-answering Natural Language Inference)

To evaluate reading comprehension, one important task is to determine whether the context sentence contains the answer to the question. The Stanford Question Answering Dataset is a question-answering dataset consisting of question-paragraph pairs. Annotators have added the label to indicate where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question.

(9.8) RTE (Recognizing Textual Entailment)

The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges. We have learned MNLI has three outcomes “neutral”, “entailment”, or “contradiction”. Here RTE has two outcomes “entailment” and “not entailment” (which is “neutral” and “contradiction”).

(9.9) WNLI (Winograd Natural Language Inference)

Similar to MNLI, WNLI is created that contains pairs of sentences for “entailment” or “not entailment”. Let’s take a look of a pair that is labeled as “entailment”:

(Text 1) “I stuck a pin through a carrot. When I pulled the pin out, it had a hole.”

(Text 2) “The carrot had a hole.”

The following pair is an example for “not entailment”:

(Text 1) “Steve follows Fred’s example in everything. He influences him hugely.”

(Text 2) “Steve influences him hugely.”

An End Note

Since data are the knowledge base, an overarching idea is that an LLM only knows up to the time that a dataset covers. For example, GPT-4 (2023) can only tell news up to 2023. If now is 2030 and you ask GPT-4 (2023) to answers news during 2024–2030, it will not know any new presidents, or any new movies after 2023.

Summary

In this post, I surveyed the datasets for training LLMs. I described the large-scale datasets CommonCrawl, WebText, C4, SQuAD 1.0 and SQuAD 2.0, SWAG, and BookCorpus. I also described the nine datasets of GLUE that are used as the benchmark for model performance. I hope this post will help your understanding of LLMs. If you are interested in Large Language Models, please click the following articles:

• Large Language Model Datasets
• Fine-tuning a GPT — Prefix-tuning
• Fine-tuning a GPT — LoRA
• GenAI model evaluation metric — ROUGE

References

• [BERT] Jacob Devlin, Ming-Wei Chang, Kenton Lee, & Kristina Toutanova. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
• [BooksCorpus] Zhu, Y., Kiros, R., Zemel, R. S., Salakhutdinov, R., Urtasun, R., Torralba, A. & Fidler, S. (2015). Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books.. ICCV (p./pp. 19–27), : IEEE Computer Society. ISBN: 978–1–4673–8391–2
• [Books3] Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., & Leahy, C. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027.
• [DeBERTa] Pengcheng He, Xiaodong Liu, Jianfeng Gao, & Weizhu Chen. (2021). DeBERTa: Decoding-enhanced BERT with Disentangled Attention.
• [DistilBERT] Victor Sanh, Lysandre Debut, Julien Chaumond, & Thomas Wolf. (2020). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.
• [Flan T5] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, & Jason Wei. (2022). Scaling Instruction-Finetuned Language Models.
• [GLUE] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
• [GLUE-MRPC] Dolan, W.B., & Brockett, C. (2005). Automatically Constructing a Corpus of Sentential Paraphrases. International Joint Conference on Natural Language Processing.
• [GLUE-CoLA] Alex Warstadt, Amanpreet Singh, & Samuel R. Bowman. (2019). Neural Network Acceptability Judgments.
• [GLUE-MNLI] Williams, A., Nangia, N., & Bowman, S. (2018). A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (pp. 1112–1122). Association for Computational Linguistics.
• [GLUE-QNLI] Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
• [GLUE-RTE] Dagan, I., Glickman, O., & Magnini, B. (2005). The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop (pp. 177–190).
• [GLUE-SST2] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP
• [GLUE STS-B] Philip May (2021). Machine translated multilingual STS benchmark dataset.
• [GLUE QQP] Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.
• [GLUE-WNLI] Levesque, H., Davis, E., & Morgenstern, L. (2012). The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
• [GPT-2] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.
• [GPT-3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, & Dario Amodei. (2020). Language Models are Few-Shot Learners.
• [GPT-4] OpenAI. (2023). GPT-4 Technical Report.
• [Kuo] Transfer learning for image classification: https://a.co/d/hLdCkMH
• [LSMDC] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, & Bernt Schiele. (2016). Movie Description.
• [MetaICL] Sewon Min, Mike Lewis, Luke Zettlemoyer, & Hannaneh Hajishirzi. (2022). MetaICL: Learning to Learn In Context.
• [MiniLM] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, & Ming Zhou. (2020). MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers.
• [PIQA] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, & Yejin Choi. (2019). PIQA: Reasoning about Physical Commonsense in Natural Language.
• [RoBERTa] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, & Veselin Stoyanov. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach.
• [SQuAD 1.1] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, & Percy Liang. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text.
• [SQuAD 2.0] Pranav Rajpurkar, Robin Jia, & Percy Liang. (2018). Know What You Don’t Know: Unanswerable Questions for SQuAD.
• [SWAG] Rowan Zellers, Yonatan Bisk, Roy Schwartz, & Yejin Choi. (2018). SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference.
• [T5] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, & Peter J. Liu. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.