This text provides a guide on using the Simple Transformers library for solving sentence pair tasks in Natural Language Processing (NLP), specifically focusing on the Semantic Textual Similarity Benchmark.
Abstract
The Simple Transformers library is built on top of the Transformers library by Hugging Face, aiming to simplify the use of Transformer models for NLP tasks. This guide demonstrates how to use Simple Transformers for sentence pair tasks, such as textual entailment and semantic similarity. The tutorial uses the Semantic Textual Similarity Benchmark dataset from the GLUE benchmark and provides step-by-step instructions on setting up the environment, preparing the data, configuring the model, and training and evaluating it. The guide also showcases the use of Weights and Biases for visualizing training progress.
Opinions
The author emphasizes the ease and simplicity of using Simple Transformers for NLP tasks.
The author suggests that BERT and other Transformer models excel at sentence pair tasks.
The author recommends using the 'roberta-base' model for this demonstration.
The author encourages the use of Weights and Biases for visualizing training progress.
The author mentions that the model training can be stopped earlier when losses are not decreasing, and scores are not improving anymore.
The author provides a link to the complete set of graphs and other information on the dashboard.
The author concludes by stating that the Simple Transformers library allows users to perform sentence pair tasks and regression tasks quickly and easily.
Solving Sentence Pair Tasks Using Simple Transformers
Want to use Transformers for Sentence Pair NLP tasks? Simple Transformers has you covered!
The Simple Transformers library is built on top of the excellent Transformers library by Hugging Face with the goal of making Transformer models quick and easy to use.
Introduction
Sentence pair tasks, as the name suggests, are Natural Language Processing (NLP) tasks where the input features consist of two pieces of text (not necessarily grammatical sentences). Textual entailment and semantic similarity are a couple of examples for such tasks.
Considering the unprecedented success of BERT and other Transformer models in many NLP tasks, it should come as no surprise that they excel at sentence pair tasks as well. This guide demonstrates how you can harness the power of Transformer models to solve Sentence Pair tasks using Simple Transformers.
All source code for Simple Transformers is available on the Github Repo. If you have any issues or questions, that’s the place to resolve them. Please do check it out!
Setup
Install Anaconda or Miniconda Package Manager from here
Create a new virtual environment and install packages.
conda create -n transformers python pandas tqdmconda activate transformers
If using cuda:
conda install pytorch cudatoolkit=10.1 -c pytorch
else:
conda install pytorch cpuonly -c pytorch
Install Apex if you are using fp16 training. Please follow the instructions here. (Installing Apex from pip has caused issues for several people.)
Once we’ve set up our Conda environment and installed all the tools, we still need a dataset on which we can train and evaluate our models. We’ll be using the Semantic Textual Similarity Benchmark which is part of the GLUE benchmark for NLP. This is the same benchmark originally used to evaluate BERT.
Extract the downloaded dataset into a directory data/.
Prepare the data
Simple Transformers expects a Pandas Dataframe containing at least 3 columns; text_a, text_b, and labels. Any additional columns will be ignored.
In my case, the tsv files are located inside data/sts-b/.
I found some unescaped quotes in the dataset which were causing errors, but I skipped them as there were only a few of them. I’m also dropping any na values that might exist.
Finally, we also need to rename the columns to reflect the names that Simple Transformers expects.
Configuration Options
There are many configuration options available in Simple Transformers to help fine-tune a model. See here for the full list. In a nutshell, we are setting up a model with 512 maximum sequence length, and we’ll be training it for 10 epochs with a batch size of 16 (reduce the batch size if you get CUDA memory errors). We’ll also be evaluating the model on the evaluation data every 50 training steps. The training progress will be automatically sent to the Weights and Biases project “sts-b-medium” where it can be easily visualized (see here for all the pretty graphs!).
Finally, we are specifying that this is a regression task where the model will be tasked with predicting a continuous output value.
Creating the Model
There are many model types and pre-trained models available but we’ll be using the roberta-base model for this demonstration.
Here, we are creating a ClassificationModel that we can use for training, evaluating, and predicting. The first parameter is the model_type, the second is the model_name, and the third is the number of labels in the data. The train_args we specified earlier is passed to the args parameter.
model_type may be one of ['bert', 'xlnet', 'xlm', 'roberta', 'distilbert', 'albert', 'camembert', 'xlmroberta'].
For a full list of pretrained models that can be used for model_name, please refer to Current Pretrained Models.
Note that num_labels has been set to 1 to indicate that the model will predict a single output value.
Training the Model
We can specify any metrics that we wish to use when performing evaluation during training. Conveniently, they will then be automatically logged to the W&B visualizations. You can also find them in the training_progress_scores.csv file in the outputs directory. Here, we’ll be using the Pearson correlation coefficient and the Spearman’s Rank Correlation Coefficient metrics.
Simple Transformers will automatically run and log the results from any function passed into the training method as keyword arguments.
The model training itself can be done by simply executing a single line (almost feels like a waste of a gist 😉).
All checkpoints will be saved into outputs/checkpoint-{global_step}/ and the final model itself will be saved into outputs/.
To load a saved model for later use, you can replace the model_name parameter with the path to a directory containing a saved model when creating a ClassificationModel.
model = ClassificationModel('roberta', 'outputs/', num_labels=1)
Visualizing Progress
The training progress can be seen live by using W&B. You can see some of the charts below.
Pearson correlation coefficient
Spearman’s Rank Correlation Coefficient metrics
Evaluation Loss
Training Loss
Notice that I could have stopped the training earlier when the graphs show that my losses are not decreasing and scores aren’t improving anymore.
The complete set of graphs and other information can be found on the dashboard here.
Evaluating the Model
Evaluation is just as easy. You can specify metrics in exactly the same way we did when training.
This returns three values.
result: The evaluation result in the form of a dict. This will include any metric functions passed into the eval_model() method.
model_outputs: A list of model outputs for each item in the evaluation dataset.
wrong_predictions: A list of InputFeature of each incorrect prediction. The text may be obtained from the InputFeature.text_a attribute. May not be particularly useful in regression tasks. (The InputFeature class can be found in the utils.py file in the repo)
With the hyperparameter values specified earlier, I obtained the following (very satisfactory) results.
pearson_corr = 0.9115
spearman_corr = 0.9099
Conclusion
With the latest addition to the Simple Transformers library, you can now perform any sentence pair task as well as regression tasks quickly, pain-free, and simply.